Lesson Introduction

Executive-function assessment presents a fundamental challenge: the cognitive processes professionals seek to measure are complex, overlapping, and strongly influenced by the circumstances in which they are observed. A student may perform adequately on a structured working-memory task while struggling to remember assignments in the classroom. Another student may receive elevated executive-function ratings from teachers but demonstrate average performance on standardized cognitive measures.

These differences are not necessarily contradictory. Performance-based tasks and informant ratings often capture different aspects of functioning. Structured assessments measure performance under specified conditions, whereas ratings typically summarize behavior across everyday activities. Both may contribute useful information, but neither should be interpreted as a complete representation of a student's executive functioning.

Sadozai and colleagues (2024) examined measurement type as a moderator of executive-function findings across neurodevelopmental conditions. Their results demonstrated substantially larger group differences on informant measures than on performance-based measures. This lesson examines what that finding means, why different methods may produce different results, and how psychologists working in schools can integrate assessment information into defensible educational decisions.

Learning Objectives

Upon completion of this lesson, participants will be able to:

1. Differentiate performance-based executive-function measures from informant ratings in terms of constructs assessed, ecological validity, strengths, and limitations.
2. Interpret measurement-related findings from the Sadozai et al. meta-analysis, including moderator effects, uncertainty, and implications for individual assessment.
3. Develop an integrated assessment interpretation using standardized measures, observations, informant reports, developmental history, and educational context.

Required Reading Assignment

What to Focus on While Reading

1. Examine how the authors distinguish performance-based executive-function measures from informant ratings. Identify the types of information each method is intended to capture.
2. Evaluate the authors' discussion of ecological validity and the relationship between standardized task performance and everyday functional behavior.
3. Compare the effect sizes for performance-based and informant measures. Consider what can and cannot be concluded from their different magnitudes.
4. Examine the moderator analyses involving age, reported sex, DSM edition, and publication year. Consider the difference between statistical significance and practical importance.
5. Review the methods used to select studies, extract outcomes, evaluate study quality, and account for multiple effect sizes from individual studies.
6. Identify limitations involving co-occurring conditions, unequal research representation, measurement variation, and the absence of extensive longitudinal evidence.
7. Consider how the findings should influence selection and interpretation of executive-function measures in school-based evaluations.

1. Two Approaches to Measuring Executive Functioning

Executive-function assessment typically draws from two broad measurement approaches: performance-based tasks and informant-based ratings. These methods are related to executive functioning but differ in their procedures, settings, response requirements, and interpretations.

Performance-based measures ask a student to complete structured activities intended to engage specific cognitive processes. Examples include inhibiting a response to a stimulus, remembering information while manipulating it, switching between task rules, or generating responses under time constraints.

Informant-based measures ask teachers, caregivers, or other observers to describe the student's behavior in everyday situations. These measures may address organization, initiation, emotional regulation, monitoring, working-memory-related behavior, and adaptability. The information reflects behavior as perceived by the respondent across particular environments.

Performance-Based Measures

Structured executive-function tasks offer several advantages. Administration procedures may be standardized, scoring can be relatively objective, and the examiner can observe the student's response patterns. Error types, response latency, persistence, and strategy changes may provide information beyond a single summary score.

However, executive-function tasks are rarely process-pure. A working-memory task may also require language comprehension, sustained attention, processing speed, and familiarity with testing procedures. A task labeled cognitive flexibility may require the student to remember changing rules, inhibit previous responses, and detect corrective feedback.

Sadozai and colleagues specifically discussed overlap between executive-function domains. They noted that a task such as the Wisconsin Card Sorting Test, commonly associated with set-shifting, may also depend on working memory. This illustrates why test names and domain labels should not be treated as evidence of isolated cognitive processes.

Informant-Based Measures

Informant ratings provide information about executive-function behaviors in everyday settings. Teachers may observe whether a student begins assignments independently, remembers materials, manages transitions, and completes multistep tasks. Caregivers may describe organization, routines, emotional regulation, and independent functioning outside school.

These observations can be especially valuable because many educational concerns arise under conditions that standardized tasks do not reproduce. A quiet individual testing environment differs substantially from a classroom requiring students to manage simultaneous instructions, competing stimuli, and social expectations.

Ratings nevertheless introduce important interpretive limitations. Respondents may differ in expectations, comparison groups, familiarity with the student, and opportunities to observe specific behaviors. A teacher working in a highly structured classroom may report fewer concerns than a teacher requiring extensive independent organization.

Such differences should not automatically be dismissed as rater error. They may reveal meaningful variation in the student's functioning across environments. The professional task is to determine which explanation best accounts for the pattern.

2. What the Meta-Analysis Revealed About Assessment Type

Sadozai and colleagues examined whether executive-function effect sizes differed according to assessment method. This question is important because findings about neurodevelopmental conditions may depend partly on how executive functioning is operationalized and measured.

In the primary analysis, performance-based measures showed a significant average difference between neurodevelopmental groups and controls, with g = 0.51. Informant-based measures showed a larger difference, with g = 1.49. These estimates came from different sets of studies and outcomes.

The authors then conducted a moderator analysis examining assessment type. This analysis included 113 studies and 2,028 outcomes. The estimated average effect was g = 0.49 for performance-based measures and g = 1.47 for informant measures. Assessment type was a statistically significant moderator.

Understanding the Results

Source: Sadozai et al. (2024), Results, "Primary outcome: neurodevelopmental groups and controls" and "Moderator analysis." Estimates from the primary and moderator analyses should not be treated as interchangeable.

The larger informant-based effect does not necessarily mean that rating scales are more accurate or more valid for every purpose. The difference may reflect the types of behaviors sampled, differences in measurement methods, and the relationship between daily functional difficulties and diagnostic characteristics.

Informant ratings often capture difficulties that emerge repeatedly across complex daily situations. Performance tasks may assess narrower cognitive operations under more supportive and controlled conditions. These methodological differences can produce different effect sizes without establishing that one method is universally superior.

Ecological Validity

Ecological validity concerns the extent to which assessment findings are relevant to functioning outside the testing situation. In executive-function assessment, this includes whether a measure captures processes involved in daily classroom behavior, independent organization, and adaptation to changing demands.

A structured task may have strong standardization and reliability but limited resemblance to everyday educational challenges. Conversely, a teacher rating may describe real-world functioning but also reflect classroom expectations, instructional structure, and observer interpretation.

Ecological validity should therefore be considered alongside construct validity, reliability, and measurement purpose. A test is not automatically ecologically valid because it resembles a classroom task, nor is a rating scale automatically invalid because it relies on subjective reports.

Other Moderators and Their Interpretation

The researchers also examined age, reported sex, publication year, and DSM edition. Age and publication year were not statistically significant moderators of the overall effect. The percentage of males in a study showed a statistically significant but very small association with effect size, accounting for little variance.

DSM edition was a statistically significant moderator. Studies using DSM-5 criteria showed larger average executive-function differences than studies using earlier editions. The authors discussed changes in diagnostic criteria and research composition as possible contributors.

These moderator findings describe relationships across studies. They do not establish that diagnostic criteria caused executive-function differences to increase in children over time. Study-level associations require cautious interpretation because multiple characteristics may vary together.

3. Integrating Discrepant Assessment Findings

Disagreement between executive-function measures is common enough to warrant deliberate interpretation rather than automatic resolution. Different instruments may sample different behaviors, require different cognitive processes, and operate under different environmental conditions.

Consider a student who receives average scores on structured working-memory tasks but elevated teacher ratings for organization and task completion. It would be inappropriate to conclude that the teacher ratings are necessarily inaccurate. The testing environment may provide structure, immediate feedback, limited distractions, and clearly defined task expectations.

Conversely, elevated ratings should not automatically establish a specific cognitive deficit. Difficulties organizing materials may reflect insufficient instruction in organizational routines, unclear expectations, inconsistent supports, emotional distress, or academic tasks that exceed the student's current skill level.

A Structured Interpretation Process

1. Define the referral question. Specify the functional concern and the educational decision the evaluation must inform.
2. Review measurement properties. Consider reliability, validity, normative appropriateness, and the constructs measured.
3. Examine task demands. Identify language, attention, memory, speed, motor, and reasoning requirements that may influence scores.
4. Compare environments. Determine whether differences in structure, expectations, sensory conditions, and available supports explain variation.
5. Integrate informant perspectives. Consider teacher, caregiver, and student reports in relation to their observation opportunities.
6. Develop testable hypotheses. Identify the most plausible explanations and determine what additional information would distinguish them.
7. Connect findings to action. Translate the integrated interpretation into individualized supports and measurable outcomes.

Developmental, Cultural, and Linguistic Considerations

Executive-function demands vary across developmental periods. A behavior considered typical for a younger student may become more concerning when older students are expected to manage long-term assignments independently. The relevant question is whether performance is appropriate for the student's developmental level and educational context.

Cultural and linguistic considerations are also essential. A verbal working-memory task may be influenced by proficiency in the language of assessment. Teacher ratings may reflect expectations about independence, communication, and classroom participation that differ across educational environments.

Assessment should therefore examine whether the student's language background, disability-related access needs, or instructional opportunities influence performance. These factors should be documented as part of the interpretation, not treated as peripheral considerations.

Evidence Quality and Research Limitations

The Sadozai et al. review followed PRISMA guidelines and included a structured study-selection and quality-appraisal process. The authors used multivariate meta-analytic models to account for nonindependence among multiple outcomes reported within studies.

Nevertheless, the review was limited by the available literature. Some neurodevelopmental conditions were represented by relatively few studies. Many included studies did not adequately control for co-occurring conditions, and longitudinal evidence was limited.

The authors also noted that IQ was not included as a primary covariate because of overlap between intellectual functioning and executive-function domains. These considerations affect how confidently results can be generalized across populations and interpreted as reflecting distinct cognitive mechanisms.

The broader lesson for professional assessment is that even well-conducted research syntheses cannot eliminate limitations in the underlying studies. Group-level evidence should inform assessment hypotheses, while individual conclusions require appropriately selected and interpreted data.

4. Applied Professional Practice

Example 1: Average Test Scores, Significant Classroom Concerns

A fictional sixth-grade student has average standardized working-memory and inhibition-task scores. Teacher ratings indicate substantial difficulties with initiation, organization, and assignment completion. Classroom observations show that the student begins work promptly during structured small-group instruction but requires repeated prompts during independent activities.

A defensible interpretation would recognize that standardized performance demonstrates adequate functioning under specific testing conditions. It does not establish that the student independently manages complex classroom demands.

The team should examine differences between supported and unsupported tasks, including the availability of visual directions, task segmentation, adult feedback, and organizational routines. Consultation may focus on identifying supports that improve independence without assuming a global cognitive impairment.

Example 2: Low Cognitive Score, Limited Everyday Concerns

A fictional fourth-grade student obtains a low score on a timed cognitive-flexibility measure. Teacher and caregiver ratings describe generally adequate flexibility and organization. During testing, the student required repeated clarification of instructions and demonstrated slow response speed across several unrelated tasks.

The low score warrants examination, but it does not independently establish a pervasive flexibility deficit. The examiner should review instruction comprehension, processing speed, task validity, and the student's behavior across other activities.

Additional evidence might include classroom observations during transitions, examples of adapting to changed instructions, and assessment tasks with different response demands. The resulting report should distinguish what the standardized measure demonstrated from what remains uncertain.

Key Takeaways

1. Performance-based measures and informant ratings assess related but different aspects of executive functioning.
2. Performance tasks provide information about cognitive performance under structured conditions, but often require multiple overlapping cognitive processes.
3. Informant ratings describe everyday functional behavior but are influenced by observation opportunities, expectations, and environmental context.
4. Sadozai et al. found substantially larger average executive-function differences on informant measures than on performance-based measures.
5. Larger group-level effects do not automatically establish greater diagnostic accuracy, reliability, or validity.
6. Discrepancies between measures should prompt examination of construct demands, environmental conditions, and functional educational impact.
7. An integrated assessment interpretation should distinguish observed findings from hypotheses, acknowledge uncertainty, and guide individualized educational decisions.

Primary reference: Sadozai et al. (2024). Nature Human Behaviour, 8, 2357–2366. DOI: 10.1038/s41562-024-02000-9. Research findings are distinguished from fictional instructional cases and professional application examples.