Monkeys and Lions: Improving the ecological validity of working memory tests to assess contextual impact on daily functioning in children with developmental disorders

Working memory is an important predictor of both behavioral and academic
functioning in typically developing children (Alloway, Gathercole, Kirkwood, &
Ellit, 2009; Peng, Barnes, Wang, Wang, Li, et al., 2018; Peng, Namkung, Barnes,
& Sun, 2016). Moreover, and perhaps more important, problems in working memory
ability have been identified in children with developmental disorders (Kasper,
Alderson, & Hudec, 2012; Kercood, Grskovic, Banda, & Begske, 2014). Working
memory problems are exhibited by children with various developmental disorders,
such as autism spectrum disorder (ASD) and attention deficit hyperactivity
disorder (ADHD)(Kasper et al., 2012; Kercood et al., 2014), although there are
some inconsistencies in findings with regard to how profound these problems are
and in which aspects of working memory (verbal or visual-spatial) these mostly
occur. Likewise, although working memory assessment is considered a crucial
part of the clinical diagnostic process when children are suspected to have
such developmental difficulties (Holmes, Gathercole, Place, Alloway, Elliot &
Hilton, 2010), neuropsychologists are often perplexed by the inconsistency
between clients performance on tests of EF and their everyday EF abilities
(Chaytor, Schmitter-Edgecombe & Burr, 2006). These inconsistencies in research
and clinical assessments have sparked continuous discussion on the ecological
validity of neuropsychological tests and it is questioned whether current
available tests provide an accurate prediction of a person*s functioning in
real-world settings (Barkley, 1991; Burgess, Alderman, Forbes, Costello,
Coates, et al., 2006).
Contributing to the limited ecological validity of neuropsychological tests is
the fact that the diagnostic testing situation is highly structured and
standardized, offering an environment stripped of the external stimuli that put
a burden on children with developmental disorders in everyday life. In other
words, a working memory deficit may cause no problems if the environment places
little demand on this skill. A recent non-clinical study on the ecological
validity of two existing working memory tests, the Lion game and the Monkey
game, compared performance in a typically developing sample of primary school
children on the tests in two settings: an individual (diagnostic-like) setting
and a classroom setting (Friso van den Bos & Van de Weijer-Bergsma, 2019).
Because instructions are embedded in the computerized tasks and given through
audio-recordings using headphone, children were able to perform the tests
self-reliantly and independently. The results from this study showed that
performance in a classroom setting was hampered when compared to the individual
setting. Most importantly, data from the same study showed that children differ
greatly in how much their performance was hampered, with some children
performing 5 standard deviations lower in their classroom assessment compared
to their individual assessment. Also, classroom assessment of verbal working
memory (but not visual-spatial working memory) proved a better predictor of
mathematics and reading attainment than verbal WM as tested in the controlled
testing situation, supporting the conclusion that adding (natural) distractors
may provide a more ecologically valid assessment of working memory functioning
(Friso van den Bos & Van de Weijer-Bergsma, 2019). Consequently, while children
who have difficulties with attention are more vulnerable to distractions than
typically developing children (Cassuto, Ben-Simon, & Berger, 2013; Geffner,
Lucker, & Koch, 1996), it is very likely that these children will show even
more profound differences between settings than typically developing children.
Indeed, several studies have shown that, in (virtual reality) classrooms,
attentional functioning in children with ADHD or ASD is hampered more by
distracters than in typically developing children (Adams, Finn, Moes, Flannery,
& Rizzo, 2009; Hanley, Khairat, Taylor, Wilson, Cole-Fletcher, & Riby, 2017;
Negut, Jurma & David, 2017). A such, the assessment of individual differences
in working memory functioning in different settings has potential clinical
utility through adding a valuable source of information that is typically not
acquired in the diagnostic process. A comparison between working memory
performance in settings that vary on a continuum from *ideal circumstances* to
*everyday circumstances* may give more insight into working memory difficulties
and under which circumstances they occur. As children are likely to differ in
the extent to which they are affected by environmental distracters, information
about such setting-specific performance and differences between settings may
provide crucial information for the diagnostic process as well as for promising
areas of intervention and support.

The objective is to examine how working memory functioning in children with
ADHD and/or ASD as measured with the Lion game and the Monkey is influenced by
context and to assess the diagnostic utility and ecological validity of the
Lion and Monkey games.

The following research questions will be examined:
* Do children with developmental disorders score lower on the Monkey and Lion
games compared typically developing children and is this moderated by the type
of setting?
* Which setting, diagnostic or classroom, yields the best working memory
predictors for everyday working memory functioning (ecological validity) and
for academic performance (predictive validity) in children with a developmental
disorder?
* What is the diagnostic utility (i.e., specificity and sensitivity) of the
working memory tests in both settings?
* Which child-level factors (e.g. EF, symptoms and severity) affect how
strongly children with ADHD and/or ASD are affected by the setting (classroom
versus diagnostic setting) in which the working memory tests are conducted?

This prospective observational study investigates working memory in two
different settings, the diagnostic setting and the classroom setting, in
children with and without developmental disorders.

The aim of the study is to assess the performance on the Lion game (Van de
Weijer-Bergsma, Kroesbergen, Prast, & van Luit, 2015) and Monkey game (Van de
Weijer-Bergsma, Kroesbergen, Jolani, & Van Luit, 2016) in a group of children
with developmental disorders (N = 75) and a control group of children without a
developmental disorder (N = 30) in two different settings (i.e., diagnostic
setting versus classroom setting).
All participants will be assessed with the Lion game and Monkey game in two
settings: a) in the diagnostic setting and b) a classroom setting. The order of
these conditions (diagnostic first versus classroom first) will be varied
between subjects to preclude condition order effects. Time between assessment
in the two settings will be 1 (minimum) to 2 (maximum) weeks.
In the classroom setting, the teacher will ask the participant to finish the
Monkey and Lion games on the class computer during a moment where classmates
are working independently. The total duration of this assessment is 20 minutes.
In the diagnostic setting, participants will finish the Monkey game and the
Lion game on a computer in the presence of a test-leader. Also, two additional
neuropsychological behavioral tasks will be administered, assessing inhibitory
control and focused attention, to examine which individual child-level factors
affect the influence of the setting. To do so, children are asked to perform
the Baseline Speed task (duration: 2 minutes), the Flanker task (duration: max.
15 minutes) and the Focused attention 4 Letters task (duration: 10 minutes)
from the Amsterdamse Neuropsychologische Taken (ANT, De Sonneville, 2014). The
total maximum duration of this assessment is 45 minutes.
In case children with a developmental disorder use psychostimulants (e.g.,
ritalinmethylphenidate, e.g. Ritalin, Concerta, Medikinet or Equasym), they
will be asked to withhold medication on the morning of both assessments. This
is standard procedure during the diagnostic process, to ensure a reliable
picture of a child*s everyday functioning and to be able to reliably assess the
ecological validity and diagnostic utility of the working memory tasks for the
diagnostic process. Any other form of psychoactive medication is not allowed
during the study, and therefore a exclusion criterion.

The (final) DSM diagnosis will be requested from the recruiting organization,
as well as information on Intelligence (IQ) scores (when available).
When additional data is available at the recruiting organization from the
following ANT tasks (GoNoGO task; Memory Search Letters; Response Organization
Objects; Sustained attention Dots; Shifting Attentional Set * Visual) these
data will also be requested.

Parents and teachers will be requested to fill out two questionnaires regarding
the child*s behavior and executive functioning (i.e., the Behavior Rating
Inventory of Executive Function (BRIEF; Goia, Isquith, Guy, & Kenworthy, 2000),
Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2001) and Teacher Report
Form (TRF; Achenbach & Rescorla, 2001). Additionally, the latest Cito
achievement scores for mathematics achievement and reading comprehension
(Feenstra, Kamphuis, Kleintjes, & Krom, 2010; Janssen, Scheltens, & Kraemer,
2005) will be requested rom teachers (duration: 10 minutes).

The risks associated with participation in this study are minimal/ negligible.
The burden to child participants will be reduced to a minimum as much as
possible and can be considered *light*. The physical or psychological integrity
of the participants will not be impinged. Children are assessed using
computerized behavioral tasks tapping into working memory ability twice: once
in their classroom during school time (max. duration: 20 minutes), and once in
a diagnostic setting (max. duration: 45 minutes). The working memory tasks are
game-like and resemble the tasks children do in school. In the diagnostic
setting two additional computerized behavioral tasks tapping into inhibitory
and attentional skills are selected to match the tasks that will be
administered in the standard diagnostic process.
The burden for teachers and parents is light, with a maximum of 30 minutes for
filling out questionnaires. If questionnaire data is available through the
diagnostic process, permission will be asked to use this data, further reducing
the burden for parents and teachers.
To be able to draw conclusions about the ecological validity and diagnostic
utility of the working memory tasks for children with developmental disorders,
it is crucial to examine these aspects in the goal population itself. A
typically developing control group is added to generate reliable reference data.
Participants do not directly benefit from participating in this study.
However, for children with ADHD and/or ASD, this study brings a detailed
characterization of cognitive functioning in different settings for each of the
participants, which can facilitate better guidance for parents, clinicians and
teachers.