Autism spectrum disorder (ASD) is a highly heritable neurodevelopmental disorder characterized by social-communication abnormalities and restricted, repetitive behaviors (RRBs). Numerous studies have documented the presence of a “broader autism phenotype” (BAP), a qualitatively similar, but milder, presentation of the defining ASD characteristics in some unaffected family members of individuals with ASD, suggesting inter-generational transmission of core ASD-related traits [1-3]. Yet, even with heritability estimates as high as 0.90 [4], our understanding of underlying pathophysiological processes and their relation to ASD traits remains limited owing, in part, to a lack of definitive biological and neurobehavioral markers of core clinical features [5].

Family studies

identifying biologically based quantitative traits present in both individuals with ASD and their unaffected family members (i.e., endophenotypes) may help delineate characteristic patterns of inter-generational transmission and build mechanistic bridges between etiological processes and clinically relevant behavioral traits [6, 7].

Neurocognitive dimensions associated with core clinical features of ASD may represent important targets in this regard, as they are quantifiable and potentially more closely related to underlying neurobiological processes than broader clinical phenomena. Still, few studies have systematically examined neurocognitive traits in family members of individuals with ASD. Deficits in cognitive control have been repeatedly documented in individuals with ASD, and they have been linked to key clinical issues [8, 9]. Cognitive control is necessary for adaptive goal-directed behavior and includes neurobehavioral processes including behavioral flexibility (i.e., the ability to change behavior in response to contextual demands) and behavioral response inhibition (i.e., the ability to inhibit contextually inappropriate prepotent behaviors). Recent findings indicate that deficits in behavioral flexibility and response inhibition each uniquely contribute to higher-order RRBs, including insistence on sameness and compulsive behaviors [10], suggesting these cognitive control abilities represent distinct targets for family studies aimed at identifying endophenotypic markers associated with ASD.

Individuals with ASD show reduced behavioral flexibility characterized by an impaired ability to maintain new behavioral responses after previously reinforced responses are no longer rewarded [11-13]. They also show a reduced ability to withhold behavioral responses and use cognitive strategies to proactively delay response onset during tests of response inhibition [14, 15]. These cognitive control deficits are associated with more severe ASD symptoms including stereotyped speech and repetitive behaviors [11, 16, 17]. Thus, behavioral inflexibility may contribute to perseverative response patterns such as repetitive questioning despite attempts at redirection. Likewise, reduced response inhibition may contribute to patients seeking out strong interests even when these interests are contextually inappropriate.

Several studies have documented deficits in behavioral flexibility and response inhibition abilities in unaffected first-degree relatives of individuals with ASD, suggesting these deficits may serve as dimensional traits linked to familial risk [18-21]. Studies of these traits primarily have used traditional neuropsychological tests (e.g., Wisconsin Card Sorting Task, Stroop) that assess multiple cognitive processes simultaneously, making it difficult to determine the contributions of specific cognitive processes, especially for parents whose deficits are subclinical. Family trio studies examining inter-relationships of discrete cognitive control impairments among biological relatives of ASD probands can help determine whether these disorder-relevant impairments represent familial traits associated with ASD. This approach also can assess the extent to which cognitive control deficits covary with subclinical features in unaffected relatives to better understand inter-generational transmission of behavioral traits associated with ASD.

In the current study, we examined behavioral flexibility and behavioral response inhibition in probands with ASD and their biological mothers and fathers using tests that we previously validated in independent ASD samples [11, 15]. Based on our prior studies, we hypothesized that both probands and parents would show more errors than typically developing controls on tests of behavioral flexibility and response inhibition. Consistent with our hypothesis that specific cognitive control deficits represent separate neurodevelopmental risk pathways associated with ASD, we predicted that both behavioral flexibility and inhibition impairments would inter-correlate among probands and their parents, but that they would not inter-correlate with each other. To determine the extent to which cognitive control abilities tracked with core clinical and subclinical issues in probands and parents, behavioral flexibility and response inhibition were examined separately in families with BAP features (BAP+) and those without (BAP-). We predicted that reductions in behavioral flexibility and response inhibition would be greater for BAP+ parents and their offspring.