Tuberous sclerosis complex (TSC) is a multisystem syndrome classically associated with the occurrence of focal brain dysplasias. We used structural magnetic resonance imaging to test for neuroradiological abnormalities in TSC (tubers, white matter lesions, and subependymal nodules) and to explore the relationships between these lesions and computational morphometric abnormalities of gray and white matter distribution. We tested memory function in TSC and investigated the relationship between memory function and both morphometric variation and lesion load. Patients demonstrated deficits bilaterally in volume of subcortical gray matter regions including thalamus, basal ganglia, insula, and cerebellum, as well as white matter deficits bilaterally in intrahemispheric tracts. Morphometric deficits could not be explained as local effects of lesions. Patients demonstrated deficits in executive working memory and recall memory, sparing recognition. Structure–function mapping showed long-term and working memory function was positively correlated with gray matter density (in thalamus, caudate nucleus, and frontal cortex) but not with lesion load. The neuroanatomical endophenotype of TSC is more extensive than previously recognized and comprises abnormalities in the distribution of gray and white matter in addition to classical lesions. Normal intelligence quotient patients with TSC show a profile of long-term and working memory impairment that is related to gray matter deficits in thalamus and basal ganglia components of fronto-striatal circuits.