Due to this enormous data gap, researchers have been skeptical that recent AI advances can tell us much about human learning and development. An ideal test for demonstrating a connection would involve training an AI model, not on massive data from the web, but on only the input that a single child receives. What would the model be able to learn then?
A team of New York University researchers ran this exact experiment. They trained a multimodal AI system through the eyes and ears of a single child, using headcam video recordings from when the child was six months and through their second birthday. They examined if the AI model could learn words and concepts present in a childโs everyday experience.
Theirย findings, reported in the latest issue of the journalย Science, showed that the model, or neural network, could, in fact, learn a substantial number of words and concepts using limited slices of what the child experienced. That is, the video only captured about 1% of the childโs waking hours, but that was sufficient for genuine language learning.
โWe show, for the first time, that a neural network trained on this developmentally realistic input from a single child can learn to link words to their visual counterparts,โ says Wai Keen Vong, a research scientist at NYUโs Center for Data Science and the paperโs first author. โOur results demonstrate how recent algorithmic advances paired with one childโs naturalistic experience has the potential to reshape our understanding of early language and concept acquisition.โ
โBy using AI models to study the real language-learning problem faced by children, we can address classic debates about what ingredients children need to learn wordsโwhether they need language-specific biases, innate knowledge, or just associative learning to get going,โ adds Brenden Lake, an assistant professor in NYUโs Center for Data Science and Department of Psychology and the paperโs senior author. โIt seems we can get more with just learning than commonly thought.โ
Vong, Lake, and their NYU colleagues, Wentao Wang and Emin Orhan, analyzed a childโs learning process captured on first-person videoโvia a light, head-mounted cameraโon a weekly basis beginning at six months and through 25 months, using more than 60 hours of footage. The footage contained approximately a quarter of a million word instances (i.e., the number of words communicated, many of them repeatedly) that are linked with video frames of what the child saw when those words were spoken and included a wide range of different activities across development, including mealtimes, reading books, and the child playing.
Share this article
NYU researchers analyzed a childโs learning process captured on first-person videoโvia a light, head-mounted camera (similiar to the one here)โand trained a multimodal AI system through the eyes and ears of a single child. Video by Jonathan King/NYU’s Office of Public Affairs.
The NYU researchers then trained a multimodal neural network with two separate modules: one that takes in single video frames (the vision encoder) and another that takes in the transcribed child-directed speech (the language encoder). These two encoders were combined and trained using an algorithm called contrastive learning, which aims to learn useful input features and their cross-modal associations. For instance, when a parent says something in view of the child, it is likely that some of the words used are likely referring to something that the child can see, meaning comprehension is instilled by linking visual and linguistic cues.
โThis provides the model a clue as to which words should be associated with which objects,โ explains Vong. โCombining these cues is what enables contrastive learning to gradually determine which words belong with which visuals and to capture the learning of a childโs first words.โ
After training the model, the researchers tested it using the same kinds of evaluations used to measure word learning in infantsโpresenting the model with the target word and an array of four different image options and asking it to select the image that matches the target word. Their results showed that the model was able to learn a substantial number of the words and concepts present in the childโs everyday experience. Furthermore, for some of the words the model learned, it could generalize them to very different visual instances than those seen at training, reflecting an aspect of generalization also seen in children when they are tested in the lab.
โThese findings suggest that this aspect of word learning is feasible from the kind of naturalistic data that children receive while using relatively generic learning mechanisms such as those found in neural networks,โ observes Lake.
