A group of Johns Hopkins computer scientists has proved that in-context learning (ICL) is a general capability that emerges whenever a large AI model is trained to predict the next element with any sufficiently complex sequence data—not an ability only tied to human language as previously believed.
The work—co-authored by Daniel Khashabi and Anqi “Angie” Liu, both assistant professors of computer science; Bloomberg Distinguished Professor Michael Schatz; PhD students Aayush Mishra and Mahler Revsine; and undergraduate student Nathan Breslow—appears in Transactions on Machine Learning Research.
Modern large language models (LLMs) use ICL to pick up patterns from just a few examples in a prompt and apply them to new problem instances without specifically being trained for them. Most prior research has assumed the emergence of ICL is tied to human language, but the Hopkins researchers weren’t so sure.
To test their theory, the computer scientists chose to evaluate DNA sequence data, which are highly complex but distinct from human language. The team compared the popular Evo2 genomic model with a standard LLM on learning patterns present in these data—and, just like the LLM, Evo2 showed the same steady performance improvement as more examples were provided, confirming that ICL is a general property of large models trained on structured sequence prediction.
“This shifts our understanding of what makes modern AI systems powerful: The same kind of flexible, example-driven reasoning of LLMs could emerge in models trained on many other types of data—DNA, proteins, chemical structures, and beyond,” says Khashabi. “This opens the door to AI systems that can quickly adapt to new scientific problems from just a few examples. In genomics, for instance, models could potentially learn to recognize new patterns in genetic data on the fly, accelerating progress in areas like disease diagnosis, personalized medicine, and biotechnology.”
Plus, this capability emerges entirely for free in large models—nobody explicitly trained these systems to reason by analogy or learn from examples; it simply arose as a consequence of training on large amounts of complex sequential data.
This work opens a window into the foundations of computational reasoning itself. Understanding why and how this happens brings us closer to answering one of the deepest questions in AI: Where does the ability to reason come from, and how can we build systems that do it reliably and safely?
“We’ve demonstrated for the first time that ICL can emerge organically outside of human language, but there is a rich and exciting research agenda that follows from this,” Khashabi explains. “Beyond genomics, our findings motivate a broader search for ICL in other complex sequential domains, such as time series data, climate models, physics simulations, and even chess games.”