Meandering on Manifolds: The Neural Geometry of Stories Over Time

Stories have shapes: a comedy rises toward joy; a tragedy falls into loss; a hero leaves home, descends into danger, and returns transformed. We usually mean this metaphorically—but inside a language model, the shape of a story may be visible more literally.

As an LLM reads sentence by sentence, its internal activations move through conceptual space, tracing a wandering path that reflects the model’s evolving sense of what kind of story it is in.

Our previous posts focus only on static geometry in neural representations, during a single forward pass of a language model. In contrast, this post explores neural geometry over time, using a simple case study: how language models track emotional dynamics as they read short stories.

The geometry of human emotion in LLMs

Before we get to temporal dynamics, we need to understand the (static) neural geometry of emotions in language models. Like the other concepts we’ve previously discussed in this series, emotions are represented along a curved structure in the activation space of large language models (LLMs).

Prior research has established that LLMs represent emotion in a geometrically meaningful way. For example, like humans, LLMs represent emotions hierarchically, with emotions like “happiness”, “joy”, and “delight” falling under one branch of the tree, and emotions like “anger”, “rage”, and “resentment” falling under another branch.Zhao, B., Okawa, M., Bigelow, E. J., Yu, R., Ullman, T., & Tanaka, H. (2024). Emergence of hierarchical emotion representations in large language models.

Recent work has also shown that LLMs organize emotions in a way that mirrors the classic valence-arousal model from psychology.Choi, B. J., & Weber, M. (2026). Latent structure of affective representations in large language models.
Sun, L., Yan, L., Lu, X., Lee, A., Zhang, J., & Shao, J. (2026). Valence-arousal subspace in llms: Circular emotion geometry and multi-behavioral control. arXiv preprint arXiv:2604.03147. The valence-arousal model organizes human emotions along two dimensions: valence (how positive or negative is the emotion?) and arousal (how energetic is the emotion?).

We found similar results using Llama 3.1 8B, a small LLM. Below, we compare how six basic emotions are mapped in valence-arousal space, using human-judgment data (i.e., according to real people), versus a manifold of emotions in Llama’s activations:

This makes sense, because emotional humans wrote the text we use to train LLMs, and consequently the geometry of human emotion is recapitulated in LLM behavior. You are what you eat!

LLMs track emotion dynamics in stories

Putting aside the question of neural geometry for a moment, we might wonder how LLMs update their understanding token by token and sentence by sentence, across an entire conversation.

This is not a new question! Prior work has studied how LLMs update their beliefs through in-context learning, in the case of both structured few-shot learningXie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.
Schubert, J. A., Jagadish, A. K., Binz, M., & Schulz, E. (2024). In-context learning agents are asymmetric belief updaters. as well as less-structured input data, like stories or chains of thought.Bogdan, P. C., Macar, U., Nanda, N., & Conmy, A. (2025). Thought Anchors: Which LLM Reasoning Steps Matter?. arXiv preprint arXiv:2506.19143.
Bigelow, E., Holtzman, A., Tanaka, H., & Ullman, T. (2025). Forking paths in neural text generation. In International Conference on Learning Representations. Recent work has also studied the internal representations that LLMs form and change during these kinds of tasks,Bigelow*, E., Wurgaft*, D., Wang, Y., Goodman, N., Ullman, T., Tanaka, H., & Lubana, E. S. (2025). Belief dynamics reveal the dual nature of in-context learning and activation steering.
Lubana*, E. S., Rager*, C., Hindupur*, S. S. R., Costa, V., Tuckute, G., Patel, O., ... & Mueller, A. (2025). Priors in Time: Missing Inductive Biases for Language Model Interpretability. arXiv preprint arXiv:2511.01836. although there is still a wide gap in our understanding of the relationship between structured conceptual representations and how models dynamically update their beliefs at inference time.

In this work, we examine the question of how representations change over time with a case study: how LLMs keep track of a story’s emotional dynamics, from tragic falls, to heroic redemptions, to unexpected twists.Kurt Vonnegut (1995). Shapes of stories. As expected, they seem to have a dynamically shifting sense of emotion from sentence to sentence.

To understand how LLMs track the emotional dynamics of a story, we simply ask them.Can we trust that the LLM’s responses reflect its true beliefs? In this case, we think so—in the next section, we probe the LLM’s internal representations without asking it about emotions, and these probes are very accurate.

After each sentence, we prompt the LLM to report the degree of surprise, disgust, anger, happiness, sadness, and fear in the story thus far on a scale from 0 to 10. These six scores form a vector that serves as a behavioral readout of what the LLM believes has happened up to a given moment.

We plot these behavioral readouts in the demo below, where you can see how the LLM follows the arc of a story. For each story, the demo plots how the model reports each of the six basic emotions at each moment.

Why do models track these dynamics? During training, LLMs read millions of storiesAlex Reisner (2025). The Unbelievable Scale of AI’s Pirated-Books Problem. while learning to predict each word from everything that came before. This task may seem simple, but sometimes the arc of a story hinges on a single word. Imagine the understanding of story, character, and emotion needed to successfully predict the next word in the line “No, I am your ___” from the screenplay of Star Wars: The Empire Strikes Back.

Combining space and time: trajectories through activation space

Armed with an understanding of both the neural geometry of emotions and how LLMs track emotional dynamics over the course of a story, we can now ask: how are emotional dynamics represented geometrically?

If you’ve read the previous posts in this series, you won’t be surprised that the dynamic emotion tracking that we can elicit via explicit prompting is also present in the model’s internal representations. As the emotional tenor of a story changes, LLM activations trace out meandering paths along the manifold of emotions.

How do we know this? In addition to asking the LLM about emotions verbally, as we did in the previous section, we also harvest the internal activations from the last token of each sentence in a story (without asking it anything). These activations serve as a snapshot of what the model represents after reading the story so far. To model the geometry of these LLM representations, we fit a manifold to these activations. In the demo below, we show how stories trace out trajectories along this representation manifold.

Conclusion

While emotions in stories are a simple case study, they provide a window into a broader phenomenon: when viewed over time, models’ internal activations appear to trace trajectories through conceptual belief spaces.

This neatly synthesizes two existing frameworks. In the Bayesian view of in-context learning, each new sentence provides evidence that reweights latent concepts according to how well they explain the context so far. In the conceptual-spaces view, those concepts are not arbitrary hypotheses, but points or regions in a geometric space where distances encode similarity.Gärdenfors, P. (2000). Conceptual Spaces: The Geometry of Thought. MIT Press. Lifting this geometry into a conceptual belief space lets us view each moment in a story as a belief state—a distribution over the underlying conceptual space—and the story itself as a path through such states.

More work is needed to characterize belief updates and temporal dynamics in LLMs. We believe that we need to understand these temporal dynamics, in tandem with neural geometry, in order to get a fuller picture of how models work – and to be able to debug and design them with intention.

Read the full paper →