Notes on working with AI
Why long chats get slow, and what to do about it
A language model does not remember your conversation. It rereads it. That one fact explains why a three-hour chat is slower, uses more of your limit, and loses early details.
The model does not remember. It rereads.
Most people picture a chat as a continuous memory. It is not. A language model keeps nothing between messages. On every turn, the whole conversation so far goes back in as input, and the model produces the next reply from all of it.
That has a direct consequence. The hundredth message is not answered in light of the last one. It is answered against the other ninety-nine, plus your new one.
What happens on the server
The work splits into two phases. First the model reads the entire input at once. Then it writes the reply one token at a time, and each token depends on the one before it, so this part cannot be parallelized.
To avoid recomputing everything for each new token, the system stores an internal representation of every token it has already processed. This is called the KV cache, short for key-value cache. It is the model's working memory for the current conversation, and it lives in GPU memory next to the model itself.
The cache grows with the length of the chat. A longer conversation takes more memory, and that memory is shared with many other users' requests on the same hardware. This is why long context is expensive to serve, and why providers build caching and scheduling around it.
What it means for you
Three practical effects, in order of certainty.
Cost. Anthropic's own usage tips for claude.ai list current conversation length as a factor affecting usage limits, and note that caches expire after a period of inactivity. A long chat drawing on a cache that has gone cold is the expensive case.
Speed. More input to read before the first word appears, so replies start later as the chat grows.
Attention. This one is a tendency, not a guarantee. When a lot of material sits above the current question, details from the beginning tend to carry less weight. If a decision from message 8 matters at message 150, do not assume it is still in focus.
The position: a chat is a working session
Treat a conversation as a session you finish, not an archive you keep adding to. Start a new chat when the topic shifts, or when the current one has grown long and you notice the answers drifting.
The tradeoff is that a fresh chat knows nothing. The fix is a handoff summary: the decisions made, the questions still open, the key numbers, the files used, and the next step, written so it can be pasted as the first message of the new chat. A summary loses nuance, so read it before you rely on it.
Material you reuse goes somewhere else. Anthropic recommends Projects for anything you reference repeatedly. Standing instructions, background documents and style guides belong there, so you stop pasting them into every conversation.
Where this breaks
Some work really does need one long thread, such as editing a single long document section by section. There the answer is not to restart but to keep the thread focused: one task, one document. And a summary is only as good as the model's grasp of what mattered, so check that the numbers survived.
What to do tomorrow
Next time a chat passes an hour or the topic changes, paste this at the end of it:
Summarize this conversation so I can paste the summary into a new chat and continue without losing anything. Include the decisions made, open questions, key numbers, files we used, and the next step. Write it as a single message that starts with: We are continuing earlier work.
Open a new chat, paste it as the first message, and compare the speed of the replies.