Today’s LLMs including the most recent and advanced models from the frontier labs are bad at context. Extraordinarily bad at context. And bigger context windows aren’t making them any better.
That can be a shocking or somewhat awkward statement to start an article, and I’ll certainly get pushback from many on this one. That’s fine. There is a distinction here that is becoming increasingly important as we move from humans chatting with models to models participating in actual business processes.
Business context is not model context. One is far more important than the other inside the enterprise.
To break it down, I’m talking about Context Competence and LLMs simply don’t have that on their own. Context Competence is supplied by humans or external systems that have codified the business context, business rules, operating conditions and, in many cases, good old-fashioned tribal knowledge. This is where the state conversation comes into play, along with the follow-on discussion around Secure State and Conditional State Propagation in agentic AI. It’s about business context, business objectives and the competence needed to actually automate the business.
And before everyone tells me how good the latest models are with million-token context windows, there’s plenty of evidence that context capacity does not equal context competence. A 2025 EMNLP study found that performance across five open and closed models degraded as input length increased, even when the models could perfectly retrieve the relevant information … that’s a big if.
So yes … bigger isn’t necessarily better, especially in business context.
Moving Beyond the Hype
This article has actually sat in dry dock for a while, partly consumed by all the current noise in the market around rogue agents, swarms and hive minds. The hive mind discussion is an interesting one (although unfortunately anthropomorphized to death) because it gets to the main topic of this article and that is Secure State.
State maintenance and state management are discussed extensively in distributed computing, enterprise workflows, automation, service buses, RPA and any number of other technology environments. None of this is new. The problem is that maintaining state across federated and distributed systems and processes is extraordinarily difficult. Securing it is equally difficult.
And guess what? Enterprise agentic AI workflows very quickly become federated and distributed processes.
So to be clear, this is not a new problem whatsoever. Autonomous agents have simply escalated the problem to a point where some of the old assumptions no longer hold.
State starts with your data, but it isn’t just your data. State is the current condition of the business process and the execution. It includes the original data, all the permutations and derived data created throughout the flow, prior decisions already made, approvals, identity, permissions, policy conditions, provenance, intermediate results, dependencies and whatever else determines what is true right now and what is allowed to happen next.
Now start splitting that workflow across agents. 😳
It is no longer enough to simply manage state. You have to securely manage it, determine what state is authoritative, and ensure the right portions of that state are effectively and securely propagated to agents and child processes across highly parallel and asynchronous execution (oh … the infamous swarms).
Where do you maintain that broader federated state? It better not be inside an agent because you have effectively placed all your eggs in one very unpredictable basket. An agent should maintain the isolated state necessary for its narrow and hopefully singular focus. The broader business state needs to be maintained under authoritative control outside the agent, with the control plane determining how that state is governed and used.
State is Not Context.
Data scientists will routinely and almost by default mix state and context together, but they are not the same thing. State is most certainly not a markdown file or scratchpad either … and those reside all over the place. They may contain representations of state and they can be useful as isolated short-term working memory, but that also makes them an exposure surface that needs to be locked down just like logs, traces and every other artifact created by an agent.
This is one of the reasons I pondered this article for such a long time. State in a business workflow should never be confused with the conversation context being fed to a model or chatbot. That would be a little like recording every conversation around the office watercooler and feeding all of it into the employee responsible for approving a wire transfer … and I’ll let you work out the business ramifications of that one.
Nevertheless, this is where the model scientists have a legitimate argument. The latest models can be very good at precision and pulling the correct reference from the context you provide them. Ahem … the context you provide them!!
That’s the key. Context Competence resides outside the model.
Context Rot
Which gets us to another term that should become increasingly important in enterprise AI … Context Rot.
The research definition generally describes context rot as the deterioration in model or agent reliability as context grows. The 2026 LOCA-bench work is particularly relevant because it looks beyond static retrieval and into long-running agents where the context dynamically grows as the agent explores, follows plans, gathers information and takes actions. Reliability therefore deteriorates as that context grows, which the researchers explicitly describe as context rot.
But for enterprise agentic systems, I think we need to take that definition much further as that is rooted in benchmark tests and not necessarily in real-world business context competence scenarios.
Context can rot simply because it gets too long. It can also rot because irrelevant information accumulates, because old information remains after authoritative state has changed, because parallel processes create conflicting information, because external information poisons or corrupts the working context, or because the correct state simply wasn’t propagated from one execution to the next.
And then there is time. That wonderful world of temporal and longitudinal intelligence that trips-up everyone.
Some of the workflows we deal with at CharliAI can execute for hours and in some cases naturally extend into days, especially where there is agent-to-human communication. An agent does some work, hands something to a human for review or approval, waits, and then picks the workflow back up later. Meanwhile other processes can continue, data can change, permissions can change and the business can simply move on.
The agent wakes back up several hours later. Which version of reality is it operating against? Did I mention time?
That’s where Context Rot stops being an interesting benchmark problem and becomes a business control problem.
An approval may have been revoked. A risk threshold may have been crossed. A customer condition may have changed. Sensitive data may have been reclassified. A parent process may have made a decision that its child processes have not yet received. An agent can therefore be operating perfectly against the context it was originally given while that context is already obsolete somewhere else in the business.
This is also where the discussion shifts from state management to state propagation.
Conditional State Propagation
Propagation is not synchronization. You cannot simply synchronize everything everywhere and call the problem solved. In fact, synchronization can create an entire world of security hurt in agentic flows.
You have to determine what state propagates, to whom, under what conditions, with what authority, for what purpose and whether that state is still valid when the receiving agent eventually acts on it.
Not every agent is authorized to see everything, even if it never intends to act on the information. Agents write information to working memory, scratchpads, logs, tools and downstream systems. They can misuse information, expose it or simply allow it to influence future actions in ways nobody intended.
That is a data exposure problem … generally frowned upon (a touch of sarcasm here).
Agents therefore need contracts around data ingress and egress. Those contracts need to determine what information the agent can receive, what it can produce, where that output can go and under what authority. The control plane enforces those contracts, assigns the appropriate credentials and permissions, and monitors the resulting behaviour against the profile of what that agent was actually supposed to do.
As processing moves from one agent to another, propagation of state needs to be purposeful. Policies need to govern partial state, sealed state, redacted state, transient state and state that should simply never leave the execution boundary where it originated.
That propagation also needs Context Competence. Policies cannot simply understand data types. They need to understand enough about the business process to know why the information exists, how it is being used and whether the conditions that allowed its use remain valid. And please don’t solve this by creating some god agent with superpowers and access to absolutely everything. That just concentrates authority, information and failure into one extremely convenient target.
This is also where my earlier You Didn’t Build Skynet, You Wrote a Bad Script argument comes back into play.
Structured State
I also need to point out that state is structured. It has to be structured and you cannot leave it up to the LLM to continuously reconstruct it from a conversation history.
Some might point out that chatbots are stateful. They remember that we had a conversation yesterday, and in some impressive chatbot implementations they can remember last week or last month. But that’s not even close to the same thing, for so many reasons.
If a business workflow fails halfway through and the entire operational state was effectively represented by a string of accumulated chat history, what exactly do you recover from? Where is the checkpoint? Which decision was authoritative? Which approval remained valid? What downstream actions had already occurred? And did I mention time?
You shouldn’t have to ask an LLM to reread the conversation and tell you what probably happened. As you might recall from the beginning of the article, LLMs are notoriously bad at context.
When state is maintained independently of the model as structured information, whether that is structured records, an event history, an explicit state machine or other deterministic representation, the workflow can be recovered from known checkpoints. Processing can resume from a known condition instead of starting again. Decisions can be replayed. Changes can be traced. And most importantly for what we do at CharliAI, you can produce forensic evidence showing exactly what the state was, what changed, who or what changed it and what happened as a result.
State is most definitely structured.
Conversational Modelling vs State
There is another uncomfortable question buried in all of this. Could the conversational modelling that makes modern LLMs so compelling for humans actually become detrimental when it is used as the operating mechanism for production business workflows?
I think it can.
We already know about sycophancy, where models can favour agreement patterns over inconvenient facts. We know about the lost-in-the-middle problem where relevant information can become harder for models to use depending on where it sits in a long context. None of that makes LLMs unsuitable for business. Quite the opposite. They are extraordinarily powerful.
But it does mean we should stop treating conversational memory as operational state. That is great when there is a human sitting in front of a chatbot. It is lousy as the system of record for a business workflow that requires reasonable determinism, recovery, security and inherent trust in the outcome.
Governance vs [AND] Orchestration
This gets back to something I have written about before and an area where I get pushback fairly regularly … or rolling of the eyes. The cybersecurity industry can no longer treat Governance and Orchestration as completely separate responsibilities inside agentic systems.
People looking at CharliAI from the outside sometimes ask why we do orchestration. The assumption is that we must somehow be trying to become another automation platform. But we aren’t.
The reason is much more fundamental. If you need to secure every piece of information through every interaction, you have to be present where those interactions occur. Human to agent. Agent to agent. Agent to tool. Agent back to human. Parent process to child process. One execution domain into another.
Governance that cannot influence the actual handoff is just policy sitting beside the process.
This also comes back to the argument I made in AI’s “Paper Governance” Blind Spot.
This is why Secure State and Conditional State Propagation matter. The control plane has to understand what state is authoritative, what can move, where it can move, under whose authority it can move and whether the conditions that allowed it to move remain true.
In the end, never trust authoritative state or state propagation to an agent or to the LLM that is the agent’s best friend and partner in crime. The model should reason over the state it is permitted to receive. It should not get to decide what the authoritative state of the enterprise is.
Where does that state belong? Under the control of the control plane of course … but that control plane needs to be architected with real rigour and discipline around policy decisioning, policy enforcement, identity, provenance and Context Competence.
Otherwise it’s fingers crossed and hope for the best.