Recently I made an interesting discovery while working with AI agents: they can’t sense the passing of time.
I feel that this is a pretty important problem for agents.
Agent vs Harness
To understand this problem, we need to first define what an agent is, and what the harness is. Here are my definitions for agent and harness:
Agent: The execution loop that wraps around the LLM.
The agent includes the context in which system prompt, user messages, assistant messages and tool calls are stored.
The agent input and output are both 1-dimensional tokens (1D).
The agent connects to the outside environment via the Harness.
Harness: The interface that the agent uses to connect to the outside environment.
The harness includes a tool call interface that includes the tool manifest and tool execution environment.
The harness includes a skills interface that includes the skills manifest and reading the skills into the agent’s context.
With the definitions out of the way, here are 3 cases where the agent can’t sense the passing of time:
1. Token Generation Speed
When generating a response (text or tool call input), the agent can’t tell how fast the underlying model is generating tokens.
From an LLM’s perspective, it only receives the tokens one by one (stateless) and it doesn’t know how fast the tokens are being generated. It can’t tell if the tokens per second (tps) is 1 or 1000. Since the agent is just a wrapper around the LLM, it also does not know the tps.
This can negatively affect the agent’s ability to perform tasks related to speed or time management.
The user might expect the task to be done within 10 minutes, but the agent can’t tell if it can finish the task. Even if the model knows how many tokens it would take to complete the task, it won’t be able to estimate the time it would take, simply because it does not know the model tps.
For example, a task that requires 20k output tokens would take about 6 minutes for a model with 50 tps, but would take 30 minutes for a model with 10 tps.
2. Tool Call Duration
When using tools, an agent also cannot sense how long a tool call takes, or how long it has been running. This can have serious consequences.
When a human executes a command, there is typically an expectation of how long it would take, and if the command takes too long or finishes too quickly, the human would catch that signal and react accordingly. For example, if a rm command takes too long, the user would probably cancel it first. And if an intensive benchmark terminates too quickly, it would signal to the user that something probably went wrong.
However, the agent does not get this kind of signal. It would only know a rm command took 5 minutes after it has finished, or timed out. That’s obviously not ideal.
The reverse is also worrying. Say the agent ran npm run heavy-data-processing, which exits silently with exit code 0. It would have no idea how long it took. Maybe it took 1 second where the expectation was actually 2-3 minutes. The agent might assume it succeeded, but in reality it ran into some error that was silently dropped.
A similar problem can happen for event subscription notifications. Say an agent is subscribed to an event that is supposed to take 10 minutes to arrive, but instead it arrives the next second (due to a bug), the agent would have no way to know that.
3. Current Session Duration
The agent interacts with the outside environment via the harness, but the environment is ever-changing while the agent runs or waits for the next prompt.
Sometimes due to errors, rate limits, or simply humans taking a break, agent sessions will be left unattended and the agent has no idea how much time has passed before the next prompt. It could be 5 minutes later, or it could be 3 days.
This has negative consequences for how well the agent performs, if the task depends on interactions or assumptions about the outside world.
For example, an agent that has access to task tracking software might assume that tasks that were in the backlog are still in the backlog during the session (a reasonable assumption). But if the session was not attended for a few days, other tasks might have already completed while the agent was waiting, leading to conflicts or tasks being worked on with outdated information.
Another example is tasks that are tied to specific times and dates or deadlines. An agent working on a task with a deadline in 5 days might assume that each subsequent prompt is within a short window (say within a day), but in fact the subsequent prompts might come 3 days after, leading the agent to incorrectly assume that it still has 4 days left, whereas in fact it only has 2 days left.
As a result, it might miss an important timing window (ask for approval at least 3 days before), or not have sufficient time to complete the task (say it depends on a data cleaning job that takes 3 days).
Root Cause and Possible Solutions
I believe the root cause for all 3 cases is the fact that LLMs, and by extension agents, live in a 1-dimensional world of tokens, without experiencing time as a dimension. They cannot sense the passing of time like humans do, and therefore any tasks related to relative duration could be tricky for them.
There are two workarounds on different abstraction levels:
Agent-initiated: the agent proactively asks for the time via tool calls:
0. user_prompt -> agent
1. date_tool_call = agent.generate_tool_call_input("bash: date")
2. start_time = harness.tool_call(date_tool_call)
3. response = agent.generate_text_response(user_prompt)
4. date_tool_call = agent.generate_tool_call_input("bash: date")
5. end_time = harness.tool_call(date_tool_call)
6. (start_time, end_time) -> agentHarness-initiated: the time is passively injected into the agent’s context by the harness:
0. user_prompt -> agent
1. start_time = harness.get_date()
2. response = agent.generate_text_response(user_prompt)
3. end_time = harness.get_date()
4. (start_time, end_time) -> agentHowever, both have their drawbacks:
Agent-initiated adds two more tool calls for each measurement, which dramatically increases cost (2 cache reads of the entire context).
Harness-initiated injects timing deterministically, even if it is not necessary or helpful.
A more fundamental solution is to embed the agent into a physical entity (a robot), and continuously feed the outside environment information as data to the agent at a predictable fps, much like how humans work with our eyes and other senses.
In this way, the continuous inflow of data acts as an approximation for flow of time: if the fps is 10, the agent can deduce naturally that 10 such inputs means 1 second has passed in the real world.