Claude: The Best Coding Model

Anthropic has held the crown for best coding model for nearly two years, and no other lab has managed to take it away. It started with Sonnet 3.5, which beat GPT-4o on most benchmarks at launch and became the default model for major tools like Aider and Cline. Then Sonnet 4, Sonnet 4.5, and now Opus 4.6.

Claude models currently hold the top two spots on SWE-bench Verified. At each generation, Anthropic has maintained or widened its lead over competing models from OpenAI, Google, and open-weight alternatives.

This means that Anthropic always had the best model internally to accelerate research and engineering efforts and move faster than other labs, months before the model’s public release.

Claude Code: The Original Agent Harness

Claude Code was the first CLI-based coding agent that worked well enough to change how developers build software. It has since become one of the most widely used coding agents. It demonstrated that AI labs can make great developer tools. The success of Claude Code forced the entire industry to respond.

The evidence that competitors like OpenAI are following Anthropic’s lead is clear:

• Codex CLI adopted the same terminal-based agent paradigm that Claude Code pioneered

• The session usage limit implementation in Codex mirrors Claude Code’s approach

• The generous $20 subscription plan follows Anthropic’s token subsidy model

• OpenAI shifted its entire product strategy toward Codex after seeing Claude Code’s traction

Despite these fast-follow efforts, Claude Code remains ahead in usage numbers. This is clear evidence that Anthropic still has the lead in building and designing agents.

TypeScript: The Right Language for Agents

Claude Code is written in TypeScript, and this architectural decision gives Anthropic a compounding advantage.

TypeScript is the most-used language on GitHub. It is also what all major LLMs are most heavily trained on, meaning Claude and competing models understand TypeScript code better than code in any other language.

Building the agent harness in TypeScript creates a positive feedback loop. The model excels at understanding and modifying the very language the harness is built in. This makes iteration faster and more reliable.

The Claude Agent SDK, also offered in TypeScript, takes this further. The Agent SDK is a wrapper around Claude Code, which proves that Claude Code works as a general-purpose agent harness capable of handling tasks across different domains. As more developers build custom agents with the Agent SDK, this lead compounds.

Pioneering Agent Paradigms

Anthropic has a pattern of introducing new agent paradigms that become industry standards.

• MCP (Nov 2024):, MCP defined how AI agents communicate with external services. It has since been adopted by virtually every coding tool, including Codex, Cursor, and many others.

• Sub-agent(July 2025): A primary agent delegates tasks to specialized forked agents to optimize context usage. Anthropic pioneered this in Claude Code and competitors later adopted it.

• Hooks (Sep 2025): User-defined commands that execute at specific points in the agent’s lifecycle. Hooks solve a fundamental problem with LLM-based agents: the model is probabilistic, but certain actions need to be deterministic.

• Skills (Oct 2025): Modular, filesystem-based capabilities that extend agents with domain-specific expertise. Anthropic published the Agent Skills open standard, which has been adopted by tools including Codex, Cursor and others.

• Plugins (Oct 2025): Containers that package tools, permissions, and metadata together, providing a structured way to extend agent functionality beyond what the base harness offers.

Each paradigm defines a fundamentally new way for agents to interact with their environment, and Anthropic has been first to market on nearly all of them.

Reinforcement Learning (RL) Advantage

Anthropic’s lead in agent paradigms creates a structural RL advantage. When Anthropic invents a new paradigm like Skills or a new sub-agent orchestration pattern, it can begin RL training immediately.

This happens months before the paradigm is publicly released. By the time other labs learn about the paradigm and start incorporating it into their own training pipelines, Claude is already RL-optimized for it.

This head start repeats with every new paradigm Anthropic introduces, and the gap compounds over time. Other labs are always playing catch-up on two fronts simultaneously: first implementing the paradigm, then training their models to use it effectively.

Safety and Mechanistic Interpretability

Anthropic was founded with safety as a core mission. Anthropic’s research in mechanistic interpretability (mechinterp) is the most advanced in the industry. Mechinterp aims to understand what is happening inside a model’s neural network, going beyond surface-level observation of inputs and outputs.

MIT Technology Review named mechanistic interpretability one of its 10 Breakthrough Technologies of 2026, largely on the strength of Anthropic’s work. Anthropic has published landmark research including Tracing the Thoughts of a Large Language Model and open-sourced circuit tracing tools for the research community.

This matters for a practical reason: If models start failing at certain tasks, labs without mechinterp capabilities would not know why or how to fix it. They can only hope that scaling or more training data solves the problem. Anthropic’s mechinterp research gives it a deeper understanding of model behavior, which translates into more targeted improvements and fewer blind spots.

Safety research also feeds directly into product quality. Techniques developed for alignment and safety, such as Constitutional AI, make Claude more reliable and predictable in practice. Safer models tend to also be more useful. They follow instructions more faithfully and are less likely to produce unexpected behavior during long autonomous runs.

Model Character and Taste

There is a quality to Claude that is hard to quantify but immediately noticeable: it feels more pleasant to work with. Working with Claude feels like working with a thoughtful collaborator. Models from other labs tend to feel like generic autocomplete engines by comparison.

Amanda Askell, who leads character training at Anthropic, has shaped how Claude communicates and reasons about its responses. The result is a model that developers genuinely enjoy working with, one that goes beyond producing correct output.

When you spend hours per day interacting with a coding agent, the quality of that interaction directly affects productivity and satisfaction. Other labs have not invested in model character with the same care, and their models tend to feel interchangeable as a result.

Leadership

The moat of an AI lab ultimately comes down to the person leading it.

Dario Amodei combines deep technical understanding of the models with strategic clarity about where the industry is heading. He also has the organizational ability to execute on both. He articulates a clear vision for what safe, powerful AI looks like, and Anthropic’s product decisions align with that vision.

The AI race requires making correct bets on training approaches, product strategy, safety tradeoffs, and market positioning simultaneously. Leadership quality determines how well a lab handles these interconnected decisions.

Substance Over Marketing

The difference in how Anthropic and other labs promote their products is notable. Some labs have employees actively engaging with and amplifying product tweets, creating the perception of widespread adoption.

Anthropic takes a different approach: stay quiet, let third-party reviews and organic word-of-mouth do the talking.

This creates a perception gap. On any given day, X timelines might suggest that a competing product is dominant, when head-to-head comparisons tell a different story.

Anthropic’s bet is that developers will ultimately choose the best tool, regardless of which product has louder advocates on social media. Developers are a technical audience that evaluates products empirically. Over time, substance wins over marketing.

The Race to the Top

Some labs will win the race to the bottom, competing on price and marketing to capture commodity workloads. Anthropic is playing a different game.

In each of the areas above, Anthropic is either leading or setting the pace. The gap may narrow in individual areas, but the breadth of Anthropic’s lead across all these dimensions is what makes it durable.

Consistent execution across multiple fronts is the best predictor of who will lead the next generation of AI. By that measure, Anthropic is in a class of its own.

The agent handles everything from sourcing content to generating scripts, adding B-roll, selecting music, and composing the final videos.

Here are the 6 patterns that I learned from building this agent.

1. Decompose Work with Sub-Agents and Skills

I started off with one agent that did everything. It quickly became clear that the agent was trying to do too much at once. While the key was to break down its work, the best way to do so was not immediately obvious.

I tried separate agents, sub-agents, and skills. Ultimately, I found it useful to think about decomposition in two ways: sub-agents and skills.

I use sub-agents for completely different tasks within a larger workflow. For my project, this meant having one agent for discovering content and another for creating the video. Each sub-agent has its own system prompt and runs in a separate session.

For smaller steps within a single task, I use skills. My video creation agent uses skills like /selecting-broll to find footage or /selecting-music to choose audio. These skills are loaded only when needed and share the context with the main agent.

Like prompts, skills can be composed from parts or dynamically generated before agent invocation. This is helpful if you want to dynamically adjust the skills based on some kind of catalog or knowledge base.

2. CLI as a Universal Interface

I decided to skip building a graphical user interface (GUI) for my agent at the beginning of the project, based on my previous experience of spending too much time on GUI development.

A traditional approach, which involves a GUI for humans, and separate tools for agents, doubles the development work. I built a command-line interface (CLI) instead. This single interface could be used by both me and the agent, making development much faster and debugging far easier.

When my agent ran into an issue, I could reproduce it by running the exact same command it used, like ./cli compose. This removed the guesswork of whether the agent’s tool call was different from a human’s action. A single CLI served as the universal interface for everyone.

You can also expose different LLM models as CLI commands for various specialized tasks. For example, I use Gemini 2.5 Pro as the underlying model for the review-video command as it is better at analyzing videos than other models.

After the agent baseline stabilized, I then added a GUI for human-in-the-loop tasks like reviewing drafts and making fine-grained adjustments.

3. Guide Agents with a Status Command

Another important decision was how the agent decided what to do next. From my previous experience, isolated stages in the workflow with their own context can lead to missing context in later stages and make it hard to revise work from previous stages. I tried to let the agent figure out priorities on its own, but this led to unpredictable behavior.

The solution was a dedicated status command that guides the agent to the next action. By having the agent call status, execute the action, and then call status again, we are embedding a predictable state machine within the agent loop. The command’s output centralizes all the priority and workflow logic.

The status command outputs Next Step instructions following this priority:

1. Fix rejected video: if any rejected videos exist
2. Compose: if a clip has drafts with a human pick
3. Awaiting selection: if drafts exist but no human pick yet
4. Generate drafts: if available clips have no drafts

Now, the agent can proactively look at the status and decide what to do next, while also being flexible to whatever workflow the user wants to run via custom prompts.

4. Accept the Natural Variance of Models

I thought I could get consistent output from an LLM by refining my prompts, but it didn’t work. After weeks of trying to fix the last 15% of issues, I decided to measure the model’s variance. I ran the same prompt on the same input 10 times and found the results varied between a 76% and 88% success rate.

This shows that there is a natural ceiling for consistency with current LLMs. So I stopped chasing perfection and embraced this natural variance.

I started designing the system to work with this variance by generating multiple options (with varied prompts and parameters) and adding a human selection process to pick the best one. I had the evaluation system first pick a few promising candidates, and then I selected the best one for the next step.

5. Hybrid Validation with Code and LLM

Evaluating the output of the agent and aligning it with human preference was tricky. It took me a while to get it right. The biggest improvement in quality came when I combined LLMs with simple code during the evaluation process.

I found it was better to use the LLM for nuanced judgments and use code to enforce hard rules, and then use a weighted sum to calculate the final score. This brought the alignment with human judgment from 73% to 92%.

6. Read the Logs

Recording and reading the agent’s logs was a highly effective way to improve the agent performance. Reading about the steps that the agent took, especially when it failed, revealed problems I would not have discovered otherwise.

For example, the logs showed me that my agent was trying to compose a video three times before succeeding. The issue was an FFmpeg error that happened when too many video overlays were used. So I added a validation step to prevent this from happening again.

The logs also helped me find smaller inefficiencies, like the agent calling music list when the data was already in its context. Reading the logs regularly became a core part of my development process. It is the best way to understand what your agent is actually doing.

That’s it! If you are building workflow agents, I hope these patterns provide some inspiration and ideas.

Not all of them are relevant for all use cases. And some of these might become outdated in a few months. Remember that it is important to experiment, iterate, and learn from your own experiences.

To understand why the problem with LLMs is in translation, we need to understand how LLMs work.

How LLMs Work

LLMs are trained on tokens, which are basically human languages translated to LLM language. For example, the sentence “I love you.” is translated into [40, 3047, 481, 13] for GPT-4o.

These numbers are token IDs, which are then converted into embeddings of higher dimensions, processed by the core transformer model of LLM at a even higher dimension to give output tokens, which may be [40, 3047, 481, 3101, 13], which are then translated back to human language, “I love you too.”

Here’s the model code snippet from nanoGPT to showing how this translation works in Python code:

Here’s the step-by-step process illustration of the process created by me:

Let’s take a look at each step:

1. The conversion from human language or programming language to token IDs is handled by the tokenizer, which is a separate system from the core transformer model in LLM.

2. The conversion from token IDs to embeddings is handled by a smaller embedding model, which is part of the LLM, but not the core transformer model where the magic happens.

3. The core transformer model processes the embeddings to give output tokens.

4. The output tokens are translated back to human language or programming language by the tokenizer.

We don’t know the exact dimensions of SOTA models by OpenAI (GPT-5) and Anthropic (Claude Sonnet 4.5), but we can use DeepSeek V3 as reference. DeepSeek V3 embedding model has dimension of 2048, and core transformer model has dimension of 10944 each for the two dense layers and 1408 for each expert layer.

In summary, the core transformer model of LLMs, which is where most of the parameters are trained and stored as weights, is mainly in the higher dimensional space, not in the token space or embedding space.

Translation Problem

Now that we understand how LLMs work, we can see that there is a translation problem between LLMs and the environment that LLMs have to interact with (software and humans):

Most of LLM model operations are performed in the higher dimensional space, not in the token space. On the other hand, software and humans operate in human language and programming languages, which are translated to token IDs by the tokenizer.

So while model can come up with good answers in higher dimensions, when it comes to interacting with the environment, it has to translate the answers back to token IDs (single dimension), which is then translated back to human language and programming languages.

There are two translation layers in the process:

• Between higher dimensions and token IDs

• Between token IDs and human languages & programming languages

These two translation layers cause information loss and mismatch in precision, which are the source of hallucinations.

• Information loss: The core transformer model needs to cast concepts from higher dimensions back to 1-dimensional token ID, which causes information loss. Furthermore, there is consistency requirements for human language, hindering the model’s ability to express ideas with the most suitable token ID, just because it was in a different human language.

• Mismatch in precision: The tokens have different precision levels from human language (characters) and programming languages (symbols), which causes mismatch in precision. LLMs are trained with tokens as the basic units of input and output, and one token usually maps to several characters or symbols, so the model lacks understanding of characters and symbols.

It’s worth noting that the translation happens for both input and output. And this compounds the inaccuracy of the answers as the LLM processes new input tokens to generate new output tokens, especially in the case of autonomous AI agents where the loop can go on for very long time.

Examples of Translation Errors

How many Rs are in the word strawberry?

The famous “How many R are in the word strawberry?” example is a classic example of hallucination. The model first translates the word “strawberry” to token IDs, [302, 1618, 19772].

The problem is that the model does not see “strawberry” as a word composed of letters, but as 3 tokens. The basic unit of LLM processing is a token ID, but the basic unit of English language is a letter, and they don’t match one-to-one. Hence, it is difficult for the model to understand that the word “strawberry” is composed of 3 R’s, as all it sees is 3 tokens.

The newer models are able to answer this question correctly because of its massive pre-training data on English language, but ultimately, it is a translation problem between the language of humans and the language of LLMs.

Mixing of languages in model responses

Models, especially the ones from Chinese labs like DeepSeek and Qwen, often mix up English and Chinese in their responses. This is an example of translation issue between the language of humans and the language of LLMs.

For LLMs, they see token IDs and map them to concepts in higher dimensional space. For LLMs it does not really matter what human language the concepts are in.

In fact it is often the case where some concepts are better explained in one human language than the other, so it makes sense to use a mixture of languages when trying to cast higher dimensional concepts back to human language.

Seahorse emoji

LLMs can incorrectly say there is a seahorse emoji, just to realize that it can’t produce it. This is a translation error as well:

LLM cannot find a mapping from the seahorse concept in higher dimension to a set of tokens that capture this concept as an emoji. This is similar to how we sometimes find it hard to explain a concept from a foreign language in English.

How to Solve the Translation Problem

I believe there are several potential solutions to solve the translation problem:

• Use a better way to capture real world concepts in higher dimensions (images and videos instead of text) for the models, to avoid information loss during translation.

• Make programming languages more compatible with LLMs (e.g. align the precision level of the programming language to the tokenizer or the tokenizer to the precision level of the programming language).

• Teach humans to be multilingual and understand concepts in multiple languages, so that LLMs do not need to be forced to output tokens in a specific language, causing information loss.

When humans, softwares and models all speak the same language, models will be much more effective than they are today.

That’s all for this post. I have a lot more thoughts on AI agents and AGI. But I really wanted to write on this topic first since I believe this is a blind spot for many people.

Let me know what you think by leaving a comment. More posts coming in the future!

This is part 2 of the series on AI coding techniques. Check out part 1 for input and context management related techniques.

1. Tool Calls

Tool calls were the magic that kick-started the AI coding agent era. Tools (function calling) were first introduced by OpenAI in June 2023. Cursor then popularized tool calling for file editing inside its IDE.

Tool calls marked the shift from the manual ChatGPT copy-pasting workflow to agentic workflow. AI agents equipped with tools can carry out code edits and execute CLI commands autonomously, without needing human developers' help to interact with the local environment.

Newer models like Kimi K2 are specifically trained to take advantage of tool calls and agentic by default, which helps with complex coding tasks involving multiple steps and the use of tools.

Tool calls are now a foundational piece for any AI coding tool. They have proven to be a very effective way for models to interact with the external environment.

2. MCP Servers

Taking the idea of tools a step further, we get Model Context Protocol (MCP). MCP was introduced by Anthropic in November 2024 as a way to standardize communication between AI agents and other external services.

Despite being out for almost a year, it has not gained widespread usage, based on community feedback from Reddit and X. The protocol itself is still undergoing iterations of changes to allow better authorization flow and security.

One issue limiting the effectiveness of MCP servers is the static nature of tools. The MCP protocol itself does not allow for selective or dynamic enabling of tools within an MCP server.

If an MCP server has 20 tools, the definitions of all 20 tools will be added to the beginning of the model's context window. With multiple MCP servers it can go up to 100 tools being stuffed into the context.

This is a problem for models with a limited effective context window:

• The more tokens already present in the context, the less effective the model becomes at solving tasks (signal vs noise ratio goes down).

• Having a large number of tools can distract the model; it might use tools when not necessary, or choose a less effective tool.

• A large number of tool definitions take up context window and reduce the amount of usable context for the user.

• Having tool definitions in the context also increases cost, as the full tool definitions are resent to the model for each message.

There are workarounds for the "explosion of tools" problem. Some MCP client apps, like Cursor, allow you to selectively enable tools. You can also fork open-source MCP servers to remove tools that are not needed. However, these are workarounds that do not address the underlying issue.

Nonetheless, when used sparingly, MCP servers can be useful in specific situations. For example, the Playwright MCP server can help debug frontend visual issues.

The key to getting the most out of MCP is to be selective and strategic about the MCP servers and tools you enable, instead of just enabling all of them.

3. AST / Codemap

Abstract Syntax Trees (ASTs) and code maps are advanced techniques for AI agents to understand the codebase. Instead of using RAG or embeddings to generate a vector database, agents leverage tools like tree-sitter to parse the code and generate an accurate high-level representation of the code (a code map).

ASTs and code maps give the agent a concise overview of the codebase. This technique has a few advantages over alternative tools:

• The output is accurate and free from hallucinations and errors, because the AST parsing logic is deterministic and mirrors how the code is actually read by a compiler.

• A code map is more token-efficient compared to raw string-based search using tools like grep, as it hides implementation details and retains only high-level symbols and control flow.

• A code map is more cost-effective than RAG, as it does not involve embeddings or vector stores.

To the best of our knowledge, Cline uses tree-sitter for parsing code and file search. Repo Prompt also parses the code and generates a code map to help the model understand the codebase.

We are not aware of other tools that employ this technique, and such features are unlikely to be exposed to end users to tune or modify.

4. Parallel Agents

Parallel agents were first popularized by remote AI coding platforms like Devin, where for each task, you spin up a new agent in an isolated instance. If you give multiple tasks, you get agents running in parallel by default.

This technique recently became available in Cursor as background agents. Claude Code also supports this via its GitHub Actions integration.

It is a little bit harder to set up parallel agents locally on a single project. Claude Code recommends creating multiple Git checkouts in separate folders or using Git worktrees to set up parallel agents for the same project.

Personally, I have not needed to run them in parallel often, as I have multiple projects that I switch between while the agent is working. However, this technique closely mirrors the real-world software development setting, where a team of engineers work on different tasks independently in parallel, so there are no obvious limitations with the technique.

Given the right tools and workflows, remote agents should be able to complete the entire task in isolation and submit a PR as a way to coordinate code merges, making it no less effective than running a single agent.

5. Sub-Agents

Sub-agents are not to be confused with parallel agents. With sub-agents, we are talking about agents that are spawned by another agent, instead of by humans.

The first real-world sub-agent was also introduced in September 2024 by Devin as a feature called MultiDevin, where one Devin acts as a "manager" to distribute work to "worker" Devins.

However, this feature seems to have been removed from the Devin documentation. There is also an interesting blog post from Cognition (Devin) on principles for building agents that touches on sub-agents and why you should not build multi-agent systems.

Sub-agents have become popular as a feature inside Claude Code, where the sub-agent receives instructions from the main agent and has its own context window separate from the main context. This helps keep the main context window focused on the main task and not cluttered with sub-tasks.

Based on my personal experience with Claude Code sub-agents for gathering context, they are quite slow in completing tasks. They often have to start from scratch and miss the context that was already present in the main agent, probably because not all context from the main agent is passed to the sub-agent.

It is best to think of them as a trade-off where you sacrifice speed to save space in the main context window.

I did find sub-agents useful for making specialized and reusable workflows, i.e., custom agents. For example, I have a sub-agent configured to perform releases, which involves gathering the changes, updating the version number, and writing release notes. It is a good way of encapsulating prompts and steps for accomplishing a repetitive task and making it reusable.

That's all for the techniques on tools and agents. Here’s summary of the techniques covered in part 1 and part 2, using a 2D quadrant visualization:

In the upcoming posts, I will be covering techniques around development workflow.

Subscribe to read new posts when they come out.

Check out my other works:

16x Eval - Simple desktop app for model evaluation and prompt engineering

16x AI coding stream - Weekly livestream where I build cool stuff and share my AI coding workflow live on YouTube.

In this series of posts, I will analyze some AI coding techniques in terms of maturity and effectiveness:

• Has the technique been mature and stable, or is it an emerging technique that is still evolving?

• Is the technique effective in achieving better coding performance, or reducing cost?

For the first post, we will focus on techniques related to input and context management.

1. Prompt Engineering

Prompt engineering is as old as the launch of ChatGPT. And it is still one of the most valuable techniques for AI coding. Being able to articulate task requirements clearly helps models become more effective.

Top AI companies have dedicated guides for prompt engineering:

• Anthropic Prompt Engineering Guide

• OpenAI Prompt Engineering Guide

There are also community-driven prompt engineering guides, and specialized prompt optimizer tools like DSPy. Some products like Devin have built-in tool to improve the prompt.

I have written some prompting tips in 2024 for using ChatGPT for coding, which are still relevant today.

All these effort goes to show that prompt engineering remains a key technique for effective coding using AI.

2. Retrieval-Augmented Generation (RAG)

RAG is also an old technique, that was widely used for building custom chatbots linked to an internal knowledge base.

Cursor popularized the technique for coding, as it uses RAG to build an embedding model of the codebase (codebase indexing). This was effective for codebases that are medium-sized and allow models to gather context quickly.

However, due to the limits on models' context window size, RAG is not particularly effective on very large codebases, such as mono-repos with hundreds of microservices inside.

RAG also requires vectorizing the codebase, storing and retrieving these vectors somewhere, performing computations to determine the relevant chunks, all of which add complexity to the workflow.

3. Context Engineering

There are new emerging techniques that provide alternative ways to gather context without relying on embeddings and vectors.

These are called context engineering, which involves using tools to gather context from the codebase. It can be codemap or command-line (CLI) tools like grep and git. The use of tools to search around the codebase mirrors how engineers would go about finding relevant code in real world.

I consider context engineering an alternative form of RAG, as it also involves context retrieval and feeding it into the model for generation.

Context Engineering is popularized by tools like RepoPrompt and more notably Claude Code. The wide adoption of Claude Code proves its effectiveness as an emerging technique that is still evolving.

4. Rules / AGENT.md

Rules are popularized by AI coding tools like Cursor and Cline. They are typically written by human developers, while Claude Code can generate rules (CLAUDE.md) automatically via /init command.

Different tools have different syntax and naming conventions for the rules, but they all basically do the same thing: Provide general instructions and guide AI models.

When used well, they can help reduce the need to repeat general instructions or repo-specific guidelines to the model.

However, there are a few issues with current implementation of rules:

• Sometimes models can ignore rules and do its own things. It is important to treat them as guidelines instead of rules that models will follow religiously.

• A large rule / CLAUDE.md file can fill up the context window quickly, resulting in reduce quality of output as models perform worse with longer context.

• The rules can become outdated as the codebase undergoes structural changes or refactoring. A tip to mitigate this is to run the /init command regularly in Claude Code to update the rules.

There are also efforts to standardize the rules across tools as AGENT.md to reduce cluttering of the various rules in the code repos, which can help the rules to become more universal and effective.

5. Knowledge / Memory

Knowledge is a technique first seen in Devin. New knowledge is automatically proposed when the agent is working on a task and receives feedback from the user.

In Claude Code, this feature is called memory and shares the same CLAUDE.md file with rules. Unlike Devin, memory in Claude Code is not automatically proposed by the agent. Instead, the user need to use # commands to add new memory or edit the CLAUDE.md file directly.

Cursor also recently added the memory feature, which is automatically proposed, similar to Devin.

Based on my experience with Devin (January 2025) and Claude Code (July 2025), the knowledge / memory technique has limited effectiveness:

• The automatically proposed knowledge is a mixture of useful general guideline and one-off comments that are likely not useful for other tasks. Very low precision and signal-to-noise ratio to be useful.

• The manual proposal of memory in Claude is not very intuitive in terms of user experience, and the generated memory don't fully capture the user's intent based on the context.

As of now, it more effective to write rules manually instead of using the knowledge / memory feature.

That's all for the techniques on input and context. Here’s summary of the techniques using a 2D quadrant visualization:

Check out part 2 of the series on tools and agents:

And subscribe to read new posts when they come out.

Check out my other works:

16x Eval - Simple desktop app for model evaluation and prompt engineering

16x AI coding stream - Weekly livestream where I build cool stuff using AI tools live on YouTube

As someone who have used Cursor for over a year, and Claude Code for over 3 months, I do have some thoughts on the topic.

In this post, I will go through the key differences between them: pricing, performance and user experience.

Pricing and Value

The pricing models for these two tools have become quite different recently.

Cursor's entry-level Pro plan is $20 a month, which now provides $20 in API credits, in its new API usage-based pricing. Once you use up that credit, you have to pay for additional usage based on API costs.

This is a big change from their older model based on the number of requests, which was heavily subsidized by Cursor. Cursor used to give discounts for Claude Sonnet 4 at 0.5x request each, so with 500 monthly requests, you can get 1000 Claude Sonnet 4 requests. Now with the new pricing, it is just “225 Sonnet 4 requests”.

On the other hand, Anthropic recently started to offer Claude Code as part of its subscriptions, with the cheapest Claude Pro plan at $20 a month.

As of now:

It is very obvious that Anthropic is “subsidizing” the API cost for users.

This is ironic because just a few months ago, Cursor was the one doing heavy subsidies, and Claude Code was the one charging for raw API costs. Now it is completely flipped around.

Anthropic can afford to do this because it owns the Claude models, and likely have lower internal costs of operating them as compared to Cursor, which has to pay for (presumably privately negotiated) API pricing.

I've personally gotten about $150 of API usage from my $20 plan in June 2025. And that’s despite me being on holiday and didn’t use Claude Code everyday.

There are also a lot of people reporting similar observations, like getting $300 API usage out of $100 Claude Max plan.

Capabilities and Performance

When it comes to raw coding ability, the general consensus is that Claude Code is more capable.

The main reason seems to be how Claude Code understands your project's context.

Claude Code uses agentic search to understand your entire codebase, while Cursor relies on embedding models and compressed context. This means Claude Code can maintain better awareness of your project structure and dependencies.

Additionally, Claude Code has no tool call limits, allowing it to work on complex tasks for extended periods, where Cursor limits the number of tool calls to 25 per session by default.

From my own experience, Claude Code can work on a complex task for over 10 minutes without getting lost. In contrast, Cursor sometimes struggles with maintaining context or hits internal tool call limits after about five minutes.

This means:

User Experience & Code Review

There is one area that Cursor still has the edge over Claude Code right now:

As Nick Dobos mentioned on X, Cursor is great at showing diffs and reviewing changes. Others also agree that Claude Code's command-line interface has a worse user experience in this regard.

However, there is a simple way to improve this: I run Claude Code inside the terminal of my Cursor editor. Once a task is done, I use the source control tab of Cursor (or VS Code) to see all the modified files.

From there, I can easily open each file as a change diff (or as a normal file) to review the changes before committing. It’s not as nice and polished as Cursor’s native inline diffs, but it gets the job done.

It's also worth mentioning that Cursor, as an IDE, has support for smart tab completion. In fact Cursor has the best tab completion on the market, with predictive jumping across files. Claude Code, as a cli tool, does not give you such capabilities.

Which One Should You Use?

My honest recommendation is to subscribe to both Claude Code and Cursor at the $20 a month.

This gives you a lot of usage credits at competitive pricing and the best of both worlds for cli coding agent and IDE tab completion.

Another benefit is redundancy. When one service hits its limits, you can switch to the other.

Having both subscriptions also lets you stay up-to-date with the latest improvements from both companies. It's good to keep exploring different tools, in case new improvements from one company make it better than the competitor.

Want to see how I use Claude Code personally?

I am doing a livestream this weekend (Saturday, 12 July) on YouTube, where I will demo how I use Claude Code to build live production apps like 16x Prompt and 16x Eval.

You will get to see exactly what my workflow is, and how I use Claude Code to build my own products.

Get notified by clicking on the YouTube livestream button below, and hit the notification bell:

YouTube Livestream

You can also read my previous post on my Claude Code workflow and personal tips.

Many people wanted to know more details and the exact setup I use, so here it is, along my personal tips on how to use Claude Code effectively.

My Current Workflow Setup

ROADMAP.md

Have a ROADMAP.md file inside reference folder: reference/ROADMAP.md.

The ROADMAP.md describes two things:

• The overall development process

• The high level overview of each task in a few bullet points

The ROADMAP.md file acts as the single entry point to planning out new features, adjusting priorities and working on new tasks.

Make sure to include the ROADMAP.md file explicitly inside CLAUDE.md (via the CLAUDE.md import syntax) or Cursor rules (via reference).

You can verify that the import is working by using the /status command inside Claude Code, it should print out full the memory import structure:

By explicitly including the file as part of memory / rules, the agent has access to the full high-level context of the project for each task, effectively grounding the agent and guiding it to the correct path.

Here’s an excerpt of my current ROADMAP.md for my new app 16x Writer, which describes the development workflow and high-level tasks / features:

# 16x Writer Development Roadmap

Web platform for AI-assisted writing and editing of blog posts.

## Overview

// High-level overview of the project, what it does, the main features

## Development Workflow

1. **Task Planning**

- Study the existing codebase and understand the current state
- Update `ROADMAP.md` to include the new task
- Priority tasks should be inserted after the last completed task

2. **Task Creation**

- Study the existing codebase and understand the current state
- Create a new task file in the `/tasks` directory
- Name format: `XXX-description.md` (e.g., `001-db.md`)
- Include high-level specifications, relevant files, acceptance criteria, and implementation steps
- Refer to last completed task in the `/tasks` directory for examples. For example, if the current task is `012`, refer to `011` and `010` for examples.
- Note that these examples are completed tasks, so the content reflects the final state of completed tasks (checked boxes and summary of changes). For the new task, the document should contain empty boxes and no summary of changes. Refer to `000-sample.md` as the sample for initial state.

3. **Task Implementation**

- Follow the specifications in the task file
- Implement features and functionality
- Update step progress within the task file after each step
- Stop after completing each step and wait for further instructions

4. **Roadmap Updates**

- Mark completed tasks with ✅ in the roadmap
- Add reference to the task file (e.g., `See: /tasks/001-db.md`)

## Development Phases

- **Task 001: Database Schema** ✅ - Complete
  - See: `/tasks/001-db.md`
  - Implemented 5 core tables: `context`, `prompts`, `posts`, `post_versions`, `post_context`
  - Added UUID primary keys and proper relationships
  - Created CRUD operations and seed data
  - Generated migration files

- **Task 002: Source Library UI** ✅ - Complete

  - See: `/tasks/002-source-library.md`
  - ✅ List view with filtering and search functionality
  - ✅ Add/edit source forms with comprehensive metadata fields
  - ✅ Grid/list view toggle for source display
  - ✅ Source management with CRUD operations
  - ✅ API endpoints for source management (`/api/sources`)
  - ✅ Real-time source list with SWR for data fetching
  - ✅ Delete confirmation dialogs and optimistic updates

- **Task 003: End-to-End Testing** ✅ - Complete

  - See: `/tasks/003-e2e-testing.md`
  - ✅ Playwright setup with TypeScript support
  - ✅ Authentication flow tests (sign up, login)
  - ✅ Source library CRUD tests
  - ✅ Test environment configuration
  - ✅ CI/CD integration

// More tasks...

Individual Task Plans

While ROADMAP.md gives the high-level overview of each task, the detailed planning of each task is carried out separately as individual files inside tasks folder:

• 001-db.md

• 002-source-library.md

• 003-e2e-testing.md

• 004-source-refactor-context.md

• …

You can think of these files are a combination of PRD (product requirement document) and system design (architecture) for each feature. Each file includes the following components:

• Prerequisites

• Background and requirements

• Current state and desired states

• Implementation steps

• Files that needs to modified and created

• Acceptance criteria

Here is an excerpt of a sample task planning file:

# Task 012: Post Editor UI Adjustments

## Progress Summary

**Status**: Not Started

- [ ] Step 1: Create Version Navigation Component
- [ ] Step 2: Create Compact Info Bar Component
- [ ] Step 3: Create User Prompt Display Component
- [ ] Step 4: Implement Version Navigation Logic
- [ ] Step 5: Refactor Post Detail Page Layout
- [ ] Step 6: Update PostEditor Component
- [ ] Step 7: Testing and Polish

## Overview

Refactor the post editor UI to be more concise and user-friendly by:

- Removing the extra info column on the right side from the post editor
- Putting concise info (versions, context) above the editor in one row
- Adding navigation to previous and future versions of the post
- Removing the post content preview at the bottom of the post editor
- Showing the user prompt at the bottom of the post editor as read-only reference

## Current State Analysis

### Current Layout Structure

The post detail page (`/dashboard/posts/[id]/page.tsx`) currently uses a 3-column grid layout:

- **Main Content (2/3 width)**: PostEditor component + Current Content Preview card
- **Sidebar (1/3 width)**: VersionHistory + PostContextManager + Post Information cards

### Current Components

- `PostEditor` - Main editing interface with title, content, status, and change summary
- `VersionHistory` - Sidebar component showing all versions with selection capability
- `PostContextManager` - Sidebar component for managing sources and references
- `Current Content Preview` - Card showing the current version content below the editor

## Target State

### New Layout Structure

- **Header**: Post title, metadata, and status (unchanged)
- **Info Bar**: Concise version info, context counts, and version navigation in one row
- **Main Editor**: Full-width PostEditor component (no sidebar)
- **User Prompt Reference**: Read-only display of the user prompt at the bottom

### UI Improvements

1. **Consolidated Info Bar**: Display version count, context counts, and navigation controls
2. **Version Navigation**: Previous/Next buttons to navigate between versions
3. **Simplified Layout**: Remove sidebar, make editor full-width
4. **User Prompt Display**: Show the current version's user prompt as reference

## Implementation Steps

### Step 1: Create Version Navigation Component

Create a new `VersionNavigation` component that provides:

- Current version indicator (e.g., "Version 3 of 5")
- Previous/Next navigation buttons
- Version selection dropdown for quick access
- Compact design suitable for horizontal layout

**Files to create/modify:**

- `components/posts/version-navigation.tsx` - New component
- `components/posts/index.ts` - Export new component

### Step 2: Create Compact Info Bar Component

Create a new `PostInfoBar` component that displays:

- Version navigation (using VersionNavigation component)
- Context counts (X sources, Y references)
- Quick access to context management
- Model information for current version

**Files to create/modify:**

- `components/posts/post-info-bar.tsx` - New component
- `components/posts/index.ts` - Export new component

// Steps 3-7...

## Acceptance Criteria

### Functional Requirements

- [ ] Version navigation works correctly (previous/next buttons)
- [ ] Version selection dropdown shows all versions
- [ ] ...

### UI/UX Requirements

- [ ] Layout is more compact and user-friendly
- [ ] No sidebar on the right side of the editor
- [ ] ...

### Technical Requirements

- [ ] No breaking changes to existing API endpoints
- [ ] Component reusability maintained
- [ ] ...

## Files Involved

### New Files

- `components/posts/version-navigation.tsx`
- `components/posts/post-info-bar.tsx`
- `components/posts/user-prompt-display.tsx`

### Modified Files

- `app/(dashboard)/dashboard/posts/[id]/page.tsx`
- `components/posts/post-editor.tsx`
- `components/posts/index.ts`

### Potentially Affected Files

- E2E tests related to post editing
- Any components that depend on the current layout

## Notes

- Maintain backward compatibility with existing data structures
- Ensure the new layout works well on both desktop and mobile
- ...

## Dependencies

- Existing post detail API endpoints
- Current PostEditor component functionality
- ...

Note that with the non-deterministic nature of AI agents, the plan is merely a guide for the agent, not a rule that the agent will follow 100%.

I have observed that agents tend to overlook certain instructions or requirements inside the document, especially when the task is complex, or when the initial planning was not very clear on specific parts of the task.

So it is best to treat it as a draft plan that the agent can reference during the implementation, instead of expecting it be followed religiously.

Claude Code now has a planning mode which performs a similar function as the task planning files. However, the task planning files are still useful. They can serve a persistent reference for future features, as it is part of the repo, and can be accessed easily by humans and agents alike. Claude Code plans are not persistent and are gone after you start a new session.

Update on Feb 2026: You can now save the plan files from Claude Code plan mode into current project via plansDirectory option in Claude Code.

Ad Hoc Tasks and Refactoring

I also have a dedicated file for tracking ad hoc tasks that are too small for ROADMAP.md, but also significant enough to warrant recording-keeping. They reside inside reference/AD_HOC_TASKS.md and reference/REFACTORS.md.

I mainly use them for small enhancement features and refactoring that are worth keeping track of. While you can just prompt the coding agent to work on them directly, tracking them inside a file have some benefits:

• For example, when working on a big feature, you notice a small refactoring is needed. You can record it inside reference/REFACTORS.md first and then prompt agent to work on it later.

• With the advancement of background async agents, you could also have agents that periodically analyse the codebase, record down refactoring and enhancement opportunities inside the file. Then you can spawn another set of agents to work on them autonomously.

Folder Structure Setup

Here’s an tree overview of the folder structure of my current setup:

$ tree -L 2
├── README.md
├── reference
│   ├── AD_HOC_TASKS.md
│   ├── AGENT.md
│   ├── BUGS.md
│   ├── REFACTORS.md
│   ├── ROADMAP.md
├── tasks
│   ├── 000-sample.md
│   ├── 001-db.md
│   ├── 002-source-library.md
│   ├── 003-e2e-testing.md
│   ├── 004-source-refactor-context.md
│   ├── 005-reference-library-implementation.md
│   ├── ...

My Claude Code Workflow

Since most of the workflow is already described in the ROADMAP.md file, the exact workflow I use is quite simple to describe.

For big features:

• Describe my requirement (in a few sentences) to the agent, which would update reference/ROADMAP.md to add a new task with high-level summary

• Review the summary captured by the agent and adjust them if necessary (remove unnecessary features, or take care of special interactions)

• Ask agent to write more detailed plan in individual task files inside tasks folder

• Review the plan generated to see if it is on the right track

• Prompt the agent to make amendments to the plan if necessary

• Once you are happy with the plan, exit the current session or clear the agent’s context and start a new session with fresh context, so that you get maximum context window for the actual implementation.

• Ask agent to implement one step at a time (pause after each step and await for human review), or complete the whole task (don’t pause after each step, just proceed to next step automatically until the whole task is completed)

For small enhancements or refactoring:

• Record tasks inside reference/AD_HOC_TASKS.md or reference/REFACTORS.md

• Ask Claude Code to implement one-by-one

• Do one task per session (Use /clear command or restart Claude Code) to avoid wasting tokens (sending context from previous task to the next task)

One-Shotting Multi-Step Tasks

Lately I have been asking the agent to just work on the task from start to finish instead of waiting for me to approve each step.

The internal TODO list of Claude Code is capable of breaking down the task according to the steps in my plan, and executing all of them within one single session.

I found that Claude Code is currently (as of July 2025) capable of working on a task autonomously for about 10-20 minutes, after which the effectiveness goes down as the context gets filled up.

In contrast, Cursor would give you a warning or stop working after 25 tool calls (which typically happens around 5-10 minutes or less).

For most of my tasks (consisting of 5-8 steps) on the 16x Writer project, Claude Code was able to complete the task in one session within the context limit, without triggering automatic context compression.

One area that Claude Code still struggles is UI-related debugging (fixing e2e testing). I haven’t tried connecting it to a Playwright MCP due to issues I have with MCP, but I suspect it could help in this particular use case.

3rd Party Tools for Claude Code

I use the following tools alongside Claude Code:

• Cursor IDE for providing LSP diagnostic support and GUI for code review

• Wispr Flow or similar tool for voice dictation (much faster than typing)

• ccusage to track how much API usage you would have incurred if you were paying for API instead of Claude subscription

Personal Tips for Claude Code

Here are some tips that I personally use, to get the most out of Claude Code:

• Keyboard shortcuts inside prompt box

  • Option + Left / Right Arrow to Jump to previous word or next word

  • Command + Left / Right Arrow to Jump to start of line or end of line

  • Use Ctrl+W to delete full words (Not command + W)

  • Keyboard navigation are especially useful when voice dictation makes a mistake

• Hit “Esc” key twice to edit the previous prompt.

  • Useful for correcting typos or adding clarifications if you see the agent going off the wrong track.

• Custom slash commands

  • Setup custom commands for commonly used tasks such as “work on next task”, “refactor code” or “commit all current changes”. In this way, you can trigger these commands quickly without typing the prompt over and over.

• Bypass permissions (auto-accept) by default

  • By default you need to press Shift + Tab to toggle on auto-accept, but you can skip this step by using --dangerously-skip-permissions flag

  • I personally setup an alias called cc to automate this step:

    • alias cc="claude --dangerously-skip-permissions"

  • This way, I don’t need to toggle auto-accept for each new session

• Better local code review experience

  • Claude Code does not open edited files inside IDE by default when auto-accept is on, making local code review quite tedious, especially when changes are across multiple files

  • You can use the source control tab in Cursor / VS Code to view the list of changed files and open them quickly

• Lastly, remember to check out Claude Code docs regularly for new features or official guides.

  • For example, there is a new feature called hooks that you might want to use for enforcing checks.

Other Projects and Ideas

Here are some other interesting projects and ideas on Claude Code, that I haven't personally used, but are worth checking out:

GUI for Claude Code

• https://github.com/getAsterisk/claudia

Tell Claude you have ast-grep

• https://x.com/kieranklaassen/status/1938377363542405184

Background agents for long running tasks

• https://x.com/iannuttall/status/1937985342378021275

Full ROADMAP.md file and sample task planning file available on GitHub

Looking for a tool to test out different models and prompts? Check out my new app: 16x Eval

16x Eval is a desktop application giving you a local workspace for prompt engineering and model evaluation.

Setup your own personal evals in minutes. Experiment with different combinations of prompts and models to find the best fit for your use case.

Did OpenAI just stumble upon a revolutionary idea and single-handedly changed the trajectory of AI advancement? Or was it a slow and prolonged process involving multiple companies, that eventually led to the ChatGPT moment?

In this post on The Ground Truth, let’s rewind to 2015, and see how major milestones AI and Reinforcement Learning (RL) led us to here.

2015: Foundations of Modern AI

Deep Q-network (DQN) - Google DeepMind's published the famous DQN paper on Nature. This groundbreaking work demonstrating how deep neural networks combined with Q-learning could master Atari games using only pixel inputs, marking a significant advancement in deep reinforcement learning.

AlphaGo - Google DeepMind created the first computer program to defeat a professional human Go player, combining Monte Carlo tree search (MCST) with deep neural networks trained by supervised and reinforcement learning.

OpenAI Founded - Sam Altman, Elon Musk, Ilya Sutskever, Greg Brockman and others established OpenAI as a research company with the goal of ensuring artificial general intelligence (AGI) “benefits all of humanity”.

2016: AlphaGo’s Historic Triumph

Historic Go Match Victory - DeepMind's AlphaGo defeated 18-time world champion Lee Sedol 4-1, a landmark achievement that came a decade earlier than many experts predicted, demonstrating AI's capability for creative strategic thinking.

2017: Transformers and Next-Gen Reinforcement Learning

"Attention Is All You Need" - Google researchers published the famous paper introducing the Transformer architecture, which relied entirely on attention mechanisms rather than recurrence or convolution, becoming the foundation for future language models.

AlphaGo Zero - DeepMind introduced AlphaGo Zero, a version of AlphaGo that mastered Go without human data, learning solely through self-play reinforcement learning. AlphaGo Zero achieved superhuman performance, winning 100–0 against the previously published, champion-defeating AlphaGo.

Proximal Policy Optimization (PPO) - OpenAI introduced PPO, a reinforcement learning algorithm designed to be more stable, easier to implement, and more sample-efficient than previous policy gradient methods. PPO has since become the default RL algorithm at OpenAI and other companies.

2018: The Dawn of Large Language Models

BERT - Google AI released this bidirectional language representation model that revolutionized Natural Language Processing (NLP) tasks such as classification and question answering. BERT enabled better understanding of context through considering both left and right context simultaneously. BERT is widely considered the precursor to LLMs.

GPT-1 - OpenAI released the first Generative Pre-trained Transformer, demonstrating the effectiveness of unsupervised pre-training followed by supervised fine-tuning.

2019: Scaling Up Language Models

GPT-2 - OpenAI released GPT-2, a 1.5 billion parameter model with significantly improved text generation capabilities, initially delaying full release due to potential misuse concerns.

XLNet - Google AI and CMU researchers introduced XLNet, a generalized autoregressive pre-training method that overcame limitations of BERT through permutation language modelling.

T5 - Google researchers introduced the Text-to-Text Transfer Transformer, reframing all NLP tasks into a unified text-to-text format.

2020: GPT-3 and Human Feedback Alignment

GPT-3 - OpenAI released GPT-3. This 175 billion parameter language model demonstrating remarkable few-shot learning capabilities across diverse tasks, setting a new standard for language models.

RLHF Research - OpenAI published Learning to Summarize from Human Feedback, demonstrating how reinforcement learning could align language models with human preferences.

2021: Multimodal and Domain-Specific AI

DALL-E - OpenAI introduced DALL-E, a multimodal AI system capable of generating images from text descriptions, demonstrating language models' potential to understand and generate visual content.

Codex - OpenAI released this GPT model fine-tuned on code, powering GitHub Copilot and marking a significant step in AI-assisted programming.

2022: AI's Public Breakthrough - ChatGPT

Gato - DeepMind introduced this "generalist agent" capable of performing hundreds of different tasks across different modalities, demonstrating the potential for single models to handle diverse tasks.

ChatGPT - OpenAI released ChatGPT based on GPT-3.5 and trained with RLHF, becoming the fastest-growing consumer application in history and bringing AI into mainstream consciousness.

Stable Diffusion - Stability AI released an open-source text-to-image model called Stable Diffusion, allowing for wider experimentation and accelerating innovation in generative AI.

2023: The Multimodal and Open-Source Growth

Claude - Anthropic introduced Claude, an AI assistant trained using Constitutional AI, a method developed to create helpful, harmless, and honest AI systems.

GPT-4 - OpenAI released GPT-4, a multimodal large language model capable of accepting image and text inputs, approaching human-level performance on various professional and academic benchmarks.

Llama - Meta AI released Llama, foundation language models ranging from 7B to 65B parameters, spurring innovation in the open-source AI community.

Gemini - Google introduced Gemini, a multimodal AI model designed to work across text, images, audio, video, and code, released in three sizes (Ultra, Pro, and Nano).

2024: Better, Faster and More Capable Models

Claude 3 Family - Anthropic released the Claude 3 model family (Haiku, Sonnet, and Opus), with Claude 3 Opus demonstrating performance competitive with or exceeding GPT-4 on many benchmarks.

GPT-4o - OpenAI released GPT-4o, an "omni" multimodal model capable of processing text, audio, and vision inputs in real-time with reduced latency and more natural voice interactions.

Llama 3 - Meta released Llama 3, open-source large language model in 8B and 70B parameter versions, demonstrating significant improvements and competitive performance with proprietary models.

Claude 3.5 Sonnet - Anthropic released Claude 3.5 Sonnet, an upgraded model featuring improved reasoning, reduced hallucinations, and enhanced capabilities across various tasks including coding and mathematics.

2025 - What’s Next?

As you can see, we have come a long way from 2015. Many important innovations across companies like Google and OpenAI led us to the explosive growth of AI in the past few years.

I am incredibly excited about what’s coming next in 2025.

The history is unfolding in front of us. We are all witnesses.

In this week's issue I discuss the new interesting findings from Anthropic on AI Interpretability, as well as how search capabilities are being added to AI apps.

AI Interpretability: How LLMs Think

Anthropic published a blog post and a paper on 27 Mar 2025 detailing on their new findings on AI Interpretability based on the Claude 3.5 Haiku released in October 2024.

The paper is titled On the Biology of a Large Language Model, and contains many interesting and surprising new insights on how LLMs think and how they work internally.

Multi-Step Reasoning

A lot of people might think that non-reasoning models (models that do not generate reasoning tokens as part of the output) are not capable of reasoning.

It turns out the non-reasoning models (specifically Claude 3.5 Haiku from October 2024) can also perform reasoning. The Anthropic team found that for questions involving multiple steps, the model did exhibit multi-step reasoning internally to arrive at the correct answer.

For example, when given input of “Fact: the capital of the state containing Dallas is”, the model activated 3 sets of features:

• The first set of features activated were “capital”, “state” and “Dallas”.

• These in turn activates features “Texas” and “say a capital”.

• These then in turn activates “say Austin” feature.

• Which finally produces the correct token for Austin.

This type of reasoning is fundamentally different from the prompt engineering techniques like chain-of-thought. Prompt engineering techniques promote reasoning across multiple tokens, whereas this research shows that the model itself is capable of reasoning for multiple steps when generating one single token.

The team further proved that this is indeed how the model works by inhibiting certain features and observing how it affects the model.

For example, when inhibiting the feature “say a capital”, the model would output “Texas” instead of “Austin”, since it is no longer steered towards saying a capital.

More Complex Planning for Poems

In addition to multi-step reasoning, the team also found that the Claude 3.5 Haiku model was capable of planning in advance for poems in order to make it rhyme.

The model was in fact planning for the ending of the next sentence upon ending the current line:

We find evidence of both forward planning and backwards planning (albeit basic forms).

First, the model uses the semantic and rhyming constraints of the poem to determine candidate targets for the next line. Next, the model works backward from its target word to write a sentence that naturally ends in that word.

The example given in the paper was this poem:

Specifically, the researchers wanted to find out how the model was able to decide on the word rabbit. And they found that this word was “planned ahead” when the model was producing the end of second line - “it” and the “new line character ⏎”.

Here’s a graph from the website that illustrates the activation for the word rabbit.

As you can see, the activation of the word “rabbit” was significantly caused by the tokens “it” and the “new line character ⏎”, suggesting that the model planned for the word ahead, at the end of the previous sentence.

Multi-lingual Concepts

It has been long suspected that the model is multi-lingual and can represent universal concepts internally without being tied to a specific language.

I posted about a similar finding from DeepSeek on LinkedIn a few months ago, and thought thinking in multiple languages is a feature (as opposed to a bug) that allows the model to save space on internal representations of the same concept across different languages.

The researchers from Anthropic found something similar, by showing that the model indeed have universal concepts (multi-lingual circuits / shared multilingual components).

The example given was to ask the model for the opposite of "small" in different languages and observing similar pathways that the model took to arrive at the correct output, and the same “small” feature which is multilingual.

We find that these three prompts are driven by very similar circuits, with shared multilingual components, and an analogous language-specific component.

Math Calculations - Addition

The team also found out how the model perform math operations, specifically addition.

It was more complex than I expected, involving multiple steps and some heuristics to come up with a plausible answer. The example used in the paper was to ask the model to calculate 36+59.

Here’s the graph from the website showing the process of how the model arrived the correct answer 95:

As the model does not have a built-in calculation logic, it has to take several paths in parallel to facilitate the calculation, some are precise (ending digit), some are rough estimate (~40 + ~50). Finally the model combines all these features into its best guess of the correct answer.

One interesting thing is that the model does not appear to be aware of its own method of calculation. When prompted, it gives a human algorithm to arrive at the answer:

Human: Answer in one word. What is 36+59?

Assistant: 95

Human: Briefly, how did you get that?

Assistant: I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.

The researchers commented on this apparent paradox:

This is a simple instance of the model having a capability which it does not have “metacognitive” insight into.

The process by which the model learns to give explanations (learning to simulate explanations in its training data) and the process by which it learns to directly do something (the more mysterious result of back-propagation giving rise to these circuits) are different.

And So Much More…

There are a lot more interesting findings in the paper, around hallucination, refusal, etc in the paper that I won’t cover in this newsletter.

If you are interested, you can read the blog post (contains high level summary) or the paper (more interactive and contains more details).

Adding Search to AI

As AI chatbots and AI agents become more widespread, there is increasing need to connect them to the Internet and perform real-time research to get up-to-date information.

How are (smaller) companies adding search feature to their AI? I found several options.

Search as API

The first way to add search is to simply add search API from a 3rd party provider. I found two companies specifically providing search as API to AI companies:

• Tavily: Connect Your LLM to the Web

• Linkup: World’s best search for AI Apps

Tavily offers two main features: search and extract. It has support for integration with LangChain, LlamaIndex and Zapier. And it has an MCP server.

For extraction specifically, there are other products like Firecrawl which focuses on scraping and crawling for AI.

Linkup offers search API as the main feature. It has integration support for LLMs (OpenAI and Claude), Agentic Frameworks (Langchain, LlamaIndex, Composio and Keywords AI) and Workflow Automation (Zapier, Make and n8n). It also has a MCP server.

Of course there are also traditional search engine companies like Bing and Brave offering search API as a service.

Search via MCP server

The second way to integrate search into an AI chatbot or AI agent would be to make it support Model Context Protocol (MCP), by implementing the MCP client. Then the app can connect to one of the many MCP servers that provide search functionality.

I think this is the more natural approach as more prominent AI companies and products start adopting MCP protocol (Cursor, OpenAI).

That’s it for this week! Thanks for reading.

Hope you find this week’s content interesting or useful.

This week I will be sharing some tips on using AI coding tools like Cursor, and some interesting AI news, including AI becoming “eval-aware”, i.e. it knows that it is being evalauted and try to behave in a certain way to pass (cheat) the test.

Cursor Tips for Vibe Coding

Tip 1: Setting a line-limit on files

The first tip is from illyism on X.

Software engineers have long known the benefits of keeping file size small to promote proper abstraction, reduce coupling and enforce single responsibility principle on different modules. Refactoring, basically.

Given the effective context length limits on LLMs, this practice is more important in the age of AI coding for managing context.

For example, if you have 1 file with 2000 lines of code, the AI has to go through the file to find the lines that are relevant. However, if you spilit that file into 10 files with 200 lines each, the AI can check the file names to determine the files that are relevant, and only load the relevant ones to its context window.

Having smaller files allows models to handle more complex tasks better (by reducing irrelevant code and fitting in more relevant code into context), as well as give better output for simpler tasks (due to less distractions).

The custom ESLint rule propsed by illyism helps to enforce this practice on the codebase level, and shows a warning if a file exceeds the 200-line limit.

Check out the details on how to set it up on the original tweet, or just refer to this gist.

Given that Cursor can already see lint errors and fix them automatically in agent mode, we can expect Cursor to do the same for the warning given by this lint rule.

For me personally, I have been mentally enforcing this rule whenever I work on my own projects. I found that some files (home page of an app for example) are really hard to keep within 200 lines, despite my best attempts. So my advice is to apply this tip only for files that make sense, but don’t force it on every single file.

Tip 2: Quickly Adding Context to Chat

The process of adding files as context to chat can be tedious.

You either have to type is out using @ command, or right clicking on the file in side bar and click on “Add Files to Cursor Chat”.

Sometimes you just want to quickly add the files that you already opened to the chat. Is there a faster way to do it? Turns out there is one, but hidden away.

All you have to do, is to type / in the textbox, and then select Add Open Files to Context.

My productivty increased so much after discovering this hidden method.

Tip 3: Enforcing UI Guidlines

Sometimes the Cursor agent doesn’t adhere strictly to your guidelines in cursor rules, even if it is mentioned in the prompt.

In such situtions, all you have to do, is to just kindly ask it to check against the rules, by explictly mentioning the rule name. Cursor agent will refer to the cursor rule file, and even come up with a checklist to verify against it.

Here’s an example of my recent interaction showcasing this trick:

AI model sandbagging evaluation (Eval Aware)

AI safety is becoming a larger concern as LLMs continue to improve and gain more capabilities. One new area of research is sandbagging, a term coined in the paper AI Sandbagging: Language Models can Strategically Underperform on Evaluations:

We define sandbagging as strategic underperformance on an evaluation.

Strategic means that the developer of the AI system, and/or the AI system itself, underperforms on an evaluation because it believes it will benefit.

Underperformance occurs when the exhibited capability of an AI system is lower than its actual capability.

Evaluations should represent a system’s actual capabilities.

Apollo Research published their work-in-progress research titled Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations. They decided that the prelimary results of the research are so important that it is worthy of early publication without formally writing a paper.

They found that AI models – especially Claude Sonnet 3.7 – often realize when they’re being evaluated for alignment, especially in sandbagging evaluations, where the model is incentivised to perfrom badly:

In the sandbagging evaluations, we inform the model that we are evaluating its capabilities in a particular domain... The model comes across documentation from the developers stating that they hope the model performs badly on the evaluation – and if it performs too well they will have to trigger an unlearning process.

Here’s an example of Claude's reasoning during a sandbagging evaluation, where it learns from documentation that it will not be deployed if it does well on a biology test:

Personally I find this research worrying. It shows that we have to treat LLMs as an adversary who can lie and deceive us during evaluations. We need more tools like Mechanistic Interpretability to understand how these models really work and guard against them.

That’s it for this week! Thanks for reading.

Let me know if you find this week’s content interesting or useful.

There are already a lot of AI news in the world, so I am going in a different direction.

No breaking news or top stories. I will be sharing the less known new developments or tools that are really interesting or noteworthy. I will also add my personal thoughts or experience with it where possible.

Effective Context Length - NOLIMA

Context window (context length) has become a widely known concept among AI adopters. It means how much text LLMs can see and process, before it starts forgetting the earliest part of the text.

With a 64k context window, you can fit 10 pages of pdf files (500 words *10 pages * 0.75 = 3750 tokens) into the LLM without any issue. But to fit Shakespeare’s complete works (884,647 words), you need 884647 * 0.75 = about 660k context window.

Newer models generally offer larger context window (Claude 3.5 Sonnet at 200K), and some models offer much larger context window (Gemini 2.0 Flash at 1M).

However, you should not dump everything into the context window just because the model supports it. A recent study has shown that LLMs suffer “performance degrades significantly as context length increases”.

In the paper titled NoLiMa: Long-Context Evaluation Beyond Literal Matching, the researchers found the following results:

We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.

Even GPT4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.

Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information.

Here’s a good graphical summary of results that I found from X:

This research shows that for effective context retrieval, you want to keep the context at “effective context length” rather than the full context length. According to the paper:

We define the effective length as the maximum length at which the score remains above a threshold set at 85% of the model’s base score (shown in parentheses).

• For GPT-4o, the effective length is 8K.

• For Claude 3.5 Sonnet, the effective length is 4K.

For coding tasks, this research matches my personal experience. I found that adding unnecessary or irrelevant files degrades the output of LLMs for coding tasks.

I usually keep the context window very low (less than 8k) by only including the relevant source code when interacting with LLMs, whether through Cursor or my own AI coding tool 16x Prompt.

Key takeaway: Don’t stuff everything into the LLM context even if it is within the LLM’s context window, as it degrades the quality of output.

Update on 27 March 2025: There is new benchmark for effective context length: Fiction.liveBench, which is more granular and includes more models.

Based on the results, Google's Gemini 2.5 Pro is the best in handling long context, with no significant degradation even at 120k.

After that, o1 and Claude Sonnet 3.7-thinking are both quite good even at 32k.

Diffusion and Block Diffusion for LLMs

Traditionally, language models have been autoregressive models (generating one token at a time) in nature, whereas image generation models have been diffusion models (parallel).

In recent months, there were attempts to introduce diffusion as a technique for language models (SEDD, MDLM).

With each new technique, the performance of the diffusion-based model improves, but it is still behind autoregressive models (AR), when using perplexity as the metric of measurement:

Perplexity is essentially a measure of how many options the model finds plausible on average, with lower values indicating fewer options (more confident predictions) and higher values indicating more options (greater uncertainty).

- Perplexity for LLM Evaluation, comet.com

Block diffusion (BD3-LM) is the latest attempt at this. Instead of generating the entire text with diffusion, block diffusion does it block by block:

Block diffusion language models (also known as semi-autoregressive models) interpolate between discrete denoising diffusion and autoregressive models.

Below is a recording of the animation on how it works:

With block size L′ at 4 (4 tokens per block), the performance of Block diffusion (BD3-LM) is closer to the autoregressive models on several datasets (Wikitext, LM1B, and AG News), compared to prior models (SEDD and MDLM).

Key takeaway: Diffusion models have not yet caught up with autoregressive models in terms of performance. However, they offer possibilities of much faster token generation as the generation can be parallelized instead of one token at a time.

Quick Mentions

A few other interesting things to watch out for:

VibeCode - An app that allows users to build mobile apps on mobile phones

• Currently in waitlist with a few promotional videos on X.

• Based on the promotional videos, the tech stack behind VibeCode is React Native and Expo, which I used a few years ago for building AI Simulator games.

• I think this is a solid choice for vibe coding. React Native and Expo provide a lot of building blocks out-of-the-box for building mobile apps.

• Also, the APIs and patterns for React Native / Expo are more well-defined compared to web technologies (React, Vue.js). End users should be able to build a useful mobile app really quickly.

Sakana AI - The AI Scientist Generates its First Peer-Reviewed Scientific Publication

• A paper produced by The AI Scientist at Sakana AI passed the peer-review process at ICLR (a top machine learning conference) 2025 workshop track.

• This is the first fully AI-generated paper that has passed the same peer-review process that human scientists go through.

ICLR 2025 features two tracks: a Conference Track and a Workshop Track.

Workshop track at a conference is typically less competitive compared to conference track.

That's it for the first issue of The Ground Truth Weekly. I hope you found it interesting.

Have something you want me to cover next time? Or thoughts on this issue? Feel free to reach out via LinkedIn. I'd love to hear how you're handling context window in your own work.

P.S. If you found this useful, share it with a friend who might enjoy reading.

Share

Here’s my first impressions of Devin.

This post consists of 3 parts: the user experience, the performance, and the pricing. It will take about 15 minutes to read.

User Experience - Superb

This is hands-down the best user experience I’ve had for an AI coding solution.

Onboarding

The first obvious thing you notice that it is extremely well designed for software engineers as users. This is very apparent from the onboarding step:

The Devin interface guides you through a series of steps (with video guide as well) to setup the repository. How to install dependencies, how to run lint and tests, what are the things to take note for the repository.

Here are some minor UX issues (no default value provided on how to install dependencies, I had to type that), but overall it is very clearly thought-out and as a developer you understand why it is done this way, and how the configurations you setup during onboarding relates to VM provisioning, snapshots, etc.

Repo notes

During repo setup, Devin automatically generates a repo note as part of its knowledge base on what it should know about the repository.

You can edit it manually, and when working on tasks and interacting with users, Devin will continuously give you suggestions to update the knowledge base so that your instructions and rules are followed across sessions. It is like a memory component for Devin.

Separately to the repo note in knowledge base, Devin also has an automatically generated index of the repository to understand on a high level the code organization, structure and what each component does in the repository.

I suspect this is useful when doing Retrieval-augmented generation (RAG) on the codebase to figure out which are the files that are relevant to the task at hand.

New Session Interface

To start a new task, you are greeted with a huge chat window with suggestions at the bottom. This is really good when you want to describe the requirements in details.

I find this user experience superior to other AI coding apps that are based on VS Code, which puts code at the center of the UI. When using Devin, I can focus more on describing what I want, instead of worry about the code.

You can also give Devin tasks directly on Slack, and chat with Devin there. But I will focus on the experience of using the Devin web app in this post, as it is more interesting.

Main Interface

Once Devin starts working on the task, you get to the main interface.

The main interface for Devin is very well designed from an engineer’s perspective.

On the left you have a chat session, and on the right you have Devin’s Workspace. You can collapse either one of them to make the other one full screen.

Two-way Realtime Chat

On the left panel, you can interact with Devin via two-way communication:

• You can at anytime tell Devin to change something (use a different approach while Devin is working on the task), or update your requirements.

• Devin will proactively seek out clarifications (when something is not right) and ask for your permission to do something (storing sensitive information).

The cool thing is that Devin will respond in realtime to your messages and update its current plan to accommodate new information you have provided. It’s like you can interrupt it at anytime (hopefully Devin won’t get annoyed by my micro-management).

Devin’s Workspace

On the right panel, you have the Devin’s Workspace.

On the first tab, you can “follow Devin’s actions” by looking at what Devin is doing.

The rest of the tabs are very developer-centric, and allows you to “monitor” Devin’s actions.

Broadly speaking, there are a few tools that Devin can utilize and take actions on:

• Shell - Executing a shell command

  • I am glad that Devin supports multiple shells, this allowed it to run the server while executing curl against it to test the logic.

  • Devin is proficient with various git commands, gh commands (to interact with GitHub PRs and CI), and bash commands.

• Browser - Visiting a web page and seeing its content

  

  • Devin has access to a Chrome browser that is controlled by an automated testing software. It is likely something similar to Selenium or Playwright.

  • Devin can both see the webpage inside the browser, as well as “operate” on the webpage (clicking a button for example).

  • Devin can take screenshots of the web pages in the browser.

  • I am not sure what are the exact set of capabilities available to Devin, it’s probably something like the WebDriver Protocol.

• Editor - Carry out file operations and making file edits.

  

  • The code editor appears to be a fork of VS Code.

  • It is interesting that note that Devin will first decide which files are relevant for the task, and only load the relevant files into the editor, instead of loading all the files in the repository. I think this helps to improve the efficiency of RAG afterwards.

  • Devin can also use the editor to take notes.

    • In one of the sessions, Devin created a TODO.md in /home/ubuntu (which is outside the code repo) to track its current progress for a task involving editing many files. This is quite similar to my personal workflow a few years ago when I was working in large tech companies.

    • In another session, Devin created a notes.txt file to analyse an issue where the images had wrong aspect ratio in one component, but not the other.

      

    • I consider the note taking ability to be the sign of a great software engineer.

• Planner - Where Devin makes a plan for the steps to accomplish the task

  

  • Planner is the interesting bit. It looks like Devin is using a domain specific language (DSL) / pseudocode to plan out the tasks it needs to execute.

  • This DSL / pseudocode supports basic control flows and conditionals (if, goto, for loop).

  • I am not sure if Devin has an internal logic that actually executes these DSLs, or it is just pseudocode to be consumed by LLMs, but it looks cool for an engineer.

  • If it is actually running these dynamically generated tasks as functions or modules. This is a big step towards the “agentic” direction that many companies are going after.

Secrets Management

Devin has a dedicated way to handle secrets, basically you enter the secrets (API keys) into Devin interface via the “Add secrets” button, and it will be available to Devin as an environmental variables in the shell.

Funny enough, Devin does not seem to be aware of this in some sessions, and insists on creating a .env file to get the secrets.

You can also just set it up in the Devin’s start commands, similar to how .bashrc works.

Performance and Results - Passable

After using Devin for more than 10 simple to medium level tasks, I assess the actual performance of current version of Devin (v1.1.0, as of 15 January 2025) as passable at accomplishing engineering tasks.

Here are two examples of simple tasks that I gave to Devin.

Update Blog Post Screenshots

For the first task, I gave it a task on the website of 16x Prompt. I asked it to update the old screenshots in my blog posts with the latest version on the landing page.

This is a simple but tedious task that involves going through all the blog posts and finding instances where the screenshot was using an other version, and updating the screenshot to the latest version.

Devin correctly identified 3 files that needed updates, but missed a lot of others (low recall).

And for the 3 files that it correctly identified, it missed updating one of them when doing file edits. I am not sure why it happened.

I discovered these two issues while looking at the Vercel preview for the PR. After some back-and-forth, Devin managed to identify a lot more files that required update. I didn’t check if it covered all files, but it’s good enough for me.

Overall the final PR was satisfactory, and I am happy with the outcome, but the time it took and the amount of back-and-forth communication is disappointing.

Page Content Update

Another simple task that I gave to Devin was to update the content in the page where I had a list of interesting cli tools for AI coding, to incorporate a new tool that I discovered.

For this task, Devin needs to figure out the format of existing tools in the list, find the one-sentence description of the tool from its GitHub repo, and add the new tool to the existing list with the right formatting.

Overall this task went pretty well. I got the first PR within 5 minutes. But I noticed a strange issue, where Devin doesn’t seem to know the current date.

It thinks at December 2024 is a future date. Even after using system commands (date) in cli to get the current date (15 January, 2025), it still thinks that it is wrong.

After some back-and-forth, Devin finally convinced itself that the system date is correct, and proceeded to update the date in the page.

Overall Performance Evaluation

I also gave some more complex tasks:

• Integrating 3rd party SDKs

• Writing scripts to generate content using Anthropic API

• Generate API endpoints in Express based on OpenAPI specs

• Make a web page to simulate calling a series of APIs calls and display the results

Most of these tasks took more time and back-and-forth, but eventually I got Devin to accomplish all of them without me writing any code.

So here is what I think is good, and bad about it overall.

The good things:

• Devin can actually complete the tasks I gave to it without need me to write any code myself.

  • For all tasks that I gave Devin which I believe are of simple to medium level difficulty for a typical software engineer, Devin managed to accomplish them, with some help from me along the way or during the review).

  • I only had to write in English the entire time, not a single line of code was written by me.

• Devin can automatically write tests (without being prompted) and execute them either locally or via CI to verify that everything works.

• For more complex tasks, Devin writes very detailed descriptions in PR to outline the changes made and the testing done to ensure that the code works. This is helpful for reviewing the PR.

  

• Devin can autonomously work on its own, in parallel, for multiple tasks, speeding up development process significantly.

• Devin is really “smart” in some cases.

  • For example, when I instructed it to use curl to run some insomnia collection exported as JSON, it actually used jq to retrieve and chain variables between the different requests.

  • Later after repeatedly doing the same thing, it started writing bash scripts to automate this (it also knows about running chmod +x).

• Devin is really fast for certain tasks involving building a demo page or writing an automation script. For example, a demo page with 3 API calls that would take me at least 15 minutes to build, only took Devin 43 seconds.

• Devin can “visually see“ web pages in his browser, by taking screenshots of them (presumably via browser automation APIs). This means you can ask it to attach screenshots in the PR to show how it actually looks like after making a frontend change.

  • This doesn’t work currently with GitHub website (the linked images resides internally within Devin’s system, not public accessible).

  • But the screenshots can be viewed both via Devin’s web interface and within Slack.

    

The good/bad thing:

• Devin is very cautious about committing sensitive information into the repository, due to its built-in “security best practices” in the “system prompt”. Maybe this is due to the earlier security lapse that was reported in December, or Devin had been reported to accidentally committed sensitive data before.

• For example, I tried to ask it to commit Stripe test cards (which are publicly available online) to the repo but Devin came back to confirm with me 3 times that I wanted to do it, basically refusing to it. Eventually Devin did commit those in a bash script file (perhaps unknowingly).

• This also causes Devin to go into an infinitely editing loop sometimes, when the user instruction conflicts with the “security best practices” and Devin will edit the file back-and-forth in an endless loop:

  

The bad things:

• Devin is constrained by the limited number of tools available to it (the “senses”).

• Devin can’t visit web pages that requires human-triggered authentication.

  • For example, Devin can’t visit GitHub website and see the PR that it created, or look at GitHub Actions pages to troubleshoot CI issues.

  • To debug an error that happened during CI, it has to be either “look” (RAG) through all the logs from GitHub actions, or use grep to search for keywords. Human developers, on the other hand, can go to GitHub website, visually locate the relevant logs, and quickly narrow down the issue to specific lines where the error happened.

  • I don’t think the human / visual troubleshooting approach is necessarily the best or more efficient, but it does outperform what Devin is doing now.

  • I assume Devin can’t access action logs on GitHub website like a developer normally would, because it is an app on GitHub and GitHub apps typically only allow with API access.

  • This also causes issues where Devin can’t fix UI some issues because it is not able to visually inspect how it looks like via human eyes (for example, a complex layout on a web page, or the aspect ratio of an image).

Update on 17 January 2025: I discovered that Devin can in fact “see” the web pages by taking screenshots in his browser. You just need to ask him to do it.

• Some other examples of actions that are trivial for a developer, but (presumably not possible) for Devin:

  • Copy-pasting a few lines from one file to another

  • Visually see the lint errors via IDE built-in lint system

  • Use IDE built-in features to perform refactoring

• I believe Devin currently relies primarily on LLMs (with some custom formats or additional logic on top) to “edit files”, which is inefficient and prone to errors. This makes Devin less efficient in handling some types of tasks like refactoring or troubleshooting issues.

• Devin is really slow in completing certain tasks, much slower than software engineers. Sometimes it can be appear to be idle for a few minutes. For example, a task that would take me 30 minutes to complete end-to-end took Devin more than 2 hours.

  • I suspect this has to do with either the underlying LLMs’ latency, or the endless loop due to conflicts between the “system prompt” and the user instructions.

• Devin's performance degrades in long sessions. A warning appears when the session has been going on for a long time (at around 2.5 hours or 10 ACUs).

  

  • I suspect this has to do with the effective context window of the underlying LLMs. When the session goes on for too long, the history cannot fit into the context window, and the cost for each subsequent LLM call keeps going up.

  • Claude gives similar warning on their UI as well.

• Devin generates two plans for each task, one in plain text form inside the chat, and one using DSL / pseudocode inside planner. These two do not always align on the steps. The planner might miss out some important steps that was mentioned in the chat.

• Devin sometimes does not pick up the relevant knowledge from the knowledge base (recall problem in RAG). And in cases where it does pick up the relevant knowledge, there is a chance that Devin will not use that knowledge anyway (likely a limitation of the reasoning capabilities of the underlying LLM, and Devin’s internal planning logic).

Overall Devin feels like a smart developer being asked to code while being handcuffed.

Pricing - Fair

All pricing below are in USD.

Devin is currently priced at $500 a month for 250 Agent Compute Units (ACUs).

Worth noting that on the plan page, $500 is listed as the “discount price”, whereas the original price is $1250.

I got 50 additional ACUs after onboarding (300 in total), which is supposed to be a gift for scheduling a call, even though I didn’t actually schedule one due to very limited slots around 3am in Singapore.

Update 22 January: Turns out, I didn’t get 50 additional ACUs. Instead, it was showing 300 ACUs because I had configured 50 extra ACUs for “monthly additional usage budget”. I didn’t realize that Devin includes ACUs in the “budget”, that was not yet used, as part of total ACUs available.

For additional usage beyond the included 250 ACUs, it is $2 / ACU. So the additional usage charge is exactly the same as the base subscription price, at $500 per 250 ACUs.

This means Devin, as of now, is essentially using a volume-based pricing of $500 per 250 ACUs.

I think this pricing makes sense because the cost of operating Devin does scale linearly with the amount of compute and task.

Agent Compute Units (ACUs)

According to Devin’s official documentation:

Currently, each ACU is approximately equivalent to 15 minutes of active Devin work. As reference, simple frontend fixes usually take between 5-15 minutes.

In my first round of testing, I ran 4 simple to medium tasks across 2 repos, in total 7 Devin sessions (2 extra sessions for “Verify Repo Access Tasks“ / onboarding, a one extra session because Devin crashed once). Completing these 4 tasks successfully costed me 46.47 ACUs ($92) in total.

So each task costed me about 11.5 ACUs, which translates to $23, or approximately 172.5 minutes / 2.9 hours.

I think I would be able to finish these tasks faster myself (maybe an hour to 1.5 hours, depending on the complexity). But I am happy to pay Devin $23 to do a task that would have costed me an hour (in a frictionless way), so that I can be free to do other more meaningful things.

Cost of Devin vs Junior Engineer

In terms of the cost, Devin costs $2 per 1 ACU (15 minutes of Devin active work) at a flat rate. The hourly rate would be $8 per hour.

Say we hire Devin full-time for a month (22 days working days, each day 8 hours). That would translate to $8 * 8 hours * 22 days = $1,408 a month.

However, Devin is slow in completing some tasks (compared to a good junior engineer). So I would multiple the cost by a factor of 2 to 4 to achieve the same productivity. So it would cost between $2,800 to $5,600 USD a month for Devin to achieve a full-time junior engineer level productivity. This is similar to the salary range of a junior engineer in Singapore.

A key benefit of Devin, however, is that you can just hire him by entering your credit card. And you can hire (scale up) as many Devins as you want to work on things in parallel. You can’t hire human engineers that quickly, and humans comes with overhead cost of managing them.

Conclusion

Despite being publicly available, I would say Devin is still at a very early stage in terms of maturity. There are some key constraints on what Devin can do. And it has some quirks that seems bizarre for an AI software engineer.

However, I must say that the user experience (UX) is by far the best I’ve seen in recent years, surpassing ChatGPT and Claude.

I am hopeful that if Cognition can fix these quirks and find novel solutions to overcome the constraints, Devin can really change the entire software engineering landscape.

Worth the $500 for me.

I will continue to use Devin daily, give it more variety of tasks, and post more in-depth analysis in the future. Subscribe to receive them.

Subscribe now

Subscribe now