How a fintech gave its engineering org a fleet of digital coworkers, and what building the platform taught us. Field notes from building Clarity, ClearTax's in-house AI agent platform.

TL;DR

ClearTax built Clarity, an internal platform that lets any team stand up production AI agents — "digital coworkers" that safely act inside real systems, not just suggest. It runs a few dozen today: a digital SRE that triages Sev1s in ~20 seconds, cross-team debugging agents, a security scanner that cut false positives (9 "HIGH" → 3 real), a data agent that turns change requests into merged PRs, and a plain-English metrics tool scoring 95%+.

• The argument. The durable advantage in enterprise AI isn't the agent loop — that's a commodity you buy — it's the platform you wrap around it (governance, integration, durability, delivery). That layer is org-specific and can't be rented.
• The shift. Copilot → operator. A copilot assists judgment; an operator executes delegated authority. Making that safe is a governance problem, not a modeling one.
• The stack. A swappable Claude Agent SDK core loop · Temporal for durable orchestration · a Cedar policy engine (observe-before-enforce, fail-open behind a deterministic threat scanner) · per-user identity tokens · MCP into ~60 systems and 500+ tools · an org-written skill library (vibes, 67 skills) · and a self-maintaining runbook where usage is maintenance.
• The surprise. A shared, governed platform became more than the sum of its agents: it made a large, siloed org legible to itself and built a commons that gets smarter every time it's used.
• Kept honest. It's confidently wrong often enough that humans stay on the loop; governance coverage, cost controls, and the compounding loops are still maturing. A digital coworker is a force multiplier, not an engineer — and one that knows when to refuse.

Buy the primitives. Build the platform.

A few numbers to set the scene:

• ~20s — Sev1 first response, in-thread
• 9 → 3 — "HIGH" scan findings that were actually real
• ~19× smaller image after one fix (1.76 GB → 94 MB)
• 95%+ on a 176-case plain-English metrics eval

A Sunday morning that no human started

At 5 a.m. one Sunday, a production alert fired: one of our services had tipped past half its pods into CrashLoopBackOff. A Sev1. On most teams the next few minutes are a scramble. A phone buzzes, someone blinks awake, fumbles for a laptop, starts pulling up dashboards.

That is not what happened. About nineteen seconds after the page, a reply appeared in the incident thread. It had already pulled the incident record and cross-referenced the runbook for that failure class. By the time the on-call engineer acknowledged the page, a working hypothesis was waiting.

The responder was not a person. It was Clarity, the internal AI agent platform we have been building at ClearTax, and specifically the agent we treat as our digital SRE. It didn't summarize the alert. It acted: it reached into PagerDuty, our observability stack, and a shared library of runbooks, and ran the first pass of the investigation.

This post is about how we built that and what we learned. It makes one strategic claim, that the durable advantage in enterprise AI is not the agent loop, which is becoming a commodity, but the platform you wrap around it, and then it shows the work: the real architecture, the tradeoffs, and the things we got wrong. Take the argument, the blueprint, or both.

The real shift is copilot → operator

Most of the conversation about agents fixates on capability: can the model plan, use tools, write the code. Those questions are getting boring; the answer is increasingly yes. The question that reorganizes how you build is authority: what an agent may do on its own.

Ivan Peychev draws the line I find most useful. The shift that matters is from copilot to operator. "The copilot assists judgment. The operator executes delegated authority." A copilot sits beside a human who still clicks the button and merges the PR; an operator changes the system. And an operator without identity, scope, policy, evidence, the ability to refuse, and an audit trail is, in his phrase, "just a tool-connected model with borrowed credentials."

That reframes the job. A process needs a workflow; a role needs an identity, a scope of what it may touch, a way to escalate, and a record of what it did. Clarity is our attempt to staff roles and make staffing them safe. Almost everything hard about it turned out to be a governance problem, not a modeling one.

What Clarity is, in one paragraph

Clarity is an internal platform that lets any team at ClearTax stand up a production AI agent, a digital coworker that can safely act inside our systems. You reach one by mentioning it in Slack or opening its chat UI; it does the work and reports back, and it acts as the human who asked, with that person's identity and permissions, inside our own cloud boundary. Today it runs a few dozen such agents: a digital SRE that triages Sev1 incidents, a data-modelling agent that turns analysts' change requests into reviewed pull requests, debugging agents embedded in each product team's support channel, and more.

Under that surface, every agent shares one spine: a Claude Agent SDK core loop for reasoning and tool use, Temporal for durable orchestration, a Cedar policy engine for fine-grained authorization, an org-wide code graph that indexes every repository we have, and the Model Context Protocol (MCP) as the universal adapter into sixty internal systems, from observability and incident management to code, data warehouses, CRMs, and our own tax-filing and reconciliation platforms, exposing well over five hundred tools.

What it does across the org

It is easy to demo an agent; it is hard to make one busy engineers reach for under pressure. That is the test we care about, and the range of jobs Clarity has taken on is wider than we expected.

It runs incidents like an SRE and debugs across team lines. Alerts stream across eight or more service lines on a busy day; Clarity triages Sev1s in-thread, first response in the tens of seconds, running a first pass against the shared runbook library. Often its most useful output is that the problem is not ours: when a government tax portal we depend on starts failing, it says so with evidence, so nobody wakes a service owner for an upstream outage. In the product support channels it goes deeper. A ticket lands ("this return is throwing a duplicate-invoice error"), and the debugging agent traces it into that team's code, cites the file and line, checks git history, and delivers a verdict. Once it called a filing failure a portal-side rejection rather than our bug; days later the portal's own decrypted response matched.

It takes the noise out of security scanning. Static scanners flag too much, so Clarity runs a static pass and then a live, non-destructive probe against production that labels each finding real or false with HTTP evidence. On one service, nine "HIGH" findings collapsed to three; the rest were already blocked by the production auth layer. Confirmed findings become regression templates so a fixed hole cannot quietly reopen.

It turns change requests into pull requests: the analytics team's near-daily "add this column, change this logic" asks become reviewed PRs, most merged the same week. And it gives the whole company its numbers in plain English. Ask "what was the GST filing success rate in India last month?" and Clarity finds the right dbt model, runs a read-only query, and returns the number with the exact SQL it ran, so an analyst can check it, or pins it as a live, shareable dashboard rather than a throwaway answer. A 176-case benchmark for one domain scores above 95%, and when a question has no model yet, the gap routes to the data team, who add it, widening self-serve coverage for everyone: "every gap you raise makes the model smarter."

It also builds and teaches. Clarity wrote our internal security-scanning tooling end to end, root-causing its own CI build failures along the way, and it ships Terraform skills that catch drift and generate cloud resources pre-checked for least-privilege IAM. My favorite example is the most ordinary. A pod-init Sev1 turned out not to be a code bug at all: a 1.76 GB container image was cold-pulling for over nine minutes on a fresh node. Clarity explained why, then walked the team through multi-stage builds and standalone bundling. Weeks later they shipped it: the image dropped to 94 MB, nearly nineteen times smaller, and CI fell from three minutes to one. The agent had not closed a ticket; it had raised a team's baseline.

The reach is not only engineering. The same platform runs a Salesforce on-call agent, a customer-360 assistant, and CRM-enrichment agents, so sales, support, and operations draw on the same substrate as the SREs and the data team.

The capabilities were the point. The effect that surprised us showed up between teams.

The collaboration bridge

Every large engineering org has the same problem: teams are islands. The team behind authentication and internal comms knows those systems cold. Data Platform knows its warehouse models. The GST team knows its filing logic. Security knows the threat model. Support knows the customer's pain but not the code. The ferries between the islands, the threads and tickets and "can someone from X take a look," are slow and lossy.

Clarity became a ferry that speaks every dialect. The same agent identity shows up in all of those channels, each instance reading that team's repositories and wired to that team's tools. Because it reads code, it translates: support staff who do not write software file a ticket in plain language and get back a root cause with the file, the line, and a verdict on whether it is a bug, a data problem, or an upstream rejection. That is a clean hand-off to the owning engineer with nobody in the middle.

Teams have started putting it ahead of their own people. One team's on-call template now tells requesters to read Clarity's answer before paging the human rota; they slotted a digital coworker in as gated tier-0 in front of scarce senior engineers. We did not push that; a team reorganized its own escalation around it. And anyone can add one: a non-platform engineer installs a new agent from a template in a single click, no code deploy and no platform-team ticket. The platform team stopped building every agent and started paving the road.

The cumulative effect is quiet: the platform makes the organization legible to itself. A support person reaches into engineering's reality. An engineer gets a security verdict without booking a security engineer's hour. A co-founder asks the same agent, in the same thread, the same kind of question a first-week engineer would, and everyone reads the same grounded answer.

But a shared surface is only as good as the knowledge behind it, and knowledge left alone rots. Solving that turned out to matter more than any single agent.

The compounding commons: a skill library, and a runbook that maintains itself

Two contribution surfaces keep Clarity getting better without the platform team as a bottleneck. Both are things the whole org writes to.

The first is vibes, our shared library of Agent Skills: sixty-seven of them at last count, one per recurring job across every team's domain, from per-service debuggers and security scanners to metrics skills and meta-skills that improve other skills. The whole org contributes by pull request; skills ship through a small CLI and are versioned like any codebase, so when one team adds a skill, every other team's agents inherit it on the next sync. The platform team owns the road; the org co-builds what rides on it. Skills matter because that is where an organization's business logic lives, and business logic is what raw data and raw models lack. When we benchmarked the plain-English metrics tool, one query, a revenue figure that needs currency conversion, was accurate with the business-logic skill loaded and off by roughly 40% without it, because the data model has no currency column. Same question, same table; the skill carried the knowledge.

The second surface is the one I am proudest of, because it beats a problem every engineering org has lost to: the self-maintaining runbook. Every runbook, RCA, and knowledge base dies the same way. The cost of keeping it current outruns the value it returns, so the pages go stale and the real "we have seen this a hundred times, here is the fix" stays trapped in Slack threads and people's heads. Carving out doc-writing time never survives a busy quarter.

So we inverted it. Usage is maintenance: the runbook is built by using it.

Packaged as a framework in vibes and modeled on Andrej Karpathy's LLM-wiki pattern, it runs two loops. In the interactive loop you ask a question ("why are 502s spiking on my service?"). The agent reads your notes, the code, and live logs, answers with file:line citations, and asks whether to file what it just learned back into the runbook. In the autonomous loop, a scheduled "gardener" harvests the on-call channel's resolved threads twice a week, follows the linked ticket or PR to the real fix, redacts secrets and names, and opens a pull request a human reviews. It never merges.

The hard part is not generation. It is trust. An LLM that confidently writes a wrong runbook is worse than no runbook. What makes the auto-grown entries safe is that the trust gate is deterministic rather than another prompt: a fail-closed check requires every entry marked Verified to carry a verbatim quote from its source thread, and downgrades anything hedged or unquoted to Hypothesis. Code, not the model's judgment, is what stops the gardener from inventing specifics. Two more choices earned their keep. Runbooks are anchored to the on-call channel rather than the service, because at debug time all you have is a symptom and the channel it landed in, so the knowledge should live where you start looking. And the engine is global while the data is per-team, one plugin with each team's runbook in its own repo. The result is institutional memory that grows where it is used and gets smarter the more it is used.

The architecture

The principle underneath everything is a separation of concerns. Reasoning is a commodity we buy. Durability, governance, integration, and delivery are the platform we build.

The core loop, and why it is swappable. Every run executes through an executor. The default drives the Claude Agent SDK through its stateful client, so a permission callback fires on every tool call; that callback is where governance lives. A non-negotiable safety preamble is prepended to every agent's prompt. The important decision is that the loop sits behind a strategy seam: several executors run in production today (the Claude Agent SDK by default, an in-house executor, and a Gemini CLI executor), chosen per-agent by config, and open-model portability is real because the in-house executor carries a provider registry of roughly seventeen providers. Adding a model is a config row. That is the answer to "aren't you locked into one vendor": the reasoning layer is a replaceable part. A run can also fan out into parallel sub-agents; we have put the same question to a fast model and a strong one at once and compared depth against latency.

Temporal: durability without heroics. Agent runs are long, messy, and failure-prone, so every asynchronous run is a Temporal workflow with a deterministic ID derived from the Slack channel, thread, and event timestamp, which de-duplicates redelivered events for free. We wrap the entire non-deterministic run in a single activity, with a one-hour backstop and a self-healing ten-minute heartbeat, so Temporal's determinism never fights the SDK's messiness; the cost is coarser replay, which we accept. Delivery is decoupled from execution: the workflow writes events to Postgres, and the browser reads them over reconnectable SSE driven by LISTEN/NOTIFY, so a user can close the laptop mid-run and pick the stream back up later. Temporal owns durability; the database is the event bus.

Cedar: governance as code, shipped carefully. This is where the platform's real work lives. As Berkeley's California Management Review puts it, once agents become actors, "rather than asking whether agents are capable, the model asks whether they are governable." Authorization runs through the Cedar policy engine, assembled in three layers at load: a base policy for everyone, capability policies an agent opts into by name (bash-read-only, no-secret-exposure), and the agent's own grants and forbids. A detail worth stealing: the base layer is a blacklist, not a whitelist, allow-all plus platform-wide forbids, the key one scoping an agent to the channel and thread that triggered it. At runtime Cedar is the third check in the permission callback, behind a tool-name allowlist and a threat scanner, and it merges the server-trusted session context last so an agent cannot forge its own scope. Two choices define its posture. It observes before it enforces: by default it logs every allow and deny but blocks nothing, so you learn what agents actually do before enforcement risks breaking real work, the "earn autonomy against evidence" idea Google's SRE team calls progressive authorization. And it fails open: if the engine errors the call is allowed, acceptable only because the allowlist and threat scanner sit beneath it. A more paranoid org would choose differently; the point is to choose deliberately, and to audit daily what the policies did.

The threat scanner: a content-level guard. Cedar decides whether an agent is allowed to call a tool; the threat scanner, the layer in front of it, decides whether the call itself is dangerous. It is a deterministic, pattern-based inspector of tool-call arguments that fires before Cedar and blocks a call outright when the payload matches a known-dangerous pattern above a configurable risk threshold. It catches what identity-based authorization alone would wave through: filesystem destruction (an rm -rf on a system path, a dd onto a raw device, a Python shutil.rmtree), remote code execution (a curl piped into bash, an inline python -c), and secret exfiltration (reading an SSH private key, or posting a credential to an outside host). Two things make it more than a blocklist. It canonicalizes the command first, so obfuscation does not slip past a naive regex, and it parses the shell to separate executed code from inert data, so a dangerous-looking string that is only being echoed is not flagged. Cedar governs by identity and scope, the threat scanner by content, and together they are why observe-and-fail-open is a defensible default rather than a reckless one.

Agent identity, and its honest gap. When an agent calls an internal system, the platform mints a short-lived, per-user token so the call carries the requesting human's identity and least-privilege permissions, not a godmode service account. Attribution is clean and blast radius is bounded by a real person's real access. This works cleanly for MCP tools, where identity flows end to end and enforcement centralizes at a gateway. It works poorly for CLI tools like git and gh run through a shell, which authenticate with a baked-in token and carry no per-user identity; there, control drops to credential scoping and command allowlists, and the principled fix is to wrap CLI operations as MCP tools. We say so because a platform that hides its soft edges is not one you should trust.

MCP and self-serve. Clarity reaches sixty systems without sixty bespoke integrations because everything speaks MCP: you add a server as a Python class, a declarative YAML config, or a remote proxy, so "give the agents access to system X" becomes config, not a project. A few dozen agent templates ship in-repo, and installing one is a click that writes a record and auto-configures tools. The one place self-serve stops is Cedar: a genuinely novel agent needs a hand-written policy, and without one it falls back to the permissive base floor. That is a deliberate speed bump; we would rather a new agent's authorization get a human's eyes than let anyone mint an ungoverned operator.

Two pictures pull the pieces together: the shape of the platform, and what happens when you @mention an agent.

The shape of the platform, end to end. A trigger — a Slack @mention, a PagerDuty auto-ingest, or the chat UI — hits a webhook handler and becomes a Temporal workflow whose deterministic ID (channel + thread + event timestamp) dedupes redelivery. The workflow runs an executor (Claude Agent SDK by default), whose agent loop is assembled from a system prompt, a safety preamble, and skills pulled from the vibes skill library. Every tool call passes the permission callback — a tool-name allowlist, then the threat scanner (blocks by payload content), then Cedar (base + capabilities + agent policy; observe/enforce; fail-open), with a daily self-audit reading the decisions back. Once allowed, the call goes out under a per-user identity token through the MCP layer (~60 servers, 500+ tools) into observability (Grafana / Coralogix / Sentry), incidents (PagerDuty), the org-wide code graph, data (Athena / warehouses), CRM (Salesforce), and ClearTax domain systems (recon / GST / ingestion).

The path a single request takes, from an @mention to a reply:

The tradeoffs, and what we chose not to build

Every honest architecture post owes you the debits.

• One durable activity versus fine-grained replay. We kept the workflow clean at the cost of coarse replay. Know your own failure modes.

• Observe-before-enforce and fail-open. We optimized for shipping governance without breaking agents and lean on layered defenses to make fail-open acceptable. It is a real risk surface, and we accept it consciously.

• A duck-typed executor seam, not a formal contract. Adding a runtime is cheap because there is no compile-time interface, and that looseness is debt we have a proposal to formalize.

• Self-host over managed. We run the whole thing inside our own cloud boundary. For a fintech handling tax identifiers under Indian data-residency expectations, that was a requirement, and it costs us operational toil a managed product would absorb.

We were also disciplined about what not to build: we did not train our own model, did not hard-enforce authorization on day one, and did not give agents a shared superuser identity.

What we're still improving

Nothing here is finished. A few things we are actively working on, and would rather name than have you find:

• It is confidently wrong often enough that you have to check it. This is the METR failure mode turned on ourselves. Clarity has produced file-and-line-cited analyses a human later disproved, and once assessed a security control at the application layer while missing that a network rule already covered it. The retraction protocol and human review catch these, but on a high-stakes call you cannot take its answer on faith.

• Governance coverage is not uniform. Cedar governs MCP tools cleanly, but not every tool enforces authorization to the same standard yet, and shell CLIs still run outside per-user identity. The control plane is strong, not complete.

• Cost governance is nascent. Agent runs are token-hungry; a single deep scan can burn a person's daily model quota, and we do not yet have mature per-team budgets or rate caps. At scale, spend becomes a governance surface of its own.

• The compounding loops still bottleneck on humans. The runbook gardener never merges, roughly 40% of one recent week's policy flags were false positives, and the agent triages nearly everything but closes few hard tickets on its own. The flywheel is real but throttled by review bandwidth.

• The long tail is brittle. Headless-browser automation flakes on some third-party portals, skill sync can overwrite a local patch, and the harness has occasionally answered a message twice or leaked internal scratchpad narration into a channel.

Two deeper questions sit underneath: how do you evaluate an agent whose surface area grows every week, and how do you weigh governance against capability when a security task legitimately needs the agent to do what the guardrails forbid. Neither is settled; the loops that keep them visible are how we make headway.

"Why build this? Just use a managed agent product."

Everyone asks this, and it deserves a straight answer.

The managed-agent market has matured. Anthropic runs a hosted harness with credential vaults and self-hosted sandboxes so tool execution can stay in your VPC; Cloudflare offers agents as durable objects at the edge; AWS Bedrock AgentCore, Google's Vertex Agent Engine, OpenAI's AgentKit, and LangGraph Platform each cover a layer. If you can buy this, why build?

Our answer is not "managed is bad." It is a distinction about what you buy and what you build: buy the commodity primitives, build the org-specific platform. Buy the inference. Buy or open-source the agent loop. But build the layer that is genuinely yours, because that layer is where the flexibility lives. Owning it means we can wire an agent into any of our sixty internal systems the week we need it, route a task to whichever model fits its cost and latency, and shape authorization to exactly how our org works, all on our own schedule instead of a vendor's roadmap. That control is the point, and you cannot rent it. It is also, not by coincidence, where the regulatory liability sits, which is the harder-nosed reason to own it: as one build-versus-buy analysis puts it, "there is no scenario where you transfer the regulatory liability." The clearest way to see where that line falls is against the two alternatives we weighed hardest: the agent loop itself, and the whole runtime.

The loop. "Buy or open-source the agent loop" was a real question for us, not a slogan. We tried adopting an open-source harness, pi.dev (the one PrimeIntellect's prime-agent builds on), in place of our Claude Agent SDK loop, and found enough gaps that we kept our default: it runs a different language stack with an embedded code-execution kernel, which adds an integration seam and a new prompt-injection surface, and its headline draw, open-model flexibility, we already have through our own provider registry. So we kept the seam and filed it as a deferred ADR.

The runtime: a Durable-Objects stack versus Clarity

The bigger question was the runtime, and Cloudflare's Agents are the strongest version of the "just buy it" case, so it is worth being specific about why we went another way. A sibling platform inside ClearTax ran the evaluation in depth against Cloudflare's "Think," a reasoning runtime built on Durable Objects.

A Durable-Objects design colocates state with compute: one durable actor per conversation, its own embedded SQLite, single-threaded execution that serializes turns. Clarity does the inverse and separates state from compute: durable orchestration in Temporal, stateless workers, and one shared store.

The actor model genuinely wins on several axes, and we concede them: per-conversation serialization for free, resumable token streaming, enormous concurrency with almost no operations work, and the fastest path to a chat product. But Clarity is an operator platform, not a chat product, and the things it is built on are the things the actor model makes hard:

• A shared store, not a silo. One store lets us run multi-step task graphs, ask cross-conversation questions, and produce a regulator-grade audit trail, and stateless workers scale independently of any hot conversation. Per-actor SQLite silos all of that.

• The control plane, owned. Durable Objects give you durable actors, not Cedar policies, per-user identity, a threat scanner, or observe-before-enforce. On a Durable-Objects stack we would have built the entire control plane anyway, and that, not the runtime, is the hard part we most needed to own.

• Our boundary, not the edge. Durable-Object state and compute live on Cloudflare's edge in TypeScript; ours live inside our own cloud boundary, next to the sixty systems we integrate, behind a language- and model-agnostic seam.

The honest summary, borrowed from that evaluation: the actor model "buys serialization, locality, and cheap streaming," while separating state from compute "buys cross-conversation orchestration and compute that scales independently of state." For a fleet of governed operators, we wanted the second set. We do not reflexively build, and we do not reflexively adopt; we test the line against real alternatives before we commit.

The counter-arguments are real: managed is faster to start, someone else patches the injection holes, and managed harnesses track new frontier models automatically. That last is the sharpest, since a homegrown platform can fall behind. But Anthropic's own engineers argue that harnesses encode assumptions that go stale as models improve, which is why they built a managed one, and that supports our design: we buy the loop that keeps pace and build the layer that is ours. We drew the line rather than rejecting the vendor, and by shipping self-hosted sandboxes so execution stays in your boundary, the vendor is validating where the line belongs.

Buy the primitives. Build the platform.

How it raised the bar, and a necessary dose of humility

Step back from the architecture and the outcome is cultural. Governance-as-code became the default, and the platform grades itself daily, reading back its own authorization decisions and scoring its incident diagnoses against past runbooks. We grade the grader. Runbooks became living assets a digital coworker maintains, and the floor rose: code-grounded answers and tens-of-seconds incident response are the baseline now. The humans, meanwhile, moved up the abstraction ladder, from writing every line to, in Addy Osmani's phrasing, conducting one agent and then orchestrating a fleet. That ladder is not gated by title. With a capable platform beneath them, engineers earlier in their careers can operate with more context and confidence than their experience alone would give them, because the platform supplies the durability, the guardrails, and the institutional memory that seniority used to stand in for.

I want to be careful here, because this is where AI writing usually loses the plot. The evidence for autonomous engineering is mixed, and pretending otherwise would undercut everything else in this post. A controlled study by METR found that experienced open-source developers were 19% slower using early-2025 AI on large codebases they knew intimately, while believing they were faster. That perception-versus-reality gap is the most important humbling fact in the field. Independent trials of "autonomous engineer" products keep finding the same failure mode: plausible, well-structured code that does the wrong thing. The honest framing, borrowed from one such review, is that a digital coworker is not an engineer. It is a force multiplier.

That is how we treat Clarity. Humans stay on the loop: they set boundaries, review the PRs, approve escalations, and read the daily audit. It is telling that the same agent that beats a human to a Sev1 will also refuse. It has declined to bypass a live authentication control ("that's a control, not a bug"), declined to rewrite git history destructively, and retracted its own conclusion after re-reading the logs disproved it. A force multiplier that knows when to stop is worth far more than one that always says yes. Which returns us to that Sunday morning. The digital SRE did not replace the on-call engineer. It did the tedious first pass so the engineer arrived to a hypothesis instead of a blank dashboard.

The surprise was not that agents could do engineering work; everyone expected that. It was that a shared, governed, deeply integrated platform became more than the sum of its agents. It made a large, siloed organization legible to itself, and it built a commons that got a little smarter every time someone used it. We set out to build digital coworkers. We ended up building a bridge.

Clarity is ClearTax's internal AI agent platform. If you are building something similar and want to compare notes on governance, durable orchestration, the trust problem in self-growing documentation, or where to draw the build-versus-buy line, we would love to hear from you.

MCP is here to stay. It's become a Schelling point for the community, and I feel it will continue to gain traction. Just as I was writing this, I noticed that Anthropic has donated stewardship of the MCP protocol to the Agentic AI Foundation.

We've been thinking deeply about our MCP integration story, and I have recently started to build strong conviction that MCP Resources are criminally underrated. To understand why resources are useful, let's first understand the problem with MCP tools.

MCP Tools have a context problem

A typical MCP integration works like this:

• The agent's harness exposes MCP tools to the LLM provider
• In response to a completion API, the LLM may invoke an MCP tool
• This tool's response is given back to the LLM

An MCP server could expose any tool. For example: a Google Drive MCP could be exposing files on a connected Google Drive; and these files could wildly vary in sizes.

What happens if the tool response is huge?

Naive solution: send the whole response to the LLM? This is expensive: maybe the tokens consumed here could be better used elsewhere? Maybe the response is larger than the LLM supports and your agent crashes?

Another option is truncation. Either the MCP server itself could truncate large results, or the agent harness should anyway truncate results over certain sizes.

This gives you safety: MCP tool calls cannot consume unlimited tokens. But this comes at the cost of context loss: who decides what to truncate? What if the most critical information needed for the agent was truncated?

MCP Resources are a solution

The MCP protocol has support for resource templates:

Resource templates allow servers to expose parameterized resources using URI templates.

Resources are exposed as URIs. Critically: a tool response could contain resource links, and we have the agent harness access the entire resource.

This means:

• Large responses from tools can be converted to resources
• Agents and agent harnesses can inspect resources directly
• Depending on your system design, you could compose these together with other LLM tools and get to interesting emergent behaviour.

Here's an example:

// Request
{
  "method": "tools/call",
  "params": {
    "name": "read_google_drive_file",
    "arguments": {
      "id": "c85b4851-bd61-4352-84a2-6e0c6b6d3dce"
    }
  }
}

// Response without Resources
{
  "content": [
    {
      "type": "text",
      "text": "...... huge text"
    }
  ]
}

// Response with Resources
{
  "content": [
    {
      "type": "text",
      "text": "...... small text snippet"
    },
    {
      "type": "resource_link",
      "uri": "gdrive://file-id-123"
    }
  ]
}

This can enable composition:

• The agent's harness could read the resource and save the file to disk
• The agent could then use other local tools (eg: read_file, grep, execute_code, etc) to further process this file.

MCP Resource Links

The MCP protocol in fact has a first-party way of doing this: resource links:

A tool MAY return links to Resources, to provide additional context or data. In this case, the tool will return a URI that can be subscribed to or fetched by the client.

Most MCP servers don't expose resources though. This part of the protocol seems really unexplored and under-appreciated.

Addendum: Code Execution

Anthropic recently wrote about a different pattern: using code execution to interact with MCP servers. This is a really powerful pattern, though I feel it's something that can be combined with MCP Resources.

• MCP servers are fast becoming a glue layer
• Any given MCP server doesn't know which type of client would connect to it.
• The server doesn't know if the client is using code execution or just LLM tool calls.
• Thus, well-behaved servers need to be defensive about context usage in tool calls, and large responses will end up being truncated somehow.
• Resources are the officially blessed way to expose data to clients.

Code execution doesn't remove the need for MCP resources. It makes MCP resources even more useful!

We're adopting this MCP resource-link pattern heavily in servers we control. Our agents will support this pattern as well. Unfortunately, MCP Server authors need to start adopting this pattern for it to gain traction though.

We're hoping that this post can spark a discussion about this.

We're building an agent system and I have been evaluating adding support for MCP.

We really care about our tool design to an unreasonable degree. I have personally spent days thinking about the right design for some of our tools.

I couldn't help but over-think exactly how MCP should be exposed to our agents. At a high level, I felt that there could be two integration patterns — I would name them the 'meta tool pattern' and 'materialised tools pattern'.

Meta Tool Pattern

The meta tool pattern: the agent sees a few meta tools like 'list available MCP servers', 'describe MCP server', 'invoke MCP tool'.

The agent can decide to explore the capabilities of an MCP server and invoke its tools, when necessary. This design is flexible and efficient, but also requires a more capable agent:

• This pattern uses less context by default. Tool definitions are lazily loaded only when required.
• The 'discovery' calls for individual MCP servers need to happen only once in a given chat thread.
• There's no guarantee that the agent will decide to explore the installed MCP servers though.

Materialised Tools Pattern

Here, whenever a MCP server is enabled – the system will automatically discover all available tools in the specific MCP server and eagerly copy them to the list of tools available to the agent. The agent sees all of these tools by default. Calling an MCP tool is just like calling any other tool.

• This pattern makes tools really explicit to the agent.
• This comes at the cost of using additional context, even when it's not necessary.
• If multiple MCPs are installed, you may fast run out of context space.

Evaluating these patterns

If you were designing an agentic platform, which option would you choose? I did a survey of some existing open source systems and here's what I found:

• codex-cli also uses the materialised tools pattern
• opencode uses the materialised tools pattern
• Cline uses a hybrid
  • All tools exposed by enabled MCP servers are copied to the system prompt
  • A single use_mcp_tool tool is used to invoke them
• OpenManus also uses the materialised tools pattern.

This was a surprise. I'm not sure why the existing implementations are so heavily skewed towards materialised tools!

Our Decision

At this point I'm inclined to go with the meta tool pattern — it seems to make sense the system we're building.

What I have noticed is that:

• Most MCP servers don't have a great agent interface.
• They expose too many fine grained tools with overlapping responsibilities.
• They are really wasteful of context tokens.

Most importantly: the meta tool pattern just intuitively feels like the right solution to me.

There are 100s of small details like this that go into building great products – and I have a feeling that these details are actually what differentiates your product when everyone is building on the same foundation models.

The Mythical Man Month famously had the observation that adding manpower to a project that's behind schedule will often delay it even further. Additionally, in 'No Silver Bullet' Fred Brooks further states that:

There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.

How does this change with the advent of AI coding agents? Can coding agents give us the mythical 10x speedup?

My thoughts below. I've divided this blog post into sections which argue both for and against transformative change.

My position: coding agents are a fundamental shift in how we'll build software over the next few decades, and we're overestimating the impact in the short term while underestimating the impact in the long term.

Headwinds

Vibe Coding vs Writing Production Software

We should differentiate between 'vibe coding' as a way to experiment / prototype, and using AI to write production software. Simon Willison has an excellent post on this.

For production quality software, all basic tenets of software engineering apply. Code reviews, tests, architecture designs, software design reviews, etc.

Most of a senior engineer's time is often spent in these activities, not just writing code.

Coding agents help a great deal here, but you won't see the same speedup as pure vibe coding — where you build software without reviewing the code your agent writes

At least for now: coding agents aren't good enough to fully autonomously build features and ship them to production without human review.

Essential Complexity vs Incidental Complexity

As Fred Brooks pointed out in the above essay, software development consists of both essential complexity and incidental complexity.

Incidental complexity could be things like figuring out how to write a Dockerfile, or learning how a specific library works, or dealing with framework specific issues. Coding agents can be a huge help here.

Essential complexity is the core problem you're trying to solve. Coding agents can definitely help here, but you still need to pay close attention — humans will remain the bottleneck here.

Amdahl's Law basically gives a ceiling for the performance gain that automation / parallelisation gives for any given task. You are only as fast as your bottleneck.

Decision Fatigue and Time Compression

Faster coding actually compresses timeframes and lets you focus on the hard decisions, on the essential complexity. Coding agents let you focus on substance of your problem.

But human capacity for deep thought is limited!

So now, your day to day working with AI coding tools is going to be a series of hard-decisions that you need to think deeply upon, decisions that require high amount of mental effort.

Decision fatigue is real. If you have to make a weeks' worth of hard decisions in a day, your decision quality will suffer.

AI coding will exhaust you if you're not careful. Human beings need to be able to step back and think about problems. We need to go for walks, ruminate on ideas and just wander through a problem space.

Effective Communication & User Skill

AI agents need engineers to be effective communicators, and this is a problem. Most engineers aren't the best communicators. Every great coder isn't automatically great at delegation.

Effective communication is a skill. Writing clearly is a skill. And using coding agents effectively is a skill.

For example, here are two recent articles that go in-depth about the craft of using AI agents to code:

• How I use Amp
• AI-assisted coding for teams that can't get away with vibes

Craftsmanship takes time. Skills take time to build. People who are great engineers today won't automatically be great at using coding agents. Getting better will require deliberate practice, and approaching this problem with a beginner's mindset.

Headwinds Summary

Given that:

• Production software (currently) requires human oversight
• Essential complexity remains
• It will take time to learn how to use the coding agents effectively

Is an immediate 10x improvement in velocity possible? It seems there is truly no silver bullet.

Tailwinds

AI Scaling will continue

Agents will keep getting better. Underlying models will keep getting better. We've learned to not bet against scaling.

According to one recent viral benchmark, the length of tasks that AI can do uninterrupted is doubling every 7 months.

From my personal experience, I know that each recent big model release (eg: Sonnet 3.5, Sonnet 3.7, Sonnet 4.0) has made building agents easier. The LLMs are getting better at following instructions, at using tools, at planning, and just at showing agency.

It's hard to predict the future, but it's definitely possible that soon, a large majority of code written won't need human review and oversight.

Most work isn't 'Deep Work'

While my arguments above hold (essential complexity remains, decision fatigue is real) — let's be real and acknowledge the fact that most of us don't do deep work 100% of the time.

A lot of time goes into glue work, into getting various subsystems to behave, dealing with broken tools, etc.

AI can be a huge accelerator for these types of work. It's possible that this itself can be a 10x improvement for many organisations!

Quantity has its own Quality

If you're working in deep tech, if you're building something complex — AI coding allows you to try more approaches. You can build quick and dirty throwaway prototypes, and validate more ideas.

Quantity has a quality of its own. If you can do more iterations, you can get to better decisions. If you can actually build multiple candidate systems, you can make more informed choices — architecture / design decisions become easier with data.

I have personally seen this pay off: while building a zero to one product, I have been able to do many parallel experiments and actually test 10s of ideas before deciding upon a plan. This has enabled me to make bold system design bets with high confidence.

Ambition & Moonshots

AI agents allow you to be more ambitious. On the margin, with higher productivity it's possible to devote more time and resources towards building something better than you would have previously.

You can think of a productivity gain as either:

• Work 10x faster
• Build something 10x better

Either option is fine! In fact, it may be the case that building something 10x better is actually more impactful.

I suspect one of the impacts of ubiquitous coding agents will be the rising baseline quality of software!

Tailwinds Summary

If you consider the facts that:

• LLMs will continue to improve
• We'll all get better at using AI agents
• We'll be able to automate away low impact work
• We'll be able to try many more iterations
• We'll be able to build more impactful, more meaningful software

How can you doubt the impact that coding agents will have?

Conclusions

I've argued both sides of this. My position is:

• We're radically underestimating coding agents
• Most of us are not ready to adopt agents at scale

I think impact and adoption will not be uniform. People will have different lived experiences with AI tools, with some dismissing AI coding as a fad, and some enthusiastically thinking of these agents as a panacea for all their problems.

I think coding agents will have a huge impact that is overrated in the short term, but underrated in the long term.

And I think that today, to get the most out of current generation agents, you have to really dive deep and uncover their limits yourself.

Should AI agents be general purpose or should they be built for a specific task?

I would say yes to both.

Special Purpose Agents

A special purpose agent is an agent built to do one (or a few things) very precisely. Examples:

• RAG on a knowledge base and answer questions only from the knowledge base.
• Convert images to very specific structured data format (eg: convert image to a contact vCard).
• Convert text to SQL on a specific table, with specific guardrails (eg: always apply page size limits).

These agents are built to do a specific job.

General Purpose Agents

A general purpose agent on the other hand would be an agent that has emergent behaviour, that can use its tools to do something it wasn't explicitly programmed for.

This is best demonstrated by an example. I recently coded up a toy agent that runs on the command line that had the following tools available:

• List directory
• Read file
• Query file (via duckdb)
• Read PDF

This agent was designed to have generic tools, and it was allowed to do multiple tool calls if needed. It wasn't given a specific goal.

The lack of specificity actually made the agent more useful! In the last few days, I have used it to do:

• K8S cost optimisation — given a CSV containing some kubernetes utilisation data, this agent helped me find low hanging cost optimisation options
• Data entry for my own personal finance needs — given some account statements, the agent was able to help me digitise them in a format I use for tracking my expenses
• Do Q&A on my meeting notes

Interestingly, the agent showed a lot more agency than I was expecting. For example, it would often execute multiple tool calls in response to a general 'hello' message!

All the recent wow moments I have had with AI are usually with general purpose agents. They can end up giving you unexpectedly rich experiences.

Use Cases & Trade-offs

If you're building an AI enabled product, both special purpose agents and general purpose agents have their place.

• You sometimes want determinism (or close to determinism) and repeatability.

  • For example: if you're building a support desk, you may want to always categorise tickets in a certain way.
  • In such scenarios, special purpose agents are really useful. You can treat them as "intelligence that's an API call away".

• Special purpose agents are limited in scope.

• General purpose agents are where the power of AI agents becomes apparent.

• General purpose agents are going to be more expensive to build.

  • You may need smarter LLMs, you may need to spend tokens on reasoning.
  • General purpose agents could use up millions of tokens.

• Shipping general purpose agents requires you to have the right underlying infrastructure.

  • The agent is as powerful as the tools it has access to. You need to build the right primitives to unshackle the AI model.
  • You need to build the right platform for the AI agents to leverage!
    • For example: if your system processes a lot of data, you may need to invest in robust parallel execution.
    • If the agent could 'ask any question' of your data, you have to think about indexing and database performance.

• The ideal end state of a general purpose agent is a coding agent — an agent that can write code for a specific task and then execute it.

  • This could lead to potential security issues, and you have to look out for other types of abuse.
  • You might need to invest in primitives like sandboxing here.

I see this as a continuum — the more general an agent, the more powerful it is, but you need to invest proportionally into building the right safeguards.

Most use cases may not need a general purpose agent. You can probably start off building an AI-enabled product by just focusing on special purpose agents.

General purpose agents are also the most valuable ones though. And building general purpose agents will require you to invest proportionally into your platform's core primitives.

We've all been amazed, and perhaps a little nervous also If I may say, by the capabilities of Large Language Models (LLMs). They can write poetry, generate code and answer complex questions. For a piece of code, supposedly, doing next token prediction, it surely excels at a lot of impressive tasks. That begs the question - do they actually just predict the next token? Are they planning ahead of time? How do they actually do it? Is the reasoning in the “reasoning model” an afterthought, or did the LLM actually follow the sequence of steps?

This opacity isn't just a matter of academic curiosity. It has real-world implications for safety, reliability, and our ability to trust these powerful tools. If we don't understand why an LLM says what it says, how can we be sure it's not biased, is not generating misinformation or more importantly we can’t control what it says.

Anthropic has taken a significant step towards demystifying these complex systems. I would highly recommend everyone to at least read their blog. They have also published a couple of papers - Circuit Tracing: Revealing Computational Graphs in Language Models and On the Biology of a Large Language Model. I am personally very surprised why more people are not talking about it.

The Challenge: Understanding the Unseen

Imagine trying to understand how a human brain forms a thought - while we have some groundbreaking research in this area, we still have a lot to learn. Our brain is an incredibly complex network of neurons firing in intricate patterns. LLMs, while inspired by neural networks, have their own unique architecture, often involving billions of parameters. Identifying specific, human-understandable concepts within these vast networks has been a very big challenge.

The Inspiration: Looking to Life Itself

The researchers draw an analogy from the world of biology, specifically Gene Regulatory Networks (GRNs). To understand this better let us think about how a single fertilized egg develops into an incredibly complex organism with specialized cells, tissues, and organs. Genes don't act in isolation. They influence each other, turning other genes on or off in intricate cascades. A GRN is essentially a map of these genetic interactions, showing which genes regulate which others, ultimately leading to specific cellular functions and observable traits. Biologists use these networks to understand development, disease, and the fundamental workings of life.

The Methodology: Building an "AI Microscope"

To tackle the challenge of understanding LLMs, researchers are developing tools analogous to those used in biology to map complex systems. At the heart of this new methodology is the idea of charting out the internal computational pathways.

What are Attribution Graphs?

Taking a cue from concepts like GRNs, an Attribution Graph is, at its core, a map of influence within a Transformer model.

• Nodes in this graph represent the key components of the model (e.g., specific attention heads in particular layers, or MLP layers).
• Edges (the connections between nodes) represent the strength and direction of influence. An edge from component A to component B means A significantly contributes to B's activation or behavior.

By constructing these graphs, researchers can start to trace the pathways of information and causation through the network. They can see how an input (like a prompt) activates certain components, which in turn influence others, leading step-by-step to the model's output. It’s like finally getting a circuit diagram for a complex electronic device, allowing us to see how different parts work together to achieve a function.

How Did They Do It?

The creation and analysis of these attribution graphs involve several sophisticated techniques. Have tried to explain them with an analogy. However it is still a very high level overview. I highly recommend the reader to go through the papers to get more details.

• Extracting Interpretable Features: A crucial first step is to identify meaningful concepts within the model. One method involves training sparse autoencoders on the internal activations of models like Claude Sonnet. These autoencoders learn to represent the LLM's internal states using a much smaller, more interpretable set of "features." In simpler terms : Imagine you have a giant paragraph (the LLM's internal activation). It's hard to quickly grasp its main points.A sparse autoencoder acts like a summarizer. It reads the giant paragraph. It identifies a few keywords (the "sparse features") that capture the essence of the paragraph. It then tries to rewrite the original paragraph using only those few keywords. If the rewrite is good, it means those few keywords are indeed very important. Researchers can then study these "keywords" (the sparse features) to understand what core concepts the LLM is using in its internal "thoughts." This helps demystify what's happening inside the LLM.

• Cross-Layer Transcoders (CLT) and Replacement Models: To make the complex model more tractable, researchers build an interpretable "replacement model." This often involves using Cross-Layer Transcoders (CLTs), which help extract features and understand their interactions across different layers of the model. Continuing our previous example - LLM isn't just one paragraph, but a whole sequence of connected paragraphs, like chapters in a book. Each "paragraph" is the activation state of a different layer, building on the previous one. CLTs are like specialized literary analysts who study the connections between different paragraphs (layers). They don't just summarize one paragraph. Instead, they try to figure out: "How do the 'keywords' we found in Paragraph 1 (an early layer) get transformed or combined to produce the 'keywords' we see in Paragraph 5 (a later layer)?". Researchers then build an "interpretable replacement model."(CliffsNotes or a detailed study guide for a specific section of the book) - making a section of the complex "book" (LLM) much easier to comprehend.

• Feature Visualization and Labeling: Once features are extracted, they are visualized and labelled with human-interpretable concepts – from concrete objects to abstract ideas. This is like finding all the sentences (input examples) where this particular "keyword" is most strongly "lit up" or active within the LLM's "thoughts." By looking at many such examples, they might see that this keyword consistently appears when the LLM is discussing, say, "ancient stone castles," "dogs playing in a park," or even more abstractly, "a sense of injustice." Then, based on these observations and the common theme across all these examples, they "label" the keyword with a human-interpretable concept. So, an initially obscure numerical feature might be labeled as "Feature 123: 'Imagery of Medieval Architecture'" (a concrete object category) or "Feature 456: 'Concept of Betrayal'" (an abstract idea).

• Linking Concepts into Circuits: Apart from identifying features, they are also linked to computational "circuits." This allows for tracing the pathway from input words to output words, showing how different concepts interact and build upon each other. Think of this as going beyond just identifying the themes ("keywords") in our book. Researchers try to discover the specific "narrative pathways" within the LLM. For instance, they might find that when the "input words" (the prompt) introduce a character described with terms that activate the "keyword" for 'ambition', this consistently leads to the activation of the "keyword" for 'risk-taking behaviour' a few "paragraphs" (layers) later. This, in turn, might frequently interact with an activated "keyword" for 'external threat', ultimately contributing to the "keyword" for 'heroic sacrifice' being prominent when the LLM generates its "output words." By tracing these connections, they are essentially drawing a "circuit diagram" of the story, showing how one concept (like 'ambition') computationally triggers and combines with other concepts (like 'risk-taking' and 'external threat') to build a coherent narrative or logical conclusion, effectively mapping the journey from the initial premise to the final output.

• Perturbation Experiments: Inspired by neuroscience, researchers conduct perturbation experiments. They modify the internal states of the model (e.g., activating or deactivating specific features) and observe the resulting changes in output. This helps validate the hypothesized role of different features and circuits. Continuing with our analogy →

  1. "Deactivate" a specific feature: Imagine our LLM is about to write a scene, and the "keyword" for "'Concept of Betrayal'" is naturally becoming active based on the story so far. The researcher might step in and say, "For this next passage, let's artificially suppress or mute the 'Betrayal' keyword. Don't let the LLM 'feel' or express betrayal right now."
  2. "Activate" a specific feature: Conversely, they could say, "Even if the current story doesn't strongly suggest it, let's artificially boost the 'Hero's Courage' keyword here and see what happens."

  It's like asking: "If we remove this thematic element (keyword) or alter this plot device (circuit), does the story change in the way we predicted?" This experimental approach allows researchers to confirm whether their understanding of how these internal concepts influence the LLM's final "narrative" (output) is accurate.

Key Discoveries: Peeking into the AI Mind

Let us now deep dive into some fascinating insights, that are highlighted in the papers →

• Identifying and Eliciting Specific Concepts: Researchers found they could pinpoint and even activate specific concepts within the model. The most cited example is the "Golden Gate Bridge" feature: as the Anthropic team states, "When we activate a feature for the Golden Gate Bridge, our model writes about the Golden Gate Bridge." They've identified thousands of such features, ranging from other concrete entities like "the Eiffel Tower" to more abstract notions such as "gender." This demonstrates that the model internally represents these concepts in a way that can be isolated and triggered.
• Multi-Step Reasoning and Sophisticated Planning: One very interesting insight was that LLMs aren't just reacting; they're planning.
  • Forward Planning: A clear illustration of this is how Claude approaches creative writing. The Anthropic blog notes, "For example, when Claude writes poetry, it often plans out rhymes far in advance of using them." This indicates an ability to think several steps ahead. The blog provides another example: "when asked to complete the sentence 'The first U.S. president was...' it activates a feature for 'George Washington' before it writes his name."
  • Backward Planning: The research also found evidence of models "working backward from goal states," a more complex form of planning where the model seems to start with a desired outcome and deduces the steps to get there.
• Shared Conceptual Space Across Different "Languages": Demonstrating a surprisingly abstract level of understanding, the research found that models like Claude might develop a universal internal representation for similar concepts, even if they appear in very different domains. For instance, the Anthropic blog explains, "our model seems to use a shared conceptual space for English and Python code—even though these two 'languages' look very different on the surface." This suggests a deeper, underlying "language of thought." This abstract understanding isn't limited to natural language and code. We see similar shared representations for concepts that appear in, say, German and English, or in different programming languages like Python and Java."
• Primitive "Metacognitive" Abilities : The models show early signs of understanding their own knowledge limitations. A common example cited by Anthropic is that "Claude’s default behavior is often to decline to speculate and only answer if it thinks it has sufficient information." This self-awareness, or primitive metacognition, is crucial for reliability.
• The Double-Edged Sword: Fabrication and Coherence Exploits:
  • Fabricating Plausible Arguments: The research uncovered that "Claude also seems to make things up. “We’ve caught our model fabricating plausible-sounding arguments, even when it knows its conclusion is false," as stated in the Anthropic blog.
  • Exploitable Coherence for "Jailbreaks": A model's helpful tendency can sometimes be its vulnerability. The Anthropic blog points out that "Claude’s usually-helpful tendency to try to maintain grammatical coherence across turns of conversation can be exploited by certain 'jailbreak' attacks, which trick it into harmful responses."

Why Does This Matter?

The implications of this research are far-reaching:

• Enhanced Interpretability and Trust: We can finally start to see how an LLM arrives at an answer, moving away from the "black box" paradigm. This transparency is fundamental to building trust. A deeper understanding of the model's internal workings can help us build more robust, reliable, and trustworthy AI systems.
• Effective Debugging: When an LLM makes a mistake or behaves unexpectedly, these techniques could help pinpoint the internal cause.
• Steerability and Control: This research could lead to more precise control over LLM behavior, guiding them towards more helpful, truthful, and safe outputs .Understanding the internal mechanisms helps in assessing the suitability of language models for various applications and identifying their limitations.

The Journey Ahead

Anthropic is quick to point out that this is still early-stage research. The complexity of these models means there's a long road ahead. However, the techniques presented in "Tracing 'Thoughts' in Language Models" represent a significant leap forward. It’s a pioneering effort in the quest for AI transparency and safety.

Building a OCR / Document AI pipeline used to be hard work.

• Training OCR models
• Building multi-step systems (eg: line segmentation, layout detection, table detection)
• Ensuring that these steps works nicely with each other
• Adding heuristics and special cases
• Doing a lot of testing to ensure your baseline is good enough

Now: you could just give a PDF to Gemini Flash and get a 'good enough' output in one API call. What used to take weeks / months can now be done in hours.

With AI models becoming more capable — you can often get very far on your problem statement if you just try. But if you are stuck in an old mindset and think about some problems as difficult or time consuming, you may not even attempt harder problems!

I don't think realisation has sunk-in yet. We still follow old patterns of behaviour, we still mostly try to build the same things as in the past.

Here's another recent example: I need to parse something reliably. The 'right way' to do this is to write a tokeniser / parser, but this is time-consuming. In the past, I would have just depended on some basic regexes and would have just got it working (and over time — fixed edge cases as they came up).

This time though: I decided to do it in the 'right way'. I worked with an AI coding agent and got a tokeniser and recursive descent parser working in around an hour. The AI wasn't perfect — I definitely needed to know the theory and know when to give the right inputs. Even though this wasn't automatic, I ended up up with at least a 10x productivity boost.

The hard lesson for me personally was:

• I needed to let go of my pre-conceived notions that a task can take days / weeks.
• I had to decide to build something ambitious.

I need to keep reminding myself of the fact that hard things are now easy, that the impossible may be actually possible.

I think this is problem where younger folks have an edge over more experience people: with experience comes caution, but now is the time to let go of caution and just build.

To everyone reading:

• I suggest partnering with one of the current frontier models
  • Use Cursor, or Claude Code, or Windsurf, or v0, or Lovable, or anything else you want)
• Try to build something ambitious today!

We all have a pretty good idea of what makes for "Software Engineering Excellence". It’s a playbook refined over decades: write clean, maintainable code, test everything early and often, ship reliably, keep an eye on things in production, and then do it all over again. It’s a solid foundation.

ML excellence is software engineering excellence, plus data excellence, plus learning-system excellence. It inherits all that great software engineering DNA, but it also has to wrestle with a whole new set of challenges:

• messy, ever-changing data,
• models that slowly forget what they’ve learned (we call it "drift"),
• the inherent fuzziness of statistics,
• and a research landscape that moves at lightning speed.

The Cornerstones of Doing ML Engineering Right

1. Staying Ahead: Proactive Maintenance & Quality

   • Don't let "technical debt" pile up – whether it's in your data pipelines, notebooks, or the infrastructure itself. Tackle it early.
   • Models and even the prompts we use for them can "drift" over time, becoming less accurate. We need to watch for this and fix it.
   • Garbage in, garbage out. Ensuring data is high quality, both where it comes from and as it flows through our systems, is paramount.

2. Building Smart: Systematic Execution & Automation

   • If it's repetitive and boring, automate it! Think data validation checks, generating new features, or trying out countless model settings (hyper-parameter sweeps).
   • Write everything down. Document your code, what version of a dataset you used, where your features came from, and what happened in each experiment.
   • Keep an eye on everything: how fast your models respond (latency), how accurate they are and how much they cost to run. Set up alerts so the right people know when something’s off.

3. Always Learning: Continuous Improvement & Innovation

   • The ML world changes fast. Keep up with new algorithms, tools, and even hardware that can make your systems better.
   • Don't just build it and forget it. Constantly look for ways to improve your metrics, make things cheaper to run, or get answers faster.

4. Defining Success: Clarity & Precision

   • Before you even start, figure out what "good" looks like. This means clear business goals (KPIs) and the technical model metrics that support them.
   • Be upfront about what your model can't do. Point out its limitations, where biases might creep in, and any weird edge cases – ideally, before your users stumble upon them.

Where ML Feels Familiar to Software Engineers

A lot of this will sound familiar if you're from a software background:

• Quality is King: We still obsess over tests, code reviews, and smooth CI/CD pipelines.
• Automate Everything: Reproducible builds and one-click deployments.
• Know What's Happening: Logs, metrics, alerts, and service level objectives (SLOs) are crucial.
• Small Steps, Big Progress: We work in small batches, stay agile, and use things like feature flags.
• If It's Not Written Down, It Didn't Happen: READMEs, decision records and design docs are essential.

But Here's Where ML Charts Its Own Course

While ML builds on software engineering, it introduces unique twists:

Let's understand a few of those key differences:

• Data is like a whole other codebase you didn't write. Every time your dataset updates, it's as if code changes. This adds a huge layer of complexity.
• "It works" isn't always a yes/no answer. Because ML deals with probabilities, You need to think in terms of statistical confidence.
• Models get old. Unlike a piece of software that might run unchanged for years, models age. You're signing up for a long-term relationship: retraining them, possibly re-labelling data, and constantly re-evaluating their performance.
• The stakes are higher when decisions are automated by machines. Ethical considerations like fairness and avoiding bias, along with privacy and regulatory compliance, become central, not just afterthoughts.

To excel in ML engineering, you need strong software skills, to manage data carefully, and to guide AI learning effectively. It’s a challenging, rapidly evolving field, but getting it right means building powerful, reliable, and responsible AI.

In HTTP, browsers are called ‘user agents’ — they were meant to act on behalf of users. This initial positioning has sort-of carried forward even now:

• Browsers are one of the few pieces of software left that allow plugins & addons, that allow customisability.

• Browsers let users override choices. You can set your own min font size, overriding any site’s CSS. You can use reader modes.

• Browsers let users inspect applications via devtools, script applications by allowing users to run code on a developer console.

Browsers are one of the last vestiges of an old-school way of building software: software that is extensible, that users can control and change to better suit their needs.

Most current software we use today is the opposite.

• Most software (whether SaaS, mobile apps or desktop apps) is not programmable.

• Decisions are taken by product builders, not users. If a product owner hasn’t added a feature that you want, you can’t go and add it yourself!

• Most software doesn’t empower users, it enforces rigid workflows.

• We have taken away most customisability. Each option in a SaaS product is expensive to support, and it’s rational as product builders to reduce surface area and remove options that 99.9% of your users don’t use.

All of this happens because today’s software users aren’t programmers: they don’t know the internals of how software, operating systems work. Most software is built to be used by the mass market and as a result it encodes a set of assumptions about how it's meant to be used.

AI agents could reverse this trend.

I can see a world where users have personal AI agents running on their behalf. It wouldn’t matter if a product is scriptable or not, or if it has an API or not — users could just ask the agent to automate a task on their behalf! The agent may use the API, or it may just resort to spinning up a browser and clicking on pixels.

What happens when most users are able to finally take control of the tools that they use daily? I think it's finally time for general purpose compute to be actually accessible by the masses.

This fills me with optimism; similar to the heady days of Web 2.0: when everyone wasn’t so jaded about empowering users, exposing APIs was the norm, mashups were possible and products like Yahoo Pipes were built.

This is a perspective to keep in mind as we build AI-first products — let's enable our users to do great things!

A large part of our daily work involves us regularly interacting with AI. Whether it is someone writing code, creating documents, generating images or simply drafting emails, AI has become an integral part of our lives. And if recent trends are any indication, their influence is going to grow even more. It makes even more sense to step back and reflect on whether we are unlocking their full potential.

We all have faced instances when you ask for something simple, and get a confusing, generic, or just plain wrong answer. The good news is, it's often not the AI's fault – it's how we're asking. Welcome to the world of Prompt Engineering – the art and science of crafting instructions that help AI understand exactly what you want. Think of it like giving directions. Vague directions lead to getting lost. Clear, specific directions get you to your destination efficiently.

We will be referring to the concepts mentioned in the article here. For additional insights, explore the original source materials on this topic.

What is Prompt Engineering Anyway?

Think of AI like a smart prediction machine. It guesses the next word based on what you give it. A "prompt" is simply your instruction. Prompt engineering is the skill of writing good instructions to get the best possible results from the AI. At its core, it's about communicating clearly with AI systems. Well-crafted prompts can save time and produce more useful results.

What Actually Goes Into a Prompt?

A prompt can have several parts working together.

• The Instruction/Task: What do you want the AI to do? (e.g., "Summarize," "Translate," "Write," "Explain," "Generate code"). This is the core action.
• Input Data: The specific text, data, or topic the AI needs to work on (e.g., the article to summarize, the sentence to translate).
• Output Indicator/Format: How do you want the answer presented? (e.g., "in bullet points," "in a friendly tone," "limit to 100 words").
• Context Setting and Specificity: Giving the AI background helps it understand the 'why' and 'who' behind your request. For example: Instead of saying "Write an email about the project update," try giving more specifics: "Write a short, professional email to the marketing team summarizing the key results from the Q3 social media campaign project update. Mention the 15% increase in engagement."

You don't always need all parts, but combining them thoughtfully makes your prompts much more powerful.

Setting the Rules: Using Constraints Effectively

Constraints help focus the AI's response in specific directions, similar to how parameters narrow down search results

• Length: "Summarize this in 50 words"
• Format: "List the ideas as bullet points"
• Tone/Style: "Explain this in a formal tone"
• Negative Constraints: "Don't use technical jargon," "Avoid mentioning price"
• Real-world example:
  • Less effective: "Suggest team-building ideas."
  • More effective: "Suggest 3 low-cost team-building activity ideas for a remote team of 10 software engineers. Focus on activities that encourage collaboration and take about 1 hour. Present them as a numbered list with a brief description for each. Avoid virtual escape rooms."

The Power of Iterative Refinement

Prompt engineering is often a process of trial, error, and refinement – think of it as a conversation rather than a one-time command:

1. Start Simple: Write your initial prompt based on your goal.
2. Analyze the Output: Did the AI understand? Was the result accurate? Was the format correct?
3. Identify Gaps: What was missing or wrong? Was the prompt too vague? Did it lack context? Did you forget a constraint?
4. Refine the Prompt: Adjust your prompt based on your analysis. Add specificity, context, constraints, or examples.
5. Repeat: Try the new prompt and continue refining until you get the desired result.

Prompting Frameworks

For those of us who want to delve deeper into formal prompting techniques, here are several powerful approaches with practical examples:

1. Basic Prompt (Zero-Shot) – This is the simplest approach with no examples:
   "Summarize this meeting transcript and list the action items"

2. Adding Specificity and Constraints – Adding details about what you want:
   "Summarize the key decisions made in the following meeting transcript in 3-4 bullet points. Then, list all action items mentioned using the format: "- [Action Item]: [Owner] - Due: [Deadline]""

3. Role Prompting – Asking the AI to adopt a specific perspective:
   "Act as an efficient Project Manager reviewing a meeting transcript. Your goal is to quickly understand outcomes and track tasks. First, summarize the key decisions made in the following meeting transcript in 3-4 concise bullet points. Second, rigorously extract all action items assigned. List them using the format: "- [Action Item]: [Owner] - Due: [Deadline]". If an owner or deadline isn't explicitly mentioned, use 'TBD'."

4. Chain of Thought – Guiding the AI through a step-by-step reasoning process:
   "Act as an efficient Project Manager reviewing a meeting transcript. Your goal is to quickly understand outcomes and track tasks. Before generating the final summary and action list, follow these steps:

   1. Read through the transcript to identify the main topics discussed.
   2. Pinpoint the key decisions reached for each topic.
   3. Scan the transcript specifically for commitments or tasks assigned (action items). Note down the task, who is assigned (Owner), and any mentioned deadline.
   4. Consolidate the key decisions into a 3-4 bullet point summary.
   5. Format the extracted action items clearly using: "- [Action Item]: [Owner] - Due: [Deadline]" (Use 'TBD' if owner/deadline is unclear).

   Now, provide the final output based on this process."

5. Reflections – Adding a layer of self-evaluation to the AI's process:
   "After extracting the action items using the CoT steps, re-read the summary of key decisions. Does each action item clearly map to a decision or discussion point? If not, double-check the transcript for that action item's context. Also, generate two versions of the action item list based on slightly different interpretations of the transcript, and present the most likely version based on the discussion flow."

Wrapping Up: The Art of Clear Communication

Prompt engineering is fundamentally about clear communication with AI systems. Specific instructions, relevant context, occasional examples, and format guidelines typically lead to better results.
Many users find an iterative approach works well: starting with simpler prompts and gradually refining them based on results.

Helpful Shortcut: Consider asking the AI itself to help craft effective prompts. For example, you might type "What would be a good prompt to generate a marketing email about our new product launch?" The AI can often suggest well-structured prompts tailored to your specific needs.