Should AI agents be general purpose or should they be built for a specific task?
I would say yes to both.
Special Purpose Agents
A special purpose agent is an agent built to do one (or a few things) very precisely. Examples:
- RAG on a knowledge base and answer questions only from the knowledge base.
- Convert images to very specific structured data format (eg: convert image to a contact vCard).
- Convert text to SQL on a specific table, with specific guardrails (eg: always apply page size limits).
These agents are built to do a specific job.
General Purpose Agents
A general purpose agent on the other hand would be an agent that has emergent behaviour, that can use its tools to do something it wasn't explicitly programmed for.
This is best demonstrated by an example. I recently coded up a toy agent that runs on the command line that had the following tools available:
- List directory
- Read file
- Query file (via duckdb)
- Read PDF
This agent was designed to have generic tools, and it was allowed to do multiple tool calls if needed. It wasn't given a specific goal.
The lack of specificity actually made the agent more useful! In the last few days, I have used it to do:
- K8S cost optimisation — given a CSV containing some kubernetes utilisation data, this agent helped me find low hanging cost optimisation options
- Data entry for my own personal finance needs — given some account statements, the agent was able to help me digitise them in a format I use for tracking my expenses
- Do Q&A on my meeting notes
Interestingly, the agent showed a lot more agency than I was expecting. For example, it would often execute multiple tool calls in response to a general 'hello' message!
All the recent wow moments I have had with AI are usually with general purpose agents. They can end up giving you unexpectedly rich experiences.
Use Cases & Trade-offs
If you're building an AI enabled product, both special purpose agents and general purpose agents have their place.
-
You sometimes want determinism (or close to determinism) and repeatability.
- For example: if you're building a support desk, you may want to always categorise tickets in a certain way.
- In such scenarios, special purpose agents are really useful. You can treat them as "intelligence that's an API call away".
-
Special purpose agents are limited in scope.
-
General purpose agents are where the power of AI agents becomes apparent.
-
General purpose agents are going to be more expensive to build.
- You may need smarter LLMs, you may need to spend tokens on reasoning.
- General purpose agents could use up millions of tokens.
-
Shipping general purpose agents requires you to have the right underlying infrastructure.
- The agent is as powerful as the tools it has access to. You need to build the right primitives to unshackle the AI model.
- You need to build the right platform for the AI agents to leverage!
- For example: if your system processes a lot of data, you may need to invest in robust parallel execution.
- If the agent could 'ask any question' of your data, you have to think about indexing and database performance.
-
The ideal end state of a general purpose agent is a coding agent — an agent that can write code for a specific task and then execute it.
- This could lead to potential security issues, and you have to look out for other types of abuse.
- You might need to invest in primitives like sandboxing here.
I see this as a continuum — the more general an agent, the more powerful it is, but you need to invest proportionally into building the right safeguards.
Most use cases may not need a general purpose agent. You can probably start off building an AI-enabled product by just focusing on special purpose agents.
General purpose agents are also the most valuable ones though. And building general purpose agents will require you to invest proportionally into your platform's core primitives.