Lessons learned from LLM agent work
Lessons learned from LLM agent work
I’ve done real work using Swarm Forge and prompts based on the roles defined in the repository.
Introduction
Say that I want to get a larger task done, you start by asking one LLM to plan the work and split up the work in tasks. In order to reduce the LLM introduced “confusion” you can use prompts inspired by Matt Pocock’s grill me. I suspect that the popularity of the skill means that newer LLMs do not need to have the skill installed (from using grill me in Claude). You have the LLM ask you to fill in what it is “unsure” about. In my experience this type of approach makes it more probable for the LLM to converge to the intended solution.
There is some research around Code Generation by Emulating Software Process Models. In 2025 I was part of a team that tried that approach. My take was that there was a need to keep a firm hand and micro manage in order for the agents not to go off the rails. Due to the amount of extra work dealing with the different agents, there was not as much use in long running unattended workflows. The roles defined boiled down to:
- Requirement Engineer
- Architect
- Developer
- Tester
- Scrum Master
Software development and prompting
In order to be able to get any sort of useful work out of LLMs for moderately complicated backend (without being burned out by the amount of testing and verification required) we have needed to add integration tests (from database to API).
For real work (2026) where I used multi agent definitions I used mainly a combination of roles such as Orchestrator, Developer, Reviewer and Tester. These roles should be more a reflection on the type of work we have done rather than a definite starting point.
Anthropomorphism
It would be wrong to think about LLMs in human terms, they are approximations of human language artifacts. The reason this is important is that LLMs do not arrive at the solution in the same way as a human in my experience, so we need to adjust our understanding based on what works well instead of trying to use too much of a human centric understanding. Some of it applies, some does not.
I have had discussions around code reviews where people have been telling me that reviewing AI code should not be different than reviewing human code. There is a sort of “that also applies to humans” that cuts off the discussion. Instead we need to abandon our anthropomorphism of LLMs and instead focus on the tool side of them.
We can have LLMs do tedious tasks. Having tons of lint rules gets tiring for humans (death by thousand paper cuts). It does not matter for machines. Having high code coverage previously was expensive enough that most teams tried to aim for “good enough” numbers. With LLMs that number can be much higher. We have seen that it is useful to have enough code coverage, to use mutation testing to make it less likely that you can break functionality, to use even more linters.
Review
The main pain point of having to deal with AI generated code is that it is easy to generate plausible looking code. The old type of signals around work done no longer apply (polished code pre LLM was an indication of effort). To review AI written code do look for new dependencies or hints that the LLM scaffolds from memory instead of using existing tools. Do not trust that it works unless tested (do run the program). Tests are often green (as in they are green without having done any useful testing). LLMs can produce a mountain when a feather could have been enough. When you realize that you have gone overboard in prompting and not done enough oversight and get huge amount of unnecessary complexity to solve that problem, throw that code away. Salvage what’s worth keeping. Try to avoid accepting slop. Rethink and re-prompt. You have not written the code, so should not have any qualms about throwing it away.
Simple skills
I get kind of annoyed reading word salad texts generated by AI. You can reduce that amount by adding some skill includes like terse prose and unslop. There is also the recommendation on using ASD-STE100 Simplified Technical English. Some use the @ayghri i-have-adhd skill to achieve the same result as using ASD-STE100. The name is narrow but the problem isn’t, since most people’s eyes probably glaze over for a similar reason.
Putting things together
How does Swarm forge fit into this? It uses a process to have the LLMs use tests, code coverage, mutation testing plus what looks like LLM specific written role descriptions for “Requirement Engineer” and “Architect”. The “Scrum Master” role has been replaced with lieutenant. Looking at the initial commit we see LupusDei, who later built adjutant. I’m getting some vibes that adjutant is a spiritual successor of the project gastown that got a lot of attention early on in this year. My reading on that there are multiple multi role setups, used by people without Steve Yegge’s token budget, is that the approach has become more practically useful this year.
I’ve been using Swarm forge roles for real work since this summer. There are some gotchas. I’ve seen that you should be careful about using too many roles when you want to fix simple things in simpler parts of the code (over-engineering). I do use grill me style prompts to make sure that a multitude of questions are asked up front.
I’ve seen similar results when I give one big prompt that includes a definition of the roles and how they should interact. The key seems to be to tell it to use the different existing developer tools (code coverage, tests etc).
I do like when LLMs commit their work and keep a record in git so that if there is a crash I can ask a harness (such as Copilot, Claude Code or Codex) to pick up the work from that point. That loops back to issue trackers integrated into git such as git-bug, ditz or ticgit. These could become more interesting in order to keep track of agent work without having to configure access to issue tracking software.
Conclusion
By having a different process for how a combination of multiple LLMs arrives at the solution, you can reduce the amount of review needed. For instance if you know that mutation testing and code coverage has been used, the probability that you have a working solution is higher.
Use the ideas and apply them in ways that make sense for your projects. You can use the roles in Codex or Claude Code directly without Swarm Forge tooling.
Tags
Do you want to send a comment or give me a hint about any issues with a blog post: Open up an issue on GitHub.
Do you want to fix an error or add a comment published on the blog? You can do a fork of this post and do a pull request on github.
Comments