Shining Some Light in Dark AI Software Factories

With the AI world changing ever so quickly, new terms are coming up weekly, and one of them we've been hearing more often in conversations with our clients is the AI Software Factory. 

In this blog. I want to make the definitions clear, share my experience setting up one of these factories (Sandcastle), explain how you can shape the assembly lines in your factory and share what I learned from it. 

We will soon publish a series of blog posts that delve deeper into the mental model of building a factory. We will cover how to create such a factory and how to monitor your budget to keep costs under control. After all, you don’t want your factory to exhaust your AI budget in just one quarter. 

What is an AI Software Factory? 

An AI Software Factory is a continuously improving, increasingly autonomous automation of the software lifecycle. 

To make this concept easier to understand, we like to compare it to a car factory. In this analogy, a work item is like a car part that enters the factory. It is then processed and developed in an assembly line, moving from one workstation to the next until it is completed. These workstations could be some AI agents (see this blog post to learn about the difference between LLMs and agents) but can also be some deterministic code. The combination of the two and the control you have over the assembly line is what makes it so powerful. A station could also be a human where you need human review or decisions. As your factory matures, you will want to reduce human efforts and focus more on overseeing your factory’s dashboards. You can gather metrics on your factory's performance, quality, and costs, start adjusting components in a workstation or change the assembly line process if any of these parameters exceed their limits.

In the factory setup, different agents take on specific roles. You'll typically encounter some or all of these agents, depending on the maturity and scope of the operation: 

The advantage of having multiple agents manage these tasks independently is that you can choose a different language model (LLM) for each one. The planning agent is responsible for the actual thinking, so it should run on your most powerful model. I initially used Fable for this purpose when it was available, as it excels in UI/UX design. When Fable was no longer accessible, I switched to Opus. Since the planner writes the specifications, the implementation can be handled by a more cost-effective and faster model like Sonnet. Finally, a strong reasoning model, such as Opus 4.8, reviews the implementations, provides feedback on the pull request, and initiates another cycle of implementation and review. 

If you set up an AI software factory, usually these agents will be provided out of the box with predefined prompts and processes. Depending on the product you choose, you can change the prompts these agents run with, or even the process for how and when they hand off work from one agent to the next. 

One thing worth saying out loud before you get excited: this isn't free. Running several agents in parallel burns tokens quickly, so set budgets and keep an eye on spending from day one. 

There are many tools available for building an AI Software Factory, each offering different levels of control and autonomy. Lightweight options such as Sandcastle on GitHub focus on sandboxing agents while giving teams direct control over the assembly line, deliberately keeping people close to the process and prioritizing visibility, control, and durability over maximum speed. More opinionated tools, such as GasCity on GitHub, provide less hands-on involvement, allowing agents to operate with greater independence and relying on isolation to contain failures when they occur. This can increase assembly line throughput, but it also requires more trust in the factory’s controls, monitoring, and recovery mechanisms.

After choosing the one that fits your need, you're ready to ask the big question: 

Am I ready to turn the lights off in my factory? 

In my opinion, this is the most challenging step of the entire process. While both solutions mentioned are easy to set up, just use ‘gc’ or ‘npm’ and follow the friendly wizard to adjust your settings. The real challenge lies in learning to trust the system. 

If you're just starting to play around, I'd recommend Sandcastle to get a feel of how one of these systems works. It's lightweight, lives inside a single .sandcastle folder in your project. At its heart is the main.ts file that controls everything, giving you a lot of say over the process, the agents, and the prompts that initiate their actions. 

Every implementation agent also writes logs about what it's currently doing. After setting mine up, I used Claude Code to iteratively refine the main.ts file, adding a global log file that I keep open while the factory runs. That log serves as my flashlight, allowing me to check the factory from time to time to see if the agents are still running smoothly or if something is going wrong. 

Then comes the fun part: gaining confidence. Connect the factory with your GitHub issues, tag the ones you want it to pick up (choose trivial issues and bugs first), run it, and review the resulting PR and the agents' PR comments. 

If you spot flaws in the implementation or the review, don't try to fix them yourself. Instead, take a step back and refine your assembly lines and workstation within your software factory. Run it again, measuring quality with each iteration. Keeping track of quality and costs of your factory is essential for building trust. Ultimately, this allows you to rely on the factory’s output without constant oversight, not looking at the outcome but monitoring the metrics of the factory, only intervening when one of these parameters exceeds its limit. You will need to balance the models used for different tickets; a bug fixing ticket will most likely not need as advanced a reasoning model compared to a new feature implementation.  

If AI software factories are coding, where does that leave us humans? 

Entry-level software engineer work that largely consisted of "here's the ticket, go implement it" can now probably be replaced by some form of AI or factory. This mundane set of tasks has never been the hard and fun part of software engineering anyway. The good part was thinking about user problems and their solutions, designing architecture and workflows, and maintaining the quality of the codebase. The true strength and benefit of junior engineers has never been about the code they produce, but rather the questions they ask to understand the system, challenge existing decisions, and provide fresh perspectives.

Those tasks will continue to be needed, and they'll shift toward a more AI-focused way of working. Instead of updating dependencies manually, you'll design agents and workflows that do it for you, safely and much faster. Instead of hunting for a bug yourself, you'll focus on writing the test that exposes it and let agents handle the search and the refactor, making sure the test passes without breaking anything else. 

Additionally, new tasks will emerge: designing these factories, prompt engineering, and enabling other business teams to use AI effectively, setting up workflows and skills that make their lives easier. 

Don't compete with agents, design them

If you're reading this and thinking it's time to upskill yourself in these topics and learn how to use AI efficiently, effectively, and safely, check out our AI trainings here. We cover topics like agentic development, designing LLM Applications and many more. 

I hope to see you there!