What a second environment found in a shared agent code factory

I originally planned to run a localised version of a friend’s agent code factory in my Kubernetes and GitOps “home lab”. Within a week, that version had diverged far enough that maintaining it separately no longer made much sense.

My friend and I decided to work on one shared project instead. His environment remained where the factory was developed and used daily. Mine became a second test bed with different infrastructure, repository conventions and delivery rules.

That arrangement turned out to be far more useful than the local fork. My lab exposed assumptions that had survived in the original environment, while the shared project meant that fixes could benefit both of us instead of becoming another layer of local code.

This was a side project, and it ate most of my free time for the month. It also produced a fairly concentrated test of what happens when software built around one real environment meets another.

The local version lasted a week

The first plan was not to create a completely separate product. I wanted a localised version that stayed reasonably close to upstream and accounted for differences in my infrastructure.

I worked interactively with Claude Code using the Superpowers skill. A nine-stage implementation plan was divided into several waves of subagents across three forked repositories.

The work looked busy and mostly plausible as it progressed. The agent regularly asked questions, usually through a long explanation followed by three choices and a recommended option. I often accepted the recommendation.

After about a week, an audit showed what those individual choices had accumulated into: 231 commits, more than 2,600 added lines under bin/, no incorporated upstream changes and invented behaviour such as an unpark feature that did not exist in the shared project.

That was far beyond the local adaptation I thought we were building. The consequences had been spread across enough prompts and subagent reports that the change in direction was difficult to see while it was happening.

My friend and I concluded that separate versions would create more work for both of us. We deleted the fork and its database, then connected my environment directly to the shared project through an environment-specific directory.

I count the fork as a lost week, but the replacement model was better: generally useful changes belonged in the shared project, while details that were genuinely specific to my estate stayed outside it.

The environments were different in useful ways

The factory used OpenProject as its workflow state, GitLab for code and review evidence, and Kubernetes Jobs for its workers. In the workflow I ran, planners, shapers, builders and reviewers used both Claude Code and Codex. A control plane dispatched work according to tracker status and merged changes only after the configured checks passed.

The required approver could be selected per project. Some projects accepted approval from the reviewer agent, while others required a human to approve the merge request before the control plane could merge it.

The basic workflow was already in daily use in my friend’s environment. My lab brought a different set of constraints: multiple Kubernetes clusters, ArgoCD, per-cluster Vault instances, a self-hosted GitLab, Cilium policies and admission rules requiring signed and digest-pinned images.

The repositories also had different delivery conventions. When I worked interactively with Claude Code or Codex, I normally committed directly to master because this is a one-person environment. Repositories that built deployable images still required merge requests because publishing an image could lead directly to deployment.

Those differences were not a compatibility checklist prepared in advance. They appeared one at a time as the factory began handling real work.

The first project found five defects

The first end-to-end workflow completed on 8 August. Two days later, we started a greenfield dashboard as the first test of whether the factory could carry a project from a brief through implementation and review.

That run exposed five control-plane defects. Review findings could disappear during rework. Superseded merge requests remained open. A blocked dependency was stored but never read back. Human-decision notes passed through the wrong schema. The merge gate could also allow an unapproved merge request through.

We documented the findings and sent them upstream. Four fixes shipped within roughly two days, and all five were available shortly afterwards.

The dashboard deployed on 12 August. On its first day it recorded 94 runs, 3,261 worker events, 91 reports and 568 artefacts.

The original dashboard view after further changes, showing current statistics on 27 August.

It was also marked done while broken. A missing DNS egress rule and a Vault password with a trailing newline both required manual correction.

The factory had completed the workflow it knew about. My environment supplied the acceptance conditions it did not yet know about.

That distinction repeated throughout the month. A secrets-migration item closed most of its children without running the live convergence test that would have established whether the migration worked. Two monitoring changes passed their defined checks and were rejected only after a manual test showed that they did not detect the failure we cared about.

These were useful findings precisely because the second environment was not an artificial test fixture. A change could satisfy the factory, pass a pipeline and still fail against Vault data, network policy, image admission or the way a repository actually reached production.

Load exposed another class of assumptions

The next set of problems appeared when we moved from individual work items to batches.

On 18 August, 17 items dispatched within about two minutes. Around ten concurrent workers then exhausted the API’s default ten-connection database pool while sending their reports.

The incident lasted 33 minutes. Five workers were killed by liveness checks, four builders completed their code but lost their reports, and seven work items were stranded.

We raised the pool limit to 30 as a temporary workaround and filed the underlying defects upstream. After the fixes shipped and survived a live wave, the workaround was removed.

A target-freshness gate produced a different failure. It correctly prevented a merge when master had changed since review, but its 1,800-second freshness window combined badly with job-based retries on a busy branch. Attempts failed at 1,805, 2,074 and 2,093 seconds.

Parent closure also slowed as the board grew. The reconciler rotated through roughly 117 candidates in batches of 20 without pruning parents that were already closed. Version 0.0.202 added pruning. On its first live run, it removed 147 stale candidates and closed seven parents in one tick.

None of these problems required an exotic second implementation. They required enough real variation and load to make assumptions visible.

The feedback loop became the useful product

We agreed on a simple division of work. Bugs found in my environment were documented and returned to the shared project. More general capabilities became feature requests. My friend remained the maintainer and decided what belonged upstream.

My earliest contributions began around version 0.0.184. By 25 August, the shared control plane had reached version 0.0.210. Versions 0.0.203 through 0.0.210 all resulted from discussions or findings that came through my environment.

Across the month, the second estate generated or contributed to roughly 30 upstream issues and eight merge requests. That does not mean I implemented every resulting change. Some were my merge requests, some were fixes made upstream, and others developed through review and discussion between us.

The speed of the loop mattered. A defect found in a live wave could be described with the work item, merge request, logs and timings that produced it. A fix could then be released upstream and tested again against the same environment. Temporary workarounds had a reason for existing and a live condition for removing them.

This also made disagreements more concrete. We were no longer discussing whether a feature might be useful in a hypothetical second installation. We had a failing worker, a stranded item or a deployment that passed its pipeline and did not work.

Throughput is not the whole accounting

The factory moved a substantial amount of work. A backlog of roughly 107 migrated items was almost cut in half over three days. One unattended night produced 62 commits and emptied the board. A wave of 24 items ran after an upgrade of the shared control plane.

I do not have measured savings from this.

The same month included the abandoned fork, repeated manual interventions, one refactoring cycle and six reverts after a long interactive Codex session running an overseer skill created work that had not been planned. Several defects consumed real work before we understood them well enough to file or compensate for them.

The outside-test-bed model did not make experimentation free. It made the cost more likely to produce something reusable.

There are still limits to that claim. Some upstream defects remain open. Several temporary rules still need clean live runs before they can be removed. Model-agnostic runtimes remain unverified, and the automated overseer is still a design rather than a working part of the factory.

What I can say is that maintaining one shared project gave us a better place to put what the second environment discovered. The local fork accumulated differences. The shared test bed produced evidence, issues, merge requests and fixes that could be exercised in both environments.

I also ran a separate research-focused project to test the factory’s wider capacity, with surprisingly good results. That is probably better treated as its own account once I have worked through what the results actually show.