Overnight pipelines

Once triggers are armed and the daemon runs as a service, LatchAI can work while you sleep. The most demanding thing it is used for is an autonomous build pipeline: a team of agents that reads a backlog and ships tested code without a human in the loop.

This page describes the pattern. The full runbook — the exact agents, transitions, and troubleshooting — lives in docs/autonomous-build-pipeline.md in the LatchAI repository, and is the reference to read before running one.

The shape of it

Two workflows, not one. A planning workflow runs on demand and turns an idea into a plan, an architecture, and a set of stories. A build workflow runs on a short cron and advances exactly one story by one step per run: survey the board, pick the next story, implement it, review it, test it.

Each step is an agent with its own persona, tool allowlist, and standards to enforce. The agents are ordinary agents/*.md files grouped in the sidebar — there is no special “pipeline” primitive in LatchAI.

The three rules that make it work are worth stating plainly, because they are what you copy, not the agent roster:

1. One step per run. A run picks up whatever the board says is next and advances it by exactly one state. Runs are short, failures are cheap, and progress is durable.
2. State lives outside the process. See below.
3. Every step has a bar it must clear. The reviewer runs the tests; QA runs the app. An agent that can only claim success will.

A minimal version you can copy

Two agents and a condition — a survey step that decides what to do, and a worker step that does it. Everything else in a real pipeline is more agents on the same skeleton.

{
  "id": "nightly-queue",
  "name": "Nightly queue",
  "nodes": [
    {
      "id": "wake",
      "type": "cron_trigger",
      "position": { "x": 40, "y": 200 },
      "config": { "schedule": "*/10 * * * *" }
    },
    {
      "id": "survey",
      "type": "agent",
      "label": "Pick the next item",
      "position": { "x": 300, "y": 200 },
      "config": {
        "agent": "queue-surveyor",
        "prompt": "Survey the board. Reply with exactly one line: 'WORK <id>' for the item to advance, or 'NONE' if nothing is ready. Prefer finishing an item already in progress over starting a new one."
      }
    },
    {
      "id": "gate",
      "type": "condition",
      "position": { "x": 560, "y": 200 },
      "config": { "input": "{{survey}}", "op": "contains", "value": "WORK" }
    },
    {
      "id": "work",
      "type": "agent",
      "label": "Advance it",
      "position": { "x": 820, "y": 140 },
      "config": {
        "agent": "worker",
        "prompt": "The surveyor said: {{survey}}\n\nAdvance that one item by one step. Move it to the next state when — and only when — its tests pass."
      }
    }
  ],
  "edges": [
    { "from": "wake", "to": "survey" },
    { "from": "survey", "to": "gate" },
    { "from": "gate", "to": "work", "when": "true" }
  ]
}

config.agent points each node at an agents/<name>.md definition, so the persona, tool allowlist, and model live in a file you can edit and version rather than inline in the graph. The when: "true" edge is what makes the “nothing to do” path a no-op instead of an error — a condition node’s false branch simply isn’t taken, and the run completes.

Wire the tracker in through MCP and give each agent the narrow tool allowlist it needs. Start with the surveyor read-only; let it propose before it writes.

The tracker is the state

There is no separate queue. An external issue tracker holds the state, and a story’s status column is its position in the pipeline. That is what makes the pipeline restartable: a run that dies mid-story leaves the story in a status the next run knows how to pick up, because the next run reads the board rather than any in-memory state.

The corollary is that transitions must be exact. Guessing a transition id in a tracker’s API tends to silently no-op rather than error, which produces a pipeline that looks busy and advances nothing. Read the ids from the tracker’s own transitions endpoint, pin them in the agent definition, and have the agent verify the state it expected after it moves something.

The same principle covers stuck work: if an item bounces between two states repeatedly, something upstream is wrong, and a pipeline with no way to park an item will loop on it all night. Count the bounces somewhere the board can see, and route a repeat offender to a refinement step rather than back to the worker.

Isolation for agent git

Built projects live outside the LatchAI workspace, reached through a mount, each its own git repository. This is deliberate: agents run real git — branch, commit, merge per story — and giving each project its own repository keeps that from ever reaching LatchAI’s own history or another project’s.

Two mechanisms back that up. run_shell sets a git ceiling at the LatchAI home, so git discovery inside the workspace cannot walk up out of it. And checkpoints are workspace-only by design — the shadow repo does not follow the symlink into external projects, which self-version through their own history instead.

The overnight checklist

An overnight run fails quietly if any of these are missing:

• The daemon is running — a cron fires nothing if the engine is down.
• The workflow is not disabled — disabled: true leaves triggers unarmed while manual runs keep working, which is exactly the state you forget you left it in.
• Any external dependency the agents use is up (a container runtime, a model server).
• The machine stays awake — a sleeping Mac drops everything mid-run. caffeinate is the blunt instrument.
• The daemon was restarted after any change to packages/engine, since the engine does not hot-reload.

The “already running → skip” guard prevents overlap when a run takes longer than the cron interval, so a short interval is safe.

Watching it afterwards

Everything the pipeline did is in the Run Monitor live, in the append-only run logs afterwards, and in the checkpoint diffs per run. If you want a human gate at one step, a review_gate node is a one-node change.

Reading a bad night, in the order that usually finds it:

1. Did it run at all? A cron that fired while the previous run was still going emits trigger.skipped — visible in the events panel, absent from run history. Lots of those means the interval is shorter than the work.
2. Where did it stop? The run’s log has the node that failed and its error; node.failed on an agent node usually means the provider or a tool refused, not that the model was confused.
3. What did it actually change? The ± diff on the run. An agent that reports success and produced no diff is the failure mode worth hunting — it means a step “completed” without doing anything.
4. Then decide: rerun (transient — a model timeout, a container that wasn’t up), refine (the item was underspecified and keeps bouncing), or park it and move on.

Troubleshooting

The cron fires, runs “complete”, and nothing advances. The daemon is running engine code from before your last edit — tsx loads packages/engine into memory at boot and never re-reads it. Restart the engine. This one is worth suspecting first, because the symptom is a pipeline that looks perfectly healthy.