Federated Forge¶

A Kenn Forge fleet gives several development machines one shared control point without moving their repositories, worktrees, or agent sessions to a central server.

Why use a fleet¶

Use a fleet when one machine is not the right place for every task. For example, you may keep routine work on a laptop, run large builds on a workstation, and use a machine with a special operating system or GPU when a task needs it.

• Open the hub to see workspaces from every machine and the node that owns each one.
• Create a repository workspace on the hub itself or any writable spoke.
• Sync pull requests, issues, and activity once instead of spending provider API budget on every machine.
• Open a spoke directly for a local-only workspace view when you want to work in that machine's environment.
• Keep each repository, worktree, process, and tmux session on its execution machine.

If one machine already handles all your work, keep Forge standalone. A fleet adds value when the execution location matters.

The mental model¶

One hub owns provider synchronization, provider changes, and the fleet-wide view. Each spoke owns its local repositories, workspaces, Git commands, processes, and tmux sessions. The hub can also own local workspaces, so it is an execution machine as well as the coordinator.

Every machine runs the same kenn-forge binary and keeps its own directly accessible UI. Forge-to-Forge requests and terminal WebSockets use authenticated HTTPS. The hub does not use SSH to start a spoke or attach to its tmux server; start and supervise each daemon on its own machine.

Fleet rules at a glance¶

The pull-request and issue buttons stay local to the Forge UI you opened. To choose a different machine, open Workspaces, select New workspace, and use Run on.

This guide uses the following example topology. Each name is a private HTTPS origin reachable from every fleet machine:

https://forge-hub.example.test  hub and local execution
https://build-a.example.test            spoke and local execution
https://build-b.example.test            spoke and local execution

Tailscale Serve is a first-class way to provide those origins, but it is not a fleet requirement. A private LAN, UniFi network, VPN, or custom reverse proxy works through the same federation protocol.

Understand the trust boundary¶

Use federation only inside one operator-controlled private network. Keep every Forge listener behind that private ingress.

Private network access is necessary, but it is not sufficient. Forge also requires HTTPS, local browser authentication, and separate credentials created during enrollment. An activated hub and its spokes form one administrative trust domain. Someone who controls an active peer should be treated as controlling fleet workspace and terminal operations.

Pending spokes have narrower access. They can resolve only the provider facts needed for preparation, hand state to the hub, and use enrollment routes. They cannot browse general provider data or perform provider writes. Those preparation routes are limited by operation, not by repository. Creating and transferring the one-time token approves that machine to become a full fleet member, so give a token only to a machine you trust with the fleet's provider data.

Before you start¶

On every machine:

1. Install the same kenn-forge build on every machine.
2. Install Git, tmux, your agent CLIs, and the language tools used by local workspaces. Authenticate the agents as the user that will run Forge.
3. Log in to the Git host used for local clone, fetch, and push operations. For GitHub, gh auth login is the normal route.
4. Choose one stable private HTTPS origin for that machine.
5. Confirm every machine can resolve and reach every origin.

fleet setup configures Forge and its service. It does not provision the operating system, install development tools, clone your repositories, or authenticate agents. SSH access is useful for provisioning and upgrades; federation itself uses HTTPS and does not require reciprocal SSH keys.

The examples below use the default config location. If your service uses a custom config, pass that same --config /path/to/config.toml to every fleet command, including setup, token creation, join, preparation, and revocation. These commands contact the local daemon selected by that config; the hub URL does not select your local instance.

Do not copy a Forge data directory between machines. Each directory receives a stable random node ID on first start.

A federation origin contains only a scheme and authority, such as https://build-a.example.test. Forge rejects cleartext HTTP origins, URL paths, query strings, fragments, and embedded credentials. Explicit ports are allowed; default :443 is normalized away.

Federation also requires Forge to use the root base path (base_path = "/"). Non-root UI prefixes are not part of the federation protocol.

Setup at a glance¶

Set up the hub and spoke services first. Then enroll one spoke at a time. A spoke stays standalone until preparation has safely moved provider ownership to the hub and a service restart activates the enrollment.

Set up the hub¶

Run setup as the operating-system user that will run Forge. The normal Tailscale path discovers the machine's certificate name and current Tailscale login, publishes Forge with Tailscale Serve, installs a per-user service, and checks the protected HTTPS API.

For the Tailscale path, first join the machine to your tailnet and enable HTTPS certificates. On Linux, an administrator can let the Forge user manage Tailscale without running Forge as root:

sudo tailscale set --operator="$USER"

This grants the current user control of the local Tailscale daemon; see the Tailscale CLI reference. Then run Forge setup as that user:

kenn-forge fleet setup hub --tailscale

On macOS, setup also finds the command-line client inside the standard Tailscale app when tailscale is not on PATH.

Review the displayed plan and confirm it. For unattended setup, inspect the same plan with --dry-run, then repeat with --yes.

If your LAN, UniFi network, VPN, or reverse proxy already provides private HTTPS, give Forge its canonical origin instead:

kenn-forge fleet setup hub \
  --origin https://forge-hub.example.test

The --origin path never calls Tailscale. It configures and supervises the same loopback Forge service, while your ingress owns DNS and TLS. It retains Forge's normal bearer and browser-cookie authentication.

Setup requires exactly one of --tailscale and --origin. It does not expose the Forge listener directly, weaken API authentication, or make the service depend on a particular network product.

The hub also needs its own provider credentials. Open Settings → Repositories on the hub and add the repositories it should synchronize. Cloning a repository on disk does not add it to this list. On a fresh hub, allow the initial sync to populate provider data before moving existing spokes.

Set up each spoke¶

Run the matching setup command on each spoke:

# Tailscale Serve
kenn-forge fleet setup spoke --tailscale

# Operator-managed private HTTPS
kenn-forge fleet setup spoke --origin https://build-a.example.test

Spoke setup deliberately leaves the daemon in standalone mode. Do not set fleet.role = "spoke" by hand. Enrollment preparation persists that role only after provider writes drain and state handoff completes.

On Linux, setup installs a systemd user service and enables lingering so Forge survives logout. On macOS, it installs a LaunchAgent under the selected user. Both service definitions execute the installed binary directly and preserve the setup user's HOME and PATH. They do not start an interactive shell or copy arbitrary exported credentials. Configure any additional credential environment through your service manager.

Run setup with a stable PATH that includes your agent CLIs and tool-manager shims. A minimal SSH command environment can omit tools that work in your interactive terminal. Verify by launching an agent from a Forge workspace after the service restarts.

Tailscale identity mode treats local processes on the Forge host as trusted, because Tailscale Serve forwards identity headers over loopback. Use it on a single-user or otherwise trusted machine. On a multi-user host, use an external origin and Forge's bearer/cookie authentication instead.

Adopt an existing service¶

If Forge already runs under a service you maintain, inspect fleet setup ... --dry-run first. Setup refuses to overwrite a service definition it does not own. For an existing federation-ready daemon with working private HTTPS, keep its service and continue at Enroll one spoke at a time. Re-enrollment does not require reinstalling the service or moving its data.

If you want setup to own the service instead, back up the current definition and configuration, stop it, and explicitly move the old definition aside before running setup. Keep its data directory and installed binary path consistent with the displayed plan.

Keep credentials separate¶

Four credential types serve different purposes:

Do not copy browser cookies or API tokens between machines. Enrollment creates a different machine credential in each direction. Forge stores inbound credentials as digests and keeps credential files private to the daemon account.

A spoke needs its own Git credential even though it does not synchronize provider data. Preparation verifies the exact Git credential route before it stores a workspace launch specification.

The hub may have more than one credential source. User-attributed provider mutations use a user credential. Forge-managed Git follows normal credential priority and can use an App credential for repositories covered by that App. Verify the route that matters to your deployment instead of assuming all hub Git traffic uses one credential.

The hub strips browser authorization, cookies, origin, and forwarding headers before it calls a spoke. It adds only the federation credential enrolled for that exact HTTPS origin and does not follow redirects.

Enroll one spoke at a time¶

Finish and verify one spoke before enrolling another. This keeps state handoff and rollback bounded to one machine.

1. Create a one-time token¶

On the hub:

umask 077
kenn-forge fleet enrollment-token \
  --base-url https://forge-hub.example.test \
  --name "Forge hub" \
  --ttl 10m > ./forge-enrollment-token

The token is printed once, expires, and can create one enrollment. Creating and transferring it approves that spoke to finish preparation and activate. Treat it as a membership credential.

Transfer the file through an approved secret-sharing channel. Do not put the token in a command-line argument or config file.

2. Join from the spoke¶

On build-a:

kenn-forge fleet join https://forge-hub.example.test \
  --base-url https://build-a.example.test \
  --name "Build spoke A" \
  --token-file ./forge-enrollment-token
rm ./forge-enrollment-token

For automation, pass the token on standard input instead of creating a spoke-side file:

secret-command | kenn-forge fleet join \
  https://forge-hub.example.test \
  --base-url https://build-a.example.test \
  --name "Build spoke A"

Interactive terminals use a hidden token prompt when neither input method is provided.

Joining records a pending enrollment. It does not change the spoke role or restart the daemon. Record the printed enrollment ID with the host and origin in your private operations inventory; revocation uses that ID. Remove any remaining token copies after a successful join.

3. Prepare the spoke¶

Run on the spoke:

kenn-forge fleet prepare-spoke

Preparation stops new provider writes, waits for admitted writes and deferred merges, drains notification acknowledgements, refreshes workspace launch information, hands review drafts and user-authored workflows to the hub, and seals local provider writes. If it reports concrete remaining work, resolve that work and run the command again.

This handoff does not copy the entire provider database or move repositories, worktrees, or agent sessions. The hub must already track the repositories needed by existing workspaces, and the spoke must be able to use their Git credentials.

The token's original deadline no longer applies after preparation starts.

4. Restart and verify activation¶

When preparation reports completion, restart Forge through the spoke's service manager. Activation happens during startup. Verify the spoke is active in the hub's Fleet settings and in the spoke's direct UI.

For services installed by fleet setup, run the command for your platform as the service user:

# Linux
systemctl --user restart kenn-forge.service
systemctl --user status kenn-forge.service --no-pager

# macOS, in the user's GUI login session
launchctl kickstart -k "gui/$(id -u)/io.kenn.forge"

For an existing custom service, use its actual unit or LaunchAgent label. A running process alone does not prove enrollment activated.

Do not enroll the next spoke yet. Complete the checks in Verify the fleet for this spoke first.

Abort or revoke an enrollment¶

Before activation, restore standalone operation from the spoke:

kenn-forge fleet abort-preparation

This revokes the pending enrollment and reopens the durable provider-write gate. On a standalone process, provider writes resume immediately. If the daemon already restarted in spoke mode, the command reports that another restart is required.

If the hub is unavailable, --force performs local recovery and prints the enrollment ID that must be revoked later. The spoke retains only a narrow revocation credential so that later hub cleanup can still complete.

To remove a pending or active enrollment, run on the hub:

kenn-forge fleet revoke ENROLLMENT_ID

The first attempt requires the spoke to be reachable so it can invalidate its local side of the relationship. Once the spoke acknowledges revocation, retries can finish hub cleanup without contacting it again. Revoke before removing a spoke from configuration. After activation, restoring a former standalone spoke requires its complete pre-enrollment unit: config, database, federation store, credential store, and binary. Do not restore only one file from that set.

Verify the fleet¶

Configuration checks are not enough. Exercise the protected paths and the behavior an operator depends on.

Transport and identity¶

• Open every direct HTTPS UI and establish its own authenticated browser session.
• Confirm every HTTPS certificate validates normally.
• Confirm the HTTPS ingress is reachable only through the intended private network.
• Confirm every daemon reports a unique node ID.
• Restart each daemon and confirm its node ID and enrollment persist.

Fleet view and local ownership¶

• Confirm every UI shows the same fleet host set and provider data.
• Create a disposable workspace on each execution host.
• Confirm each spoke UI shows fresh local state for its own workspace and does not list workspaces owned by the hub or another spoke.
• Confirm the hub shows each workspace under its owning spoke.
• Start a tmux-backed session and attach through the spoke's direct UI.
• Attach to the same session through the hub's WebSocket proxy.
• Remove the disposable spoke workspace from the hub and confirm the spoke performs the deletion.

A spoke UI keeps the fleet host directory in the Forge selector, but its workspace surface is local-only. Use the selector to open another spoke, or use the hub for the fleet-wide workspace view and remote mutations.

Provider and Git ownership¶

• Observe provider synchronization on the hub.
• Confirm spokes run no provider synchronization worker.
• Start one safe provider mutation from a spoke UI and observe it complete through the hub.
• Clone or fetch a private repository on every spoke and confirm that machine's Git credential route is used.
• Confirm provider API request budget is not independently consumed by every spoke.

Failure behavior¶

• Stop one spoke. Healthy machines and their workspaces should remain visible; only the stopped spoke should become unavailable.
• Stop the hub. Each spoke should retain local workspace authority, keep provider data already loaded in that browser tab visible behind a stale-data banner, disable Sync and pull-request Refresh, and mark aggregate data incomplete.
• Reload a spoke UI while the hub remains stopped. The new page has no persistent provider cache, so provider pages cannot repopulate; local workspaces must remain usable.
• Restart the hub. Spokes should reconnect, replay events, reconcile provider state, and restore the fleet view without a browser reload.
• Restart each spoke. Its direct UI, local workspaces, enrollment, and remote terminal attachment should recover.

Use the fleet¶

Use the Forge selector in the top bar to open any fleet member directly. It is an ordinary link, so browser modifiers open another tab or window. Each origin keeps its own browser state, filters, searches, terminal connections, and event cursors; changing one Forge tab does not retarget another.

Open the hub UI for the aggregate workspace view and fleet-wide operations. Supported actions call the owning spoke's HTTPS API:

• create a repository workspace on the hub or a selected writable spoke;
• inspect, refresh, or remove a workspace on its owning machine;
• launch or stop a runtime session;
• inspect local Git state;
• open a terminal through the WebSocket bridge.

Every hub workspace row shows its execution-node badge. The Run on selector in New workspace defaults to the hub, keeps unavailable spokes visible with their reason, and sends the create request directly to the selected owner.

The hub never creates a local proxy process for a remote terminal. Input, output, resize messages, and close events pass between the two WebSocket connections.

Open a spoke UI when you want that machine's local execution context while still seeing the aggregate fleet and global provider data. If the hub is down, the spoke remains useful for local workspace operations but cannot provide current hub-owned provider data.

Understand Sync, Refresh, and hub outages¶

Provider controls keep the same owner regardless of which Forge UI you opened:

Opening a pull request normally reads the hub's stored detail first. If the repository is tracked but that pull request is missing from the stored projection, Forge performs one targeted pull-request sync automatically. This repairs a cold or partially completed repository sync without starting another full sync. If the repository itself is not configured, Forge directs you to Settings → Repositories; add it there, then use Sync current repo or reopen the pull request.

When a spoke loses its hub connection, it keeps provider data already loaded in that browser tab visible and labels it as stale. Sync and pull-request Refresh are disabled, provider changes are unavailable, and local workspace and terminal operations continue. New shell and agent sessions in an existing workspace also remain local; generated agent context uses the last validated workspace facts. A fresh browser load has no durable provider cache, so it cannot reconstruct provider pages until the hub returns.

Reconnection is automatic. The spoke replays hub events and reconciles the visible provider projection before removing the stale-data banner. Do not reload merely to recover a connection.

Installing a provider API credential on a spoke does not create automatic failover. In spoke role, Forge uses local credentials for Git clone, fetch, pull, and push, but it does not construct an independent provider syncer. Automatic multi-writer provider failover is not currently supported.

Advanced: publish with your own reverse proxy¶

The setup command always binds Forge to loopback and validates the public authority with trust_reverse_proxy = false. Your proxy must preserve the incoming Host header and forward to the configured loopback port. It must not rewrite Host to the upstream address.

For example:

https://build-a.example.test {
    bind 192.0.2.20
    reverse_proxy 127.0.0.1:8091
}

Use your real private address and a certificate trusted by every fleet machine. Verify the HTTPS URL with the normal operating-system trust store. Do not use curl -k; it hides certificate and hostname mistakes that also break federation.

--origin does not manage this proxy, its certificate, DNS, or browser login. That separation is intentional: the federation contract is canonical HTTPS, not Tailscale or Caddy.

Replace the hub¶

Do not replace the hub by changing DNS or copying one credential. An enrollment is bound to the hub's stable node ID and canonical origin. The hub also stores provider state alongside its own local execution state.

Choose the recovery you need before changing enrollment:

Move to a fresh hub¶

This procedure rebuilds provider data through synchronization. It preserves each retained machine's local identity and workspaces; it does not promise to reproduce the old hub's full archive. Preparation transfers review drafts and user-authored workflows when each machine joins.

The old hub and spokes must remain reachable for revocation. If the old hub was lost in a reinstall, restore it from its matching backup first. A replacement at the same hostname with an empty data directory is still a different hub.

1. Record and back up the old fleet. Record each machine's node ID, origin, enrollment ID, config and data paths, service definition, binary version, and local workspace list. Pause fleet changes. Stop the Forge services while taking a consistent backup of their databases and state, including any SQLite WAL files, enrollment and credential stores, configuration, binaries, and service definitions. Keep these as matching rollback sets, with access restricted to the operator. Back up local repositories and worktrees separately if they are outside those paths. Restart the old fleet and verify it before proceeding.
2. Prepare the new hub. Use a new data directory and its own private HTTPS origin. Follow Set up the hub, authenticate its provider and Git access, and configure its repository list. Check actual repository and pull-request reads, plus any local agent tools the hub will run.
3. Move one spoke. On the old hub, run kenn-forge fleet revoke ENROLLMENT_ID using that spoke's old enrollment ID. Then create a new token on the new hub and follow join, prepare, restart, and verify on the spoke. Use the new hub origin and retain the spoke's existing origin, config, and data directory. Expect provider access to pause between revocation and activation. Confirm the spoke's node ID and local workspace list are unchanged, and that the new hub can reach its workspaces and terminals, before moving the next spoke.
4. Move the old hub last, if it will become a spoke. Revoke all its old members first; a hub with members cannot join another hub. Keep its existing service, origin, and data directory, then run the same join, prepare, and restart sequence against the new hub. Its local workspaces remain on that machine. Do not set its role by hand or run spoke setup merely to change the role. If retiring the old hub instead, stop its service after the cutover.
5. Verify and record the final fleet. Check every direct HTTPS UI and the new hub's aggregate workspace view. Exercise a local agent and a terminal through the hub on each execution machine. Record the new enrollment IDs, service restart commands, and verification results. Keep the pre-cutover backups until the replacement has passed your recovery checks.

Roll back a cutover¶

Rollback restores a coordinated fleet state, not just the old hub's database. Stop the new hub and affected Forge services first. Preserve current state and any work created since the cutover, then restore the old hub and affected members from their matching pre-cutover sets. Restore the service definitions and original ingress bindings along with state. Start the old hub, then its spokes, and verify their original identities, enrollment, and workspace access. Keep the replacement hub stopped while this restored fleet is authoritative.

Database rollback can discard changes made after the backup. Do not overwrite new worktree contents as part of restoring Forge state. Revoking a new enrollment alone does not restore the old enrollment or provider ownership.

Upgrade federation protocol 3 to 4¶

A protocol-3 fleet needs a one-time data migration before protocol-4 binaries can use its enrollment. Keep a matching rollback backup of every host's database, enrollment and credential stores, configuration, and binary.

1. Stop the Forge service on every fleet member. Keep its worktrees and tmux sessions in place.
2. On each host, run the new binary against that host's existing configuration:

/path/to/new/kenn-forge fleet migrate-protocol --config ~/.kenn/forge/config.toml

1. Install the new binary, start the hub, then start its spokes. Verify the fleet workspace list and terminal access on each member.

The migration retains node identities, enrollment, credentials, preparation seals, local workspaces, and provider data. It does not enable communication between protocol-3 and protocol-4 peers. Repeating the command after an interruption resumes the migration; already migrated and unenrolled hosts need no changes. To roll back, stop the affected services and restore every host's matching pre-migration state and binary together.

Replace an older fleet¶

Older key-based and shell-relay entries are not accepted. Before starting the new binary:

1. Remove the old fleet entries from every config file.
2. Enable API authentication and select the hub.
3. Start each daemon with a reachable HTTPS origin.
4. Enroll every spoke again with a new one-time token.

Do not copy old tokens, node IDs, relay sockets, or daemon data directories. There is no automatic translation because the older entries did not establish the identity and credential pair required by federation.

Troubleshooting¶

Setup fails its HTTPS check¶

Read the setup error and any rollback errors before retrying. Setup attempts to restore its previous config, service, and managed publication when a later step fails. Confirm what is actually running; do not assume a failed setup left a usable service behind.

For Tailscale, check that the device is logged in, the Forge user can manage Serve, and HTTPS certificates are enabled. First-time certificate issuance can outlast Forge's readiness check. Inspect Tailscale's certificate and Serve status, resolve the reported problem, then rerun the setup plan. For external ingress, check the proxy route, preserved Host header, and certificate trust.

Agents work in SSH but not in Forge¶

Check the service's PATH and the service user's agent authentication. Shell startup files are not loaded by the generated service. For a service owned by setup, rerun the appropriate setup command with the intended stable tool paths; for a custom service, update its environment through your service manager. Restart and launch an agent from a workspace to verify the result.

A fresh hub is missing repositories or pull requests¶

Check Settings → Repositories on the new hub first. Enrollment does not import the old hub's repository selection. Then inspect sync status: a large first sync can consume the available request budget before all repositories finish. Let the budget recover and use Sync current repo for the repository you need, rather than repeatedly starting a full sync. Verify a real repository or pull-request read separately from fleet reachability.

A spoke is unreachable¶

From the hub machine, verify the spoke's HTTPS origin and certificate. Confirm the URL exactly matches the enrolled origin. A changed hostname, port, or certificate route requires deliberate re-enrollment rather than a redirect.

If the daemon is down, restart it through its local service manager. The hub does not log in to start it.

A request is forbidden¶

Read the full degraded-host message in the hub. Forge preserves the peer's authorization reason instead of reducing it to an HTTP status.

If the message says fleet authorization is inactive, inspect the spoke service's startup log and confirm it can resolve and reach the hub. An active spoke revalidates its sealed enrollment with the hub at startup and every six hours. If the hub is temporarily unavailable, the spoke can use its last confirmed activation lease for up to 24 hours. Once that lease expires, hub requests are rejected until the spoke reaches the hub and renews it.

The lease handshake is versioned separately from fleet snapshots. During a rolling upgrade, upgrade the hub first, then each spoke. A lease-unaware spoke is isolated from fleet reads and writes until it is upgraded and restarted; its existing enrollment is reused, so you do not need to re-enroll it. If a spoke was upgraded before its hub, upgrade the hub and restart that spoke so it can complete the lease handshake.

If the peer rejects the hub identity, check that the enrollment is active and has not been revoked. Re-enroll instead of widening or copying a token by hand. A valid credential can still be forbidden when its direction does not grant the requested operation.

A terminal does not open¶

Confirm the spoke is reachable and the local session still exists. The hub attaches to the spoke's terminal WebSocket; it does not attach to the spoke's tmux server directly.

A config no longer loads¶

Remove old fleet keys and relay entries, then use the enrollment workflow. Forge fails closed instead of guessing how an old host label maps to a stable spoke identity.