Automations lifecycle

The coven.automations.v1 contract defines four state machines, and the invariants they preserve, in spec/coven-automations/v1/state-machines.json. The diagrams and tables on this page render from a byte-for-byte copy of that file at the reviewed upstream commit, so they cannot drift from it.

Proposed contract, partial implementation

The contract is proposed, not certified. Its conformance manifest records releaseState: "proposed" and productionReady: false. At the reviewed commit, upstream tests compare only the occurrence transitions of an operator retry against these machines; nothing checks the rest of the daemon against them. The daemon records a subset of these states, and in places a different one; each section below says which. Branch on the values the daemon returns, described in Coven automations.

Reading the machines

Clients never write state. Command handlers, the scheduler, the dispatcher, and workers commit every transition:

on names the cause.
actor names who may commit the transition.
guard names the condition that must hold.

Terminal states are drawn in bold, and no transition leaves one. Acceptance, dispatch, and success are separate states: an accepted run has not launched anything, a dispatched attempt has not proved anything, and only runtime evidence settles a run succeeded.

Definition

Rendering diagram…

definition.v1: starts in draft; terminal states are bold.

States: draft, paused, active, disabled, invalid, tombstoned (terminal).

definition.v1 transitions
Transition	On	Actor	Guard
`draft` → `paused`	`definition_revise_committed`	`command_handler`	definition validated; revision incremented
`draft` → `tombstoned`	`definition_tombstoned`	`command_handler`
`paused` → `active`	`definition_activated`	`command_handler`	familiar, authority, and runtime requirements validated
`paused` → `disabled`	`definition_disabled`	`command_handler`
`paused` → `tombstoned`	`definition_tombstoned`	`command_handler`
`active` → `paused`	`definition_paused`	`command_handler`
`active` → `disabled`	`definition_disabled`	`command_handler`
`active` → `invalid`	`definition_invalidated`	`command_handler`	a revise committed an invalid body or dependencies (familiar/runtime/authority) became unsatisfiable
`active` → `tombstoned`	`definition_tombstoned`	`command_handler`
`disabled` → `paused`	`definition_reenabled`	`command_handler`	explicit re-enable; disabled never auto-reactivates
`disabled` → `tombstoned`	`definition_tombstoned`	`command_handler`
`invalid` → `paused`	`definition_revise_committed`	`command_handler`	a revise repaired the body and validation passed
`invalid` → `tombstoned`	`definition_tombstoned`	`command_handler`

What the daemon records today:

lifecycleState is draft, paused, active, disabled, or invalid. A definition created with definition.create.v1, or imported from a legacy source, starts in draft, and definition.revise.v1 moves it to paused.
Tombstoning sets tombstonedAt instead of a lifecycle value. Events report the change as tombstoned.
invalid comes only from the one-time migration of a stored legacy definition that cannot be digested. No command moves an active definition to invalid.
No versioned command re-enables a disabled definition yet: definition.activate.v1 requires paused.

Occurrence

Rendering diagram…

occurrence.v1: starts in planned; terminal states are bold.

States: planned, eligible, claimed, dispatching, running, recovering, recovery_required, succeeded (terminal), failed (terminal), cancelled (terminal), timed_out (terminal), skipped (terminal), superseded (terminal).

occurrence.v1 transitions
Transition	On	Actor	Guard
`planned` → `eligible`	`eligibility_passed`	`scheduler`
`planned` → `skipped`	`misfire`	`scheduler`	misfire policy disposes this slot (e.g. overlap_forbid, paused, invalid)
`planned` → `superseded`	`definition_superseded`	`scheduler`	a newer revision replaced the definition before this slot was claimed
`planned` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation command committed before any claim
`eligible` → `claimed`	`claim`	`scheduler`	compare-and-set on state='eligible' or 'planned' with lease set; increments fence generation
`eligible` → `skipped`	`misfire`	`scheduler`
`eligible` → `superseded`	`definition_superseded`	`scheduler`
`eligible` → `cancelled`	`cancel_acknowledged`	`command_handler`
`claimed` → `planned`	`operator_retry`	`command_handler`	attempt.retry.v1 names the latest immutable failed attempt of an ordinary held run; original deadline remains open, fewer than ten attempts exist, quarantine is released, and no cancellation or stop ownership is unresolved; opens a new adopted attempt
`claimed` → `planned`	`automatic_retry`	`dispatcher`	proven pre-ownership failure is retryable under the pinned policy; opens the next attempt with prior disposition and backoff; the original run deadline still bounds dispatch
`claimed` → `failed`	`run_settled`	`dispatcher`	known pre-ownership failure has no automatic retry or operator hold window, or its held run deadline expires; prior terminal attempt stays immutable
`claimed` → `dispatching`	`attempt_adopted`	`dispatcher`	attempt created with adoptionKey; fence generation matches the claim
`claimed` → `recovering`	`lease_expired`	`scheduler`	leaseExpiresAt < now and no runtime evidence recorded
`claimed` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation reconciled before dispatch
`dispatching` → `running`	`runtime_started`	`dispatcher`	runtime acknowledged the session start (attempt started)
`dispatching` → `failed`	`launch_refused`	`dispatcher`	launch failed deterministically before any side effect; occurrence settles failed with reason launch_refused
`dispatching` → `recovering`	`lease_expired`	`scheduler`	no runtime evidence before lease expiry
`dispatching` → `cancelled`	`cancel_acknowledged`	`command_handler`	no attempt reached started; otherwise cancellation waits for reconciliation
`running` → `planned`	`operator_retry`	`command_handler`	attempt.retry.v1 names the latest immutable failed attempt of an ordinary held run; original deadline remains open, fewer than ten attempts exist, quarantine is released, and no cancellation or stop ownership is unresolved; opens a new adopted attempt
`running` → `succeeded`	`run_settled`	`dispatcher`	runtime evidence proves completion; receipt recorded
`running` → `failed`	`run_settled`	`dispatcher`	runtime evidence proves failure and no operator hold window remains, or the held run deadline expires; prior terminal attempt stays immutable
`running` → `timed_out`	`timeout`	`scheduler`	perRunMinutes elapsed without settled evidence
`running` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation acknowledged by runtime or reconciled by recovery
`running` → `recovery_required`	`evidence_lost`	`scheduler`	runtime lost mid-run: outcome cannot be determined automatically
`recovering` → `failed`	`recovery_determined`	`scheduler`	no side effects were possible (sideEffectClass bounds proven)
`recovering` → `recovery_required`	`recovery_ambiguous`	`scheduler`	side effects cannot be ruled out; requires explicit operator recovery
`recovery_required` → `failed`	`operator_recovery_failed`	`command_handler`	occurrence.recover.v1 with evidenceDetermination=failed_deterministic
`recovery_required` → `dispatching`	`operator_recovery_attempt`	`command_handler`	occurrence.recover.v1 with evidenceDetermination=retry_with_new_attempt; opens a new attempt carrying priorDisposition=ambiguous

What the daemon records today:

Occurrences are planned, claimed, running, or recovery_required, and settle succeeded, failed, cancelled, or skipped. skipped marks a slot collapsed by the misfire policy.
eligible is a read view over due planned occurrences (occurrence.list.v1 with view: "eligible"), not a stored state.
The daemon does not record dispatching, recovering, or timed_out on an occurrence. A claim moves straight to running when the runtime starts, a claim whose lease expires before dispatch settles failed, and a timeout settles the occurrence failed.
superseded appears only on occurrences migrated from the legacy store, where every legacy skipped row became superseded.
occurrence.recover.v1 with retry_with_new_attempt returns the occurrence to planned for dispatch rather than to dispatching.
occurrence.cancel.v1 and attempt.cancel.v1 settle a planned or claimed occurrence cancelled directly, before anything is dispatched, as the contract's cancel_acknowledged transitions allow.
When an ordinary run is held for an operator retry, its occurrence stays claimed (the attempt failed before launch) or running (it failed after start). attempt.retry.v1 returns it to planned, the contract's operator_retry transitions; if the run's deadline passes first, it settles failed.

Run

Rendering diagram…

run.v1: starts in accepted; terminal states are bold.

States: accepted, running, succeeded (terminal), failed (terminal), cancelled (terminal), timed_out (terminal), ambiguous (terminal).

run.v1 transitions
Transition	On	Actor	Guard
`accepted` → `running`	`attempt_started`	`dispatcher`	command adoption committed before any side effect; currentAttemptId set
`accepted` → `failed`	`launch_refused`	`dispatcher`	every attempt failed deterministically before evidence
`accepted` → `cancelled`	`cancel_acknowledged`	`command_handler`
`running` → `succeeded`	`attempt_settled`	`dispatcher`	an attempt settled succeeded; receipt recorded
`running` → `failed`	`attempt_settled`	`dispatcher`	an attempt settled failed, retry policy opens no further attempt, and no operator hold window remains; held ordinary runs retain running until cancellation or their original deadline
`running` → `timed_out`	`timeout`	`scheduler`
`running` → `cancelled`	`cancel_acknowledged`	`command_handler`
`running` → `ambiguous`	`attempt_ambiguous`	`dispatcher`	the current attempt settled ambiguous and no operator recovery has resolved it

What the daemon records today, in the run's status field:

A run is created running. The daemon has no separate accepted status.
Runs settle succeeded, failed, or cancelled. A timeout settles the run failed, and its attempt records timed_out.
Cancelling an attempt that is still adopted and waiting for its retry settles the attempt and its run cancelled with no session to stop.
An ordinary run whose attempt failed with no automatic retry left stays running, held for an operator retry, until its deadline; it then settles failed. The failed attempt stays failed either way. See Operator retry.
The daemon never records ambiguous on a run. Ambiguous evidence either settles the run failed or leaves it running with an ambiguous attempt while its occurrence waits in recovery_required; see Manual runs and health.

Attempt

Rendering diagram…

attempt.v1: starts in adopted; terminal states are bold.

States: adopted, dispatching, started, observing, succeeded (terminal), failed (terminal), cancelled (terminal), timed_out (terminal), ambiguous (terminal).

attempt.v1 transitions
Transition	On	Actor	Guard
`adopted` → `dispatching`	`dispatch`	`dispatcher`	dispatch generation incremented; exactly one dispatch in flight per attempt
`adopted` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation committed before dispatch
`dispatching` → `started`	`runtime_started`	`dispatcher`	runtime session bound; a second session bind is ILLEGAL_TRANSITION
`dispatching` → `failed`	`launch_refused`	`dispatcher`	deterministic pre-side-effect refusal
`dispatching` → `ambiguous`	`dispatch_unconfirmed`	`dispatcher`	dispatch sent but no deterministic ack before lease expiry; terminal for the attempt
`dispatching` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation acknowledged before runtime start
`started` → `observing`	`evidence_stream_open`	`worker`	output/event cursors established
`started` → `failed`	`run_settled`	`worker`	terminal failure evidence without an observing phase; terminal attempt remains immutable even when the ordinary run holds for operator retry
`started` → `succeeded`	`run_settled`	`worker`	terminal evidence without an observing phase
`started` → `ambiguous`	`evidence_lost`	`scheduler`	session vanished before evidence
`observing` → `succeeded`	`run_settled`	`worker`	verified completion evidence
`observing` → `failed`	`run_settled`	`worker`	verified failure evidence
`observing` → `timed_out`	`timeout`	`scheduler`	perRunMinutes elapsed
`observing` → `cancelled`	`cancel_acknowledged`	`command_handler`	cancellation acknowledged by runtime
`observing` → `ambiguous`	`evidence_lost`	`scheduler`	heartbeat/lease evidence expired mid-observation

What the daemon records today:

All nine attempt states, and a terminal attempt row can be neither updated nor deleted.
A few transitions the contract does not list: a dispatch rolled back before launch returns the attempt to adopted, and a confirmed cancellation or timeout can settle a dispatching or started attempt directly.

Invariants

adoption-before-side-effects: Command adoption commits before any consequential side effect is issued; a run is accepted (run.v1 accepted) before dispatch begins.
one-fence-one-run: One occurrence fence generation cannot own two accepted runs: acceptance requires a compare-and-set on the occurrence fence generation, and a second acceptance with the same generation is a typed conflict.
one-attempt-one-session: One attempt cannot bind two runtime sessions; a second session bind on the same attempt is ILLEGAL_TRANSITION and commits nothing.
terminal-no-regress: Terminal states never regress; no transition in any machine above leaves a terminal state.
no-evidence-no-success: Absence of runtime evidence can never become success; the only transitions out of evidence loss are failed, recovery paths, ambiguous, or timed_out.
cancellation-is-a-request: Cancellation is a request until acknowledged or reconciled; CANCEL_PENDING is returned while a running attempt has not acknowledged, and the occurrence settles cancelled only after acknowledgment or reconciliation.
retry-new-attempt: Retry creates a new attempt with attemptNumber incremented and priorDisposition set; the prior attempt's record is never rewritten.
ambiguous-never-auto-retried: Ambiguous mutating work is not automatically retried; only occurrence.recover.v1 with an explicit operator determination can open work after an ambiguous disposition.
revision-pins-history: Definition revision changes never rewrite historical occurrences, runs, attempts, or receipts; each pins the automationRevision and definition digest it was created and executed against.
tombstone-preserves-history: Deleting a definition tombstones it without erasing required history; tombstoned definitions never plan, claim, or run, and their records remain readable.

These are the contract's normative statements. Upstream's conformance manifest lists a state-machine-property-tests suite for them, but that suite does not exist at the reviewed commit, so read them as the design target rather than verified daemon behavior. Upstream's vector-driven suites cover parts of them: attempt terminal immutability, run terminal monotonicity, occurrence fence uniqueness, occurrence lease recovery, and definition lifecycle transitions (without draft or invalid).

Coven automations for the actions, their results, and recovery.
Automations conformance for what upstream tests against these machines and the other profiles.
Coven local API for transport and trust rules.

Reading the machinesCopy Anchor Link

DefinitionCopy Anchor Link

OccurrenceCopy Anchor Link

RunCopy Anchor Link

AttemptCopy Anchor Link

InvariantsCopy Anchor Link

RelatedCopy Anchor Link

On this page

Reading the machines

Definition

Occurrence

Run

Attempt

Invariants

Related