State machines

atif-sql · State machines

No entity in this workspace carries a status enum with declared transitions. The enums the workspace declares aren’t lifecycles either. RecordType (packages/atif-converter/src/atif_converter/domain/fidelity.py:26) and FidelityGap (:55) are closed classification vocabularies whose members never move into one another. OutputFormat (packages/atif-cli/src/atif_cli/output.py:61) has one resolution: resolve_format maps AUTO to TABLE on a TTY and JSON otherwise (:84), but that is an idempotent function evaluated per emit, a no-op for every explicit format (:82-83), whose result is never stored and never advances again. One irreversible resolution with no persisted state is not a lifecycle.

The machines below are durable-state lifecycles instead: the state lives in a row, a sidecar file, or the presence of a directory, and named functions move an entity between named states. State names and transition labels are verbatim source text from each machine’s Defined at: file.

Corpus session materialization

Every pass over the raw transcript corpus reclassifies each discovered session from scratch, so the state is not a stored field; it is what the watermark records, whether the session’s artifact directory exists, and what its meta.json (or the empty-session record) says about the generation and the source. The classification is the if / elif / else at packages/atif-corpus/src/atif_corpus/domain/sessions.py:218-223, and the outcome states are the counters and id tuples of MaterializationReport (packages/atif-corpus/src/atif_corpus/application/materialize.py:220-287).

• skipped_live is the entry state for a session still being written, not an error state; a session first scanned after it went quiet enters to_materialize directly. A transcript is appended to many times per turn, and a session qualifies to convert only once its newest source mtime is at least quiesce_seconds old, 300 by default (packages/atif-corpus/src/atif_corpus/domain/sessions.py:84). force overrides staleness but never quiescence, because converting a half-written transcript produces a wrong artifact rather than a stale one (:183-185).
• failures --> to_materialize is the retry, and it works only because _advance_watermark retains the failed session’s entries. Staleness is “recorded set differs from scanned set”, so the retained entry is the retry signal; dropping it would classify the session up_to_date forever (packages/atif-corpus/src/atif_corpus/application/materialize.py:1324-1331).
• up_to_date --> to_materialize: _unmaterialized_session_ids is the crash-recovery edge. It is reachable exactly one way: a pass killed inside the directory swap, after the previous generation was renamed aside and before the new one landed. The watermark records source mtimes only and cannot express a missing artifact directory, so the directory check is what force-replans it (:1025-1031).
• empty_session_ids isn’t a failure state. A converter that raises EmptySourceError found a well-formed transcript with nothing to convert, so its watermark entries advance and empty_sessions.json records the generation it was checked under (packages/atif-corpus/src/atif_corpus/application/materialize.py:1742-1745, :1360-1384). The next pass counts it up_to_date until its source or the converter schema moves.
• unreadable_session_ids is a parking state, and a session can starve in it. A transient stat failure clears next pass; a permanent one (a side-file left at mode 000) leaves the session never materialized, never up_to_date, never skipped_live, and deliberately never marked source-removed, which is why the report carries the ids at all (:253-261).
• No state here deletes a session. removed_session_ids lists the sessions whose source vanished this pass: _retain_sourceless_sessions keeps the corpus session directory and rewrites its meta.json once with source_present: false and source_removed_at (packages/atif-corpus/src/atif_corpus/application/materialize.py:1255-1260), because the agent expires old transcripts and the corpus holds the only copy left (:46-49). Later passes count the session in retained_count. _advance_watermark drops its entries, so a source that comes back reads as stale and converts again from the live file (:1333-1336). A retained session whose recorded generation is stale re-converts from its raw source archive, when it has one, and stays marked (:1270-1307). A session whose sources merely could not be read is never marked source-removed (:1202-1208), and a scan that found zero sessions over a non-empty corpus raises SuspiciousEmptyScanError instead of marking every session (:1661-1667).
• Transition sites: scan_sources at :1622; QuiescencePolicy at :1699; _write_session at :777; MaterializationFailure at :1740; _retain_sourceless_sessions (it keeps and marks, never deletes); _advance_watermark at :1778.

Defined at: packages/atif-corpus/src/atif_corpus/application/materialize.py:1497

Embed store schema version

The vector store’s state is a schema generation recorded in a schema_version.json sidecar next to the Lance table, written on create and checked on open. A sidecar rather than a table column, because the version describes the schema and reading it must not require the schema to be readable (packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:85-89). SCHEMA_VERSION is 2 (:83).

• v1 and unrecorded have no incoming edge on purpose. This code never writes either state; it discriminates a store it finds on disk. v1 is the five-column shape lacking text_hash and truncated; unrecorded is a store whose columns are already current but whose sidecar is absent, which read_schema_version reports as None (:342-355).
• v1 --> v2 is online and leaves the vectors alone. migrate_pre_stamp_table adds the two columns with ALTER TABLE ... ADD COLUMN ... DEFAULT and a literal default, then marks each NOT NULL: no rows dropped, no vectors re-embedded, and readers keep working throughout (:365-401). text_hash backfills to _PRE_STAMP_SENTINEL, the literal <pre-stamp> (:130), whose angle brackets make it impossible to equal a real blake2b digest; every migrated row therefore mismatches its corpus hash in the discovery anti-join and re-embeds incrementally through the ordinary staleness path (:126-129).
• v2 --> v2: has_table is idempotent reopening. migrate_pre_stamp_table returns early when nothing is missing (:392-393) and the sidecar stamp is left alone.
• The machine has no terminal state in source. v2 is absorbing; the store is never dropped here. A breaking change (a provider or dimension switch) is refused rather than migrated, and stays fail-loud through embedding_guard (:33-35).

Defined at: packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:421

Retry-queue row

One row per (pipeline, unit_id) in the retry_queue table of the analytics state.db. The state is the triple “is completed_at NULL”, “attempts against max_attempts”, and “next_attempt_at against now”. The transition table is written as source at packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:12-16, and the columns holding the state are declared at :45-56.

• The first enqueue lands in backoff, never due: it stamps next_attempt_at at now plus _backoff_delta(1), which is two minutes (:78-81, :123). Backoff is exponential in minutes (2, 4, 8, 16, 32), and _BACKOFF_CAP_MIN clamps it at 60 (:43, :80). That clamp is defensive rather than reachable at the shipped default: drain admits a unit only while attempts < max_attempts (:162), so with MAX_ATTEMPTS_DEFAULT of 5 (:42) the largest value _backoff_delta receives is 5 and the longest real wait is 32 minutes.
• backoff and exhausted together are exactly what blocked_units returns, and due is what drain returns; the two are complementary partitions of the live rows (:176-207, :145-173). Together they make the queue the single re-admission gate for a failed unit: a failed unit is not checkpointed, so without the blocked_units subtraction the checkpoint path would re-admit and re-bill it on every run (:188-192).
• exhausted is terminal in effect and has no outgoing edge. drain never returns it, so the pipeline never dispatches it, so it never fails again and is never marked done. The guard is keep -= blocked at packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:139, spelled already |= blocked in packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:250, whose unit is a message uuid rather than a session id. exhausted is a parking state rather than a deleted row: the module issues no DELETE, and raising max_attempts on a drain call re-admits it.
• completed --> backoff is real, and the attempt counter survives it. enqueue clears completed_at back to NULL on conflict (packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:128-133) while reading the prior attempts with no filter on completed_at (:115-118), so a unit that failed four times, succeeded, then failed again re-enters at attempts = 5 and is exhausted immediately under the default MAX_ATTEMPTS_DEFAULT of 5 (:42). Every state name here is a row predicate, not a stored string; the table has no status column (:45-56).

Every pipeline named by PIPELINE_NAMES (packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:37-42) fires the identical call sequence (drain, then blocked_units, then enqueue on failure and mark_done on success):

pipeline	drain	blocked_units	enqueue	mark_done
classify	packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:124	:133	:269	:294
conflicts	packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py:139	:146	:268	:335
user_friction	packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:236	:244	:450	:470
perceived	packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:164	:171	:291	:340

Defined at: packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:12

See also

• processes
• module map
• business logic
• contract map
• debugging guide