Skip to main content

Documentation Index

Fetch the complete documentation index at: https://s2.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

​
Snapshot and follow

Many applications store state as an ordered stream of deltas: each record says what changed. A reader can recover current state by folding those records from the head, then keep the read open to follow new records. However, that replay cost grows with the stream. The snapshot-and-follow pattern bounds it by materializing state at a known stream position, or a cursor. The cursor is the first sequence number not covered by the snapshot — it can come from a writer’s current state, or a separate snapshotting process can checkTail to ensure its knowledge is current.

​
Where snapshots live

​
External snapshots

For snapshots stored outside the stream, such as in object storage, attach the cursor to the snapshot itself. For example, include it in the object key or metadata. To restore, load the snapshot, then read from the cursor and follow the stream. Trimming the stream to the cursor is optional; do it when you want to discard history already covered by the snapshot. For example, a snapshot at cursor 1000 covers records before 1000, so recovery resumes at 1000:
  1. A snapshotting process folds sequence numbers 0 through 999 into state.
  2. It writes snapshots/orders/1000.json, where 1000 is the cursor.
  3. It optionally trims the stream to 1000 to discard the covered records.
  4. A reader loads that object, reads from sequence number 1000, and follows from there.

​
In-stream snapshots

Storing the snapshot in the stream lets readers recover from the stream alone. When paired with trimming, the snapshot record or first fragment can become the new head, so readers simply start from the first record of the stream, apply the snapshot, then follow later deltas. Because the snapshot is written to the stream, make the first append conditional on the cursor. If the stream has moved, rebuild or extend the snapshot and retry.

​
Single-record snapshot

If the snapshot and trim command will fit in a 1 MiB batch, use a single atomic batch to append the trim command and snapshot record. The command that advances the head and the snapshot that remains there become durable together.
  1. Choose cursor 1000.
  2. Build or serialize the snapshot for that position.
  3. Append one batch with match_seq_num = 1000:
    • a trim command record to 1001
    • a snapshot record, e.g. { "type": "snapshot", "covers_before": 1000, "state": ... }
  4. On success, the trim command lands at sequence number 1000 and the snapshot lands at 1001. After trimming takes effect, the command record is removed and the snapshot is the stream head.
  5. Readers start at the head, apply the snapshot, and follow later deltas.

​
Framed snapshot

For snapshots too large for one record, split the snapshot into ordered fragments. The fragment records should be distinguishable from normal deltas, usually with headers or body fields that identify a snapshot ID, chunk order, and final marker. Every fragment is written before trimming.
  1. Choose cursor 1000.
  2. Build or serialize the snapshot for that position.
  3. Append the first snapshot fragment with match_seq_num = 1000.
  4. Append the remaining fragments and a final marker in order, for example by using an append session or by waiting for each append before sending the next one.
  5. Append a trim command record to 1000.
After trimming takes effect, the first snapshot fragment is the stream head. Unlike the single-record case, the trim command remains in the retained stream. Readers can skip it when applying application deltas. While a reader reconstructs the snapshot from its fragments, it may encounter delta records appended concurrently with the snapshot fragments. The reader should buffer those deltas, finish applying the snapshot, then apply the buffered deltas in order and continue following the stream.

​
See also

Trimming

Discard records already covered by a snapshot.

Reads

Read from a cursor and follow new records.

Concurrency control

Use conditional appends with match sequence numbers.