Skip to main content

Documentation Index

Fetch the complete documentation index at: https://s2.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

Clients interact with S2 at its public https endpoints, primarily with JSON bodies conforming to its OpenAPI spec.
Prefer to use an available SDK.

​
Authentication

The API requires a Bearer access token to authenticate requests. S2 SDKs take care of supplying it automatically. If you are using curl, you can provide it with -H "Authorization: Bearer ${S2_ACCESS_TOKEN}".
The first usage of an access token on a connection may experience a latency overhead for verification, typically tens of milliseconds. Subsequent requests will not incur such an overhead.

​
Compression

To optimize network usage, S2 supports compressed request and response bodies with the following algorithms:
  • zstd preferred
  • gzip
Standard HTTP semantics around Content-Encoding and Accept-Encoding apply.

​
Data Plane

There are 3 core data operations on a stream:

Append records

POST /streams/{stream}/records

Read records

GET /streams/{stream}/records?seq_num=42&count=100

Check the tail

GET /streams/{stream}/records/tail

​
Data in JSON

The following pieces of record data are stored as bytes:
  • Header name
  • Header value
  • Body
The header s2-format is used to indicate the desired encoding of these bytes when records are represented in JSON.

raw

s2-format: raw or omit header.Use when your record data is valid Unicode. Zero overhead, human-readable. Cannot handle binary data safely.

base64

s2-format: base64Use when you are working with arbitrary bytes. Always safe. 33% overhead over the wire.
You can write using one format and read with another. When reading raw, S2 is interpreting the stored bytes as UTF-8. This will be a potentially-lossy conversion if it was not also written as raw, or as base64-encoded valid UTF-8.

​
Protobuf messages

Data plane endpoints to append and read records also support protobuf bodies. This helps avoid the base64 encoding tax compared to binary data in JSON messages. To send and receive protobuf:
  • Set the Content-Type header to application/protobuf and send a protobuf-encoded payload.
  • Set the Accept header to application/protobuf to receive a protobuf-encoded response. The response will include the Content-Type: application/protobuf header if the server returns a protobuf.
Type definitions are available in git and Buf.

Append

POST ../{stream}/records AppendInput AppendAck

Read

GET ../{stream}/records?.. ReadBatch
Sending Accept: application/protobuf request header only guarantees a protobuf response in case of a success (HTTP 200). Other status codes are always accompanied by JSON bodies.

​
Sessions

S2S (S2-Session) is a minimal binary protocol used by S2 SDKs to encapsulate streaming append and read session semantics over , at the same endpoints as “unary” calls. It is meant for S2 SDKs, but documented for the adventurous.

​
Setup

  • Content-Type: s2s/proto signals that a session is being requested.
  • Accept-Encoding signals which compression algorithms are supported (service supports zstd and gzip). Content-Encoding is not sent as message-level compression is used.
  • 200 OK response establishes a session.

​
Message framing

All integers use big-endian byte order. Messages smaller than 1KiB should not be compressed.Length prefix (3 bytes): Total message length (flag + body)Flag byte (1 byte): [T][CC][RRRRR]
  • T (bit 7): Terminal flag (1 = stream ends after this message)
  • CC (bits 6-5): Compression (00=none, 01=zstd, 10=gzip)
  • RRRRR (bits 4-0): Reserved
Body (variable):
  • Regular message is a Protobuf
  • Terminal message contains a 2-byte status code, followed by JSON error information (corresponding to unary response behavior)

​
Data flow

Append sessions are a bi-directional stream of AppendInput messages from Client → Server, and AppendAck messages from Server → Client.Read sessions are a uni-directional stream of ReadBatch messages from Server → Client. When waiting for new records, an empty batch is sent as a heartbeat at least every 15 seconds.
Read sessions are also supported over SSE which has the benefit of being usable in the browser context.