Skip to main content
S2 API is available as s2.dev the managed service, and s2-lite for self-hosting.

​
s2.dev

s2.dev is designed around a truly serverless developer experience. S2 resources can be created instantly, scale seamlessly, and require no operational overhead from users.
You don’t need to know any of this to use the service.
s2.dev has a cellular architecture. A cell is a physically isolated set of cloud resources which helps contain the blast radius of changes (… and cloud provider quotas). Basin and stream endpoints use DNS to route to the right cell. There is exactly one home cell for an account (i.e. your organization) in a given scope (e.g. aws:us-east-1). This makes it straightforward to setup private networking for the data plane against the correct endpoint.
New accounts default to a multi-tenant cell with transparent scale-to-zero pricing.Dedicated cells can be provisioned with a minimum commitment — contact us.
Architecture of a cell

Control plane

  • cop (“channel operator”) ensures resilience and elasticity of stateful pieces:
    • supervises and creates/destroys channels
    • rebalances load by migrating streams
  • bgtask (“background task”) workers partitioned on role and keyspace handle async jobs, such as garbage collection.

Data plane

  • frontend replicas are an elastic deployment that fronts all API requests, acting as a gateway layer.
  • cachey replicas form an elastic read-through cache per zone for all reads from object storage.
  • channel pods sequence writes with a WAL maintained as sequential chunks, and flush batched stream segments accumulated over a longer timeframe.
Our read-through cache, cachey, is an open source project.

​
Write path

The frontend talks to the assigned channel for the stream to append records. If necessary it will first activate the stream by assigning to a channel using power of 2 choices for load balancing. Channels multiplex stream records in chunks on object storage (effectively a WAL), flushing based on a combination of elapsed time and accumulated size. Appends are only acknowledged when associated records are durable in a chunk.
Once a stream is active, metadata storage is not in the hot-path of writes — only object storage.

​
Read path

The frontend uses stream metadata to generate a concatenated iteration over relevant stream segments. This is further chained with pointers into chunks (the channel WAL), if the client is reading recently-written data. It fetches relevant object byte ranges via a cachey pod in its own zone, using rendezvous hashing with bounded load to optimize hit rate without hotspotting. Concurrent reads are combined wherever possible:
  • Tailing readers are coalesced on the frontend as well as channel
  • cachey will coalesce reads for the same “page” of an object
s2.dev effectively implements a purpose-built CDN for streaming data.

​
Stream lifecycle

Streams are made up of a chain of streamlets. There is usually only one, but multiple may exist when the stream is being migrated between channels, or migration(s) recently completed. Streamlets take up a tiny amount of memory on channels — and only stick around if there have been recent writes. Dormant streams live only as metadata and segment data.

​
Storage classes

The choice of storage class affects latency of appending and reading recently-written records. In AWS,
  • Standard is powered by channels operating against a S3 Standard chunk storage bucket, which guarantees writes are durable in multiple zones.
  • Express is powered by channels operating against a quorum of three S3 Express zonal chunk storage buckets – so writes are still guaranteed to be durable in multiple zones, but with much lower latency.

​
Quality of service

Admission control and load balancing logic is embedded into components in the online request path. Auto-scaling and stream migrations handle sustained shifts. To optimize throughput, the data plane employs pipelining, i.e. multiple in-flight operations with an ordering dependency. This is not just a server-side concern, S2 client SDKs will also pipeline when using the append and read session APIs. Stream migrations use a speculation-based protocol that keeps the write pipeline flowing to minimize disruption.

​
Correctness

Beyond unit and integration test, we practice deterministic simulation testing – both with our Rust framework, as well as Antithesis. We also verify linearizability.

​
s2-lite

s2-lite is an open source server implementation of core S2 APIs. To make for a simple self-hosting experience, it is a single binary expected to run as a single instance. Thanks to its SlateDB engine, you can enjoy the same guarantee as the s2.dev managed service — every write durable on object storage. See the README’s design section for more on s2-lite internals.