We can increase certain limits if user feedback suggests they are too restrictive. Let us know!

​
Records

  • Each record may be up to 1 MiB in metered bytes.
  • Batches may contain up to 1000 records, and cumulatively total up to 1 MiB in metered bytes.

S2 uses the following formula for metering record size:

metered_bytes = lambda record: 8 + 2 * len(record.headers) \
  + sum((len(h.key) + len(h.value)) for h in record.headers) \
  + len(record.body)

​
Throughput

Aggregate throughput per stream is rate limited as follows:

  • 125 MiBps to append records.
  • 500 MiBps to read records written within the last 20 seconds.

  • MiBps to read records older than the last 20 seconds.

Clients can optimize for throughput with TCP tuning. We recommend bbr as a congestion control algorithm over cubic.

If you are not in the same cloud region as your S2 basin, you can also consider increasing TCP send and receive buffer sizes.

​
Retention

Records may stay visible for a brief period after their retention window expires, typically on the order of minutes.

Temporary limit for preview period: maximum retention of 28 days.

​
Reconfiguration

Storage class changes (i.e. from Standard to Express, or vice versa) only take effect when a stream has been inactive for up to 10 minutes. This will become a live migration in the coming weeks.

The StreamSecurity