Drop-in S3-compatible storage gateway with GPU-accelerated transparent compression. Reduces S3 storage bytes 50–80% for compressible payloads (logs, JSON, Parquet/ORC) without changing application code. Total bill impact depends on workload mix — request cost / egress / GPU compute are unchanged.

Headline numbers (RTX 4070 Ti SUPER + Ryzen 9 9950X, single-pass roundtrip through s4-codec, last benchmarked 2026-05-13 on nvCOMP 5.2.0.10 / CUDA 13.2 driver 595.58.03; full table + reproduction recipe below):

Codec selection is not always GPU (#96 #97). The dispatcher samples entropy + magic bytes and routes per object:

• text / log → cpu-zstd-3 (often beats GPU codecs both on ratio AND throughput at the input size where everything fits in L3)
• columnar integers (Parquet / postings / time-series) → nvcomp-bitcomp (GPU's strength on integer/columnar layouts). Two modes:
  • explicit: --codec nvcomp-bitcomp always picks Bitcomp regardless of sample content
  • automatic: --prefer-columnar-gpu (opt-in) lets the sampling dispatcher detect a u32 / u64 LE integer column via per-stride byte-position entropy and route to Bitcomp once the body is >= --gpu-min-bytes. Default is off so v0.8.11-or-earlier deployments are bit-for-bit unchanged
• already-compressed (mp4 / jpeg / parquet-with-zstd-block-codec / .gz detected by magic byte) → passthrough (no harm done)
• non-GPU build OR no GPU at runtime → CPU codecs end-to-end

Observe per-codec request distribution via PromQL sum by (codec) (rate(s4_requests_total[5m])) (the codec label on s4_requests_total carries the chosen codec name) Prometheus counter, or per-PUT in the structured JSON access log ({"codec_chosen":"..."}). GPU is a multiplier on the integer/columnar side of mixed workloads, not a blanket "compress with GPU" claim.

Translated to AWS S3 Standard at $0.023/GB/month: 1 TiB of nginx log data → ~6.6 GiB stored → $0.15/month vs $23.55/month uncompressed (99% storage savings, single-pass). Mixed-content Parquet workloads see ~50% storage savings.

What this number does and doesn't cover (#95): storage-bytes only. PUT/GET request cost is unchanged (1 PUT in = 1 PUT out, plus a small .s4index sidecar PUT for indexed range-read). Egress is unchanged (GET serves the decompressed payload). GPU compute is a separate cost (c. EC2 g4dn / g5 hourly) — pays for itself on TB-scale, not GB-scale, ingest. See Cost savings — does S4 make sense for your bill? below for the break-even maths.

S4 (Squished S3) is an S3-compatible storage gateway written in Rust that sits between your applications (boto3 / aws-sdk / aws-cli / Spark / Trino / DuckDB / anything S3) and your real S3 bucket — and transparently compresses each object with a codec the dispatcher picks per-payload: GPU (NVIDIA nvCOMP zstd / Bitcomp / GDeflate) for integer/columnar data, CPU zstd / gzip for text/log, passthrough (no codec cost) for already-compressed inputs. See the codec verdict table above for the routing rules.

                        endpoint: s4.example.com
   your application ──────────────────────────▶  S4 (this project)
   (boto3, Spark,                                       │
    Trino, ...)                                         ▼
                                            (compress with GPU)
                                                        │
                                                        ▼
                                                 AWS S3 (real bucket)

• No app changes: same S3 wire protocol, same SigV4 auth, same SDK calls
• Transparent: PUT compresses, GET decompresses; clients see the original bytes
• Open format, no lock-in: stop the gateway and the compressed objects + S4IX sidecars remain S3-native — readable by stock aws-cli / boto3 / any S3 client. The original payload then requires s4-codec (CLI tool), s4-codec-py (pip), or s4-codec-wasm (browser) to decompress — all Apache-2.0, ~1k LOC of pure decode, no gateway runtime needed. The wire format (S4F2 frame + S4IX sidecar) is documented in the source: crates/s4-codec/src/multipart.rs (frame layout) and crates/s4-codec/src/index.rs (sidecar layout)

S4 ships under the Semantic Versioning contract as of v1.0.0. That means the items below are stable for the v1.x line — any incompatible change to them ships under a v2.0.0 release with migration guidance, not a v1.x patch.

s4-server ships 30 pub mod declarations from crates/s4-server/src/lib.rs so the s4 binary (which is a separate crate) + the integration tests + the example binaries can reach the surface they need. Five modules contribute frozen items above: repair, service, sse, streaming, and service_arc (the last contributes only SharedService; the rest of service_arc's contents are not frozen).

Library consumers MAY use s4_server::<other_module>::*; — Rust visibility allows it — but those imports are not frozen and may break in any v1.x minor release without notice. The other 25 modules (access_log, acme, audit_log, blob, cors, inventory, kms, lifecycle, lock_recovery, metrics, mfa, multipart_state, notifications, object_lock, policy, rate_limit, replication, routing, select, sigv4a, state_loader, streaming_checksum, tagging, tls, versioning) exist as pub mod for binary-and-tests' needs, not as a published surface.

If you depend on one of these unfrozen modules, pin a precise =1.x.y (rather than ^1) and treat any minor bump as a manual integration step. If you would like an item promoted to the frozen surface, please file an issue with the use case.

compat-matrix.yml runs a 1 PUT + 1 GET + sidecar HEAD round-trip per backend through a live s4-server, on a weekly schedule and via workflow_dispatch. CI-verified backends as of v1.0:

The compat-matrix job's start-step always gates (provisioning failure = workflow failure); the round-trip step is continue-on-error only for the two backends above. The status here is the source of truth — if a backend isn't in this table as ✓ gating, treat the README's other compat claims as "should work given S3 wire compatibility, not asserted by CI."

• Compression ratios + throughput numbers: these are workload-dependent and benchmark conditions are published, not promised SLAs.
• Default values for --max-body-bytes, --sse-chunk-size, --gpu-min-bytes, and similar runtime tunables: defaults may shift in a minor release if a clear correctness / safety reason warrants it (the v0.9 #106-32bit fix that clamped to isize::MAX on 32-bit is an example of a default the SemVer-stable contract did not protect).
• Implementation details inside frozen modules (private functions, struct field reordering, internal trait impls): the v1.0 freeze pins the items listed above, not "every line in service.rs". Re-arranging request-routing internals is fine in a minor.
• Backend behavior beyond S3-wire-spec compliance (e.g. how a specific backend handles a particular SigV4 edge case): we test the documented backends (see §"Backend compatibility matrix"), but breakage caused by a backend-side change is not a v2.0 trigger on our end.
• Experimental flags marked --allow-legacy-* or surfaced as unstable in --help: explicitly opt-in to behavior that may change.
• Cross-region replication and the replication.* config surface: shipped as experimental scaffolding in v0.6 with the wire path stubbed in but no production-grade reconciliation. Excluded from v1.0 freeze; promotion to first-class (with Jepsen-class consistency tests) is on the v1.x roadmap below.
• Security advisories accepted as risk-with-mitigation: see docs/security/cargo-audit-ignores.md for the 4 currently-ignored RUSTSEC advisories, each with rationale, mitigation, and upstream-tracking links. The ignore list is part of CI (cargo audit is a merge-block); changes to the list are visible in the diff.

• Chunked SSE-KMS envelope (provisional S4E7 magic) + chunked SSE-C (S4E8) → Range GET partial-fetch fast-path for SSE-KMS / SSE-C, parallel to the v0.9 #106 work that enabled it for SSE-S4 chunked (S4E6).
• S4F3 streaming frame format → enables streaming PUT checksum verify for multipart upload_part (= closes the codec-API constraint documented in docs/security/streaming-checksum-coverage.md).
• 32-bit s4-server runtime promotion from advisory to required CI smoke (currently advisory per v0.11 #A4).
• Per-action SHA pinning on the GHA workflows (supply chain hardening; v0.11 #A5 ended at the floating-major tag policy).
• Cross-region replication promoted from experimental scaffolding to production-grade, with Jepsen-style consistency tests.
• Re-introducing Garage + Ceph as ✓ gating in the backend compat matrix once the upstream signature-interop drifts are resolved.
• Additional codec backends (Snappy, LZ4 if user demand emerges).

• Adding a new codec / SSE envelope / sidecar version / CLI subcommand / lib function is additive = ships in a minor release. The v0.9 verify-sidecar subcommand + the v3 sidecar variant + the S4E6 chunked envelope are examples of minor-release additions.
• Changing the wire format of an existing magic (e.g. shrinking S4F2's header) is breaking = ships in a major release.
• Removing a CLI subcommand or a pub function is breaking = ships in a major release after a deprecation cycle.
• Default value drifts for runtime tunables — not breaking per the carve-out above, but always called out in CHANGELOG ### Changed.

The audit trail for what counts as breaking lives in CHANGELOG.md per the Keep a Changelog format S4 uses end-to-end. Migration recipes for any future v2.0 will live in docs/migration/<from>-to-<to>.md; no such file exists today because no breaking change is on the v1.x roadmap.

cargo install s4-server                                  # CPU build
s4 --endpoint-url https://s3.us-east-1.amazonaws.com     # binary is `s4`, not `s4-server`

Caveats (v0.8.8, #98):

• Requires Rust 1.92+ (rustup update stable first).
• The default cargo install builds CPU codecs only. GPU codecs (nvcomp-zstd / Bitcomp / GDeflate) require cargo install s4-server --features nvcomp-gpu, which needs the CUDA toolchain and NVCOMP_HOME pointing at an extracted nvCOMP SDK at build time. Without these the build fails at link time with an nvcomp lib not found error.
• The installed binary is s4 (not s4-server); check with which s4.

git clone https://github.com/abyo-software/s4 && cd s4
docker compose up -d                    # MinIO + S4 server on localhost:8014

# Generate a sample object so the cp lines have something to upload.
head -c 100M /dev/urandom | base64 > big.log    # ~135 MiB of text, compresses well

# Use any S3 client. Below uses aws-cli; replace endpoint with anything.
aws --endpoint-url http://localhost:8014 s3 mb s3://demo
aws --endpoint-url http://localhost:8014 s3 cp big.log s3://demo/big.log
aws --endpoint-url http://localhost:8014 s3 cp s3://demo/big.log ./big.log.roundtrip

# Inspect the compressed object directly on MinIO (different endpoint, bypasses S4).
aws --endpoint-url http://localhost:9000 s3 cp s3://demo/big.log ./big.log.compressed
ls -la big.log big.log.compressed big.log.roundtrip
# Expected: big.log == big.log.roundtrip (lossless), big.log.compressed is much smaller.

# Requires NVIDIA Container Toolkit + a CUDA-capable GPU
docker compose -f docker-compose.gpu.yml up -d
aws --endpoint-url http://localhost:8014 s3 cp parquet-file.parq s3://demo/

See docker-compose.gpu.yml for details.

Official container images are published to GitHub Container Registry on every v*.*.* release tag — ghcr.io/abyo-software/s4:<version> (CPU, multi-arch amd64 + arm64) and ghcr.io/abyo-software/s4:<version>-gpu (nvCOMP GPU build, amd64). The package is public; no imagePullSecrets needed.

helm install s4 ./charts/s4 \
  --set image.tag=0.9.0 \
  --set backend.endpointUrl=https://s3.us-east-1.amazonaws.com \
  --set backend.region=us-east-1
kubectl port-forward svc/s4 8014:8014

For the GPU image, override image.tag with the -gpu suffix and turn on GPU scheduling:

helm install s4 ./charts/s4 \
  --set image.tag=0.9.0-gpu \
  --set codec=nvcomp-zstd \
  --set gpu.enabled=true \
  --set backend.endpointUrl=https://s3.us-east-1.amazonaws.com

The chart in charts/s4/ ships a stateless Deployment + Service (ClusterIP, port 8014), optional GPU node selector (gpu.enabled=true for nvCOMP), inline or cert-manager TLS, and bucket-policy ConfigMap. See charts/s4/README.md for the full values table and .github/workflows/docker.yml for the image build / publish pipeline.

The published image + chart pair is exercised in CI on every push that touches the distribution surface (.github/workflows/docker-smoke.yml — v0.10 wave-2 #B2): helm lint + helm template against charts/s4 with a placeholder backend URL (catches values-schema / template regressions), docker compose config against both compose files (catches reference / image-tag drift), and docker pull + s4 --help / s4 --version against the latest published ghcr.io tag (tolerates the not-yet-published case via continue-on-error). Operators can reproduce the same checks locally before deploying:

# Helm chart sanity (with placeholder so backend.endpointUrl is satisfied)
helm lint ./charts/s4 --set backend.endpointUrl=https://s3.example.com
helm template s4 ./charts/s4 --set backend.endpointUrl=https://s3.example.com \
  | kubectl apply --dry-run=client -f -

# Compose file syntax + image-ref validation
docker compose -f docker-compose.yml config > /dev/null
docker compose -f docker-compose.gpu.yml config > /dev/null

# Image smoke (run this after a release lands on ghcr.io)
docker pull ghcr.io/abyo-software/s4:0.9.0
docker run --rm ghcr.io/abyo-software/s4:0.9.0 --help
docker run --rm ghcr.io/abyo-software/s4:0.9.0 --version

For ML / ETL pipelines that just want the codec without the gateway:

from s4_codec import CpuZstd, CpuGzip, gpu_available
codec = CpuZstd(level=3)
compressed, original_size, crc = codec.compress(data_bytes)
roundtrip = codec.decompress(compressed, original_size, crc)

PyO3 bindings live in crates/s4-codec-py/ — build with maturin build --release (and --features nvcomp-gpu for GPU).

For frontend apps that read S4-compressed objects directly from S3 over a presigned URL, no S4 server in the read path:

rustup target add wasm32-unknown-unknown
wasm-pack build --release --target web crates/s4-codec-wasm  # → pkg/

The bundle exports decompressFramed / decompressSingle for the CPU codec subset (passthrough, cpu-zstd, cpu-gzip). See crates/s4-codec-wasm/README.md for the API and a 10-line example.

cargo build --release --workspace                       # CPU-only
NVCOMP_HOME=/path/to/nvcomp cargo build --release --workspace --features s4-server/nvcomp-gpu

target/release/s4 --endpoint-url https://s3.us-east-1.amazonaws.com \
    --host 0.0.0.0 --port 8014 --codec cpu-zstd --log-format json

Runtime-tested platform is x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu (CI matrix). The 32-bit i686-unknown-linux-gnu target builds clean for s4-codec / s4-config / s4-server as of v0.9 #106 (default-bytes constants are now target_pointer_width cfg-gated so the 5 GiB AWS S3 single-PUT ceiling no longer const-overflows usize on 32-bit). v0.10 wave-2 #A4 adds a per-push CI job that (a) executes the s4-codec + s4-config test suites under --target i686-unknown-linux-gnu and (b) builds the s4 binary itself for i686 + invokes s4 --help / s4 --version as a runtime smoke. v0.11 #A4 extends the same job with an end-to-end PUT/GET round-trip — the i686 s4 binary runs in front of a stock MinIO container and the AWS CLI puts then gets a small object back through it, byte-equality-checked. The round-trip step lands in CI as advisory (continue-on-error: true) so a first-time 32-bit runtime bug surfaces in the job log without turning the badge red while a fix lands in a follow-up v0.11.x commit; promotion to a required gate happens once a stretch of green main pushes is observed. Operators running on i686 should still treat --max-body-bytes carefully (auto-clamps to isize::MAX as usize ≈ 2 GiB on 32-bit — Rust caps any single Vec / Bytes allocation at isize::MAX, so a higher gateway guard would let oversized requests panic inside the SSE buffered-decrypt pre-alloc path).

The wasm32-unknown-unknown target is the public release channel for the browser decoder (s4-codec-wasm); the criterion regression-tracking suite and cargo check --target wasm32-unknown-unknown keep it green on every CI push to main.

(MinIO / Garage license cells link to upstream LICENSE files; project licenses can change between releases. Do not treat as legal advice. See #103.)

S4 implements the parts of the S3 API needed to act as a transparent compression proxy in front of an existing bucket. It is not a complete S3 implementation — operations marked "—" return NotImplemented and should not be called against an S4 endpoint. PRs welcome on the matrix rows you need.

Range GET caveat (#99): the S4IX sidecar gives a per-frame index, so range maps to a contiguous read of the covering frames and a decode that's sliced at the boundaries the caller asked for. Parquet/ORC readers (arrow-rs, datafusion, duckdb's parquet reader) that issue suffix-range GET against the footer work out of the box. Parallel range reads against overlapping frame extents do extra decode work and are not yet optimized; see #99 for the parquet/ORC reader cross-validation harness on the roadmap.

Test status per major S3 client. "Tested" means a green E2E run in CI or documented manual verification; "Should work" means the wire shape is satisfied but no explicit test covers it yet; "Known issue" links to the relevant issue.

Endpoint URL style (#101): S4 accepts both virtual-hosted-style (https://my-bucket.s4.example.com/key) and path-style (https://s4.example.com/my-bucket/key); the backend aws-sdk-s3 client uses whatever the operator's --endpoint-url configuration specifies. If your client is fussy about this, set --path-style on the s4 server side or --force-path-style on the AWS SDK side.

S4 is a transparent compression proxy in front of an S3-compatible backend. Each row below is the verification posture S4 holds for that backend — what CI actually exercises, not "should work" claims. v0.11 #A7 added the weekly compat-matrix.yml workflow that drives the docker-tier verifications (and the real-cloud rows when operators provide credentials).

Each compat-matrix job runs a 1 PUT + 1 GET + sidecar HEAD against the live backend through an s4 --codec cpu-zstd --dispatcher always server — sidecar HEAD on the backend asserts the second backend round- trip (sidecar PUT) lands the way s4 expects, which is where most S3-API-shape divergences would surface (PutObject without Content-MD5, aws-chunked encoding, etc.).

S4 is a TLS-terminating S3-compatible proxy. The boundaries you should think about:

• Authentication scope: S4 verifies SigV4 / SigV4a on incoming requests using credentials operators configure (--credentials FILE or --sigv4a-credentials DIR). The S4 server then turns around and speaks to the backend bucket using its own AWS credentials (AWS_ACCESS_KEY_ID etc. from the standard SDK chain). Client identity is not delegated to the backend; the backend sees S4 as one principal regardless of which incoming client made the request. If you need per-client backend identity, run one S4 instance per client and use distinct backend credentials.
• TLS termination: S4 terminates TLS at its own listener (--tls-cert / --tls-key, or ACME via --acme). The connection to the backend uses the SDK's own TLS (rustls with the system root CA store). If your security model requires end-to-end TLS without intermediate decryption, S4 is the wrong shape — use a different proxy or run S4 colocated with the backend so the second TLS hop doesn't leave the same host.
• Bucket policy enforcement at the S4 layer: when --bucket-policy FILE is set, S4 evaluates AWS-style JSON Allow / Deny rules before forwarding to the backend. The backend's own bucket policy still applies on top. Two policies in series; both must permit. We do not parse every IAM Condition operator — see crates/s4-server/src/policy.rs for the supported subset.
• Body-size limits / request smuggling: hyper limits enforced (--max-header-bytes, default 64 KiB; --max-concurrent-connections, default 1024; --read-timeout-seconds, default 30s — see v0.8.5 #84). HTTP/2 is off by default (--http2 to opt in); the S3 API is HTTP/1.1 in practice and h2 adds DoS surface (stream-multiplexing abuse) that doesn't pay off for our workload.
• Tenant isolation: S4 is single-tenant by design — one S4 instance per security boundary. We do not enforce cross-bucket isolation at the S4 layer beyond what the backend's IAM enforces. Multi-tenant deployments should run one S4 instance per tenant with separate backend credentials.
• Non-goals: S4 is not an IDS / WAF, does not log request bodies (only headers + length), does not implement S3's ObjectACL Grant-by-CanonicalUser semantics beyond canned ACLs, does not proxy IAM API calls.

For incident reporting see SECURITY.md.

┌──────────────────────────────────────────────────────────────────┐
│                          S4 server                               │
│  ┌──────────────────┐  ┌─────────────────┐  ┌────────────────┐   │
│  │ s3s framework    │→ │ S4Service       │→ │ s3s_aws::Proxy │ → │ → backend (AWS S3 / MinIO)
│  │ (HTTP + SigV4)   │  │ (compress hook) │  │ (aws-sdk-s3)   │   │
│  └──────────────────┘  └────────┬────────┘  └────────────────┘   │
│                                 ▼                                │
│  ┌─────────────────────────────────────────────────────────┐     │
│  │ s4-codec::CodecRegistry  (multi-codec dispatch by id)   │     │
│  │   ├─ Passthrough          (no compression)              │     │
│  │   ├─ CpuZstd              (zstd-rs, streaming)          │     │
│  │   ├─ NvcompZstd           (nvCOMP, GPU, per-chunk)      │     │
│  │   ├─ NvcompBitcomp        (nvCOMP, integer columns)     │     │
│  │   └─ NvcompGDeflate       (nvCOMP, DEFLATE-family GPU)  │     │
│  └─────────────────────────────────────────────────────────┘     │
│  ┌─────────────────────────────────────────────────────────┐     │
│  │ s4-codec::CodecDispatcher                               │     │
│  │   ├─ AlwaysDispatcher                                   │     │
│  │   └─ SamplingDispatcher  (entropy + 12 magic bytes)     │     │
│  └─────────────────────────────────────────────────────────┘     │
└──────────────────────────────────────────────────────────────────┘
        ▲              ▲              ▲                ▲
        │              │              │                │
   /health         /ready         /metrics         OTLP traces
   (probe)        (probe)       (Prometheus)       (Jaeger / X-Ray)

Single-pass roundtrip through s4-codec. Hardware: RTX 4070 Ti SUPER 16 GB

• nvCOMP 5.2.0.10 + CUDA 13.2 driver 595.58.03 + Ryzen 9 9950X. Throughput is reported as uncompressed bytes per second (the convention nvCOMP / lz4 / zstd publish). Last benchmarked 2026-05-13 (v0.8 #53, crates/s4-codec/examples/bench_codecs.rs).

v0.3 → v0.8 throughput delta (compress GB/s on the same hardware, nvCOMP 5.0.x → 5.2.0.10, no source-code changes — pure runtime / driver gains):

Reading the table:

• cpu-zstd-3 dominates on text — 155× on nginx logs is hard to beat.
• nvcomp-bitcomp is the killer for typed numeric columns: 11.93× on sorted u32 posting lists (vs ~1.5× for everything else), 2.95× on monotonic i64 timestamps. The data_type hint is critical (Char on numeric data degrades to ~1.2×); see [s4_codec::nvcomp::BitcompDataType] for the typed constructors.
• nvcomp-zstd is competitive on Parquet-like / mixed workloads and frees the CPU for serving requests in parallel.
• nvcomp-gdeflate sits between zstd and "no compression" — useful when you need DEFLATE-format wire compat (in v0.3 the gunzip-compatible wrapper will make this codec serve Content-Encoding: gzip to any HTTP client).
• Already-compressed inputs are correctly bypassed at ratio 1.0× by every codec — S4 never makes a file bigger.

Throughput note: nvCOMP runs through the FCG1-framed batched API at the default 64 KiB chunk size, so per-call overhead dominates the 64 MiB input cases. Production deployments using larger chunks via streaming_compress_to_frames (v0.2 #1) push GPU compress >5 GB/s on highly compressible inputs. The full head-to-head bench vs MinIO S2 / Garage zstd is tracked in issue #14; the latest CSV captured on 2026-05-13 lives at benches/comparison/result-2026-05-13.csv (MinIO + s4-cpu only; Garage's auto-issued keys and the s4-gpu image require manual setup outside the driver script).

Multipart streaming note (v0.2 #1, surfaced again by the v0.8 #53 comparison run): per-part S4F2 framing (4 MiB chunks) means a 64 MiB nginx-log multipart upload reports ~1.6× ratio at the storage layer instead of the 155× single-pass ratio above — each chunk is too small for zstd's longest-match window to amortize across the whole object. Ratio scales back to single-pass numbers once cargo install users configure larger multipart chunk sizes via the AWS SDK multipart_chunksize knob (S4 itself stays at the 4 MiB default for Range-GET granularity). The CSV captures end-to-end PUT/GET wall-clock including framing overhead.

The single-pass numbers above are captured manually on the maintainer's workstation; for per-commit regression detection S4 also runs a criterion bench suite on every push to main (.github/workflows/bench.yml), stores the timing history in the gh-pages branch via benchmark-action/github-action-benchmark, and comments on a commit when any tracked target gets ≥ 1.1× slower than its previous best. The targets cover the CPU hot paths every default-build deployment runs through:

• crates/s4-codec/benches/codec_roundtrip.rs — cpu-zstd (levels 1 / 3 / 22) / cpu-gzip / passthrough compress + decompress at 1 KiB / 1 MiB / 16 MiB.
• crates/s4-codec/benches/frame_codec.rs — write_frame and the FrameIter walker, with the padding-skip branch exercised.
• crates/s4-codec/benches/index_codec.rs — S4IX sidecar encode_index / decode_index / lookup_range across 128 / 1024 / 4096 frame counts.

GPU codecs (nvcomp-*) are intentionally not in the regression suite because GitHub-hosted runners have no CUDA-capable GPU; the manual table above remains the canonical source for those numbers.

The rendered trend chart lives at https://abyo-software.github.io/s4/dev/bench/ after the first successful CI run on main initialises the gh-pages branch.

S4's server-side encryption (--sse-s4-key) goes through the aes-gcm crate, which selects the AES-NI hardware path automatically on x86_64 hosts where the aes + pclmulqdq CPU features are present. v0.8 #50 adds (a) a boot log line confirming which backend is live, (b) a s4_sse_aes_backend{kind="aes-ni"|"neon"|"software"} Prometheus gauge stamped at startup, and (c) the bench_sse_throughput example below that measures the resulting encrypt / decrypt throughput.

Numbers below are from the same Ryzen 9 9950X host as the codec table. Reproduce with cargo run --release -p s4-server --example bench_sse_throughput (AES-NI is the default; force the software backend with RUSTFLAGS="--cfg aes_force_soft --cfg polyval_force_soft" and a clean target dir).

AES-NI delivers ~8.7× throughput on 64 KiB / 1 MiB bodies (the regime that dominates real S3 object traffic). The 100 MiB row's narrower gap (~5.2×) is the buffer allocator + page-fault floor — aes-gcm uses a single contiguous Vec for the ciphertext, so 100 MiB cases charge a mmap per iteration that's not on the AES path. Operators running on hosts without AES-NI (very old / virtualized x86 or non-x86 hardware) should expect ~190 MB/s encrypt / decrypt as the sustained ceiling for SSE-S4 — still ahead of the network for most deployments, but worth knowing when sizing CPU headroom.

Detecting which backend is live: the boot log emits S4 AES-NI feature detection ... aes_ni_available=true (or false), and curl -s localhost:9100/metrics | grep s4_sse_aes_backend shows the gauge with the active kind label.

Reproducing locally (requires CUDA + nvCOMP):

NVCOMP_HOME=/opt/nvcomp LD_LIBRARY_PATH=/opt/nvcomp/lib \
  cargo run --release --example bench_codecs \
    -p s4-codec --features nvcomp-gpu

# Streaming pipeline bench (1 GiB highly-compressible, in-flight chunks):
NVCOMP_HOME=/opt/nvcomp LD_LIBRARY_PATH=/opt/nvcomp/lib \
  cargo run --release --example bench_pipeline \
    -p s4-server --features nvcomp-gpu

# Comparison vs MinIO / Garage (Docker required):
docker compose -f benches/comparison/docker-compose.yml up -d
AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
  ./benches/comparison/run.sh benches/comparison/result-$(date +%F).csv

S4 is not worth deploying for everyone. The economics depend on (a) your AWS S3 bill, (b) how compressible your data is, (c) the cost of the EC2 GPU instance running S4. Here's an honest table to self-diagnose:

Notes:

• "Likely savings 50–80%" is the typical range for log-heavy workloads (cpu-zstd-3 155×) and Parquet (nvcomp-zstd ~2× plus better Range GET efficiency). For pure-numeric column-store data with nvcomp-bitcomp on sorted posting lists, the ratio swings to >10× — savings closer to 90%+.
• EC2 prices are us-east-1 on-demand, May 2026. Spot instances cut these by ~70%, breakeven at ~$300/mo S3 bill instead of $1,000.
• S4 itself is open source (Apache-2.0) — the only cost is the EC2 instance and your time.
• If your monthly S3 bill is under $1,000 and you're not already running GPUs for other work, don't bother. Use S4's cpu-zstd codec on a small CPU instance, or front your bucket with nginx + gzip — both will give most of the savings without GPU hardware.

Honest list of workloads where S4 doesn't pay off:

• Already-compressed payloads (mp4, jpeg, gzip-of-anything, parquet with column-level codec already on, lz4 / zstd-prepacked archives) — S4's dispatcher detects + routes to passthrough so there's no harm done, but you're paying for the round-trip without getting savings.
• Small objects (< 16 KiB) — the S4F2 frame header (28 bytes) + S4IX sidecar (32–96 bytes per object) eats the compression ratio before you start. Break-even is workload-dependent; rule of thumb is objects > 1 MiB make the math comfortable, < 16 KiB make it negative. The dispatcher does not yet skip-compress small objects automatically (#105 follow-up).
• Metadata-ops dominant workloads — heavy ListObjects / HeadObject / CopyObject against millions of small keys add S4 hop latency without touching the codec. S4 is on-path for those, so you pay the second TLS hop + s3s framework overhead.
• Ultra-low-latency tail SLOs (sub-10ms p99 GET) — S4's streaming GET adds decoder warm-up + S4IX sidecar fetch (one extra round-trip for the index when not cached). Fine for analytics / archival / bulk; not fine for an OLTP-style hot read path.
• Single-region cold-storage-only (everything goes straight to Glacier) — Glacier already prices low enough that the storage savings rarely pay for the compute / operational cost of running S4.
• Strict regulatory environments without third-party audit on file — v1.0 freezes the wire + API surface, but S4 has no SOC2 / ISO27001 / FedRAMP audit trail yet. If your compliance team's bar is "must have third-party audit on file", S4 isn't there.
• As the only copy of irreplaceable data, before a production reference is on file — until at least one public production deployment reference lands (we're collecting them under issue label production-reference), pair S4 with backend-native versioning + replication. The v1.0 freeze is a contract on surface stability, not a substitute for the operational track record that a reference deployment provides.

A PUT goes through three S3 calls behind one client-visible request:

1. PUT <key> — the compressed S4F2-framed body (atomic single-PUT for objects under the multipart threshold; otherwise an S3 multipart upload with per-part frames).
2. PUT <key>.s4index — the S4IX sidecar with per-frame offset + original-size + crc32c entries.
3. (multipart only) CompleteMultipartUpload — finalises the main object atomically; the sidecar is written after this completes.

The main object PUT is the commit point; the sidecar exists to optimise Range GET and is treated as recoverable / rebuildable from the main object (next section).

crc32c is computed over the decompressed original payload of each frame and stored in both the frame header and the sidecar entry. This catches:

• Mid-flight corruption at the backend storage layer
• Codec backend bugs that decode to subtly wrong bytes
• Forged manifest attacks where the attacker replaces the compressed body

It does not catch:

• A correctly-encoded malicious payload from a tampered backend (the CRC verifies the bytes match what was encoded, not that what was encoded was the originally-PUT bytes) — that's what S4's SigV4 auth on the PUT side covers
• Lost frames from a truncated multipart that nonetheless committed (the per-part Complete API itself is the integrity check there)

v0.9 #106 shipped three sidecar-maintenance subcommands on the s4 binary. All three point at the backend (not the S4 gateway) — the gateway hides .s4index from listings and decompresses bodies on GET, both of which break this tooling:

# Read-only check. Exits 0 on Ok / LegacyV1 / MissingHarmless
# (single-frame object, no sidecar by design) / MissingUnknown (body
# exceeds the deep-scan cap, can't classify); exits 1 on
# MissingDivergent / StaleEtag / StaleSize / DecodeError /
# EncryptedSidecarUnsupported (SSE-S4 chunked, see follow-up below).
s4 verify-sidecar bucket/key --endpoint-url https://s3.example.com

# Re-scan the main object and overwrite the sidecar. Default body cap
# is 5 GiB (matches --max-body-bytes); pass --max-body-bytes to raise.
# Does NOT yet support SSE-S4 chunked encrypted objects from the CLI
# (operator needs the SSE keyring; v0.10 roadmap is to plumb
# `--sse-s4-key <path>` through). Until then, re-PUT the object via
# the v0.9+ gateway to regenerate the v3 sidecar.
s4 repair-sidecar bucket/key --endpoint-url https://s3.example.com

# Find dangling `.s4index` whose pair is missing or stale. Dry-run by
# default; --delete actually removes them. The default --delete only
# removes pair-bound orphans (PairedMissing / PairedEtagMismatch /
# PairedSizeMismatch); SidecarUndecodable entries stay until you
# escalate with --delete-undecodable (guards against deleting legacy
# reserved-name user data under --allow-legacy-reserved-key-reads).
s4 sweep-orphan-sidecars bucket --endpoint-url https://s3.example.com [--delete] [--delete-undecodable]

The manual fallback (DELETE the sidecar — Range GET drops to the full-read path) still works for one-offs without the CLI handy. See docs/orphan-sidecar-recovery.md for the v0.8.15 H-g cleanup recipe using s4 sweep-orphan-sidecars.

Measurement conditions for the numbers below (#107): RTX 4070 Ti SUPER + Ryzen 9 9950X, single-pass 256 MiB compressible input, codec cpu-zstd-3 (or as noted), single concurrent request, S4 colocated with backend (no network RTT to amortise). TTFB excludes TLS handshake

• SigV4 verification (those add 5–15 ms once per connection).

• Streaming GET for non-multipart cpu-zstd / passthrough objects: TTFB 8–20 ms under the conditions above, memory ≈ zstd window (8 MiB at level 3) + 64 KiB buffer
• Streaming PUT for the same codecs: input never fully buffered, peak memory ≈ compressed size (5 GB → ~50 MB at 100× ratio). Client-supplied whole-body checksums (Content-MD5, x-amz-checksum-{crc32, crc32c, sha1, sha256, crc64nvme}) are verified in-stream via a tee-into-hasher wrapper (v0.9 #106): mismatched bytes surface as 400 BadDigest without buffering the body. GPU codecs and multipart UploadPart keep the buffered per-body / per-part verify path (the bytes are already in memory there for framing / padding) — see docs/security/streaming-checksum-coverage.md for the full coverage matrix and the codec-API constraint that makes this a fundamental property of those branches, not deferred plumbing
• GPU streaming compress (v0.2): nvCOMP zstd / gdeflate PUTs run a per-chunk pipeline so a 10 GB highly-compressible upload peaks at ~210 MB host RAM instead of buffering the full input
• Single-PUT framed format unification (v0.2): every compressed PUT now uses the same S4F2 multi-frame format multipart uploads use, with an optional <key>.s4index sidecar. Range GET partial-fetch optimisation applies to single-PUT objects too, not just multipart
• Multipart per-part compression: each part compressed and frame-encoded (S4F2 magic), per-frame codec dispatch (mixed codecs in one object)
• Multipart final-part padding trim (v0.2): the final part of a multipart with a tiny highly-compressible tail skips S4P1 padding (saves up to ~5 MiB per object on highly compressible workloads)
• Range GET via sidecar <key>.s4index: only the needed compressed bytes are fetched from backend, decoded, and sliced. Falls back to full read when sidecar is absent
• Encryption-aware Range GET fast-path (v0.9 #106): SSE-S4 chunked (--sse-chunk-size > 0, S4E6 frame) Range GETs now partial-fetch just the enclosing S4E6 chunks from backend instead of pulling the full encrypted body. The v3 <key>.s4index sidecar carries the per-PUT salt + chunk geometry so the GET path can compute the encrypted byte range without re-fetching the header. SSE-KMS / SSE-C / SSE-S4 buffered (--sse-chunk-size 0) keep the v0.8.12 #120 buffered fallback (= full decrypt → frame-parse → slice); covering them needs separate plumbing (KMS DEK envelope shape, customer-key per-request material) and is on the v0.10+ roadmap
• Byte-range aware upload_part_copy (v0.2): when the source is S4-framed, the user-visible byte range is what gets copied (decompressed and re-framed), not raw compressed bytes

S4 supports four SSE modes (table below). The Range GET fast-path introduced in v0.9 #106 partial-fetches only the enclosing encrypted chunks for a given byte range instead of pulling the full body — but it only works for SSE-S4 chunked (--sse-chunk-size > 0, S4E6 wire envelope). The other three modes fall back to the v0.8.12 #120 buffered path (full decrypt → frame-parse → slice).

Why only chunked SSE-S4? Non-chunked envelopes (S4E2 / S4E3 / S4E4) wrap the entire body under one AES-256-GCM authentication tag. AEAD decrypt is only defined over the full ciphertext + AAD + tag quadruple — there is no "verify just the prefix" mode — so partial plaintext cannot be exposed without fetching and tag-verifying the whole body. This is the AEAD security contract, not an optimization deferment. The S4E6 chunked envelope (v0.8 #52, refined in v0.8.1 #57) explicitly slices the plaintext into fixed-size chunks and emits one tag per chunk with a nonce derived from a per-PUT salt + chunk index, which is what makes chunk-aligned partial decrypt well-defined. Full per-mode walkthrough lives in docs/security/sse-partial-fetch-constraint.md.

Operator recommendation: for Range-GET-heavy workloads on large objects (parquet / ORC footer reads, video segment seeks, log-line slice reads) where SSE is required, scope your data to SSE-S4 chunked to keep the fast-path. The 1 MiB default chunk size matches the typical parquet row-group read pattern; smaller chunks give finer-grained partial fetch at higher tag overhead, larger chunks reduce on-disk tag bytes but do more wasted decrypt per Range GET.

s4-server \
  --sse-s4-key /etc/s4/sse.key \
  --sse-chunk-size 1048576 \
  ...

If SSE-KMS or SSE-C is required by your key-management posture, either accept the buffered Range GET cost or restructure the data into smaller objects so the buffered fetch is bounded. Chunked-KMS (provisional S4E7) and chunked-SSE-C (provisional S4E8) envelopes are v0.11+ roadmap candidates, not promised features.

• /health — liveness probe, always 200 OK
• /ready — readiness probe, runs ListBuckets against the backend
• /metrics — Prometheus text format (s4_requests_total{op,codec,result}, s4_bytes_in_total, s4_bytes_out_total, s4_request_latency_seconds, s4_policy_denials_total{action,bucket})
• Structured JSON logs (--log-format json) with per-request fields: op, bucket, key, codec, bytes_in, bytes_out, ratio, latency_ms, ok
• OpenTelemetry traces (--otlp-endpoint http://collector:4317) — each PUT/GET emitted as s4.put_object / s4.get_object span with semantic attributes; export to Jaeger / Tempo / Grafana / AWS X-Ray.

• Native HTTPS / TLS (v0.2) — --tls-cert / --tls-key for direct termination via tokio-rustls + ring, ALPN advertises h2 then http/1.1. No reverse-proxy required for HTTPS deployments.
• Bucket policy enforcement at the gateway (v0.2) — --policy <path> accepts an AWS-style bucket policy JSON; every PUT / GET / DELETE / List / Copy / UploadPartCopy is evaluated with explicit Deny > explicit Allow > implicit Deny semantics (matches AWS). Subset: Effect, Action (e.g. s3:GetObject / s3:*), Resource with glob, Principal (SigV4 access-key match). Denials are bumped on s4_policy_denials_total{action,bucket}.

• CRC32C stored per-object (single PUT) or per-frame (multipart), verified on GET
• copy_object S4-aware: source's s4-* metadata is preserved across MetadataDirective: REPLACE (prevents silent corruption of the destination)
• Zstd decompression bomb hardening: Decoder + take(manifest.original_size + 1024) caps the decode at the manifest's declared size (+ a small overshoot margin) so a zero-size manifest paired with a high-ratio frame surfaces as a typed Io("bomb detected") instead of unbounded RAM growth. The cap is still bound by the manifest claim itself — a 5 GiB manifest is honored up to 5 GiB, so operators must additionally enforce a per-request memory ceiling at the listener (--max-body-bytes / a future per-frame cap) for adversarial uploads

• Each compressed object is stored as <key> + <key>.s4index sidecar. S3 lifecycle rules must move both files together — a split pair breaks Range GET (sidecar in IA + main in Glacier ⇒ InvalidObjectState).
• Recommended: "Filter": {} (whole bucket) or a Filter.Prefix rule that covers both foo/... and foo/....s4index. Avoid size- or suffix-scoped filters that catch one but not the other.
• See docs/storage-class-transitions.md for two example lifecycle JSONs (IA-after-30d and prefix→Glacier-after-60d), the anti-pattern walkthrough, and a head-object drift-audit recipe.

• Compression hook: put_object, get_object, upload_part
• Range GET: full S3 spec (bytes=N-M, bytes=-N, bytes=N-)
• Multipart: create_multipart_upload, upload_part, complete_multipart_upload, abort_multipart_upload, list_parts, list_multipart_uploads
• Phase 2 delegations (passthrough): ACL, Tagging, Lifecycle, Versioning, Replication, CORS, Encryption, Logging, Notification, Website, Object Lock, Public Access Block, ...
• Hidden: *.s4index sidecars are filtered from list_objects[_v2] responses

125 default tests + 15 ignored (Docker / GPU / AWS env required) = 140 tests, plus PROPTEST_CASES=10000 stress run on every push (~73 sec, 380K fuzz cases), 1M cases × 38 properties nightly (~6 h, 38M+ fuzz cases).

Two real bugs already caught by fuzz infrastructure:

1. FrameIter infinite-loop on 1-byte input (DoS) — fixed with fused: bool
2. cpu_zstd::decompress could OOM on attacker-controlled manifest claim — fixed with Decoder + take(limit)

cargo test --workspace                   # default
cargo test --workspace -- --ignored --test-threads=1   # E2E (Docker required)
PROPTEST_CASES=100000 cargo test --workspace --release --test fuzz_parsers --test fuzz_server --test fuzz_advanced
NVCOMP_HOME=... cargo test --workspace --features s4-server/nvcomp-gpu -- --ignored
./scripts/soak/run.sh                    # 24 h soak (Marketplace pre-release)

AWS credentials are read from the standard AWS chain (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_PROFILE / IAM role on EC2).

S4 can terminate TLS itself — no fronting reverse proxy required:

s4 --endpoint-url https://s3.us-east-1.amazonaws.com \
   --host 0.0.0.0 --port 8443 \
   --tls-cert /etc/ssl/s4.crt --tls-key /etc/ssl/s4.key
aws --endpoint-url https://localhost:8443 s3 ls

Backed by tokio-rustls + ring. ALPN advertises h2 then http/1.1, so HTTP/2 is negotiated automatically with capable clients. Without these flags, S4 serves plain HTTP (the default).

Cert hot-reload (v0.3): rotate --tls-cert / --tls-key files on disk and kill -HUP <pid> to swap the active cert without dropping any in-flight connections. Re-read failures keep the previous cert in effect so a bad deploy never causes a listener outage.

ACME / Let's Encrypt (v0.3): for public deployments, fetch and renew certs automatically with --acme:

s4 --endpoint-url https://s3.us-east-1.amazonaws.com \
   --host 0.0.0.0 --port 443 \
   --acme s4.example.com,api.example.com \
   --acme-contact ops@example.com \
   --acme-staging   # remove for production after first-run validation

Uses TLS-ALPN-01 challenge handled inline on the listening port — no separate port-80 listener required. Background renewal at the standard ~60-day interval; s4_acme_renewal_total{result} Prometheus counter

• s4_acme_cert_expiry_seconds gauge for monitoring.

Pass an AWS-style bucket policy JSON to --policy to gate requests at the gateway:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Sid": "ReadOnly",  "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"],
     "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]},
    {"Sid": "DenyDelete", "Effect": "Deny",  "Action": "s3:DeleteObject",
     "Resource": "arn:aws:s3:::my-bucket/*"}
  ]
}

Supported subset (v0.3):

• Effect / Action / Resource / Principal (v0.2 baseline): SigV4 access-key match for Principal, glob matching for Resource.
• Condition clauses (v0.3 #13): IpAddress / NotIpAddress (CIDR), StringEquals / StringNotEquals / StringLike / StringNotLike, DateGreaterThan / DateLessThan, Bool. Supports the well-known AWS context keys aws:SourceIp (taken from the X-Forwarded-For header — set this at your reverse proxy / load balancer), aws:UserAgent, aws:CurrentTime, aws:SecureTransport (true when the listener is --tls-cert or --acme).

Decision order is the standard AWS one: explicit Deny > explicit Allow > implicit Deny. Conditions are AND-combined within a Statement. Denials are exposed as the s4_policy_denials_total{action,bucket} Prometheus counter.

For STS / AssumeRole chains and cross-account delegation (still out of scope), front S4 with an IAM-aware proxy and use this flag for the in-gateway last-mile checks.

S4 stores data as either:

S3 metadata holds the manifest:

x-amz-meta-s4-codec:           passthrough | cpu-zstd | nvcomp-zstd | ...
x-amz-meta-s4-original-size:   <decoded bytes>
x-amz-meta-s4-compressed-size: <stored bytes, includes S4F2 framing>
x-amz-meta-s4-crc32c:          <CRC32C of original bytes>

Since v0.2 #4 the body is the same S4F2 framed format multipart uploads use (one frame per DEFAULT_S4F2_CHUNK_SIZE = 4 MiB chunk). Small objects (< 4 MiB) produce a single S4F2 frame and pay a constant +28 byte wire overhead vs the raw compressed bytes — see footnote ¹.

x-amz-meta-s4-multipart: true
x-amz-meta-s4-codec:     <default codec for the object>

Object body is a sequence of:

┌──────────── 28-byte frame header ────────────┐
│ "S4F2" │ codec_id u32 │ orig u64 │ comp u64  │ crc32c u32 │  payload (comp bytes)
└────────────────────────────────────────────────┘

(optional) ┌──── padding ────┐
           │ "S4P1" │ len u64 │ <len zero bytes>
           └─────────────────┘

A sidecar object <key>.s4index (binary, S4IX magic) maps decompressed byte ranges to compressed byte offsets — used by Range GET to fetch only the needed bytes from S3.

Status: v1.0 — stable surface, no public production deployment reference yet. v1.0 is the SemVer-stable freeze of the wire formats, library API surface, CLI subcommands, s3s 0.13 HTTP trait set, and Helm values.yaml key shape enumerated in the §"Stability" section above. It is not a marketing claim that "S4 has been battle-tested at every Fortune 500." The freeze means downstream consumers can pin s4-server = "1" (or ghcr.io/abyo-software/s4:1, or s4 = "1" in a Cargo.toml) and rely on the surface not changing under them; first public production deployment references are still being collected. If you're putting S4 into a TB-scale workload, please file an issue tagged production-reference so we can list your deployment alongside the audit + fuzz evidence below.

• Release line: CHANGELOG.md has the full per-version history; the GitHub Releases page has the cut-points. Cumulative scope through v1.0 is 714+ workspace tests + 14+ production milestones covering S3-compatible PUT / GET / multipart / Select / SSE-S3 / SSE-KMS / SSE-C / IAM Conditions / bucket policy / versioning / object-lock / lifecycle / inventory / notifications (Webhook / SQS / SNS) / CORS / tagging / MFA delete / SigV4 + SigV4a, plus Python (s4-codec-py) and browser (s4-codec-wasm) bindings, all on crates.io as the s4-server / s4-codec / s4-config trio. Cross-region replication ships as experimental scaffolding (config surface + wire stub) and is intentionally excluded from the v1.0 freeze — promotion to production-grade is on the v1.x roadmap.
• Audit history: three rounds of deep audit (第一弾 / 第二弾 / 第三弾) closed in v0.8.2 → v0.8.5; pre-launch audit (claude + codex cross-review, tracker #111) in v0.8.7 → v0.8.8; integrated audit rounds R1–R6 across v0.9 / v0.10 / v0.11 cuts; v1.0 readiness audit (Opus + Codex adversarial review) drove 13 surfaced findings to closure — including the v1.0 stability section in this README, the #[non_exhaustive] annotations on every public enum, gating test-only helpers out of the public API contract, and qualifying the backend compatibility matrix above. Findings spanned CRITICAL pre-auth state-machine bugs, HTTP wire hardening, GPU codec safety, binding correctness, background-task lifecycle, README claim accuracy, and v1.0 freeze surface completeness. CVE clean (cargo audit, see CI security-audit job); 4 advisories accepted as risk-with-mitigation per docs/security/cargo-audit-ignores.md.
• Continuous fuzz farm (v0.8.6) — 5 bolero targets running 24/7 under a systemd-user slice budgeted at 8 cores / 30 GiB (1/4 of the build host). Coverage compounds across Restart=always wakeups; any crash auto-files a GitHub issue (label fuzz-crash, deduped by SHA1 of the input). First catch: #89 (CpuZstd / CpuGzip alloc-before-validate) found within seconds, fixed and shipped same-day in v0.8.6.
• Real-GPU validation done on RTX 4070 Ti SUPER + nvCOMP 5.x: streaming zstd 1 GiB roundtrip + GDeflate roundtrip both green; OMB bench runs on EC2 c7gd.8xlarge (latest v0.8 perf chart at docs/perf-v0.8.png).
• Suitable for log archival, data lake / parquet/ORC analytics, drop-in transparent-compression proxy in front of any S3-compatible backend. The v1.0 surface freeze means you can integrate against a stable contract; the "no public production reference yet" caveat means we still recommend pairing with backend-native replication / versioning for irreplaceable data until at least one production reference is published.
• Roadmap is driven by audit findings + continuous fuzz rather than feature checklists; file issues at https://github.com/abyo-software/s4/issues to influence it.

Pull requests are welcome! See CONTRIBUTING.md for the development setup, coding conventions, and the test/fuzz/soak protocol.

By contributing, you agree your contributions will be licensed under Apache-2.0 (no separate CLA required).

Found a vulnerability? Please do not open a public issue. Instead, follow SECURITY.md for coordinated disclosure.

Licensed under the Apache License, Version 2.0 (LICENSE). See NOTICE for third-party attributions including the vendored ferro-compress (Apache-2.0 OR MIT) and the optional NVIDIA nvCOMP SDK (proprietary, BYO).

Full third-party license disclosure (auto-generated from cargo about across the three target triples we ship to — x86_64-unknown-linux-gnu / aarch64-unknown-linux-gnu / wasm32-unknown-unknown) lives at docs/THIRD_PARTY_LICENSES.html. ~350 transitive crates, all permissive (Apache-2.0 / MIT / BSD-{2,3}-Clause / ISC / Zlib / Unicode / 0BSD / MPL-2.0 / OpenSSL / CDLA-Permissive-2.0). Regenerate with cargo about generate about.hbs --output-file docs/THIRD_PARTY_LICENSES.html (requires cargo install cargo-about --features cli).

The optional nvcomp-gpu feature pulls the proprietary NVIDIA nvCOMP SDK at build time. nvCOMP is not bundled with S4 distributions; operators set NVCOMP_HOME to a locally extracted SDK from the NVIDIA Developer Zone. nvCOMP redistribution is subject to NVIDIA's SLA — confirm with NVIDIA in writing before bundling into a downstream AMI / container image.

"S4" and "Squished S3" are unregistered trademarks of abyo software 合同会社. "Amazon S3" and "AWS" are trademarks of Amazon.com, Inc. S4 is not affiliated with, endorsed by, or sponsored by Amazon.

• abyo software 合同会社 — sponsoring organization, commercial AMI distribution
• masumi-ryugo — original author / maintainer