How Arcois Gateway works — architecture, write path and volume format

One rule

Clients address the gateway and never a backend. The blobs a backend receives are written once and never modified in place. All S3 meaning lives in the gateway's metadata plane.

S3 meaning is everything a client can observe about an object apart from its bytes: what it is called, which version is current, what a bucket contains, the ETag that identifies the stored content, and the metadata the client attached. A backend holds none of it. On the request path a backend is asked to store a blob, and later to return a byte range from it.

Aggregation, mobility and tenancy are not three features bolted together. They are three consequences of that single separation.

What the rule buys you

Listings get fast and stay fast: A listing is a range scan over the gateway's own ordered rows. No backend is ever asked to enumerate anything.
Your stores become interchangeable: A backend that holds only bytes can be swapped, mirrored or added to without your applications knowing.
Metadata can be rebuilt from the data: Every volume carries a list of what is inside it, so the object records can be repopulated from the volumes themselves.

Three tiers

The gateway scales by adding pods. Pods coordinate through leases taken in the metadata plane, not through an elected leader. Every pod therefore runs the same code, and one instance of each background pass is active at a time.

Tier	What it holds	How it grows
Gateway	The S3 endpoint, served by a set of pods behind one address. Each request is answered from the metadata plane and the backends.	Add pods.
Metadata plane	Every bucket, object version, multipart upload, credential, policy and volume record.	Add nodes; key ranges redistribute across them.
Backends	Opaque blobs and nothing else. One or more S3-compatible object stores.	Register another and bind it into a placement policy.
Pod-local disk	A staging area for volumes not yet uploaded, plus a write-ahead sidecar for each open volume.	Sized from the write rate and the retention window, not from how much you store.

The write path for a small object

Two paths exist and the choice is made per object. This is the aggregated one.

Append to an open volume

The object's bytes go straight into a volume file open on the pod's local disk. There is no temporary file in between.

Record volume, offset and length

The position goes into the metadata plane in the same transaction as the object's version row and its current- version pointer. A client therefore never observes half a write.

Rotate when full or old enough

Both limits are configurable, as is a size floor below which age alone will not rotate a volume. New appends are refused and appends in progress drain.

Seal with a manifest

A manifest of every object in the file is appended at the end. The header written at creation is never rewritten, so every offset already recorded stays valid. The manifest travels with the data, so the volume describes itself.

Upload to the targets the placement policy selects

Volumes go under a reserved key namespace of their own. A client write whose key would fall inside that namespace is refused, so a client object can neither collide with a volume nor be presented as one.

Mark it backend-resident and release the local copy

The metadata plane records the volume as living on the backend, and the local file is removed after a configurable retention period.

Large objects skip aggregation, because packing them would gain nothing. The gateway writes such an object straight to each selected backend, as one backend object. Above a further configurable size, it streams the body to each target as a multipart upload. A bucket whose placement policy disables aggregation takes the direct path whatever the object's size.

What a sealed volume looks like

A volume is self-describing on purpose. Given the file and nothing else, the gateway can list what is inside it and read any of it back.

header
length

JSON header
written once

object bytes, one region per fragment

manifest
JSON

trailer
length, CRC32C, magic

The header is fixed at creation, so the offsets recorded when each object was appended stay valid for the life of the volume. The manifest is a footer appended at rotation, and the trailer lets the gateway find and verify it with a size probe and one explicit byte range.

Why the sidecar matters

While a volume is open for writing, a write-ahead sidecar beside it records each object as it is appended. Sealing appends the manifest and deletes the sidecar, so a surviving sidecar proves the volume was never sealed.

Recovery after a crash reads the sidecar first and never trusts manifest-shaped bytes at the end of an unsealed file. Without that ordering, a torn write could be read back as a complete one.

What this buys you

Metadata can be rebuilt from the data: Once the configuration records are in place, a documented procedure lists the volumes on each backend, reads their manifests and repopulates the object rows.
No suffix ranges: The footer is read with a size probe and an explicit byte range, and the number of decoded entries is capped.

The metadata plane

A distributed, transactional key-value database built on TiKV. The records the gateway answers from are rows in it: bucket configuration, one row per object version with a pointer to the current version, multipart upload state, credentials and policies, and volume state. Tenant rows are scoped by realm, so one tenant's records occupy their own part of the key space.

The key space is ordered. That is what makes listing predictable. A listing is a bounded range scan over one bucket's rows, returned in key order, and it asks no backend to enumerate anything. Versioning, multipart state and the ETag a client is given are answered the same way.

Records that must agree are written in one transaction. An object's version row, its current-version pointer and the indexes recording where its bytes live commit together.

Where per-object cost lives

One record per object version: It still exists, and it moves off the backend's index into a plane designed to grow by adding nodes.
Sizing follows record count: Not stored bytes. It is planned separately from the storage backend.
Expansion redistributes key ranges: Rather than reconstructing erasure-coded fragments, so that work falls on the metadata nodes and not on the devices serving object reads.

Placement, durability and health

A placement policy is a named topology of backend targets, defined within a realm. A bucket binds to exactly one when it is created, and the binding holds for the bucket's life.

Replication across groups, sharding within one

Every group receives a write, so writing across groups replicates. Inside a group, the group's replica count decides how many of its targets take a given object and the selection spreads objects over its active targets.

Acknowledgement is part of the policy

Under all-copies acknowledgement the gateway answers only once every target in every group holds the object. Under first-group acknowledgement it answers as soon as the first complete group is durable, and records every missing copy as an owed copy.

An owed copy is never forgotten

It is recorded in the same transaction as the object, so an interruption cannot lose the record of what is missing. A background healer completes owed copies and never abandons one: a copy that cannot be made now is retried on later passes. A backlog that stops shrinking is a standing signal for the operator.

Health reorders reads, it never drops them

Each backend carries a state of up, degraded or down, derived from probes and from ordinary requests. Reads rank an object's locations by health and use a down backend as a last resort, so a stale health signal cannot make readable data unreachable.

Retiring a target is drain-gated

A draining target takes no new writes while reads continue to be served from it. Removing it outright is refused while any object or volume still references it, and the index answering that question is maintained as writes happen rather than estimated.

Physical buckets provision themselves

The gateway creates the physical buckets a policy needs. Because each bucket name is calculated from the logical bucket and the target, pods provisioning at the same moment always end up with the same name.

Moving data is an operation you can watch

Data movement is exposed as five typed operations rather than a scripting exercise. Each is a plan of phases held in the metadata plane and advanced one step at a time, so a restart resumes it. A phase an operator must see before the work continues is a checkpoint, where the operation waits for approval. A failed phase can be retried, and an operation can be cancelled.

Cutover enumerates the old backend and never deletes its bytes. Cancelling before the final target-removal phase leaves the old backend exactly as it was. That is what makes a provider change reversible rather than one-way.

The five operations

drain-target: Empty one of a policy's targets so the backend behind it can be retired.
rebalance: Spread a bucket's directly written objects across a group's targets after one is added.
import-bucket: Bring a bucket that already exists on an S3-compatible store under the gateway.
convert-bucket: Repack a bucket's directly written objects into volumes.
cutover: Move a bucket to a different backend with near-zero downtime.

Bring your own object count and we will work through the arithmetic

You already know your object count and size distribution. That is everything needed to work out what your backend would hold after aggregation, before anyone signs anything.

Talk to us