webtessera/mirror

Copies a log into S3-compatible storage or any ObjectStore.

It ports the Go package tessera/cmd/experimental/mirror. Its source is src/mirror/index.ts.

Functions

isRecoverable

isRecoverable reports whether err was not marked with unrecoverable, like retry-go's IsRecoverable.

newS3Sink

newS3Sink returns a Sink that stores objects in a bucket of any service that speaks the S3 API, signing requests with AWS Signature Version 4: AWS S3, Cloudflare R2, Google Cloud Storage (XML API, with HMAC keys), Backblaze B2, MinIO, Ceph, Wasabi and the like.

newSinkTarget

newSinkTarget returns the mirror Target that writes each resource to the sink under its tlog-tiles path (checkpoint, tile/0/x001/234, tile/entries/000.p/7, ...), so that the sink ends up holding a static tlog-tiles log that any file server, CDN or public bucket can serve as it is.

newSourceFetch

newSourceFetch returns a fetch function for reading a log that is not trusted, to hand to newHTTPFetcher as the source of a mirror: - It does not follow redirects. tlog-tiles says its resources "MUST NOT serve redirect responses", and following one would let the source send the mirror anywhere. - It refuses any response body larger than maxBytes, from its declared length or while streaming, so that a hostile source cannot exhaust the mirror's memory.

newVerifiedMirror

newVerifiedMirror returns a Mirror that copies a log only as far as it can verify it: every resource is checked against the source's signed checkpoint, and that checkpoint against the target's, before anything is written, and the checkpoint is written last.

newVerifyingSource

newVerifyingSource wraps a mirror Source so that it returns only verified resources: - readCheckpoint returns the source's checkpoint once it carries a valid signature from verifier for origin, the source's partial tiles hash to its root, and (with a target) it is consistent with the mirror's current checkpoint; - readTile returns a tile once it is proven part of that tree: a partial tile by the root itself, a full one by its hash in the verified tile above it; - readEntryBundle returns an entry bundle once its entries hash to the verified level-0 tile.

signV4

signV4 signs a request with AWS Signature Version 4, in the form S3 specifies: the path URI-encoded once and not normalised ("Each path segment must be URI-encoded twice" holds for every other AWS service, not for S3), the query parameters sorted by encoded name then value, every header the request carries signed, and the payload hash both signed and, by convention of the caller, sent as x-amz-content-sha256.

unrecoverable

unrecoverable marks err as one retry must not retry, the counterpart of retry-go's retry.Unrecoverable: a verification failure, say, which would only fail again.

Classes

Mirror

Mirror is a struct which knows how to use the Src and Store functions to copy a tlog-tiles compliant log from one location to another.

RetryError

RetryError is what retry throws once every attempt has failed: the port of retry-go's Error (a []error), with the same message, and the same Is, which matches if any attempt's error does.

S3Error

S3Error reports a request the service refused.

S3Sink

S3Sink is a Sink backed by an S3-compatible bucket.

SinkTarget

SinkTarget is a mirror Target, and a Source, over a Sink.

VerifyingSource

VerifyingSource is a Source that verifies.

Interfaces

AwsCredentials

AwsCredentials are the credentials a request is signed with.

MirrorProgress

MirrorProgress is what Mirror.progress returns.

S3SinkOptions

S3SinkOptions configures newS3Sink.

SignedV4

SignedV4 is the outcome of signing: the headers to add, and the intermediate values.

SignV4Input

SignV4Input is a request to sign.

Sink

Sink is the minimal storage a log can be mirrored into: something that stores bytes under slash-separated keys.

SinkTargetOptions

SinkTargetOptions configures newSinkTarget.

Source

Source describes a type which can fetch static resources from a source log, like the .*Fetcher implementations in the client package.

SourceFetchOptions

SourceFetchOptions configures newSourceFetch.

Target

Target describes a type which can store log static resources.

VerifiedMirrorOptions

VerifiedMirrorOptions configures newVerifiedMirror.

VerifyingSourceOptions

VerifyingSourceOptions configures newVerifyingSource.

Types

SinkObject

SinkObject is what a sink's get may resolve to: the bytes themselves, or a body to read them from.

Constants and variables

MaxCheckpointBytes

MaxCheckpointBytes caps the checkpoints a VerifyingSource accepts.

MaxResourceBytes

MaxResourceBytes is the size of the largest resource a tlog-tiles log can serve: a full entry bundle of 256 entries of 65535 bytes, each with its two-byte length prefix.