swift-tangled
A practical Tangled CLI nnabeyang.tngl.sh/swift-tangled
tangled atproto cli
Star 8
Fork 0
Watch

Artifact tag resolution is affected by a shared upstream pre-authentication rate limit #5

Open nnabeyang.tngl.sh
Open opened by nnabeyang.tngl.sh

Summary #

Artifact upload, view, and download resolve the annotated Git tag through Bobbin's sh.tangled.repo.tags endpoint before transferring any blob. When that Git read is not served by the mirror, Bobbin proxies it to the repository's Knot, where it can be affected by a shared pre-authentication rate-limit bucket.

The observed error is:

API error: rate limited: too many pre-authentication requests, retry shortly

The current evidence points to the upstream Bobbin-to-Knot proxy and deployment path rather than an error-classification bug in tng.

Observed behavior #

  • Using tng 0.6.0
  • The error occurred during tng artifact upload
  • Waiting before retrying sometimes produced the same error
  • Upload succeeded immediately after tng auth login
  • Artifact view and download returned the same error immediately after that successful upload
  • An anonymous download with a temporary HOME containing no stored session returned the same error
  • The same error has not been observed while creating Pull Requests or Issues
  • The frequency appears to vary over time

Upstream findings #

Source of the error #

Knot 2 emits this exact message only when its pre-auth token bucket has no token available, returning HTTP 429 RateLimitExceeded.

Per-peer or global in-flight saturation takes a separate branch and returns HTTP 503 Overloaded. Internal and upstream failures also retain 5xx classifications. Knot does not broadly convert arbitrary internal failures into this HTTP 429 response.

Knot 2 defaults to:

  • burst: 20 requests
  • refill: one token every 100 ms
  • key: the effective peer IP resolved from the proxy configuration

Why artifact operations reach the Knot limiter #

Artifact upload, view, and download share these preliminary steps:

  1. Resolve the repository record.
  2. Resolve the annotated tag through Bobbin's sh.tangled.repo.tags endpoint.
  3. Look up artifact records.
  4. For upload, write the blob and record to the uploader's PDS.
  5. For download, retrieve the blob from its owner's PDS.

Bobbin can serve sh.tangled.repo.tags through a Git mirror. If the mirror is unset, unavailable, or declines the request, Bobbin falls back to the Knot. Consequently, an operation reported as an artifact upload or download failure can actually fail during tag resolution, before any blob transfer starts.

Known upstream issue #

Upstream core issue #713 reports that Bobbin did not forward client addresses to Knots, causing API consumers behind Bobbin to share one rate-limit bucket.

The following fixes were merged into upstream master on 2026-07-31:

  • Bobbin forwards a verified client address to the Knot.
  • Knot walks the trusted proxy chain to select the actual client address.
  • Knot warns operators when one address receives a concentrated majority of pre-auth refusals.

These fixes require the trusted proxy chain to be configured correctly on both Bobbin and Knot.

Production configuration hypothesis #

The tracked worker configuration does not appear to pass BOBBIN_TRUSTED_PROXIES or BOBBIN_MIRROR_URL into the Bobbin container.

If that reflects production, either or both of the following can occur:

  • Bobbin forwards the front proxy's socket address as the client address, pooling multiple users into one Knot bucket.
  • The Git mirror is unused, so every tag lookup falls back to the Knot.

The actual production environment variables, reverse-proxy topology, and Bobbin/Knot logs have not been inspected. This is therefore a strong verification target, not a confirmed production configuration.

Separate rate limit #

The api.tangled.org Cloudflare Worker also has a limit of 300 requests per 60 seconds per IP. It returns a different response:

too many requests, slow down
Retry-After: 60

That does not match the observed Knot pre-authentication message and is not the direct source of this failure.

tng assessment #

tng 0.6.0 correctly maps the upstream HTTP 429 response to TangledError.rateLimited.

For Bobbin requests, it retries HTTP 429, 502, 503, 504, and transient network failures up to three attempts. Without Retry-After, a 429 is retried after two and four seconds. Exhausting those retries is consistent with a shared bucket being continuously consumed by other traffic.

No tng code change is planned at this point.

Upstream verification needed #

  • Production Bobbin's BOBBIN_TRUSTED_PROXIES
  • Production Bobbin's BOBBIN_MIRROR_URL
  • Knot's xrpc.trusted_proxy_header and xrpc.trusted_proxies
  • The effective client address Bobbin forwards to Knot
  • Knot's concentrated pre-auth refusal warning
  • Whether artifact upload, view, and download still reproduce after correcting the configuration
  • Whether Knot should attach Retry-After to its 429 response

Expected behavior #

  • Preliminary artifact tag resolution does not share a rate-limit bucket across unrelated users.
  • Eligible Git reads use the mirror instead of unnecessarily falling back to Knot.
  • Real rate limits remain distinguishable from overload and upstream failures.
  • A rate-limit response communicates when the caller should retry.
Sign up or Sign in to add to the discussion.