Monorepo for Tangled tangled.org

Indexer keeps waow.tech pinned to pds2.mock.tangled.org after relaunch; all new records dropped #4654

Closed zzstoatzz.io
Closed opened by zzstoatzz.io

After the relaunch, index.tangled.network still records waow.tech (did:plc:siv7zbedip4vcet4v67piibr) as hosted on https://pds2.mock.tangled.org/. PLC has pointed it at https://pds.zat.dev since 2026-10-23. The indexer appears to drop every new commit from the real PDS, so nothing the account writes reaches bobbin or tangled.org. Repo creation succeeds on the knot and the PDS, but the page 404s, and signals from the PDS side do not correct the stored host.

Another account on the same PDS (zat.dev) is indexed with the correct host, so this looks like per-account state left by the relaunch import, not a host problem. Possibly related: #590 (repos gone after moving PDS, on the old appview).

Ask: re-resolve or resync did:plc:siv7zbedip4vcet4v67piibr so its PDS becomes pds.zat.dev. More generally: accounts that the import pinned to the mock PDS seem to have no way to recover on their own.

Evidence (checked 2026-11-01, about 20:27 UTC)

The indexer's view:

$ curl 'https://index.tangled.network/xrpc/systems.gaze.hydrant.describeRepo?identifier=waow.tech'
{"did":"did:plc:siv7zbedip4vcet4v67piibr","handle":"waow.tech",
 "pds":"https://pds2.mock.tangled.org/",
 "collections":["sh.tangled.actor.profile","sh.tangled.repo","sh.tangled.publicKey"]}

$ curl 'https://index.tangled.network/xrpc/com.atproto.sync.getRepoStatus?did=did:plc:siv7zbedip4vcet4v67piibr'
{"error":"RepoNotFound"}

PLC is correct:

$ curl https://plc.directory/did:plc:siv7zbedip4vcet4v67piibr
alsoKnownAs: ["at://waow.tech"]   atproto_pds: https://pds.zat.dev

The same PDS, indexed correctly for another account:

$ curl '.../systems.gaze.hydrant.describeRepo?identifier=zat.dev'
{"handle":"zat.dev","pds":"https://pds.zat.dev/", ...}

The declaration exists on the PDS, but bobbin never sees it:

$ curl 'https://pds.zat.dev/xrpc/com.atproto.repo.getRecord?repo=did:plc:siv7zbedip4vcet4v67piibr&collection=org.tangled.repo.declaration&rkey=remind'
{"uri":"at://did:plc:siv7zbedip4vcet4v67piibr/org.tangled.repo.declaration/remind", ...}

$ curl 'https://api.tangled.org/xrpc/org.tangled.repo.getRepoByRepoDid?repoDid=did:plc:xx5y2tpzhpdlbcmixqyddfnd'
{"error":"RecordNotFound","message":"record not found"}

$ curl 'https://api.tangled.org/xrpc/org.tangled.bobbin.awaitRecord?uri=<declaration uri>&cid=<cid from putRecord>'
{"error":"NotSettled","message":"record has not been ingested yet"}

The knot has the repo and its manifest (waow.tech/remind accepts pushes and clones from knot1.tangled.sh). org.tangled.publicKey.listKeys?subject=did:plc:siv7zbedip4vcet4v67piibr returns [], even though an org.tangled.key.sshKey was published earlier the same day.

What was tried from the PDS side
  • com.atproto.server.activateAccount on the already active account. The PDS (zds) emits #account active, #identity and #sync and notifies its crawlers. The stored host did not change.
  • Fresh putRecords of the declaration with new CIDs. awaitRecord returned NotSettled for each.
  • requestCrawl for pds.zat.dev against index.tangled.network, following the knot upgrade docs. It was accepted, and a new account on the same PDS became indexed within a minute. waow.tech still reports the mock PDS.
  • tg repo create remind-probe --account waow.tech printed repository saved, but indexing is pending: ... NotSettled, and the page 404s. The same flow for an account on another PDS indexed immediately.
  • Deleting and re-creating the repo is blocked: the knot returns 409 Conflict: repository with that record key already exists, and tg repo delete cannot resolve the repo through bobbin.
Suspected mechanism (inferred from hydrant's source, not from indexer logs)

In src/ingest/relay.rs, check_host_authority compares a commit's source host with the stored repo_state.pds. On a mismatch it calls refresh_doc once and re-checks. Only then does it reject with WrongHost, and it caches that rejection per (DID, host). For waow.tech the stored PDS never moves off pds2.mock.tangled.org. That suggests the re-resolution also returns the mock host for this DID, perhaps from an identity override or mirror seeded by the import, so every commit from pds.zat.dev is rejected. The stored collections are only legacy sh.tangled.*, which fits state seeded from the import and never updated live.

was this account migrated by logging into the web client?

yes!

hmm well, its still marked as a liar account and the records dont seem to have migrated properly... did you get any errors while doing it? what happens if you logout and login? does it show migration again?

also it seems like your pds doesnt send rebase field so hydrant is rejecting the commit and thus disconnecting from the firehose. so thats maybe why. though it doesnt explain why the records didnt get migrated properly

(at least in the draft spec, rebase doesnt exist anymore. but the lexicon still requires it to be sent)

actually now that i see zds doesnt even send since? it leaves it out while its supposed to send it (as null for the first commit of a repo). rebase idm since the draft spec doesnt mention it but that one should be fixed over on zds

oof i see that now! thanks will update zds and close this ty

but to answer the question, no, no errors during migration. and prompt comes back after logout and re-running it doesn't appear to help

zds now sends since/rebase/tooBig (v0.7.2) and stored history is fixed (v0.7.3); hydrant seems caught up on pds.zat.dev. but getRepoStatus for waow.tech (did:plc:siv7zbedip4vcet4v67piibr) is still RepoNotFound after new commits. is that something you could resync?

i think you might just have to requestCrawl it again its most likely stuck in backoff for the firehose conn

Sign up or Sign in to add to the discussion.