When refreshing identities if a did doc does not pass PartialMiniDoc validation Slingshot logs a warning and continues. This loops and it tries the doc again, which of course fails. On my server this happens about 600 times per minute, and I've had ~14M of this log in the last 30 days. It only gets unstuck when the doc becomes unreachable or is fixed. Because the queue is stuck, it just keeps growing, which potentially accounts for the memory growth that got me looking into this in the first place.
The log in question is WARN slingshot::identity: converting mini doc failed followed by "no valid atproto signing key found in verification methods". dropping refresh or some other variation. This doesn't look very recoverable to me! At least not by slingshot.
This PR switches the behavior to drop bad identities, letting the queue continue processing without interruption, and removing the identity from the cache so we don't pretend it's valid when the validation failed.
The possible errors on that line are:
did doc missing also_known_as,
no valid atproto handles in also_known_as,
no valid pds service found,
no verification methods found: verificationMethod,
no valid atproto signing key found in verification methods
Either way the doc is not valid and slingshot should reasonably stop serving it as valid.
EDIT:
Although I haven't seen this one in production, a few lines above we have the same issue where if the did doesn't match the doc id it continues and goes back to the same top item and fetches it again, which will fail again. So I've fixed that the same way.
Also want to highlight that while stuck on an identity slingshot will hammer the did doc it's checking because it keeps refetching it with no backoff. Looking at my log history my slingshot does about 10 requests per second when it has been stuck on a document. Over several days.