500 on large layer upload #28

Closed

Closed opened by

Is there a limit on the size of an individual layer? I have an image with a 500MB+ layer that has failed to upload on several retries, failing anywhere between 50-95% of the way to completion.

I'm pushing with podman using an app password. Is this a situation in which I need to set up my own hold and BYOS?

Appreciate your work on this. The UI is great.

❯ podman push 18c docker://atcr.io/rtw.run/graphhopper:latest
Getting image source signatures
Copying blob a40c811f6e7d skipped: already exists
Copying blob ccd42f8a1fa7 skipped: already exists
Copying blob e1b542319d62 skipped: already exists
Copying blob 6ac452ae67e2 skipped: already exists
Copying blob fc85f377dd10 skipped: already exists
Copying blob 136422f031b9 skipped: already exists
Copying blob 4b5b3b133830 skipped: already exists
Copying blob 5f1a69da2bf7 skipped: already exists
Copying blob da5b9fd2815c skipped: already exists
Copying blob 6cb4f7a41e41 skipped: already exists
Copying blob af9f85353387 skipped: already exists
Copying blob 88766d9ef403 skipped: already exists
Copying blob 939cb7537ac9 skipped: already exists
Copying blob 76832a3218f9 skipped: already exists
Copying blob dffea5f8e79d skipped: already exists
Copying blob 2bdabb3b4f5b skipped: already exists
Copying blob db2b5631ca89 skipped: already exists
Copying blob f2d59192676e skipped: already exists
Copying blob 1f7a33df9b76 skipped: already exists
Copying blob f9dc4b1d8e06 skipped: already exists
Copying blob 11cbf9b4153a skipped: already exists
Copying blob 2f77e140721c skipped: already exists
Copying blob 0054f62bc345 skipped: already exists
Copying blob 48ab267d8705 skipped: already exists
Copying blob 263acc0a47b1 skipped: already exists
Copying blob f155e86958ff skipped: already exists
Copying blob 58be7844b36e skipped: already exists
Copying blob 11a751cf0361 skipped: already exists
Copying blob a8745f86f3d0 skipped: already exists
Copying blob 91402cd4d25d skipped: already exists
Copying blob f0ddcc902a29 skipped: already exists
Copying blob 65790258d28e skipped: already exists
Copying blob 7f0f76267981 skipped: already exists
Copying blob 9c20b4f1058f skipped: already exists
Copying blob a75caac628b5 skipped: already exists
Copying blob 9c9c41b804c8 skipped: already exists
Copying blob 3946439e785e skipped: already exists
Copying blob 8ce60522a10d skipped: already exists
Copying blob 422e12f802fc skipped: already exists
Copying blob e4cc7ec8dbfb skipped: already exists
Copying blob b75e7f81b859 skipped: already exists
Copying blob d585d26f2038 skipped: already exists
Copying blob 40366953e9fd skipped: already exists
Copying blob 588eb365cc97 [=====================>----------------] 331.0MiB / 565.3MiB | 152.6 MiB/s
Copying blob 6142c8dbf1a2 skipped: already exists
Copying blob 3039cfca6548 skipped: already exists
Copying blob bea9929ac568 skipped: already exists
Copying blob a8158c6bc32f skipped: already exists
Copying blob b095f580febd skipped: already exists
Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error

rtw.run 2mo ago

Just one update - I broke this up into smaller layers and optimized it for size, keeping the layer sizes to max of around 150MB. I was able to upload one container image for a single architecture that way, but pushing the second architecture also ran into a 500 error with a bit more information:

Copying blob f9ba4655140d [============>-------------------------] 33.0MiB / 99.5MiB | 70.3 MiB/s
Copying blob 348246d2f4e3 done   |
Copying blob d9ef38382ba3 done   |
Copying blob a14f65c6ec74 [===============>----------------------] 40.0MiB / 97.4MiB | 73.9 MiB/s
Copying blob 0697027cd881 done   |
Copying blob 1011de26cd7b done   |
FATA[0165] writing blob: uploading layer to https://atcr.io/v2/rtw.run/graphhopper/blobs/uploads/upload-1787138895237987599?_
state=qQOGfyjh6sRvTH9WXA1Ti6sY6ldkWOHJQdoAPRKjT6d7Ik5hbWUiOiJydHcucnVuL2dyYXBoaG9wcGVyIiwiVVVJRCI6InVwbG9hZC0xNzg3MTM4ODk1MjM
3OTg3NTk5IiwiT2Zmc2V0IjoxNzA3NDEwNCwiU3RhcnRlZEF0IjoiMjAyNi0wOC0xOVQxMToyODoxNS4zMDM0NDI4NjNaIn0%3D&digest=sha256%3A1e223af8c
2aa30ed645da11ca2f1f77e7e31ff19bc079aa641bf65988e27b3e7: received unexpected HTTP status: 500 Internal Server Error

evan.jarrett.net 2mo ago

No, there's no size limit on individual layers, and this isn't a timeout. Nothing wrong with your setup, and BYOS wouldn't help since your own hold would reject the same expired token.

The short answer is that you are hitting the wall on upload size/time.

The long answer: Every one of your failed pushes shows the same thing in our logs: the hold rejecting the appview's request with service token authentication failed: token has expired. That's the short-lived service auth token the appview mints from your PDS to talk to blob storage on your behalf. The bug is that we capture it once when the upload starts and reuse it for the whole layer, instead of refreshing it as the upload progresses. Anything that takes longer than the token's remaining life dies partway through, which is why it failed at a different percentage each retry and why smaller layers only made it less likely.

rtw.run 2mo ago

Damn my slow internet. Thanks for the explanation. It's good to know some of these gotchas.

evan.jarrett.net 2mo ago

It’s fixable, but will be a bit before I can deploy any changes. There aren’t many users trying to upload 500mb+ images.