git.listLanguages has been one of the method that fails most often.
Opening multiple git repos simultaneously can eaily cause OOM and
language indexing itself usually takes super long.
So several changes:
- use
gitea.CatFileBatchinstead of go-git to avoid OOM - skip files larger than 16KB
- sync HEAD ref language stats in knotmirror db
- cache language stats info by commits (30d TTL)
When syncing HEAD ref language stats, we do indexing on background.
KnotMirror maintains internal "repo_stats_update" queue and right after
doResync is done, enqueue the language stat indexing job so we can
pre-index the language stats of HEAD ref.
It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily.
Signed-off-by: Seongmin Lee git@boltless.me