The compaction and training processes don't log much (at the INFO level, at least), so it is often unclear when they start and end and what the progress is in between. For compaction, the only logs consist of Starting major compaction from lsm_tree (I think one of these log messages for each internal tree?). For training, there is also no overall log that training has started; each keyspace that is trained logs training zstd dictionary for keyspace __ and after saved zstd dictionary for keyspace __, but the former comes only after the keyspace's samples have been collected. Ideally, the logs should indicate when a compaction or training request has been received, when the overall compaction is started, and when all of the compaction is completed (and that all of the ingestion modules have resumed).
A somewhat separate but related issue is that training can practically hang indefinitely on large databases, I think because it takes a very long time to collect the relevant samples for the blocks keyspace at such scale (hundreds of millions of records). This logging issue is relevant because it isn't even clear that the training has begun for the blocks keyspace because the samples take so long to collect. This issue is further exacerbated by the fact that the whole training (and compaction too, but that isn't necessarily problematic in my experience) operation is wrapped with with_ingestion_paused, which lacks a timeout/cancellation mechanism, so it can look like the training completed once the other two keyspace dictionaries are saved but ingestion will still be paused forever until the blocks training finally completes.
yeah apparently the sampling was O(db) basically... i thought it was not but i did a silly it turns out. anyway it should be fixed now i believe