recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Index only the first 1,000 comments on any object

Summary:
Depends on D19502. Ref T13151. See PHI719. An install ended up with an object with 111,000+ comments on it because someone wrote a script to treat it like a logfile.

Although we seem to do mostly okay with this (locally, it only takes about 30s to index a similar object) we'll hit a wall somewhere (since we need to hold everything in memory), and it's hard to imagine a legitimate object with more than 1,000 comments. Just ignore comments past the first thousand.

(Conpherence threads may legitimately have more than 1,000 comments, but go through a different indexer.)

Test Plan:
- Piped some comments into `maniphest.edit` in a loop to create a task with 100K comments.
- Ran `bin/search index Txxx --force` to reindex it, with `--trace`.
- Before: task indexed in about 30s.
- After: script loaded comments with LIMIT 1000 and indexed in a couple seconds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19503

epriestley 14e911a0 cbc71e75

+19 -3
+19 -3
src/applications/transactions/engineextension/PhabricatorTransactionsFulltextEngineExtension.php
··· 22 22 return; 23 23 } 24 24 25 - $xactions = $query 25 + $query 26 26 ->setViewer($this->getViewer()) 27 27 ->withObjectPHIDs(array($object->getPHID())) 28 28 ->withComments(true) 29 - ->needComments(true) 30 - ->execute(); 29 + ->needComments(true); 30 + 31 + // See PHI719. Users occasionally create objects with huge numbers of 32 + // comments, which can be slow to index. We handle this with reasonable 33 + // grace: at time of writing, we can index a task with 100K comments in 34 + // about 30 seconds. However, we do need to hold all the comments in 35 + // memory in the AbstractDocument, so there's some practical limit to what 36 + // we can realistically index. 37 + 38 + // Since objects with more than 1,000 comments are not likely to be 39 + // legitimate objects with actual discussion, index only the first 40 + // thousand comments. 41 + 42 + $query 43 + ->setOrderVector(array('-id')) 44 + ->setLimit(1000); 45 + 46 + $xactions = $query->execute(); 31 47 32 48 foreach ($xactions as $xaction) { 33 49 if (!$xaction->hasComment()) {