recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Implement smart waits for rarely updated repositories

Summary:
Ref T4605. When figuring out how long to wait to update a repository, factor in when it was last pushed. For rarely updated repositories, wait longer between updates.

(A slightly funky thing about this is that empty repos update every 15 seconds, but that seems OK for the moment.)

Test Plan:
Ran `bin/phd debug pulllocal` and saw sensible calculations and output:

```
...
<VERB> PhabricatorRepositoryPullLocalDaemon Last commit to repository "rPOEMS" was 1,239,608 seconds ago; considering a wait of 6,198 seconds before update.
>>> [79] <query> SELECT * FROM `repository` r ORDER BY r.id DESC
<<< [79] <query> 514 us
>>> [80] <query> SELECT * FROM `repository_statusmessage` WHERE statusType = 'needs-update'
<<< [80] <query> 406 us
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIH" is not due for an update for 8,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rDUCK" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTESTX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rQWER" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBT" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVNX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIG" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBTX" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGX" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTX" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPOEMS" is not due for an update for 6,198 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHY" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIS" is not due for an update for 6,894 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARCLINT" is not due for an update for 21,599 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rLPHX" is not due for an update for 1,979 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARC" is not due for an update for 1,824 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIHG" is not due for an update for 21,599 second(s).
...
```

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T4605

Differential Revision: https://secure.phabricator.com/D8782

epriestley c402d7d3 5671c4b2

+43
+43
src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php
··· 346 346 } 347 347 348 348 $sleep_for = (int)$repository->getDetail('pull-frequency', $min_sleep); 349 + 350 + // Smart wait: pull rarely used repositories less frequently. Find the 351 + // most recent commit which is older than the current time (this keeps us 352 + // from spinning on repositories with a silly commit post-dated to some time 353 + // in 2037), and adjust how frequently we pull based on how frequently this 354 + // repository updates. 355 + 356 + $table = id(new PhabricatorRepositoryCommit()); 357 + $last_commit = queryfx_one( 358 + $table->establishConnection('w'), 359 + 'SELECT epoch FROM %T 360 + WHERE repositoryID = %d AND epoch <= %d 361 + ORDER BY epoch DESC LIMIT 1', 362 + $table->getTableName(), 363 + $repository->getID(), 364 + time() + $min_sleep); 365 + if ($last_commit) { 366 + $time_since_commit = (time() + $min_sleep) - $last_commit['epoch']; 367 + 368 + // Wait 0.5% of the time since the last commit before we pull. This gives 369 + // us these wait times: 370 + // 371 + // 50 minutes or less: 15 seconds 372 + // about 3 hours: 1 minute 373 + // about 16 hours: 5 minutes 374 + // about 2 days: 15 minutes 375 + // 50 days or more: 6 hours 376 + 377 + $smart_wait = ($time_since_commit / 200); 378 + $smart_wait = min($smart_wait, phutil_units('6 hours in seconds')); 379 + 380 + $this->log( 381 + pht( 382 + 'Last commit to repository "%s" was %s seconds ago; considering '. 383 + 'a wait of %s seconds before update.', 384 + $repository->getMonogram(), 385 + new PhutilNumber($time_since_commit), 386 + new PhutilNumber($smart_wait))); 387 + 388 + $smart_wait = max(15, $smart_wait); 389 + $sleep_for = max($smart_wait, $sleep_for); 390 + } 391 + 349 392 if ($sleep_for < $min_sleep) { 350 393 $sleep_for = $min_sleep; 351 394 }