recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

When waiting for long-running Harbormaster futures to resolve, close idle database connections

Summary:
Ref T13216. See PHI916. Harbormaster builds may be long-running, particularly if they effectively wrap `ssh ... ./run-huge-build.sh`. If we spend more than a few seconds waiting for futures to resolve, close idle database connections.

The general goal here is to reduce the held connection load for installs with a very large number of test runners.

Test Plan: Added debugging code to `phlog()` closures, saw connections closed while running builds.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19824

epriestley 43cf4edf a0d4b6da

+47 -10
+2 -2
src/applications/feed/query/PhabricatorFeedQuery.php
··· 78 78 79 79 $where[] = qsprintf( 80 80 $conn, 81 - 'ref.chronologicalKey IN (%Q)', 82 - implode(', ', $keys)); 81 + 'ref.chronologicalKey IN (%LQ)', 82 + $keys); 83 83 } 84 84 85 85 // NOTE: We may not have 64-bit PHP, so do the shifts in MySQL instead.
+24 -4
src/applications/harbormaster/step/HarbormasterBuildStepImplementation.php
··· 253 253 HarbormasterBuildTarget $target, 254 254 array $futures) { 255 255 256 + $did_close = false; 257 + $wait_start = PhabricatorTime::getNow(); 258 + 256 259 $futures = new FutureIterator($futures); 257 260 foreach ($futures->setUpdateInterval(5) as $key => $future) { 258 - if ($future === null) { 259 - $build->reload(); 260 - if ($this->shouldAbort($build, $target)) { 261 - throw new HarbormasterBuildAbortedException(); 261 + if ($future !== null) { 262 + continue; 263 + } 264 + 265 + $build->reload(); 266 + if ($this->shouldAbort($build, $target)) { 267 + throw new HarbormasterBuildAbortedException(); 268 + } 269 + 270 + // See PHI916. If we're waiting on a remote system for a while, clean 271 + // up database connections to reduce the cost of having a large number 272 + // of processes babysitting an `ssh ... ./run-huge-build.sh` process on 273 + // a build host. 274 + if (!$did_close) { 275 + $now = PhabricatorTime::getNow(); 276 + $elapsed = ($now - $wait_start); 277 + $idle_limit = 5; 278 + 279 + if ($elapsed >= $idle_limit) { 280 + LiskDAO::closeIdleConnections(); 281 + $did_close = true; 262 282 } 263 283 } 264 284 }
+2 -2
src/applications/search/index/PhabricatorIndexEngine.php
··· 141 141 queryfx( 142 142 $conn_w, 143 143 'INSERT INTO %T (objectPHID, extensionKey, version) 144 - VALUES %Q 144 + VALUES %LQ 145 145 ON DUPLICATE KEY UPDATE version = VALUES(version)', 146 146 $table->getTableName(), 147 - implode(', ', $sql)); 147 + $sql); 148 148 } 149 149 150 150 }
+2 -2
src/applications/search/ngrams/PhabricatorSearchNgrams.php
··· 102 102 if ($sql) { 103 103 queryfx( 104 104 $conn_w, 105 - 'INSERT INTO %T (objectID, ngram) VALUES %Q', 105 + 'INSERT INTO %T (objectID, ngram) VALUES %LQ', 106 106 $this->getTableName(), 107 - implode(', ', $sql)); 107 + $sql); 108 108 } 109 109 110 110 return $this;
+17
src/infrastructure/storage/lisk/LiskDAO.php
··· 1652 1652 1653 1653 $now = PhabricatorTime::getNow(); 1654 1654 foreach ($connections as $key => $connection) { 1655 + // If the connection is not idle, never consider it inactive. 1656 + if (!$connection->isIdle()) { 1657 + continue; 1658 + } 1659 + 1655 1660 $last_active = $connection->getLastActiveEpoch(); 1656 1661 1657 1662 $idle_duration = ($now - $last_active); ··· 1668 1673 $connections = self::$connections; 1669 1674 1670 1675 foreach ($connections as $key => $connection) { 1676 + self::closeConnection($key); 1677 + } 1678 + } 1679 + 1680 + public static function closeIdleConnections() { 1681 + $connections = self::$connections; 1682 + 1683 + foreach ($connections as $key => $connection) { 1684 + if (!$connection->isIdle()) { 1685 + continue; 1686 + } 1687 + 1671 1688 self::closeConnection($key); 1672 1689 } 1673 1690 }