In taskmaster daemons, only close connections which were not used recently

@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge

phorge phabricator

fork

Summary:
Ref T11458. Depends on D16388. Currently, we're very aggressive about closing connections in the taskmaster daemons.

This can end up taking up a lot of resources. In particular, because the outgoing port for outbound connections normally can not be reused for 60 seconds after a connection closes, we may exhaust outbound ports on the host if there's a big queue full of stuff that's being processed very quickly.

At a minimum, we //always// are holding open a `worker` connection, which we always need again right away. So even in the best case we end up opening/closing this about once per second and each daemon takes up about ~60 outbound ports when it should take up ~1.

So, make two adjustments:

- First, only close connections which we haven't issued a query on in the last 60 seconds. This should prevent us from closing connections that we'll need again immediately in most cases. In the worst case, we shouldn't be eating up any extra ports under default TCP behavior.
- Second, explicitly close connections. We were relying on implicit/GC behavior (maybe as a holdover from very long ago, before we got connection wrappers in place?), which probably did about the same thing but isn't as predictable and can't be profiled or instrumented.

Test Plan:
This is somewhat difficult to test completely convincingly in isolation since the problem behavior depends on production scales and the workload, and to some degree on configuration.

I tested that this stuff baiscally works by adding logging to connect/close and running the daemons, verifying that they churned connections a lot before this change (e.g., ~1/s even at no load) and churn rarely afterward (e.g., almost never at no load).

I ran some workload through them to make sure I didn't completely break anything.

The best real test is just seeing how production responds. Current inbound/outbound connections on `secure001` are 1,200:

```
secure001 $ netstat -t | grep :mysql | wc -l
1164
```

Current outbound from `repo001` are 18,600:

```
repo001 $ netstat -t | grep :mysql | wc -l
18663
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11458

Differential Revision: https://secure.phabricator.com/D16389

epriestley 9 years ago 5e3efca0 3b45608c

+47 -2

2 changed files

expand all

src

infrastructure

daemon

PhabricatorDaemon.php

storage

lisk

LiskDAO.php

+1 -1

src/infrastructure/daemon/PhabricatorDaemon.php

··· 11 11 } 12 12 13 13 protected function willSleep($duration) { 14 - LiskDAO::closeAllConnections(); 14 + LiskDAO::closeInactiveConnections(60); 15 15 return; 16 16 } 17 17

+46 -1

src/infrastructure/storage/lisk/LiskDAO.php

··· 1621 1621 return (bool)self::$transactionIsolationLevel; 1622 1622 } 1623 1623 1624 + /** 1625 + * Close any connections with no recent activity. 1626 + * 1627 + * Long-running processes can use this method to clean up connections which 1628 + * have not been used recently. 1629 + * 1630 + * @param int Close connections with no activity for this many seconds. 1631 + * @return void 1632 + */ 1633 + public static function closeInactiveConnections($idle_window) { 1634 + $connections = self::$connections; 1635 + 1636 + $now = PhabricatorTime::getNow(); 1637 + foreach ($connections as $key => $connection) { 1638 + $last_active = $connection->getLastActiveEpoch(); 1639 + 1640 + $idle_duration = ($now - $last_active); 1641 + if ($idle_duration <= $idle_window) { 1642 + continue; 1643 + } 1644 + 1645 + self::closeConnection($key); 1646 + } 1647 + } 1648 + 1649 + 1624 1650 public static function closeAllConnections() { 1625 - self::$connections = array(); 1651 + $connections = self::$connections; 1652 + 1653 + foreach ($connections as $key => $connection) { 1654 + self::closeConnection($key); 1655 + } 1656 + } 1657 + 1658 + private static function closeConnection($key) { 1659 + if (empty(self::$connections[$key])) { 1660 + throw new Exception( 1661 + pht( 1662 + 'No database connection with connection key "%s" exists!', 1663 + $key)); 1664 + } 1665 + 1666 + $connection = self::$connections[$key]; 1667 + unset(self::$connections[$key]); 1668 + 1669 + $connection->close(); 1626 1670 } 1671 + 1627 1672 1628 1673 /* -( Utilities )---------------------------------------------------------- */ 1629 1674

Configure Feed

Configure Feed