@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Allow persistent connections to be configured per database host
Summary: Ref T11044. Fixes T11672. In T11672, persistent connections seem to work fine, but they can require `max_connections` and other settings to be raised. Since most users don't need them, make them an advanced option.
Test Plan: Configured persistent connections, loaded some pages, observed persistent connections get used.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T11044, T11672
Differential Revision: https://secure.phabricator.com/D16913
+79
-19
+39
-1
src/docs/user/cluster/cluster_partitioning.diviner
+39
-1
src/docs/user/cluster/cluster_partitioning.diviner
···
31
31
should review and understand how replication works before you partition. For
32
32
details, see @{Cluster:Databases}.
33
33
34
+
Databases also support some advanced configuration options. Briefly:
35
+
36
+
- `persistent`: Allows use of persistent connections, reducing pressure on
37
+
outbound ports.
38
+
39
+
See "Advanced Configuration", below, for additional discussion.
40
+
34
41
35
42
What Partitioning Does
36
43
======================
···
145
152
To add a new partition, follow these steps:
146
153
147
154
- Set up the new database host or hosts.
148
-
- Add the new database to `cluster.database`, but keep its "partition"
155
+
- Add the new database to `cluster.databases`, but keep its "partition"
149
156
configuration empty (just an empty list). If this is the first time you
150
157
are partitioning, you will need to configure your existing master as the
151
158
new "default". This will let Phabricator interact with it, but won't send
···
194
201
Phabricator does not perform joins across logical databases, so there are no
195
202
meaningful differences in runtime behavior if two applications are on the same
196
203
physical host or different physical hosts.
204
+
205
+
206
+
Advanced Configuration
207
+
======================
208
+
209
+
Separate from partitioning, some advanced configuration is supported. These
210
+
options must be set on database specifications in `cluster.databases`. You can
211
+
configure them without actually building a cluster by defining a cluster with
212
+
only one master.
213
+
214
+
`persistent` //(bool)// Enables persistent connections. Defaults to off.
215
+
216
+
With persitent connections enabled, Phabricator will keep a pool of database
217
+
connections open between web requests and reuse them when serving subsequent
218
+
requests.
219
+
220
+
The primary benefit of using persistent connections is that it will greatly
221
+
reduce pressure on how quickly outbound TCP ports are opened and closed. After
222
+
a TCP port closes, it normally can't be used again for about 60 seconds, so
223
+
rapidly cycling ports can cause resource exuastion. If you're seeing failures
224
+
because requests are unable to bind to an outbound port, enabling this option
225
+
is likely to fix the issue. This option may also slightly increase performance.
226
+
227
+
The cost of using persistent connections is that you may need to raise the
228
+
MySQL `max_connections` setting: although Phabricator will make far fewer
229
+
connections, the connections it does make will be longer-lived. Raising this
230
+
setting will increase MySQL memory requirements and may run into other limits,
231
+
like `open_files_limit`, which may also need to be raised.
232
+
233
+
Persistent connections are enabled per-database. If you always want to use
234
+
them, set the flag on each configured database in `cluster.databases`.
197
235
198
236
199
237
Next Steps
+1
src/infrastructure/cluster/PhabricatorClusterDatabasesConfigOptionType.php
+1
src/infrastructure/cluster/PhabricatorClusterDatabasesConfigOptionType.php
+35
-2
src/infrastructure/cluster/PhabricatorDatabaseRef.php
+35
-2
src/infrastructure/cluster/PhabricatorDatabaseRef.php
···
40
40
private $applicationMap = array();
41
41
private $masterRef;
42
42
private $replicaRefs = array();
43
+
private $usePersistentConnections;
43
44
44
45
public function setHost($host) {
45
46
$this->host = $host;
···
169
170
170
171
public function getIsDefaultPartition() {
171
172
return $this->isDefaultPartition;
173
+
}
174
+
175
+
public function setUsePersistentConnections($use_persistent_connections) {
176
+
$this->usePersistentConnections = $use_persistent_connections;
177
+
return $this;
178
+
}
179
+
180
+
public function getUsePersistentConnections() {
181
+
return $this->usePersistentConnections;
172
182
}
173
183
174
184
public function setApplicationMap(array $application_map) {
···
582
592
->setPort($default_port)
583
593
->setIsIndividual(true)
584
594
->setIsMaster(true)
585
-
->setIsDefaultPartition(true);
595
+
->setIsDefaultPartition(true)
596
+
->setUsePersistentConnections(false);
586
597
}
587
598
588
599
public static function getAllReplicaDatabaseRefs() {
···
672
683
'database' => null,
673
684
'retries' => $default_retries,
674
685
'timeout' => $default_timeout,
686
+
'persistent' => $this->getUsePersistentConnections(),
675
687
);
676
688
677
-
return self::newRawConnection($spec);
689
+
$is_cli = (php_sapi_name() == 'cli');
690
+
691
+
$use_persistent = false;
692
+
if (!empty($spec['persistent']) && !$is_cli) {
693
+
$use_persistent = true;
694
+
}
695
+
unset($spec['persistent']);
696
+
697
+
$connection = self::newRawConnection($spec);
698
+
699
+
// If configured, use persistent connections. See T11672 for details.
700
+
if ($use_persistent) {
701
+
$connection->setPersistent($use_persistent);
702
+
}
703
+
704
+
// Unless this is a script running from the CLI, prevent any query from
705
+
// running for more than 30 seconds. See T10849 for details.
706
+
if (!$is_cli) {
707
+
$connection->setQueryTimeout(30);
708
+
}
709
+
710
+
return $connection;
678
711
}
679
712
680
713
public static function newRawConnection(array $options) {
+4
-1
src/infrastructure/cluster/PhabricatorDatabaseRefParser.php
+4
-1
src/infrastructure/cluster/PhabricatorDatabaseRefParser.php
···
58
58
$role = $server['role'];
59
59
$is_master = ($role == 'master');
60
60
61
+
$use_persistent = (bool)idx($server, 'persistent', false);
62
+
61
63
$ref = id(new PhabricatorDatabaseRef())
62
64
->setHost($host)
63
65
->setPort($port)
64
66
->setUser($user)
65
67
->setPass($pass)
66
68
->setDisabled($disabled)
67
-
->setIsMaster($is_master);
69
+
->setIsMaster($is_master)
70
+
->setUsePersistentConnections($use_persistent);
68
71
69
72
if ($is_master) {
70
73
$master_count++;
-15
src/infrastructure/storage/lisk/PhabricatorLiskDAO.php
-15
src/infrastructure/storage/lisk/PhabricatorLiskDAO.php
···
73
73
$connection->setReadOnly(true);
74
74
}
75
75
76
-
// Unless this is a script running from the CLI:
77
-
// - (T10849) Prevent any query from running for more than 30 seconds.
78
-
// - (T11672) Use persistent connections.
79
-
if (php_sapi_name() != 'cli') {
80
-
81
-
// TODO: For now, disable this until after T11044: it's better at high
82
-
// load, but causes us to use slightly more connections at low load and
83
-
// is pushing users over limits like MySQL "max_connections".
84
-
$use_persistent = false;
85
-
86
-
$connection
87
-
->setQueryTimeout(30)
88
-
->setPersistent($use_persistent);
89
-
}
90
-
91
76
return $connection;
92
77
}
93
78