@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Don't apply `security.require-https` to intracluster requests
Summary:
Ref T10784. Currently, if you terminate SSL at a load balancer (very common) and use HTTP beyond that, you have to fiddle with this setting in your premable or a `SiteConfig`.
On the balance I think this makes stuff much harder to configure without any real security benefit, so don't apply this option to intracluster requests.
Also document a lot of stuff.
Test Plan: Poked around locally but this is hard to test outside of a production cluster, I'll vet it more thoroughly on `secure`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10784
Differential Revision: https://secure.phabricator.com/D15696
+157
-14
+12
src/aphront/configuration/AphrontApplicationConfiguration.php
+12
src/aphront/configuration/AphrontApplicationConfiguration.php
···
345
345
346
346
if ($site->shouldRequireHTTPS()) {
347
347
if (!$request->isHTTPS()) {
348
+
349
+
// Don't redirect intracluster requests: doing so drops headers and
350
+
// parameters, imposes a performance penalty, and indicates a
351
+
// misconfiguration.
352
+
if ($request->isProxiedClusterRequest()) {
353
+
throw new AphrontMalformedRequestException(
354
+
pht('HTTPS Required'),
355
+
pht(
356
+
'This request reached a site which requires HTTPS, but the '.
357
+
'request is not marked as HTTPS.'));
358
+
}
359
+
348
360
$https_uri = $request->getRequestURI();
349
361
$https_uri->setDomain($request->getHost());
350
362
$https_uri->setProtocol('https');
+9
src/aphront/site/PhabricatorSite.php
+9
src/aphront/site/PhabricatorSite.php
···
3
3
abstract class PhabricatorSite extends AphrontSite {
4
4
5
5
public function shouldRequireHTTPS() {
6
+
// If this is an intracluster request, it's okay for it to use HTTP even
7
+
// if the site otherwise requires HTTPS. It is common to terminate SSL at
8
+
// a load balancer and use plain HTTP from then on, and administrators are
9
+
// usually not concerned about attackers observing traffic within a
10
+
// datacenter.
11
+
if (PhabricatorEnv::isClusterRemoteAddress()) {
12
+
return false;
13
+
}
14
+
6
15
return PhabricatorEnv::getEnvConfig('security.require-https');
7
16
}
8
17
+15
-8
src/applications/config/option/PhabricatorClusterConfigOptions.php
+15
-8
src/applications/config/option/PhabricatorClusterConfigOptions.php
···
34
34
PhabricatorEnv::getDoclink('Cluster: Databases'),
35
35
pht('Cluster: Databases')));
36
36
37
+
38
+
$intro_href = PhabricatorEnv::getDoclink('Clustering Introduction');
39
+
$intro_name = pht('Clustering Introduction');
40
+
37
41
return array(
38
42
$this->newOption('cluster.addresses', 'list<string>', array())
39
43
->setLocked(true)
40
44
->setSummary(pht('Address ranges of cluster hosts.'))
41
45
->setDescription(
42
46
pht(
43
-
'To allow Phabricator nodes to communicate with other nodes '.
44
-
'in the cluster, provide an address whitelist of hosts that '.
45
-
'are part of the cluster.'.
47
+
'Define a Phabricator cluster by providing a whitelist of host '.
48
+
'addresses that are part of the cluster.'.
46
49
"\n\n".
47
-
'Hosts on this whitelist are permitted to use special cluster '.
48
-
'mechanisms to authenticate requests. By default, these '.
49
-
'mechanisms are disabled.'.
50
+
'Hosts on this whitelist have special powers. These hosts are '.
51
+
'permitted to bend security rules, and misconfiguring this list '.
52
+
'can make your install less secure. For more information, '.
53
+
'see **[[ %s | %s ]]**.'.
50
54
"\n\n".
51
55
'Define a list of CIDR blocks which whitelist all hosts in the '.
52
-
'cluster. See the examples below for details.',
56
+
'cluster and no additional hosts. See the examples below for '.
57
+
'details.'.
53
58
"\n\n".
54
59
'When cluster addresses are defined, Phabricator hosts will also '.
55
-
'reject requests to interfaces which are not whitelisted.'))
60
+
'reject requests to interfaces which are not whitelisted.',
61
+
$intro_href,
62
+
$intro_name))
56
63
->addExample(
57
64
array(
58
65
'23.24.25.80/32',
+8
-5
src/applications/config/option/PhabricatorSecurityConfigOptions.php
+8
-5
src/applications/config/option/PhabricatorSecurityConfigOptions.php
···
80
80
pht(
81
81
"If the web server responds to both HTTP and HTTPS requests but ".
82
82
"you want users to connect with only HTTPS, you can set this ".
83
-
"to true to make Phabricator redirect HTTP requests to HTTPS.\n\n".
84
-
83
+
"to `true` to make Phabricator redirect HTTP requests to HTTPS.".
84
+
"\n\n".
85
85
"Normally, you should just configure your server not to accept ".
86
86
"HTTP traffic, but this setting may be useful if you originally ".
87
87
"used HTTP and have now switched to HTTPS but don't want to ".
88
88
"break old links, or if your webserver sits behind a load ".
89
89
"balancer which terminates HTTPS connections and you can not ".
90
-
"reasonably configure more granular behavior there.\n\n".
91
-
90
+
"reasonably configure more granular behavior there.".
91
+
"\n\n".
92
92
"IMPORTANT: Phabricator determines if a request is HTTPS or not ".
93
93
"by examining the PHP `%s` variable. If you run ".
94
94
"Apache/mod_php this will probably be set correctly for you ".
95
95
"automatically, but if you run Phabricator as CGI/FCGI (e.g., ".
96
96
"through nginx or lighttpd), you need to configure your web ".
97
97
"server so that it passes the value correctly based on the ".
98
-
"connection type.",
98
+
"connection type.".
99
+
"\n\n".
100
+
"If you configure Phabricator in cluster mode, note that this ".
101
+
"setting is ignored by intracluster requests.",
99
102
"\$_SERVER['HTTPS']"))
100
103
->setBoolOptions(
101
104
array(
+75
-1
src/docs/user/cluster/cluster.diviner
+75
-1
src/docs/user/cluster/cluster.diviner
···
30
30
and "Cluster Recipes" at the bottom of this document.
31
31
32
32
33
+
Preparing for Clustering
34
+
========================
35
+
36
+
To begin deploying Phabricator in cluster mode, set up `cluster.addresses`
37
+
in your configuration.
38
+
39
+
This option should contain a list of network addess blocks which are considered
40
+
to be part of the cluster. Hosts in this list are allowed to bend (or even
41
+
break) some of the security and policy rules when they make requests to other
42
+
hosts in the cluster, so this list should be as small as possible. See "Cluster
43
+
Whitelist Security" below for discussion.
44
+
45
+
If you are deploying hardware in EC2, a reasonable approach is to launch a
46
+
dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster,
47
+
and then deploy only Phabricator services into that VPC.
48
+
49
+
If you have additional auxiliary hosts which run builds and tests via Drydock,
50
+
you should //not// include them in the cluster address definition. For more
51
+
detailed discussion of the Drydock security model, see @{Drydock User Guide:
52
+
Security}.
53
+
54
+
Most other clustering features will not work until you define a cluster by
55
+
configuring `cluster.addresses`.
56
+
57
+
58
+
Cluster Whitelist Security
59
+
========================
60
+
61
+
When you configure `cluster.addresses`, you should keep the list of trusted
62
+
cluster hosts as small as possible. Hosts on this list gain additional
63
+
capabilities, including these:
64
+
65
+
**Trusted HTTP Headers**: Normally, Phabricator distrusts the load balancer
66
+
HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be
67
+
client-controlled and can be set to arbitrary values by an attacker if no load
68
+
balancer is deployed. In particular, clients can set `X-Forwarded-For` to any
69
+
value and spoof traffic from arbitrary remotes.
70
+
71
+
These headers are trusted when they are received from a host on the cluster
72
+
address whitelist. This allows requests from cluster loadbalancers to be
73
+
interpreted correctly by default without requiring additional custom code or
74
+
configuration.
75
+
76
+
**Intracluster HTTP**: Requests from cluster hosts are not required to use
77
+
HTTPS, even if `security.require-https` is enabled, because it is common to
78
+
terminate HTTPS on load balancers and use plain HTTP for requests within a
79
+
cluster.
80
+
81
+
**Special Authentication Mechanisms**: Cluster hosts are allowed to connect to
82
+
other cluster hosts with "root credentials", and to impersonate any user
83
+
account.
84
+
85
+
The use of root credentials is required because the daemons must be able to
86
+
bypass policies in order to function properly: they need to send mail about
87
+
private conversations and import commits in private repositories.
88
+
89
+
The ability to impersonate users is required because SSH nodes must receive,
90
+
interpret, modify, and forward SSH traffic. They can not use the original
91
+
credentials to do this because SSH authentication is asymmetric and they do not
92
+
have the user's private key. Instead, they use root credentials and impersonate
93
+
the user within the cluster.
94
+
95
+
These mechanisms are still authenticated (and use asymmetric keys, like SSH
96
+
does), so access to a host in the cluster address block does not mean that an
97
+
attacker can immediately compromise the cluster. However, an overbroad cluster
98
+
address whitelist may give an attacker who gains some access additional tools
99
+
to escalate access.
100
+
101
+
Note that if an attacker gains access to an actual cluster host, these extra
102
+
powers are largely moot. Most cluster hosts must be able to connect to the
103
+
master database to function properly, so the attacker will just do that and
104
+
freely read or modify whatever data they want.
105
+
106
+
33
107
Cluster: Databases
34
108
=================
35
109
···
39
113
disaster recovery.
40
114
41
115
Configuring replicas allows Phabricator to run in read-only mode if you lose
42
-
the master, and to quickly promote the replica as a replacement.
116
+
the master and to quickly promote the replica as a replacement.
43
117
44
118
For details, see @{article:Cluster: Databases}.
45
119
+12
src/docs/user/cluster/cluster_databases.diviner
+12
src/docs/user/cluster/cluster_databases.diviner
···
301
301
302
302
TODO: Make `bin/storage dump` replica-aware. See T10758.
303
303
304
+
With recent versions of MySQL, it is also possible to configure a //delayed//
305
+
replica which intentionally lags behind the master (say, by 12 hours). In the
306
+
event of a bad mutation, this could give you a larger window of time to
307
+
recognize the issue and recover the lost data from the delayed replica (which
308
+
might be quick) without needing to restore backups (which might be very slow).
309
+
310
+
Delayed replication is outside the scope of this document, but may be worth
311
+
considering as an additional data security step on top of backup snapshots
312
+
depending on your resources and needs. If you configure a delayed replica, do
313
+
not add it to the `cluster.databases` configuration: Phabricator should never
314
+
send traffic to it, and does not need to know about it.
315
+
304
316
305
317
Next Steps
306
318
==========
+26
src/docs/user/cluster/cluster_repositories.diviner
+26
src/docs/user/cluster/cluster_repositories.diviner
···
58
58
version check and fetch if necessary, then allow the write to continue.
59
59
60
60
61
+
HTTP vs HTTPS
62
+
=============
63
+
64
+
Intracluster requests (from the daemons to repository servers, or from
65
+
webservers to repository servers) are permitted to use HTTP, even if you have
66
+
set `security.require-https` in your configuration.
67
+
68
+
It is common to terminate SSL at a load balancer and use plain HTTP beyond
69
+
that, and the `security.require-https` feature is primarily focused on making
70
+
client browser behavior more convenient for users, so it does not apply to
71
+
intracluster traffic.
72
+
73
+
Using HTTP within the cluster leaves you vulnerable to attackers who can
74
+
observe traffic within a datacenter, or observe traffic between datacenters.
75
+
This is normally very difficult, but within reach for state-level adversaries
76
+
like the NSA.
77
+
78
+
If you are concerned about these attackers, you can terminate HTTPS on
79
+
repository hosts and bind to them with the "https" protocol. Just be aware that
80
+
the `security.require-https` setting won't prevent you from making
81
+
configuration mistakes, as it doesn't cover intracluster traffic.
82
+
83
+
Other mitigations are possible, but securing a network against the NSA and
84
+
similar agents of other rogue nations is beyond the scope of this document.
85
+
86
+
61
87
Backups
62
88
======
63
89