@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Clean up some old cluster-ish documentation
Summary:
Ref T10751. We currently have a placeholder Almanac document, and a fairly-bad-advice section in Daemons.
Pull these into the modern cluster documentation.
Test Plan: 17 phabricator PHDs
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10751
Differential Revision: https://secure.phabricator.com/D15689
+234
-69
+1
-2
src/applications/almanac/application/PhabricatorAlmanacApplication.php
+1
-2
src/applications/almanac/application/PhabricatorAlmanacApplication.php
+1
-1
src/applications/almanac/controller/AlmanacController.php
+1
-1
src/applications/almanac/controller/AlmanacController.php
+125
-1
src/docs/user/cluster/cluster.diviner
+125
-1
src/docs/user/cluster/cluster.diviner
···
26
26
The remainder of this document summarizes how to add redundancy to each
27
27
service and where your efforts are likely to have the greatest impact.
28
28
29
+
For additional guidance on setting up a cluster, see "Overlaying Services"
30
+
and "Cluster Recipes" at the bottom of this document.
31
+
29
32
30
33
Cluster: Databases
31
34
=================
···
44
47
Cluster: Repositories
45
48
=====================
46
49
47
-
Configuring multiple repository hosts is complex.
50
+
Configuring multiple repository hosts is complex, but is required before you
51
+
can add multiple daemon or web hosts.
48
52
49
53
Repository replicas are important for availability if you host repositories
50
54
on Phabricator, but less important if you host repositories elsewhere
···
55
59
the entire history.
56
60
57
61
For details, see @{article:Cluster: Repositories}.
62
+
63
+
64
+
Cluster: Daemons
65
+
================
66
+
67
+
Configuring multiple daemon hosts is straightforward, but you must configure
68
+
repositories first.
69
+
70
+
With daemons running on multiple hosts, you can transparently survive the loss
71
+
of any subset of hosts without an interruption to daemon services, as long as
72
+
at least one host remains alive. Daemons are stateless, so spreading daemons
73
+
across multiple hosts provides no resistance to data loss.
74
+
75
+
For details, see @{article:Cluster: Daemons}.
76
+
77
+
78
+
Cluster: Web Servers
79
+
====================
80
+
81
+
Configuring multiple web hosts is straightforward, but you must configure
82
+
repositories first.
83
+
84
+
With multiple web hosts, you can transparently survive the loss of any subset
85
+
of hosts as long as at least one host remains alive. Web hosts are stateless,
86
+
so putting multiple hosts in service provides no resistance to data loss.
87
+
88
+
For details, see @{article:Cluster: Web Servers}.
89
+
90
+
91
+
Overlaying Services
92
+
===================
93
+
94
+
Although hosts can run a single dedicated service type, certain groups of
95
+
services work well together. Phabricator clusters usually do not need to be
96
+
very large, so deploying a small number of hosts with multiple services is a
97
+
good place to start.
98
+
99
+
In planning a cluster, consider these blended host types:
100
+
101
+
**Everything**: Run HTTP, SSH, MySQL, repositories and daemons on a single
102
+
host. This is the starting point for single-node setups, and usually also the
103
+
best configuration when adding the second node.
104
+
105
+
**Everything Except Databases**: Run HTTP, SSH, repositories and daemons on one
106
+
host, and MySQL on a different host. MySQL uses many of the same resources that
107
+
other services use. It's also simpler to separate than other services, and
108
+
tends to benefit the most from dedicated hardware.
109
+
110
+
**Just Databases**: Separating MySQL onto dedicated nodes
111
+
112
+
Database nodes tend to benefit the most from
113
+
114
+
**Repositories and Daemons**: Run repositories and daemons on the same host.
115
+
Repository hosts //must// run daemons, and it normally makes sense to
116
+
completely overlay repositories and daemons. These services tend to use
117
+
different resources (repositories are heavier on I/O and lighter on CPU/RAM;
118
+
daemons are heavier on CPU/RAM and lighter on I/O).
119
+
120
+
Repositories and daemons are also both less latency sensitive than other
121
+
service types, so there's a wider margin of error for underprovisioning them
122
+
before performance is noticably affected.
123
+
124
+
These nodes tend to use system resources in a balanced way. Individual nodes
125
+
in this class do not need to be particularly powerful.
126
+
127
+
**Frontend Servers**: Run HTTP and SSH on the same host. These are easy to set
128
+
up, stateless, and you can scale the pool up or down easily to meet demand.
129
+
Routing both types of ingress traffic through the same initial tier can
130
+
simplify load balancing.
131
+
132
+
These nodes tend to need relatively little RAM.
133
+
134
+
135
+
Cluster Recipes
136
+
===============
137
+
138
+
This section provides some guidance on reasonable ways to scale up a cluster.
139
+
140
+
The smallest possible cluster is **two hosts**. Run everything (web, ssh,
141
+
database, repositories, and daemons) on each host. One host will serve as the
142
+
master; the other will serve as a replica.
143
+
144
+
Ideally, you should physically separate these hosts to reduce the chance that a
145
+
natural disaster or infrastructure disruption could disable or destroy both
146
+
hosts at the same time.
147
+
148
+
From here, you can choose how you expand the cluster.
149
+
150
+
To improve **scalability and performance**, separate loaded services onto
151
+
dedicated hosts and then add more hosts of that type to increase capacity. If
152
+
you have a two-node cluster, the best way to improve scalability by adding one
153
+
host is likely to separate the master database onto its own host.
154
+
155
+
Note that increasing scale may //decrease// availability by leaving you with
156
+
too little capacity after a failure. If you have three hosts handling traffic
157
+
and one datacenter fails, too much traffic may be sent to the single remaining
158
+
host in the surviving datacenter. You can hedge against this by mirroring new
159
+
hosts in other datacenters (for example, also separate the replica database
160
+
onto its own host).
161
+
162
+
After separating databases, separating repository + daemon nodes is likely
163
+
the next step.
164
+
165
+
To improve **availability**, add another copy of everything you run in one
166
+
datacenter to a new datacenter. For example, if you have a two-node cluster,
167
+
the best way to improve availability is to run everything on a third host in a
168
+
third datacenter. If you have a 6-node cluster with a web node, a database node
169
+
and a repo + daemon node in two datacenters, add 3 more nodes to create a copy
170
+
of each node in a third datacenter.
171
+
172
+
You can continue adding hosts until you run out of hosts.
173
+
174
+
175
+
Next Steps
176
+
==========
177
+
178
+
Continue by:
179
+
180
+
- learning how Phacility configures and operates a large, multi-tenant
181
+
production cluster in ((cluster)).
+59
src/docs/user/cluster/cluster_daemons.diviner
+59
src/docs/user/cluster/cluster_daemons.diviner
···
1
+
@title Cluster: Daemons
2
+
@group intro
3
+
4
+
Configuring Phabricator to use multiple daemon hosts.
5
+
6
+
Overview
7
+
========
8
+
9
+
WARNING: This feature is a very early prototype; the features this document
10
+
describes are mostly speculative fantasy.
11
+
12
+
You can run daemons on multiple hosts. The advantages of doing this are:
13
+
14
+
- you can completely survive the loss of multiple daemon hosts; and
15
+
- worker queue throughput may improve.
16
+
17
+
This configuration is simple, but you must configure repositories first. For
18
+
details, see @{article:Cluster: Repositories}.
19
+
20
+
Since repository hosts must run daemons anyway, you usually do not need to do
21
+
any additional work and can skip this entirely.
22
+
23
+
24
+
Adding Daemon Hosts
25
+
===================
26
+
27
+
After configuring repositories for clustering, launch daemons on every
28
+
repository host according to the documentation in
29
+
@{article:Cluster: Repositories}. These daemons are necessary: repositories
30
+
will not fetch, update, or synchronize properly without them.
31
+
32
+
If your repository clustering is redundant (you have at least two repsoitory
33
+
hosts), these daemons are also likely to be sufficient in most cases. If you
34
+
want to launch additional hosts anyway (for example, to increase queue capacity
35
+
for unusual workloads), see "Dedicated Daemon Hosts" below.
36
+
37
+
38
+
Dedicated Daemon Hosts
39
+
======================
40
+
41
+
You can launch additional daemon hosts without any special configuration.
42
+
Daemon hosts must be able to reach other hosts on the network, but do not need
43
+
to run any services (like HTTP or SSH). Simply deploy the Phabricator software
44
+
and configuration and start the daemons.
45
+
46
+
Normally, there is little reason to deploy dedicated daemon hosts. They can
47
+
improve queue capacity, but generally do not improve availability or increase
48
+
resistance to data loss on their own. Instead, consider deploying more
49
+
repository hosts: repository hosts run daemons, so this will increase queue
50
+
capacity but also improve repository availability and cluster resistance.
51
+
52
+
53
+
Next Steps
54
+
==========
55
+
56
+
Continue by:
57
+
58
+
- returning to @{article:Clustering Introduction}; or
59
+
- configuring repositories first with @{article:Cluster: Repositories}.
+42
src/docs/user/cluster/cluster_webservers.diviner
+42
src/docs/user/cluster/cluster_webservers.diviner
···
1
+
@title Cluster: Web Servers
2
+
@group intro
3
+
4
+
Configuring Phabricator to use multiple web servers.
5
+
6
+
Overview
7
+
========
8
+
9
+
WARNING: This feature is a very early prototype; the features this document
10
+
describes are mostly speculative fantasy.
11
+
12
+
You can run Phabricator on multiple web servers. The advantages of doing this
13
+
are:
14
+
15
+
- you can completely survive the loss of multiple web hosts; and
16
+
- performance and capacity may improve.
17
+
18
+
This configuration is simple, but you must configure repositories first. For
19
+
details, see @{article:Cluster: Repositories}.
20
+
21
+
22
+
Adding Web Hosts
23
+
================
24
+
25
+
After configuring repositories in cluster mode, you can add more web hosts
26
+
at any time: simply deploy the Phabricator software and configuration to a
27
+
host, start the web server, and then add the host to the load balancer pool.
28
+
29
+
Phabricator web servers are stateless, so you can pull them in and out of
30
+
production freely.
31
+
32
+
You may also want to run SSH services on these hosts, since the service is very
33
+
similar to HTTP, also stateless, and it may be simpler to load balance the
34
+
services together.
35
+
36
+
37
+
Next Steps
38
+
==========
39
+
40
+
Continue by:
41
+
42
+
- returning to @{article:Clustering Introduction}.
-50
src/docs/user/configuration/cluster.diviner
-50
src/docs/user/configuration/cluster.diviner
···
1
-
@title User Guide: Phabricator Clusters
2
-
@group config
3
-
4
-
Guide on scaling Phabricator across multiple machines.
5
-
6
-
Overview
7
-
========
8
-
9
-
IMPORTANT: Phabricator clustering is in its infancy and does not work at all
10
-
yet. This document is mostly a placeholder.
11
-
12
-
IMPORTANT: DO NOT CONFIGURE CLUSTER SERVICES UNLESS YOU HAVE **TWENTY YEARS OF
13
-
EXPERIENCE WITH PHABRICATOR** AND **A MINIMUM OF 17 PHABRICATOR PHDs**. YOU
14
-
WILL BREAK YOUR INSTALL AND BE UNABLE TO REPAIR IT.
15
-
16
-
See also @{article:Almanac User Guide}.
17
-
18
-
19
-
Managing Cluster Configuration
20
-
==============================
21
-
22
-
Cluster configuration is managed primarily from the **Almanac** application.
23
-
24
-
To define cluster services and create or edit cluster configuration, you must
25
-
have the **Can Manage Cluster Services** application permission in Almanac. If
26
-
you do not have this permission, all cluster services and all connected devices
27
-
will be locked and not editable.
28
-
29
-
The **Can Manage Cluster Services** permission is stronger than service and
30
-
device policies, and overrides them. You can never edit a cluster service if
31
-
you don't have this permission, even if the **Can Edit** policy on the service
32
-
itself is very permissive.
33
-
34
-
35
-
Locking Cluster Configuration
36
-
=============================
37
-
38
-
IMPORTANT: Managing cluster services is **dangerous** and **fragile**.
39
-
40
-
If you make a mistake, you can break your install. Because the install is
41
-
broken, you will be unable to load the web interface in order to repair it.
42
-
43
-
IMPORTANT: Currently, broken clusters must be repaired by manually fixing them
44
-
in the database. There are no instructions available on how to do this, and no
45
-
tools to help you. Do not configure cluster services.
46
-
47
-
If an attacker gains access to an account with permission to manage cluster
48
-
services, they can add devices they control as database servers. These servers
49
-
will then receive sensitive data and traffic, and allow the attacker to
50
-
escalate their access and completely compromise an install.
+6
-15
src/docs/user/configuration/managing_daemons.diviner
+6
-15
src/docs/user/configuration/managing_daemons.diviner
···
113
113
- See @{article:Diffusion User Guide} for details about tuning the repository
114
114
daemon.
115
115
116
-
== Multiple Machines ==
117
116
118
-
If you have multiple machines, you should use `phd launch` to tweak which
119
-
daemons launch, and split daemons across machines like this:
117
+
Multiple Hosts
118
+
==============
120
119
121
-
- `PhabricatorRepositoryPullLocalDaemon`: Run one copy on any machine.
122
-
On each web frontend which is not running a normal copy, run a copy
123
-
with the `--no-discovery` flag.
124
-
- `PhabricatorTriggerDaemon`: Run one copy on any machine.
125
-
- `PhabricatorTaskmasterDaemon`: Run as many copies as you need to keep
126
-
tasks from backing up. You can run them all on one machine or split them
127
-
across machines.
120
+
For information about running daemons on multiple hosts, see
121
+
@{article:Cluster: Daemons}.
128
122
129
-
A gratuitously wasteful install might have a dedicated daemon machine which
130
-
runs `phd start` with a large pool of taskmasters set in the config, and then
131
-
runs `phd launch PhabricatorRepositoryPullLocalDaemon -- --no-discovery` on each
132
-
web server. This is grossly excessive in normal cases.
133
123
134
-
= Next Steps =
124
+
Next Steps
125
+
==========
135
126
136
127
Continue by:
137
128