@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Adjust and wordsmith Search documentation
Summary:
Ref T12450. General adjustments:
- Try to make "Cluster: Search" more about "stuff in common + types" instead of pretty much all being Elastic-specific, so we can add Solr or whatever later.
- Provide guidance about rebuilding indexes after making a change.
- Simplify the basic examples, then provide a more advanced example at the ed.
- Really try to avoid suggesting anyone configure Elasticsearch ever for any reason.
Test Plan: Read documents, previewed in remarkup.
Reviewers: chad, 20after4
Reviewed By: 20after4
Maniphest Tasks: T12450
Differential Revision: https://secure.phabricator.com/D17602
+191
-69
+7
-19
src/docs/user/cluster/cluster.diviner
+7
-19
src/docs/user/cluster/cluster.diviner
···
47
47
| **SSH Servers** | Minimal | Low | No Risk | Low
48
48
| **Web Servers** | Minimal | **High** | No Risk | Moderate
49
49
| **Notifications** | Minimal | Low | No Risk | Low
50
-
| **Fulltext Search** | Moderate | **High** | Minimal Risk | Moderate
50
+
| **Fulltext Search** | Minimal | Low | No Risk | Low
51
51
52
52
See below for a walkthrough of these services in greater detail.
53
53
···
241
241
Cluster: Fulltext Search
242
242
========================
243
243
244
-
At a certain scale, you may begin to bump up against the limitations of MySQL's
245
-
built-in fulltext search capabilities. We have seen this with very large
246
-
installations with several million objects in the database and very many
247
-
simultaneous requests. At this point you may consider adding Elasticsearch
248
-
hosts to your cluster to reduce the load on your MySQL hosts.
249
-
250
-
Elasticsearch has the ability to spread the load across multiple hosts and can
251
-
handle very large indexes by sharding.
252
-
253
-
Search does not involve any risk of data lost because it's always possible to
254
-
rebuild the search index from the original database objects. This process can
255
-
be very time consuming, however, especially when the database grows very large.
244
+
Configuring search services is relatively simple and has no pre-requisites.
256
245
257
-
With multiple Elasticsearch hosts, you can survive the loss of a single host
258
-
with minimal disruption as Phabricator will detect the problem and direct
259
-
queries to one of the remaining hosts.
246
+
By default, Phabricator uses MySQL as a fulltext search engine, so deploying
247
+
multiple database hosts will effectively also deploy multiple fulltext search
248
+
hosts.
260
249
261
-
Phabricator supports writing to multiple indexing servers. This Simplifies
262
-
Elasticsearch upgrades and makes it possible to recover more quickly from
263
-
problems with the search index.
250
+
Search indexes can be completely rebuilt from the database, so there is no
251
+
risk of data loss no matter how fulltext search is configured.
264
252
265
253
For details, see @{article:Cluster: Search}.
266
254
+184
-50
src/docs/user/cluster/cluster_search.diviner
+184
-50
src/docs/user/cluster/cluster_search.diviner
···
4
4
Overview
5
5
========
6
6
7
-
You can configure phabricator to connect to one or more fulltext search clusters
8
-
running either Elasticsearch or MySQL. By default and without further
9
-
configuration, Phabricator will use MySQL for fulltext search. This will be
10
-
adequate for the vast majority of users. Installs with a very large number of
11
-
objects or specialized search needs can consider enabling Elasticsearch for
12
-
better scalability and potentially better search results.
7
+
You can configure Phabricator to connect to one or more fulltext search
8
+
services.
9
+
10
+
By default, Phabricator will use MySQL for fulltext search. This is suitable
11
+
for most installs. However, alternate engines are supported.
12
+
13
13
14
14
Configuring Search Services
15
15
===========================
16
16
17
-
To configure an Elasticsearch service, use the `cluster.search` configuration
18
-
option. A typical Elasticsearch configuration will probably look similar to
19
-
the following example:
17
+
To configure search services, adjust the `cluster.search` configuration
18
+
option. This option contains a list of one or more fulltext search services,
19
+
like this:
20
+
21
+
```lang=json
22
+
[
23
+
{
24
+
"type": "...",
25
+
"hosts": [
26
+
...
27
+
],
28
+
"roles": {
29
+
"read": true,
30
+
"write": true
31
+
}
32
+
}
33
+
]
34
+
```
35
+
36
+
When a user makes a change to a document, Phabricator writes the updated
37
+
document into every configured, writable fulltext service.
38
+
39
+
When a user issues a query, Phabricator tries configured, readable services
40
+
in order until it is able to execute the query successfully.
41
+
42
+
These options are supported by all service types:
43
+
44
+
| Key | Description |
45
+
|---|---|
46
+
| `type` | Constant identifying the service type, like `mysql`.
47
+
| `roles` | Dictionary of role settings, for enabling reads and writes.
48
+
| `hosts` | List of hosts for this service.
49
+
50
+
Some service types support additional options.
51
+
52
+
Available Service Types
53
+
=======================
54
+
55
+
These service types are supported:
56
+
57
+
| Service | Key | Description |
58
+
|---|---|---|
59
+
| MySQL | `mysql` | Default MySQL fulltext index.
60
+
| Elasticsearch | `elasticsearch` | Use an external Elasticsearch service
61
+
62
+
63
+
Fulltext Service Roles
64
+
======================
65
+
66
+
These roles are supported:
67
+
68
+
| Role | Key | Description
69
+
|---|---|---|
70
+
| Read | `read` | Allows the service to be queried when users search.
71
+
| Write | `write` | Allows documents to be published to the service.
72
+
73
+
74
+
Specifying Hosts
75
+
================
76
+
77
+
The `hosts` key should contain a list of dictionaries, each specifying the
78
+
details of a host. A service should normally have one or more hosts.
79
+
80
+
When an option is set at the service level, it serves as a default for all
81
+
hosts. It may be overridden by changing the value for a particular host.
82
+
83
+
84
+
Service Type: MySQL
85
+
==============
86
+
87
+
The `mysql` service type does not require any configuration, and does not
88
+
need to have hosts specified. This service uses the builtin database to
89
+
index and search documents.
90
+
91
+
A typical `mysql` service configuration looks like this:
20
92
21
93
```lang=json
22
94
{
23
-
"cluster.search": [
24
-
{
25
-
"type": "elasticsearch",
26
-
"hosts": [
27
-
{
28
-
"host": "127.0.0.1",
29
-
"roles": { "write": true, "read": true }
30
-
}
31
-
],
32
-
"port": 9200,
33
-
"protocol": "http",
34
-
"path": "/phabricator",
35
-
"version": 5
36
-
},
37
-
],
95
+
"type": "mysql"
38
96
}
39
97
```
40
98
41
-
Supported Options
42
-
-----------------
43
-
| Key | Type |Comments|
44
-
|`type` | String |Engine type. Currently, 'elasticsearch' or 'mysql'|
45
-
|`protocol`| String |Either 'http' or 'https'|
46
-
|`port`| Int |The TCP port that Elasticsearch is bound to|
47
-
|`path`| String |The path portion of the url for phabricator's index.|
48
-
|`version`| Int |The version of Elasticsearch server. Supports either 2 or 5.|
49
-
|`hosts`| List |A list of one or more Elasticsearch host names / addresses.|
50
99
51
-
Host Configuration
52
-
------------------
53
-
Each search service must have one or more hosts associated with it. Each host
54
-
entry consists of a `host` key, a dictionary of roles and can optionally
55
-
override any of the options that are valid at the service level (see above).
100
+
Service Type: Elasticsearch
101
+
======================
56
102
57
-
Currently supported roles are `read` and `write`. These can be individually
58
-
enabled or disabled on a per-host basis. A typical setup might include two
59
-
elasticsearch clusters in two separate datacenters. You can configure one
60
-
cluster for reads and both for writes. When one cluster is down for maintenance
61
-
you can simply swap the read role over to the backup cluster and then proceed
62
-
with maintenance without any service interruption.
103
+
The `elasticsearch` sevice type supports these options:
104
+
105
+
| Key | Description |
106
+
|---|---|
107
+
| `protocol` | Either `"http"` (default) or `"https"`.
108
+
| `port` | Elasticsearch TCP port.
109
+
| `version` | Elasticsearch version, either `2` or `5` (default).
110
+
| `path` | Path for the index. Defaults to `/phabriator`. Advanced.
111
+
112
+
A typical `elasticsearch` service configuration looks like this:
113
+
114
+
```lang=json
115
+
{
116
+
"type": "elasticsearch",
117
+
"hosts": [
118
+
{
119
+
"protocol": "http",
120
+
"host": "127.0.0.1",
121
+
"port": 9200
122
+
}
123
+
]
124
+
}
125
+
```
63
126
64
127
Monitoring Search Services
65
128
==========================
66
129
67
-
You can monitor fulltext search in {nav Config > Search Servers}. This interface
68
-
shows you a quick overview of services and their health.
130
+
You can monitor fulltext search in {nav Config > Search Servers}. This
131
+
interface shows you a quick overview of services and their health.
69
132
70
133
The table on this page shows some basic stats for each configured service,
71
134
followed by the configuration and current status of each host.
72
135
73
-
NOTE: This page runs its diagnostics //from the web server that is serving the
74
-
request//. If you are recovering from a disaster, the view this page shows
75
-
may be partial or misleading, and two requests served by different servers may
76
-
see different views of the cluster.
136
+
137
+
Rebuilding Indexes
138
+
==================
139
+
140
+
After adding new search services, you will need to rebuild document indexes
141
+
on them. To do this, first initialize the services:
142
+
143
+
```
144
+
phabricator/ $ ./bin/search init
145
+
```
146
+
147
+
This will perform index setup steps and other one-time configuration.
148
+
149
+
To populate documents in all indexes, run this command:
150
+
151
+
```
152
+
phabricator/ $ ./bin/search index --force --background --type all
153
+
```
154
+
155
+
This initiates an exhaustive rebuild of the document indexes. To get a more
156
+
detailed list of indexing options available, run:
157
+
158
+
```
159
+
phabricator/ $ ./bin/search help index
160
+
```
161
+
162
+
163
+
Advanced Example
164
+
================
165
+
166
+
This is a more advanced example which shows a configuration with multiple
167
+
different services in different roles. In this example:
168
+
169
+
- Phabricator is using an Elasticsearch 2 service as its primary fulltext
170
+
service.
171
+
- An Elasticsearch 5 service is online, but only receiving writes.
172
+
- The MySQL service is serving as a backup if Elasticsearch fails.
173
+
174
+
This particular configuration may not be very useful. It is primarily
175
+
intended to show how to configure many different options.
176
+
177
+
178
+
```lang=json
179
+
[
180
+
{
181
+
"type": "elasticsearch",
182
+
"version": 2,
183
+
"hosts": [
184
+
{
185
+
"host": "elastic2.mycompany.com",
186
+
"port": 9200,
187
+
"protocol": "http"
188
+
}
189
+
]
190
+
},
191
+
{
192
+
"type": "elasticsearch",
193
+
"version": 5,
194
+
"hosts": [
195
+
{
196
+
"host": "elastic5.mycompany.com",
197
+
"port": 9789,
198
+
"protocol": "https"
199
+
"roles": {
200
+
"read": false,
201
+
"write": true
202
+
}
203
+
}
204
+
]
205
+
},
206
+
{
207
+
"type": "mysql"
208
+
}
209
+
]
210
+
```