@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Provide general documentation on how to use perfomance tools
Summary: Ref T8617. Provide general documentation with tools for debugging hangs and slow pages. Update DarkConsole docs and discuss how to use Services and XHProf. Explain what Multimeter is for and how to use it. Update XHProf docs and provide some usage hints.
Test Plan: Read documentation.
Reviewers: joshuaspence, btrahan
Reviewed By: joshuaspence, btrahan
Subscribers: joshuaspence, epriestley
Maniphest Tasks: T8617
Differential Revision: https://secure.phabricator.com/D13359
+574
-114
+9
src/applications/multimeter/application/PhabricatorMultimeterApplication.php
+9
src/applications/multimeter/application/PhabricatorMultimeterApplication.php
+3
src/docs/book/user.book
+3
src/docs/book/user.book
-60
src/docs/contributor/darkconsole.diviner
-60
src/docs/contributor/darkconsole.diviner
···
1
-
@title Using DarkConsole
2
-
@group developer
3
-
4
-
Enabling and using the built-in debugging console.
5
-
6
-
= Overview =
7
-
8
-
DarkConsole is a debugging console built into Phabricator which exposes
9
-
configuration, performance and error information. It can help you detect,
10
-
understand and resolve bugs and performance problems in Phabricator
11
-
applications.
12
-
13
-
DarkConsole was originally implemented as part of the Facebook Lite site; its
14
-
name is a bit of play on that (and a reference to the dark color palette its
15
-
design uses).
16
-
17
-
= Warning =
18
-
19
-
Because DarkConsole exposes some configuration and debugging information, it is
20
-
disabled by default (and **you should not enable it in production**). It has
21
-
some simple safeguards to prevent leaking credential information, but enabling
22
-
it in production may compromise the integrity of an install.
23
-
24
-
= Enabling DarkConsole =
25
-
26
-
You enable DarkConsole in your configuration, by setting `darkconsole.enabled`
27
-
to `true`, and then turning it on in `Settings` -> `Developer Settings`. Once
28
-
DarkConsole is enabled, you can show or hide it by pressing ##`## on your
29
-
keyboard.
30
-
31
-
Since the setting is not available to logged-out users, you can also set
32
-
`darkconsole.always-on` if you need to access DarkConsole on logged-out pages.
33
-
34
-
DarkConsole has a number of tabs, each of which is powered by a "plugin". You
35
-
can use them to access different debugging and performance features.
36
-
37
-
= Plugin: Error Log =
38
-
39
-
The "Error Log" plugin shows errors that occurred while generating the page,
40
-
similar to the httpd `error.log`. You can send information to the error log
41
-
explicitly with the @{function@libphutil:phlog} function.
42
-
43
-
If errors occurred, a red dot will appear on the plugin tab.
44
-
45
-
= Plugin: Request =
46
-
47
-
The "Request" plugin shows information about the HTTP request the server
48
-
received, and the server itself.
49
-
50
-
= Plugin: Services =
51
-
52
-
The "Services" plugin lists calls a page made to external services, like
53
-
MySQL and the command line.
54
-
55
-
= Plugin: XHProf =
56
-
57
-
The "XHProf" plugin gives you access to the XHProf profiler. To use it, you need
58
-
to install the corresponding PHP plugin -- see instructions in the
59
-
@{article:Installation Guide}. Once it is installed, you can use XHProf to
60
-
profile the runtime performance of a page.
-54
src/docs/contributor/installing_xhprof.diviner
-54
src/docs/contributor/installing_xhprof.diviner
···
1
-
@title Installing XHProf
2
-
@group developer
3
-
4
-
Describes how to install XHProf, a PHP profiling tool.
5
-
6
-
Overview
7
-
========
8
-
9
-
You can install XHProf to activate the XHProf tab in DarkConsole and the
10
-
`--xprofile` flag from the CLI. This will allow you to generate performance
11
-
profiles of pages and scripts, which can be tremendously valuable in identifying
12
-
and fixing slow code.
13
-
14
-
Installing XHProf
15
-
=================
16
-
17
-
XHProf is a PHP profiling tool. You don't need to install it unless you are
18
-
developing Phabricator and making performance changes.
19
-
20
-
You can install xhprof with:
21
-
22
-
$ pecl install xhprof
23
-
24
-
If you have a PEAR version prior to 1.9.3, you may run into a `phpize` failure.
25
-
If so, you can download the source and build it with:
26
-
27
-
$ cd extension/
28
-
$ phpize
29
-
$ ./configure
30
-
$ make
31
-
$ sudo make install
32
-
33
-
You may also need to add `extension=xhprof.so` to your php.ini.
34
-
35
-
See <https://bugs.php.net/bug.php?id=59747> for more information.
36
-
37
-
Using XHProf: Web
38
-
=================
39
-
40
-
To profile a web page, activate DarkConsole and navigate to the XHProf tab.
41
-
Use the **Profile Page** button to generate a profile.
42
-
43
-
Using XHProf: CLI
44
-
=================
45
-
46
-
From the command line, use the `--xprofile <filename>` flag to generate a
47
-
profile of any script.
48
-
49
-
Next Steps
50
-
==========
51
-
52
-
Continue by:
53
-
54
-
- enabling DarkConsole with @{article:Using DarkConsole}.
+162
src/docs/user/field/darkconsole.diviner
+162
src/docs/user/field/darkconsole.diviner
···
1
+
@title Using DarkConsole
2
+
@group fieldmanual
3
+
4
+
Enabling and using the built-in debugging and performance console.
5
+
6
+
Overview
7
+
========
8
+
9
+
DarkConsole is a debugging console built into Phabricator which exposes
10
+
configuration, performance and error information. It can help you detect,
11
+
understand and resolve bugs and performance problems in Phabricator
12
+
applications.
13
+
14
+
15
+
Security Warning
16
+
================
17
+
18
+
WARNING: Because DarkConsole exposes some configuration and debugging
19
+
information, it is disabled by default and you should be cautious about
20
+
enabling it in production.
21
+
22
+
Particularly, DarkConsole may expose some information about your session
23
+
details or other private material. It has some crude safeguards against this,
24
+
but does not completely sanitize output.
25
+
26
+
This is mostly a risk if you take screenshots or copy/paste output and share
27
+
it with others.
28
+
29
+
30
+
Enabling DarkConsole
31
+
====================
32
+
33
+
You enable DarkConsole in your configuration, by setting `darkconsole.enabled`
34
+
to `true`, and then turning it on in {nav Settings > Developer Settings}.
35
+
36
+
Once DarkConsole is enabled, you can show or hide it by pressing ##`## on your
37
+
keyboard.
38
+
39
+
Since the setting is not available to logged-out users, you can also set
40
+
`darkconsole.always-on` if you need to access DarkConsole on logged-out pages.
41
+
42
+
DarkConsole has a number of tabs, each of which is powered by a "plugin". You
43
+
can use them to access different debugging and performance features.
44
+
45
+
46
+
Plugin: Error Log
47
+
=================
48
+
49
+
The "Error Log" plugin shows errors that occurred while generating the page,
50
+
similar to the httpd `error.log`. You can send information to the error log
51
+
explicitly with the @{function@libphutil:phlog} function.
52
+
53
+
If errors occurred, a red dot will appear on the plugin tab.
54
+
55
+
56
+
Plugin: Request
57
+
===============
58
+
59
+
The "Request" plugin shows information about the HTTP request the server
60
+
received, and the server itself.
61
+
62
+
63
+
Plugin: Services
64
+
================
65
+
66
+
The "Services" plugin lists calls a page made to external services, like
67
+
MySQL and subprocesses.
68
+
69
+
The Services tab can help you understand and debug issues related to page
70
+
behavior: for example, you can use it to see exactly what queries or commands a
71
+
page is running. In some cases, you can re-run those queries or commands
72
+
yourself to examine their output and look for problems.
73
+
74
+
This tab can also be particularly useful in understanding page performance,
75
+
because many performance problems are caused by inefficient queries (queries
76
+
with bad query plans or which take too long) or repeated queries (queries which
77
+
could be better structured or benefit from caching).
78
+
79
+
When analyzing performance problems, the major things to look for are:
80
+
81
+
**Summary**: In the summary table at the top of the tab, are any categories
82
+
of events dominating the performance cost? For normal pages, the costs should
83
+
be roughly along these lines:
84
+
85
+
| Event Type | Approximate Cost |
86
+
|---|---|
87
+
| Connect | 1%-10% |
88
+
| Query | 10%-40% |
89
+
| Cache | 1% |
90
+
| Event | 1% |
91
+
| Conduit | 0%-80% |
92
+
| Exec | 0%-80% |
93
+
| All Services | 10%-75% |
94
+
| Entire Page | 100ms - 1000ms |
95
+
96
+
These ranges are rough, but should usually be what you expect from a page
97
+
summary. If any of these numbers are way off (for example, "Event" is taking
98
+
50% of runtime), that points toward a possible problem in that section of the
99
+
code, and can guide you to examining the related service calls more carefully.
100
+
101
+
**Duration**: In the Duration column, look for service calls that take a long
102
+
time. Sometimes these calls are just what the page is doing, but sometimes they
103
+
may indicate a problem.
104
+
105
+
Some questions that may help understanding this column are: are there a small
106
+
number of calls which account for a majority of the total page generation time?
107
+
Do these calls seem fundamental to the behavior of the page, or is it not clear
108
+
why they need to be made? Do some of them seem like they could be cached?
109
+
110
+
If there are queries which look slow, using the "Analyze Query Plans" button
111
+
may help reveal poor query plans.
112
+
113
+
Generally, this column can help pinpoint these kinds of problems:
114
+
115
+
- Queries or other service calls which are huge and inefficient.
116
+
- Work the page is doing which it could cache instead.
117
+
- Problems with network services.
118
+
- Missing keys or poor query plans.
119
+
120
+
**Repeated Calls**: In the "Details" column, look for service calls that are
121
+
being made over and over again. Sometimes this is normal, but usually it
122
+
indicates a call that can be batched or cached.
123
+
124
+
Some things to look for are: are similar calls being made over and over again?
125
+
Do calls mostly make sense given what the page is doing? Could any calls be
126
+
cached? Could multiple small calls be collected into one larger call? Are any
127
+
of the service calls clearly goofy nonsense that shouldn't be happening?
128
+
129
+
Generally, this column can help pinpoint these kinds of problems:
130
+
131
+
- Unbatched queries which should be batched (see
132
+
@{article:Performance: N+1 Query Problem}).
133
+
- Opportunities to improve performance with caching.
134
+
- General goofiness in how service calls are woking.
135
+
136
+
If the services tab looks fine, and particularly if a page is slow but the
137
+
"All Services" cost is small, that may indicate a problem in PHP. The best
138
+
tool to understand problems in PHP is XHProf.
139
+
140
+
141
+
Plugin: XHProf
142
+
==============
143
+
144
+
The "XHProf" plugin gives you access to the XHProf profiler. To use it, you need
145
+
to install the corresponding PHP plugin.
146
+
147
+
Once it is installed, you can use XHProf to profile the runtime performance of
148
+
a page. This will show you a detailed breakdown of where PHP spent time. This
149
+
can help find slow or inefficient application code, and is the most powerful
150
+
general-purpose performance tool available.
151
+
152
+
For instructions on installing and using XHProf, see @{article:Using XHProf}.
153
+
154
+
155
+
Next Steps
156
+
==========
157
+
158
+
Continue by:
159
+
160
+
- installing XHProf with @{article:Using XHProf}; or
161
+
- understanding and reporting performance issues with
162
+
@{article:Troubleshooting Performance Problems}.
+179
src/docs/user/field/performance.diviner
+179
src/docs/user/field/performance.diviner
···
1
+
@title Troubleshooting Performance Problems
2
+
@group fieldmanual
3
+
4
+
Guide to the troubleshooting slow pages and hangs.
5
+
6
+
Overview
7
+
========
8
+
9
+
This document describes how to isolate, examine, understand and resolve or
10
+
report performance issues like slow pages and hangs.
11
+
12
+
This document covers the general process for handling performance problems,
13
+
and outlines the major tools available for understanding them:
14
+
15
+
- **Multimeter** helps you understand sources of load and broad resource
16
+
utilization. This is a coarse, high-level tool.
17
+
- **DarkConsole** helps you dig into a specific slow page and understand
18
+
service calls. This is a general, mid-level tool.
19
+
- **XHProf** gives you detailed application performance profiles. This
20
+
is a fine-grained, low-level tool.
21
+
22
+
Performance and the Upstream
23
+
============================
24
+
25
+
Performance issues and hangs will often require upstream involvement to fully
26
+
resolve. The intent is for Phabricator to perform well in all reasonable cases,
27
+
not require tuning for different workloads (as long as those workloads are
28
+
generally reasonable). Poor performance with a reasonable workload is likely a
29
+
bug, not a configuration problem.
30
+
31
+
However, some pages are slow because Phabricator legitimately needs to do a lot
32
+
of work to generate them. For example, if you write a 100MB wiki document,
33
+
Phabricator will need substantial time to process it, it will take a long time
34
+
to download over the network, and your browser will proably not be able to
35
+
render it especially quickly.
36
+
37
+
We may be able to improve perfomance in some cases, but Phabricator is not
38
+
magic and can not wish away real complexity. The best solution to these problems
39
+
is usually to find another way to solve your problem: for example, maybe the
40
+
100MB document can be split into several smaller documents.
41
+
42
+
Here are some examples of performance problems under reasonable workloads that
43
+
the upstream can help resolve:
44
+
45
+
- {icon check, color=green} Commenting on a file and mentioning that same
46
+
file results in a hang.
47
+
- {icon check, color=green} Creating a new user takes many seconds.
48
+
- {icon check, color=green} Loading Feed hangs on 32-bit systems.
49
+
50
+
The upstream will be less able to help resolve unusual workloads with high
51
+
inherent complexity, like these:
52
+
53
+
- {icon times, color=red} A 100MB wiki page takes a long time to render.
54
+
- {icon times, color=red} A turing-complete simulation of Conway's Game of
55
+
Life implented in 958,000 Herald rules executes slowly.
56
+
- {icon times, color=red} Uploading an 8GB file takes several minutes.
57
+
58
+
Generally, the path forward will be:
59
+
60
+
- Follow the instructions in this document to gain the best understanding of
61
+
the issue (and of how to reproduce it) that you can.
62
+
- In particular, is it being caused by an unusual workload (like a 100MB
63
+
wiki page)? If so, consider other ways to solve the problem.
64
+
- File a report with the upstream by following the instructions in
65
+
@{article:Contributing Bug Reports}.
66
+
67
+
The remaining sections in this document walk through these steps.
68
+
69
+
70
+
Understanding Performance Problems
71
+
==================================
72
+
73
+
To isolate, examine, and understand performance problems, follow these steps:
74
+
75
+
**General Slowness**: If you are experiencing generally poor performance, use
76
+
Multimeter to understand resource usage and look for load-based causes. See
77
+
@{article:Multimeter User Guide}. If that isn't fruitful, treat this like a
78
+
reproducible performance problem on an arbitrary page.
79
+
80
+
**Hangs**: If you are experiencing hangs (pages which never return, or which
81
+
time out with a fatal after some number of seconds), they are almost always
82
+
the result of bugs in the upstream. Report them by following these
83
+
instructions:
84
+
85
+
- Set `debug.time-limit` to a value like `5`.
86
+
- Reproduce the hang. The page should exit after 5 seconds with a more useful
87
+
stack trace.
88
+
- File a report with the reproduction instructions and the stack trace in
89
+
the upstream. See @{article:Contributing Bug Reports} for detailed
90
+
instructions.
91
+
- Clear `debug.time-limit` again to take your install out of debug mode.
92
+
93
+
If part of the reproduction instructions include "Create a 100MB wiki page",
94
+
the upstream may be less sympathetic to your cause than if reproducing the
95
+
issue does not require an unusual, complex workload.
96
+
97
+
In some cases, the hang may really just a very large amount of processing time.
98
+
If you're very excited about 100MB wiki pages and don't mind waiting many
99
+
minutes for them to render, you may be able to adjust `max_execution_time` in
100
+
your PHP configuration to allow the process enough time to complete, or adjust
101
+
settings in your webserver config to let it wait longer for results.
102
+
103
+
**DarkConsole**: If you have a reproducible performance problem (for example,
104
+
loading a specific page is very slow), you can enable DarkConsole (a builtin
105
+
debugging console) to examine page performance in detail.
106
+
107
+
The two most useful tabs in DarkConsole are the "Services" tab and the
108
+
"XHProf" tab.
109
+
110
+
The "Services" module allows you to examine service calls (network calls,
111
+
subprocesses, events, etc) and find slow queries, slow services, inefficient
112
+
query plans, and unnecessary calls. Broadly, you're looking for slow or
113
+
repeated service calls, or calls which don't make sense given what the page
114
+
should be doing.
115
+
116
+
After installing XHProf (see @{article:Using XHProf}) you'll gain access to the
117
+
"XHProf" tab, which is a full tracing profiler. You can use the "Profile Page"
118
+
button to generate a complete trace of where a page is spending time. When
119
+
reading a profile, you're looking for the overall use of time, and for anything
120
+
which sticks out as taking unreasonably long or not making sense.
121
+
122
+
See @{article:Using DarkConsole} for complete instructions on configuring
123
+
and using DarkConsole.
124
+
125
+
**AJAX Requests**: To debug Ajax requests, activate DarkConsole and then turn
126
+
on the profiler or query analyzer on the main request by clicking the
127
+
appropriate button. The setting will cascade to Ajax requests made by the page
128
+
and they'll show up in the console with full query analysis or profiling
129
+
information.
130
+
131
+
**Command-Line Hangs**: If you have a script or daemon hanging, you can send
132
+
it `SIGHUP` to have it dump a stack trace to `sys_get_temp_dir()` (usually
133
+
`/tmp`).
134
+
135
+
Do this with:
136
+
137
+
```
138
+
$ kill -HUP <pid>
139
+
```
140
+
141
+
You can use this command to figure out where the system's temporary directory
142
+
is:
143
+
144
+
```
145
+
$ php -r 'echo sys_get_temp_dir()."\n";'
146
+
```
147
+
148
+
On most systems, this is `/tmp`. The trace should appear in that directory with
149
+
a name like `phabricator_backtrace_<pid>`. Examining this trace may provide
150
+
a key to understanding the problem.
151
+
152
+
**Command-Line Performance**: If you have general performance issues with
153
+
command-line scripts, you can add `--trace` to see a service call log. This is
154
+
similar to the "Services" tab in DarkConsole. This may help identify issues.
155
+
156
+
After installing XHProf, you can also add `--xprofile <filename>` to emit a
157
+
detailed performance profile. You can `arc upload` these files and then view
158
+
them in XHProf from the web UI.
159
+
160
+
Next Steps
161
+
==========
162
+
163
+
If you've done all you can to isolate and understand the problem you're
164
+
experiencing, report it to the upstream. Including as much relevant data as
165
+
you can, including:
166
+
167
+
- reproduction instructions;
168
+
- traces from `debug.time-limit` for hangs;
169
+
- screenshots of service call logs from DarkConsole (review these carefully,
170
+
as they can sometimes contain sensitive information);
171
+
- traces from CLI scripts with `--trace`;
172
+
- traces from sending HUP to processes; and
173
+
- XHProf profile files from `--xprofile` or "Download .xhprof Profile" in
174
+
the web UI.
175
+
176
+
After collecting this information:
177
+
178
+
- follow the instructions in @{article:Contributing Bug Reports} to file
179
+
a report in the upstream.
+122
src/docs/user/field/xhprof.diviner
+122
src/docs/user/field/xhprof.diviner
···
1
+
@title Using XHProf
2
+
@group fieldmanual
3
+
4
+
Describes how to install and use XHProf, a PHP profiling tool.
5
+
6
+
Overview
7
+
========
8
+
9
+
XHProf is a profiling tool which will let you understand application
10
+
performance in Phabricator.
11
+
12
+
After you install XHProf, you can use it from the web UI and the CLI to
13
+
generate detailed performance profiles. It is the most powerful tool available
14
+
for understanding application performance and identifying and fixing slow code.
15
+
16
+
Installing XHProf
17
+
=================
18
+
19
+
You are likely to have the most luck building XHProf from source:
20
+
21
+
$ git clone https://github.com/phacility/xhprof.git
22
+
23
+
From any source distribution of the extension, build and install it like this:
24
+
25
+
$ cd xhprof/
26
+
$ cd extension/
27
+
$ phpize
28
+
$ ./configure
29
+
$ make
30
+
$ sudo make install
31
+
32
+
You may also need to add `extension=xhprof.so` to your php.ini.
33
+
34
+
You can also try using PECL to install it, but this may not work well with
35
+
recent versions of PHP:
36
+
37
+
$ pecl install xhprof
38
+
39
+
Once you've installed it, `php -i` should report it as installed (you may
40
+
see a different version number, which is fine):
41
+
42
+
$ php -i | grep xhprof
43
+
...
44
+
xhprof => 0.9.2
45
+
...
46
+
47
+
48
+
Using XHProf: Web UI
49
+
====================
50
+
51
+
To profile a web page, activate DarkConsole and navigate to the XHProf tab.
52
+
Use the **Profile Page** button to generate a profile.
53
+
54
+
For instructions on activating DarkConsole, see @{article:Using DarkConsole}.
55
+
56
+
57
+
Using XHProf: CLI
58
+
=================
59
+
60
+
From the command line, use the `--xprofile <filename>` flag to generate a
61
+
profile of any script.
62
+
63
+
You can then upload this file to Phabricator (using `arc upload` may be easiest)
64
+
and view it in the web UI.
65
+
66
+
67
+
Analyzing Profiles
68
+
==================
69
+
70
+
Understanding profiles is as much art as science, so be warned that you may not
71
+
make much headway. Even if you aren't able to conclusively read a profile
72
+
yourself, you can attach profiles when submitting bug reports to the upstream
73
+
and we can look at them. This may yield new insight.
74
+
75
+
When looking at profiles, the "Wall Time (Inclusive)" column is usually the
76
+
most important. This shows the total amount of time spent in a function or
77
+
method and all of its children. Usually, to improve the performance of a page,
78
+
we're trying to find something that's slow and make it not slow: this column
79
+
can help identify which things are slowest.
80
+
81
+
The "Wall Time (Exclusive)" column shows time spent in a function or method,
82
+
excluding time spent in its children. This can give you hint about whether the
83
+
call itself is slow or it's just making calls to other things that are slow.
84
+
85
+
You can also get a sense of this by clicking a call to see its children, and
86
+
seeing if the bulk of runtime is spent in a child call. This tends to indicate
87
+
that you're looking at a problem which is deeper in the stack, and you need
88
+
to go down further to identify and understand it.
89
+
90
+
Conversely, if the "Wall Time (Exclusive)" column is large, or the children
91
+
of a call are all cheap, there's probably something expesive happening in the
92
+
call itself.
93
+
94
+
The "Count" column can also sometimes tip you off that something is amiss, if
95
+
a method which shouldn't be called very often is being called a lot.
96
+
97
+
Some general thing to look for -- these aren't smoking guns, but are unusual
98
+
and can lead to finding a performance issue:
99
+
100
+
- Is a low-level utility method like `phutil_utf8ize()` or `array_merge()`
101
+
taking more than a few percent of the page runtime?
102
+
- Do any methods (especially high-level methods) have >10,00 calls?
103
+
- Are we spending more than 100ms doing anything which isn't loading data
104
+
or rendering data?
105
+
- Does anything look suspiciously expensive or out of place?
106
+
- Is the profile for the slow page a lot different than the profile for a
107
+
fast page?
108
+
109
+
Some performance problems are obvious and will jump out of a profile; others
110
+
may require a more nuanced understanding of the codebase to sniff out which
111
+
parts are suspicious. If you aren't able to make progress with a profile,
112
+
report the issue upstream and attach the profile to your report.
113
+
114
+
115
+
Next Steps
116
+
==========
117
+
118
+
Continue by:
119
+
120
+
- enabling DarkConsole with @{article:Using DarkConsole}; or
121
+
- understanding and reporting performance problems with
122
+
@{article:Troubleshooting Performance Problems}.
+99
src/docs/user/userguide/multimeter.diviner
+99
src/docs/user/userguide/multimeter.diviner
···
1
+
@title Multimeter User Guide
2
+
@group userguide
3
+
4
+
Using Multimeter, a sampling profiler.
5
+
6
+
Overview
7
+
========
8
+
9
+
IMPORTANT: This document describes a prototype application.
10
+
11
+
Multimeter is a sampling profiler that can give you coarse information about
12
+
Phabricator resource usage. In particular, it can help quickly identify sources
13
+
of load, like bots or scripts which are making a very large number of requests.
14
+
15
+
Configuring and Using Multimeter
16
+
================================
17
+
18
+
To access Multimeter, go to {nav Applications > Multimeter}.
19
+
20
+
By default, Multimeter samples 0.1% of pages. This should be a reasonable rate
21
+
for most installs, but you can increase or decrease the rate by adjusting
22
+
`debug.sample-rate`. Increasing the rate (by setting the value to a lower
23
+
number, like 100, to sample 1% of pages) will increase the granualrity of the
24
+
data, at a small performance cost.
25
+
26
+
Using Multimeter
27
+
================
28
+
29
+
Multimeter shows you what Phabricator has spent time doing recently. By
30
+
looking at the samples it collects, you can identify major sources of load
31
+
or resource use, whether they are specific users, pages, subprocesses, or
32
+
other types of activity.
33
+
34
+
By identifying and understanding unexpected load, you can adjust usage patterns
35
+
or configuration to make better use of resources (for example, rewrite bots
36
+
that are making too many calls), or report specific, actionable issues to the
37
+
upstream for resolution.
38
+
39
+
The main screen of Multimeter shows you everything Phabricator has spent
40
+
resources on recently, broken down by action type. Categories are folded up
41
+
by default, with "(All)" labels.
42
+
43
+
To filter by a dimension, click the link for it. For example, from the main
44
+
page, you can click "Web Request" to filter by only web requests. To expand a
45
+
grouped dimension, click the "(All)" link.
46
+
47
+
For example, suppose we suspect that someone is running a bot that is making
48
+
a lot of requests and consuming a lot of resources. We can get a better idea
49
+
about this by filtering the results like this:
50
+
51
+
- Click {nav Web Request}. This will show only web requests.
52
+
- Click {nav (All)} under "Viewer". This will expand events by viewer.
53
+
54
+
Recent resource costs for web requests are now shown, grouped and sorted by
55
+
user. The usernames in the "Viewer" column show who is using resources, in
56
+
order from greatest use to least use (only administrators can see usernames).
57
+
58
+
The "Avg" column shows the average cost per event, while the "Cost" column
59
+
shows the total cost.
60
+
61
+
If the top few users account for similar costs and are normal, active users,
62
+
there may be nothing amiss and your problem might lie elsewhere. If a user like
63
+
`slowbot` is in the top few users and has way higher usage than anyone else,
64
+
there might be a script running under that account consuming a disproportionate
65
+
amount of resources.
66
+
67
+
Assuming you find a user with unusual usage, you could dig into their usage
68
+
like this:
69
+
70
+
- Click their name (like {nav slowbot}) to filter to just their requests.
71
+
- Click {nav (All)} under "Label". This expands by request detail.
72
+
73
+
This will show exactly what they spent those resources doing, and can help
74
+
identify if they're making a lot of API calls or scraping the site or whatever
75
+
else.
76
+
77
+
This is just an example of a specific kind of problem that Multimeter could
78
+
help resolve. In general, exploring Multimeter data by filtering and expanding
79
+
resource uses can help you understand how resources are used and identify
80
+
unexpected uses of resources. For example:
81
+
82
+
- Identify a problem with load balancing by filtering on {nav Web Request}
83
+
and expanding on {nav Host}. If hosts aren't roughly even, DNS or a load
84
+
balancer are misconfigured.
85
+
- Identify which pages cost the most by filtering on {nav Web Request}
86
+
and expanding on {nav Label}.
87
+
- Find outlier pages by filtering on {nav Web Request} and expanding on
88
+
{nav ID}.
89
+
- Find where subprocess are invoked from by filtering on {nav Subprocesses},
90
+
then expanding on {nav Context}.
91
+
92
+
93
+
Next Steps
94
+
==========
95
+
96
+
Continue by:
97
+
98
+
- understanding and reporting performance issues with
99
+
@{article:Troubleshooting Performance Problems}.