@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Add a repository import troubleshooting guide
Summary: Fixes T8615. Basic overview of the pipeline, where to look for errors, and how to identify/understand/fix/force your way through.
Test Plan: Read documentation.
Reviewers: btrahan, joshuaspence
Reviewed By: joshuaspence
Subscribers: epriestley
Maniphest Tasks: T8615
Differential Revision: https://secure.phabricator.com/D13376
+281
-4
+247
src/docs/user/field/repository_imports.diviner
+247
src/docs/user/field/repository_imports.diviner
···
1
+
@title Troubleshooting Repository Imports
2
+
@group fieldmanual
3
+
4
+
Guide to the troubleshooting repositories which import incompletely.
5
+
6
+
Overview
7
+
========
8
+
9
+
When you first import an external source code repository (or push new commits to
10
+
a hosted repository), Phabricator imports those commits in the background.
11
+
12
+
While a repository is initially importing, some features won't work. While
13
+
individual commits are importing, some of their metadata won't be available in
14
+
the web UI.
15
+
16
+
Sometimes, the import process may hang or fail to complete. This document can
17
+
help you understand the import process and troubleshoot problems with it.
18
+
19
+
20
+
Understanding the Import Pipeline
21
+
=================================
22
+
23
+
Phabricator first performs commit discovery on repositories. This examines
24
+
a repository and identifies all the commits in it at a very shallow level,
25
+
then creates stub objects for them. These stub objects primarily serve to
26
+
assign various internal IDs to each commit.
27
+
28
+
Commit discovery occurs in the update phase, and you can learn more about
29
+
updates in @{article:Diffusion User Guide: Repository Updates}.
30
+
31
+
After commits are discovered, background tasks are queued to actually import
32
+
commits. These tasks do things like look at commit messages, trigger mentions
33
+
and autoclose rules, cache changes, trigger Herald, publish feed stories and
34
+
email, and apply Owners rules. You can learn more about some of these steps in
35
+
@{article:Diffusion User Guide: Autoclose}.
36
+
37
+
Specifically, the import pipeline has four steps:
38
+
39
+
- **Message**: Parses the commit message and author metadata.
40
+
- **Change**: Caches the paths the commit affected.
41
+
- **Owners**: Runs Owners rules.
42
+
- **Herald**: Runs Herald rules and publishes notifications.
43
+
44
+
These steps run in sequence for each commit, but all discovered commits import
45
+
in parallel.
46
+
47
+
48
+
Identifying Missing Steps
49
+
=========================
50
+
51
+
There are a few major pieces of information you can look at to understand where
52
+
the import process is stuck.
53
+
54
+
First, to identify which commits have missing import steps, run this command:
55
+
56
+
```
57
+
phabricator/ $ ./bin/repository importing rXYZ
58
+
```
59
+
60
+
That will show what work remains to be done. Each line shows a commit which
61
+
is discovered but not imported, and the import steps that are remaining for
62
+
that commit. Generally, the commit is stuck on the first step in the list.
63
+
64
+
Second, load the Daemon Console (at `/daemon/` in the web UI). This will show
65
+
what work is currently being done and waiting to complete. The most important
66
+
sections are "Queued Tasks" (work waiting in queue) and "Leased Tasks"
67
+
(work currently being done).
68
+
69
+
Third, run this command to look at the daemon logs:
70
+
71
+
```
72
+
phabricator/ $ ./bin/phd log
73
+
```
74
+
75
+
This can show you any errors the daemons have encountered recently.
76
+
77
+
The next sections will walk through how to use this information to understand
78
+
and resolve the issue.
79
+
80
+
81
+
Handling Permanent Failures
82
+
===========================
83
+
84
+
Some commits can not be imported, which will permanently stop a repository from
85
+
fully importing. These are rare, but can be caused by unusual data in a
86
+
repository, version peculiarities, or bugs in the importer.
87
+
88
+
Permanent failures usually look like a small number of commits stuck on the
89
+
"Message" or "Change" steps in the output of `repository importing`. If you
90
+
have a larger number of commits, it is less likely that there are any permanent
91
+
problems.
92
+
93
+
In the Daemon console, permanent failures usually look like a small number of
94
+
tasks in "Leased Tasks" with a large failure count. These tasks are retrying
95
+
until they succeed, but a bug is permanently preventing them from succeeding,
96
+
so they'll rack up a large number of retries over time.
97
+
98
+
In the daemon log, these commits usually emit specific errors showing why
99
+
they're failing to import.
100
+
101
+
These failures are the easiest to identify and understand, and can often be
102
+
resolved quickly. Choose some failing commit from the output of `bin/repository
103
+
importing` and use this command to re-run any missing steps manually in the
104
+
foreground:
105
+
106
+
```
107
+
phabricator/ $ ./bin/repository reparse --importing --trace rXYZabcdef012...
108
+
```
109
+
110
+
This command is always safe to run, no matter what the actual root cause of
111
+
the problem is.
112
+
113
+
If this fails with an error, you've likely identified a problem with
114
+
Phabricator. Collect as much information as you can about what makes the commit
115
+
special and file a bug in the upstream by following the instructions in
116
+
@{article:Contributing Bug Reports}.
117
+
118
+
If the commit imports cleanly, this is more likely to be caused by some other
119
+
issue.
120
+
121
+
122
+
Handling Temporary Failures
123
+
===========================
124
+
125
+
Some commits may temporarily fail to import: perhaps the network or services
126
+
may have briefly been down, or some configuration wasn't quite right, or the
127
+
daemons were killed halfway through the work.
128
+
129
+
These commits will retry eventually and usually succeed, but some of the retry
130
+
time limits are very conserative (up to 24 hours) and you might not want to
131
+
wait that long.
132
+
133
+
In the Daemon console, temporarily failures usually look like tasks in the
134
+
"Leased Tasks" column with a large "Expires" value but a low "Failures" count
135
+
(usually 0 or 1). The "Expires" column is showing how long Phabricator is
136
+
waiting to retry these tasks.
137
+
138
+
In the daemon log, these temporary failures might have created log entries, but
139
+
might also not have. For example, if the failure was rooted in a network issue
140
+
that probably will create a log entry, but if the faiulre was rooted in the
141
+
daemons being abruptly killed that may not create a log entry.
142
+
143
+
You can follow the instructions from "Handling Permanent Failures" above to
144
+
reparse steps individually to look for an error that represents a root cause,
145
+
but sometimes this can happen because of some transient issue which won't be
146
+
identifiable.
147
+
148
+
The easiest way to fix this is to restart the daemons. When you restart
149
+
daemons, all task leases are immediately expired, so any tasks waiting for a
150
+
long time will run right away:
151
+
152
+
```
153
+
phabricator/ $ ./bin/phd restart
154
+
```
155
+
156
+
This command is always safe to run, no matter what the actual root cause of
157
+
the problem is.
158
+
159
+
After restarting the daemons, any pending tasks should be able to retry
160
+
immediately.
161
+
162
+
For more information on managing the daemons, see
163
+
@{article:Managing Daemons with phd}.
164
+
165
+
166
+
Forced Parsing
167
+
==============
168
+
169
+
In rare cases, the actual tasks may be lost from the task queue. Usually, they
170
+
have been stolen by gremlins or spritied away by ghosts, or someone may have
171
+
been too ambitious with running manual SQL commands and deleted a bunch of
172
+
extra things they shouldn't have.
173
+
174
+
There is no normal set of conditions under which this should occur, but you can
175
+
force Phabricator to re-queue the tasks to recover from it if it does occur.
176
+
177
+
This will look like missing steps in `repository importing`, but nothing in the
178
+
"Queued Tasks" or "Leased Tasks" sections of the daemon console. The daemon
179
+
logs will also be empty, since the tasks have vanished.
180
+
181
+
To re-queue parse tasks for a repository, run this command, which will queue
182
+
up all of the missing work in `repository importing`:
183
+
184
+
```
185
+
phabricator/ $ ./bin/repository reparse --importing --all rXYZ
186
+
```
187
+
188
+
This command may cause duplicate work to occur if you have misdiagnosed the
189
+
problem and the tasks aren't actually lost. For example, it could queue a
190
+
second task to perform publishing, which could cause Phabricator to send a
191
+
second copy of email about the commit. Other than that, it is safe to run even
192
+
if this isn't the problem.
193
+
194
+
After running this command, you should see tasks in "Queued Tasks" and "Leased
195
+
Tasks" in the console which correspond to the commits in `repository
196
+
importing`, and progress should resume.
197
+
198
+
199
+
Forced Imports
200
+
==============
201
+
202
+
In some cases, you might want to force a repository to be flagged as imported
203
+
even though the import isn't complete. The most common and reasonable case
204
+
where you might want to do this is if you've identified a permanent failure
205
+
with a small number of commits (maybe just one) and reported it upstream, and
206
+
are waiting for a fix. You might want to start using the repository immediately,
207
+
even if a few things can't import yet.
208
+
209
+
You should be cautious about doing this. The "importing" flag controls
210
+
publishing of notifications and email, so if you flag a repository as imported
211
+
but it still has a lot of work queued, it may send an enormous amount of email
212
+
as that work completes.
213
+
214
+
To mark a repository as imported even though it really isn't, run this
215
+
command:
216
+
217
+
```
218
+
phabricator/ $ ./bin/repository mark-imported rXYZ
219
+
```
220
+
221
+
If you do this by mistake, you can reverse it later by using the
222
+
`--mark-not-imported` flag.
223
+
224
+
225
+
General Tips
226
+
============
227
+
228
+
Broadly, `bin/repository` contains several useful debugging commands which
229
+
let you figure out where failures are occuring. You can add the `--trace` flag
230
+
to any command to get more details about what it is doing. For any command,
231
+
you can use `help` to learn more about what it does and which flag it takes:
232
+
233
+
```
234
+
phabricator/ $ bin/repository help <command>
235
+
```
236
+
237
+
In particular, you can use flags with the the `repository reparse` command to
238
+
manually run parse steps in the foreground, including re-running steps and
239
+
running steps out of order.
240
+
241
+
242
+
Next Steps
243
+
==========
244
+
245
+
Continue by:
246
+
247
+
- returning to the @{article:Diffusion User Guide}.
+18
-4
src/docs/user/userguide/diffusion.diviner
+18
-4
src/docs/user/userguide/diffusion.diviner
···
94
94
95
95
= Next Steps =
96
96
97
-
- Learn about creating a symbol index at
97
+
Continue by:
98
+
99
+
- learning how to creating a symbol index at
98
100
@{article:Diffusion User Guide: Symbol Indexes}; or
99
-
- set up repository hosting with
101
+
- setting up repository hosting with
100
102
@{article:Diffusion User Guide: Repository Hosting}; or
101
-
- understand daemons in detail with @{article:Managing Daemons with phd}; or
102
-
- give us feedback at @{article:Give Feedback! Get Support!}.
103
+
- managing repository hooks with
104
+
@{article:Diffusion User Guide: Commit Hooks}; or
105
+
- understanding daemons in more detail with
106
+
@{article:Managing Daemons with phd}.
107
+
108
+
If you're having trouble getting things working, these topic guides may be
109
+
helpful:
110
+
111
+
- get details about automatically closing tasks and revisions in response
112
+
to commits in @{article:Diffusion User Guide: Autoclose}; or
113
+
- understand how Phabricator updates repositories with
114
+
@{article:Diffusion User Guide: Repository Updates}; or
115
+
- fix issues with repository imports with
116
+
@{article:Troubleshooting Repository Imports}.
+8
src/docs/user/userguide/diffusion_autoclose.diviner
+8
src/docs/user/userguide/diffusion_autoclose.diviner
···
58
58
no containing branch was configured to autoclose.
59
59
- //Field Not Present// This commit was processed before we implemented
60
60
this diagnostic feature, and no information is available.
61
+
62
+
Next Steps
63
+
==========
64
+
65
+
Continue by:
66
+
67
+
- troubleshooting in greater depth with
68
+
@{article:Troubleshooting Repository Imports}.
+8
src/docs/user/userguide/diffusion_updates.diviner
+8
src/docs/user/userguide/diffusion_updates.diviner
···
113
113
114
114
To catch potential issues with permissions, run this command as the same user
115
115
that the daemons run as.
116
+
117
+
Next Steps
118
+
==========
119
+
120
+
Continue by:
121
+
122
+
- troubleshooting in greater depth with
123
+
@{article:Troubleshooting Repository Imports}.