Unify intracluster sync and Drydock working copy construction timeouts as a repository "copy time limit"

Summary:
Depends on D19814. Ref T13216. See PHI885. For various eldritch reasons, `git fetch` can hang. Although we'd probably like to fix this with `git fetch --require-sustained-network-transfer-rate=512KB/5s` or similar, that flag doesn't exist and we don't have a reasonable way to build it.

Short of that, move toward formalizing a repository "copy time limit": the longest amount of time anything may spend trying to make a copy of this repository.

This grows out of the existing intracluster sync limit, which is effectively the same thing. Here, apply it to `git clone` and `git fetch` in Drydock working copy construction, too. A future change may make it configurable.

Test Plan:
- Set the limit to 0.001.
- Tried to build and lease working copies, got sensible timeout errors (see D19815).

```
<Activation Failed> Lease activation failed: [CommandException] Command killed by timeout after running for more than 0.001 seconds.
COMMAND
ssh '-o' 'LogLevel=quiet' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l '********' -p '2222' -i '********' '127.0.0.1' -- '(cd '\''/var/drydock/workingcopy-163/repo/spellbook/'\'' && git clean -d --force && git fetch && git reset --hard)'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19816

epriestley 7 years ago cb033673 933462b4

+50 -14

5 changed files

expand all

src

applications

diffusion

protocol

DiffusionCommandEngine.php

drydock

blueprint

DrydockWorkingCopyBlueprintImplementation.php

storage

DrydockSlotLock.php

repository

storage

PhabricatorRepository.php

PhabricatorRepositoryURIIndex.php

+2 -1

src/applications/diffusion/protocol/DiffusionCommandEngine.php

··· 138 138 // See T13108. By default, don't let any cluster command run indefinitely 139 139 // to try to avoid cases where `git fetch` hangs for some reason and we're 140 140 // left sitting with a held lock forever. 141 - $future->setTimeout(phutil_units('15 minutes in seconds')); 141 + $repository = $this->getRepository(); 142 + $future->setTimeout($repository->getCopyTimeLimit()); 142 143 143 144 return $future; 144 145 }

+28 -6

src/applications/drydock/blueprint/DrydockWorkingCopyBlueprintImplementation.php

··· 173 173 174 174 $map = $resource->getAttribute('repositories.map'); 175 175 176 + $futures = array(); 176 177 $repositories = $this->loadRepositories(ipull($map, 'phid')); 177 178 foreach ($map as $directory => $spec) { 178 179 // TODO: Validate directory isn't goofy like "/etc" or "../../lol" ··· 181 182 $repository = $repositories[$spec['phid']]; 182 183 $path = "{$root}/repo/{$directory}/"; 183 184 184 - // TODO: Run these in parallel? 185 - $interface->execx( 185 + $future = $interface->getExecFuture( 186 186 'git clone -- %s %s', 187 187 (string)$repository->getCloneURIObject(), 188 188 $path); 189 + 190 + $future->setTimeout($repository->getCopyTimeLimit()); 191 + 192 + $futures[$directory] = $future; 193 + } 194 + 195 + foreach (new FutureIterator($futures) as $key => $future) { 196 + $future->resolvex(); 189 197 } 190 198 191 199 $resource ··· 240 248 $map = $lease->getAttribute('repositories.map'); 241 249 $root = $resource->getAttribute('workingcopy.root'); 242 250 251 + $repositories = $this->loadRepositories(ipull($map, 'phid')); 252 + 243 253 $default = null; 244 254 foreach ($map as $directory => $spec) { 255 + $repository = $repositories[$spec['phid']]; 256 + 245 257 $interface->pushWorkingDirectory("{$root}/repo/{$directory}/"); 246 258 247 259 $cmd = array(); ··· 271 283 $arg[] = $branch; 272 284 } 273 285 274 - $this->execxv($interface, $cmd, $arg); 286 + $this->newExecvFuture($interface, $cmd, $arg) 287 + ->setTimeout($repository->getCopyTimeLimit()) 288 + ->resolvex(); 275 289 276 290 if (idx($spec, 'default')) { 277 291 $default = $directory; ··· 295 309 $arg[] = $ref_ref; 296 310 297 311 try { 298 - $this->execxv($interface, $cmd, $arg); 312 + $this->newExecvFuture($interface, $cmd, $arg) 313 + ->setTimeout($repository->getCopyTimeLimit()) 314 + ->resolvex(); 299 315 } catch (CommandException $ex) { 300 316 $display_command = csprintf( 301 317 'git fetch %R %R', ··· 509 525 DrydockCommandInterface $interface, 510 526 array $commands, 511 527 array $arguments) { 528 + return $this->newExecvFuture($interface, $commands, $arguments)->resolvex(); 529 + } 530 + 531 + private function newExecvFuture( 532 + DrydockCommandInterface $interface, 533 + array $commands, 534 + array $arguments) { 512 535 513 536 $commands = implode(' && ', $commands); 514 537 $argv = array_merge(array($commands), $arguments); 515 538 516 - return call_user_func_array(array($interface, 'execx'), $argv); 539 + return call_user_func_array(array($interface, 'getExecFuture'), $argv); 517 540 } 518 - 519 541 520 542 }

+2 -2

src/applications/drydock/storage/DrydockSlotLock.php

··· 140 140 try { 141 141 queryfx( 142 142 $conn_w, 143 - 'INSERT INTO %T (ownerPHID, lockIndex, lockKey) VALUES %Q', 143 + 'INSERT INTO %T (ownerPHID, lockIndex, lockKey) VALUES %LQ', 144 144 $table->getTableName(), 145 - implode(', ', $sql)); 145 + $sql); 146 146 } catch (AphrontDuplicateKeyQueryException $ex) { 147 147 // Try to improve the readability of the exception. We might miss on 148 148 // this query if the lock has already been released, but most of the

+13

src/applications/repository/storage/PhabricatorRepository.php

··· 1890 1890 1891 1891 1892 1892 /** 1893 + * Time limit for cloning or copying this repository. 1894 + * 1895 + * This limit is used to timeout operations like `git clone` or `git fetch` 1896 + * when doing intracluster synchronization, building working copies, etc. 1897 + * 1898 + * @return int Maximum number of seconds to spend copying this repository. 1899 + */ 1900 + public function getCopyTimeLimit() { 1901 + return phutil_units('15 minutes in seconds'); 1902 + } 1903 + 1904 + 1905 + /** 1893 1906 * Retrieve the service URI for the device hosting this repository. 1894 1907 * 1895 1908 * See @{method:newConduitClient} for a general discussion of interacting

+5 -5

src/applications/repository/storage/PhabricatorRepositoryURIIndex.php

··· 48 48 49 49 queryfx( 50 50 $conn_w, 51 - 'DELETE FROM %T WHERE repositoryPHID = %s', 52 - $table->getTableName(), 51 + 'DELETE FROM %R WHERE repositoryPHID = %s', 52 + $table, 53 53 $repository_phid); 54 54 55 55 if ($sql) { 56 56 queryfx( 57 57 $conn_w, 58 - 'INSERT INTO %T (repositoryPHID, repositoryURI) VALUES %Q', 59 - $table->getTableName(), 60 - implode(', ', $sql)); 58 + 'INSERT INTO %R (repositoryPHID, repositoryURI) VALUES %LQ', 59 + $table, 60 + $sql); 61 61 } 62 62 63 63 $table->saveTransaction();

Configure Feed

Configure Feed