recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Free task leases on "phd start"

Summary:
Fixes T5154. Currently, "phd stop" terminates daemons relatively abruptly (and other things do too, like killing them). This can leave them with long leases that won't expire any time soon. Normally this isn't a big deal, since it just means an email or an import takes a bit longer (often 2 hours, but up to 24 hours) to run. However:

- We've increased default lease durations a lot fairly recently -- the 2 hours used to be 15 minutes.
- Harbormaster and Drydock add new types of tasks which are more dependent on other tasks, so waiting 2 hours for something to free up can hold up more stuff in queue.

When `phd start` is run, we can be confident (at least, in normal circumstances) that leases are safe to free, since we do a check. This undoes any damage done by abrupt stops in "phd stop" or by users or systems killing stuff.

(It would be nice to make "phd stop" more graceful at some point, but we always have to deal with abrupt termination in some cases no matter how gentle "phd stop" is.)

One sort-of-questionable thing here is that we don't distinguish between tasks which had an active lease and tasks which had been released, since the system itself does not make a distiction. So, for example, if you have a task that retries 5 times and waits an hour between retries, you'll get a retry on every `phd start` now, and could exhaust them all in a few minutes if you cycle `phd start` aggressively. I think this is OK. In the future, we could try to distinguish between these types of tasks, and only free the ones with active leases.

Test Plan:
- Used `phd start` normally, saw it free leases.
- Used `phd start`, killed it real quick so no taskmasters spawned, ran it again an saw no leases freed.
- Used `phd start --keep-leases`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5154

Differential Revision: https://secure.phabricator.com/D9256

epriestley 5e7b316f 890ae77a

+32 -3
+10 -2
src/applications/daemon/management/PhabricatorDaemonManagementStartWorkflow.php
··· 11 11 'Start the standard configured collection of Phabricator daemons. '. 12 12 'This is appropriate for most installs. Use **phd launch** to '. 13 13 'customize which daemons are launched.')) 14 - ->setArguments(array()); 14 + ->setArguments( 15 + array( 16 + array( 17 + 'name' => 'keep-leases', 18 + 'help' => pht( 19 + 'By default, **phd start** will free all task leases held by '. 20 + 'the daemons. With this flag, this step will be skipped.'), 21 + ), 22 + )); 15 23 } 16 24 17 25 public function execute(PhutilArgumentParser $args) { 18 - return $this->executeStartCommand(); 26 + return $this->executeStartCommand($args->getArg('keep-leases')); 19 27 } 20 28 21 29 }
+22 -1
src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php
··· 224 224 /* -( Commands )----------------------------------------------------------- */ 225 225 226 226 227 - protected function executeStartCommand() { 227 + protected function executeStartCommand($keep_leases = false) { 228 228 $console = PhutilConsole::getConsole(); 229 229 230 230 $running = $this->loadRunningDaemons(); ··· 244 244 $console->writeErr("%s\n", $message); 245 245 exit(1); 246 246 } 247 + } 248 + 249 + if ($keep_leases) { 250 + $console->writeErr("%s\n", pht('Not touching active task queue leases.')); 251 + } else { 252 + $console->writeErr("%s\n", pht('Freeing active task leases...')); 253 + $count = $this->freeActiveLeases(); 254 + $console->writeErr( 255 + "%s\n", 256 + pht('Freed %s task lease(s).', new PhutilNumber($count))); 247 257 } 248 258 249 259 $daemons = array( ··· 350 360 } 351 361 352 362 return 0; 363 + } 364 + 365 + private function freeActiveLeases() { 366 + $task_table = id(new PhabricatorWorkerActiveTask()); 367 + $conn_w = $task_table->establishConnection('w'); 368 + queryfx( 369 + $conn_w, 370 + 'UPDATE %T SET leaseExpires = UNIX_TIMESTAMP() 371 + WHERE leaseExpires > UNIX_TIMESTAMP()', 372 + $task_table->getTableName()); 373 + return $conn_w->getAffectedRows(); 353 374 } 354 375 355 376 }