Make Drydock lease infrastructure more nimble

Summary:
Ref T9252. Currently, Harbormaster does this when trying to acquire a working copy:

- Ask for a working copy.
- Yield for 15 seconds.
- Check if we have a working copy yet.

That's OK, but Drydock takes ~1s to acquire a working copy lease if a resource is already available, so we end up doing this:

- T+0: Ask for a working copy.
- T+0: Yield for 15 seconds.
- T+1: Working copy lease activates.
- T+15: Working copy lease is used.
- T+16: Build finishes.

So we end up spending about 2 seconds doing work and 14 seconds sleeping.

One way to fix this would be to fiddle with the yield duration, so we yield for 1, 2, 4, ... seconds or something. This probably isn't a bad idea for longer leases (i.e., wait for 15, 30, 45 ... seconds or similar) but it implies a lot of churn for short leases.

Instead, let tasks "awaken" other tasks when they complete. The "awaken" operation means: if a task is in a yielded state (no failures, no owner, explicitly yielded, future expires time), pretend it only yielded until right now instead of whenever it really yielded to.

Basically, this rewrites history so that even though Harbormaster did a `yield(15)`, we pretend it did a `yield(4)` after we activate the lease if lease activation took 4 seconds.

If this misses, it's fine: we fall back to the normal yield behavior and things move forward normally a few seconds later.

If it hits, we get a more nimble process pretty cleanly.

Test Plan:
- Restarted a build plan (lease working copy + run `ls`) with this patch no-op'd, took about 16 seconds.
- Restarted a build plan with this patch active, took about 1 second.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14178

epriestley 10 years ago 9b29d46e cd2dd2a0

+106 -2

7 changed files

expand all

src

applications

drydock

storage

DrydockLease.php

DrydockResource.php

harbormaster

step

HarbormasterBuildStepImplementation.php

HarbormasterLeaseWorkingCopyBuildStepImplementation.php

worker

HarbormasterTargetWorker.php

infrastructure

daemon

workers

PhabricatorWorker.php

storage

PhabricatorWorkerActiveTask.php

+11 -1

src/applications/drydock/storage/DrydockLease.php

··· 334 334 ), 335 335 array( 336 336 'objectPHID' => $this->getPHID(), 337 - 'delayUntil' => $epoch, 337 + 'delayUntil' => ($epoch ? (int)$epoch : null), 338 338 )); 339 + } 340 + 341 + public function setAwakenTaskIDs(array $ids) { 342 + $this->setAttribute('internal.awakenTaskIDs', $ids); 343 + return $this; 339 344 } 340 345 341 346 private function didActivate() { ··· 358 363 $expires = $this->getUntil(); 359 364 if ($expires) { 360 365 $this->scheduleUpdate($expires); 366 + } 367 + 368 + $awaken_ids = $this->getAttribute('internal.awakenTaskIDs'); 369 + if (is_array($awaken_ids) && $awaken_ids) { 370 + PhabricatorWorker::awakenTaskIDs($awaken_ids); 361 371 } 362 372 } 363 373

+1 -1

src/applications/drydock/storage/DrydockResource.php

··· 218 218 ), 219 219 array( 220 220 'objectPHID' => $this->getPHID(), 221 - 'delayUntil' => $epoch, 221 + 'delayUntil' => ($epoch ? (int)$epoch : null), 222 222 )); 223 223 } 224 224

+10

src/applications/harbormaster/step/HarbormasterBuildStepImplementation.php

··· 6 6 abstract class HarbormasterBuildStepImplementation extends Phobject { 7 7 8 8 private $settings; 9 + private $currentWorkerTaskID; 10 + 11 + public function setCurrentWorkerTaskID($id) { 12 + $this->currentWorkerTaskID = $id; 13 + return $this; 14 + } 15 + 16 + public function getCurrentWorkerTaskID() { 17 + return $this->currentWorkerTaskID; 18 + } 9 19 10 20 public static function getImplementations() { 11 21 return id(new PhutilClassMapQuery())

src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php

··· 54 54 ->setAttribute('repositoryPHID', $repository_phid) 55 55 ->setAttribute('commit', $commit); 56 56 57 + $task_id = $this->getCurrentWorkerTaskID(); 58 + if ($task_id) { 59 + $lease->setAwakenTaskIDs(array($task_id)); 60 + } 61 + 57 62 $lease->queueForActivation(); 58 63 59 64 $build_target

src/applications/harbormaster/worker/HarbormasterTargetWorker.php

··· 59 59 } 60 60 61 61 $implementation = $target->getImplementation(); 62 + $implementation->setCurrentWorkerTaskID($this->getCurrentWorkerTaskID()); 62 63 $implementation->execute($build, $target); 63 64 64 65 $next_status = HarbormasterBuildTarget::STATUS_PASSED;

+75

src/infrastructure/daemon/workers/PhabricatorWorker.php

··· 8 8 private $data; 9 9 private static $runAllTasksInProcess = false; 10 10 private $queuedTasks = array(); 11 + private $currentWorkerTask; 11 12 12 13 // NOTE: Lower priority numbers execute first. The priority numbers have to 13 14 // have the same ordering that IDs do (lowest first) so MySQL can use a ··· 18 19 const PRIORITY_BULK = 3000; 19 20 const PRIORITY_IMPORT = 4000; 20 21 22 + /** 23 + * Special owner indicating that the task has yielded. 24 + */ 25 + const YIELD_OWNER = '(yield)'; 21 26 22 27 /* -( Configuring Retries and Failures )----------------------------------- */ 23 28 ··· 77 82 return null; 78 83 } 79 84 85 + public function setCurrentWorkerTask(PhabricatorWorkerTask $task) { 86 + $this->currentWorkerTask = $task; 87 + return $this; 88 + } 89 + 90 + public function getCurrentWorkerTask() { 91 + return $this->currentWorkerTask; 92 + } 93 + 94 + public function getCurrentWorkerTaskID() { 95 + $task = $this->getCurrentWorkerTask(); 96 + if (!$task) { 97 + return null; 98 + } 99 + return $task->getID(); 100 + } 101 + 80 102 abstract protected function doWork(); 81 103 82 104 final public function __construct($data) { ··· 104 126 $task_class, 105 127 $data, 106 128 $options = array()) { 129 + 130 + PhutilTypeSpec::checkMap( 131 + $options, 132 + array( 133 + 'priority' => 'optional int|null', 134 + 'objectPHID' => 'optional string|null', 135 + 'delayUntil' => 'optional int|null', 136 + )); 107 137 108 138 $priority = idx($options, 'priority'); 109 139 if ($priority === null) { ··· 206 236 */ 207 237 final public function getQueuedTasks() { 208 238 return $this->queuedTasks; 239 + } 240 + 241 + 242 + /** 243 + * Awaken tasks that have yielded. 244 + * 245 + * Reschedules the specified tasks if they are currently queued in a yielded, 246 + * unleased, unretried state so they'll execute sooner. This can let the 247 + * queue avoid unnecessary waits. 248 + * 249 + * This method does not provide any assurances about when these tasks will 250 + * execute, or even guarantee that it will have any effect at all. 251 + * 252 + * @param list<id> List of task IDs to try to awaken. 253 + * @return void 254 + */ 255 + final public static function awakenTaskIDs(array $ids) { 256 + if (!$ids) { 257 + return; 258 + } 259 + 260 + $table = new PhabricatorWorkerActiveTask(); 261 + $conn_w = $table->establishConnection('w'); 262 + 263 + // NOTE: At least for now, we're keeping these tasks yielded, just 264 + // pretending that they threw a shorter yield than they really did. 265 + 266 + // Overlap the windows here to handle minor client/server time differences 267 + // and because it's likely correct to push these tasks to the head of their 268 + // respective priorities. There is a good chance they are ready to execute. 269 + $window = phutil_units('1 hour in seconds'); 270 + $epoch_ago = (PhabricatorTime::getNow() - $window); 271 + 272 + queryfx( 273 + $conn_w, 274 + 'UPDATE %T SET leaseExpires = %d 275 + WHERE id IN (%Ld) 276 + AND leaseOwner = %s 277 + AND leaseExpires > %d 278 + AND failureCount = 0', 279 + $table->getTableName(), 280 + $epoch_ago, 281 + $ids, 282 + self::YIELD_OWNER, 283 + $epoch_ago); 209 284 } 210 285 211 286 }

src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php

··· 141 141 $worker = null; 142 142 try { 143 143 $worker = $this->getWorkerInstance(); 144 + $worker->setCurrentWorkerTask($this); 144 145 145 146 $maximum_failures = $worker->getMaximumRetryCount(); 146 147 if ($maximum_failures !== null) { ··· 174 175 $result->setExecutionException($ex); 175 176 } catch (PhabricatorWorkerYieldException $ex) { 176 177 $this->setExecutionException($ex); 178 + 179 + $this->setLeaseOwner(PhabricatorWorker::YIELD_OWNER); 177 180 178 181 $retry = $ex->getDuration(); 179 182 $retry = max($retry, 5);

Configure Feed

Configure Feed