Make taskmaster consumption of failed tasks more FIFO-ey

Summary:
Ref T1049. See discussion in D7745. We have some specific interest in this for D7745, but generally we want to consume tasks with expired leases in roughly FIFO order, just like we consume new tasks in roughly FIFO order. Currently, when we select an expired task we order them by `id`, but this is the original insert order, not lease expiration order. Instead, order by `leaseExpires`.

This query is actually much better than the old one was, since the WHERE part is `leaseExpries < VALUE`.

Test Plan: Ran `EXPLAIN` on the query. Ran a taskmaster in debug mode and saw it lease new and expired tasks successfully.

Reviewers: hach-que, btrahan

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7746

epriestley 12 years ago 0ed281d2 8fd256a1

+40 -5

2 changed files

expand all

src

infrastructure

daemon

workers

__tests__

PhabricatorWorkerTestCase.php

query

PhabricatorWorkerLeaseQuery.php

+18 -1

src/infrastructure/daemon/workers/__tests__/PhabricatorWorkerTestCase.php

··· 49 49 $this->expectNextLease($task1); 50 50 } 51 51 52 - 53 52 public function testExecuteTask() { 54 53 $task = $this->scheduleAndExecuteTask(); 55 54 ··· 153 152 )); 154 153 155 154 $this->assertEqual(true, ($task->getLeaseExpires() - time()) > 1000); 155 + } 156 + 157 + public function testLeasedIsOldestFirst() { 158 + // Tasks which expired earlier should lease first, all else being equal. 159 + 160 + $task1 = $this->scheduleTask(); 161 + $task2 = $this->scheduleTask(); 162 + 163 + $task1->setLeaseOwner('test'); 164 + $task1->setLeaseExpires(time() - 100000); 165 + $task1->forceSaveWithoutLease(); 166 + 167 + $task2->setLeaseOwner('test'); 168 + $task2->setLeaseExpires(time() - 200000); 169 + $task2->forceSaveWithoutLease(); 170 + 171 + $this->expectNextLease($task2); 172 + $this->expectNextLease($task1); 156 173 } 157 174 158 175 private function expectNextLease($task) {

+22 -4

src/infrastructure/daemon/workers/query/PhabricatorWorkerLeaseQuery.php

··· 60 60 'SELECT id, leaseOwner FROM %T %Q %Q %Q', 61 61 $task_table->getTableName(), 62 62 $this->buildWhereClause($conn_w, $phase), 63 - $this->buildOrderClause($conn_w), 63 + $this->buildOrderClause($conn_w, $phase), 64 64 $this->buildLimitClause($conn_w, $limit - $leased)); 65 65 66 66 // NOTE: Sometimes, we'll race with another worker and they'll grab ··· 102 102 $task_table->getTableName(), 103 103 $taskdata_table->getTableName(), 104 104 $lease_ownership_name, 105 - $this->buildOrderClause($conn_w), 105 + $this->buildOrderClause($conn_w, $phase), 106 106 $this->buildLimitClause($conn_w, $limit)); 107 107 108 108 $tasks = $task_table->loadAllFromArray($data); ··· 188 188 189 189 } 190 190 191 - private function buildOrderClause(AphrontDatabaseConnection $conn_w) { 192 - return qsprintf($conn_w, 'ORDER BY id ASC'); 191 + private function buildOrderClause(AphrontDatabaseConnection $conn_w, $phase) { 192 + switch ($phase) { 193 + case self::PHASE_UNLEASED: 194 + // When selecting new tasks, we want to consume them in roughly 195 + // FIFO order, so we order by the task ID. 196 + return qsprintf($conn_w, 'ORDER BY id ASC'); 197 + case self::PHASE_EXPIRED: 198 + // When selecting failed tasks, we want to consume them in roughly 199 + // FIFO order of their failures, which is not necessarily their original 200 + // queue order. 201 + 202 + // Particularly, this is important for tasks which use soft failures to 203 + // indicate that they are waiting on other tasks to complete: we need to 204 + // push them to the end of the queue after they fail, at least on 205 + // average, so we don't deadlock retrying the same blocked task over 206 + // and over again. 207 + return qsprintf($conn_w, 'ORDER BY leaseExpires ASC'); 208 + default: 209 + throw new Exception(pht('Unknown phase "%s"!', $phase)); 210 + } 193 211 } 194 212 195 213 private function buildLimitClause(AphrontDatabaseConnection $conn_w, $limit) {

Configure Feed

Configure Feed