sched_ext: Call wakeup_preempt() in local_dsq_post_enq()

There are several edge cases (see linked thread) where an IMMED task
can be left lingering on a local DSQ if an RT task swoops in at the
wrong time. All of these edge cases are due to rq->next_class being idle
even after dispatching a task to rq's local DSQ. We should bump
rq->next_class to &ext_sched_class as soon as we've inserted a task into
the local DSQ.

To optimize the common case of rq->next_class == &ext_sched_class,
only call wakeup_preempt() if rq->next_class is below EXT. If next_class
is EXT or above, wakeup_preempt() is a no-op anyway.

This lets us also simplify the preempt_curr() logic a bit since
wakeup_preempt() will call preempt_curr() for us if next_class is
below EXT.

Link: https://lore.kernel.org/all/DHZPHUFXB4N3.2RY28MUEWBNYK@google.com/
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

authored by

Kuba Piecuch and committed by

Tejun Heo 1 week ago 163f8b7f deb7b2f9

+39 -5

1 changed file

expand all

kernel

sched

ext.c

+39 -5

kernel/sched/ext.c

··· 1402 1402 struct task_struct *p, u64 enq_flags) 1403 1403 { 1404 1404 struct rq *rq = container_of(dsq, struct rq, scx.local_dsq); 1405 - bool preempt = false; 1406 1405 1407 1406 call_task_dequeue(sch, rq, p, 0); 1407 + 1408 + /* 1409 + * Note that @rq's lock may be dropped between this enqueue and @p 1410 + * actually getting on CPU. This gives higher-class tasks (e.g. RT) 1411 + * an opportunity to wake up on @rq and prevent @p from running. 1412 + * Here are some concrete examples: 1413 + * 1414 + * Example 1: 1415 + * 1416 + * We dispatch two tasks from a single ops.dispatch(): 1417 + * - First, a local task to this CPU's local DSQ; 1418 + * - Second, a local/remote task to a remote CPU's local DSQ. 1419 + * We must drop the local rq lock in order to finish the second 1420 + * dispatch. In that time, an RT task can wake up on the local rq. 1421 + * 1422 + * Example 2: 1423 + * 1424 + * We dispatch a local/remote task to a remote CPU's local DSQ. 1425 + * We must drop the remote rq lock before the dispatched task can run, 1426 + * which gives an RT task an opportunity to wake up on the remote rq. 1427 + * 1428 + * Both examples work the same if we replace dispatching with moving 1429 + * the tasks from a user-created DSQ. 1430 + * 1431 + * We must detect these wakeups so that we can re-enqueue IMMED tasks 1432 + * from @rq's local DSQ. scx_wakeup_preempt() serves exactly this 1433 + * purpose, but for it to be invoked, we must ensure that we bump 1434 + * @rq->next_class to &ext_sched_class if it's currently idle. 1435 + * 1436 + * wakeup_preempt() does the bumping, and since we only invoke it if 1437 + * @rq->next_class is below &ext_sched_class, it will also 1438 + * resched_curr(rq). 1439 + */ 1440 + if (sched_class_above(p->sched_class, rq->next_class)) 1441 + wakeup_preempt(rq, p, 0); 1408 1442 1409 1443 /* 1410 1444 * If @rq is in balance, the CPU is already vacant and looking for the 1411 1445 * next task to run. No need to preempt or trigger resched after moving 1412 1446 * @p into its local DSQ. 1447 + * Note that the wakeup_preempt() above may have already triggered 1448 + * a resched if @rq->next_class was idle. It's harmless, since 1449 + * need_resched is cleared immediately after task pick. 1413 1450 */ 1414 1451 if (rq->scx.flags & SCX_RQ_IN_BALANCE) 1415 1452 return; ··· 1454 1417 if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr && 1455 1418 rq->curr->sched_class == &ext_sched_class) { 1456 1419 rq->curr->scx.slice = 0; 1457 - preempt = true; 1458 - } 1459 - 1460 - if (preempt || sched_class_above(&ext_sched_class, rq->curr->sched_class)) 1461 1420 resched_curr(rq); 1421 + } 1462 1422 } 1463 1423 1464 1424 static void dispatch_enqueue(struct scx_sched *sch, struct rq *rq,

Configure Feed

Configure Feed