tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

sched: Update rq->avg_idle when a task is moved to an idle CPU

Currently, rq->idle_stamp is only used to calculate avg_idle during
wakeups. This means other paths that move a task to an idle CPU such as
fork/clone, execve, or migrations, do not end the CPU's idle status in
the scheduler's eyes, leading to an inaccurate avg_idle.

This patch introduces update_rq_avg_idle() to provide a more accurate
measurement of CPU idle duration. By invoking this helper in
put_prev_task_idle(), we ensure avg_idle is updated whenever a CPU
stops being idle, regardless of how the new task arrived.

Testing on an 80-core Ampere Altra (ARMv8) with 6.19-rc5 baseline:
- Hackbench : +7.2% performance gain at 16 threads.
- Schbench: Reduced p99.9 tail latencies at high concurrency.

Signed-off-by: Shubhang Kaushik <shubhang@os.amperecomputing.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Shubhang Kaushik <shubhang@os.amperecomputing.com>
Link: https://patch.msgid.link/20260121-v8-patch-series-v8-1-b7f1cbee5055@os.amperecomputing.com

authored by

Shubhang Kaushik and committed by
Peter Zijlstra 4b603f15 bb332a9e

+14 -12
+12 -12
kernel/sched/core.c
··· 3613 3613 trace_sched_wakeup(p); 3614 3614 } 3615 3615 3616 + void update_rq_avg_idle(struct rq *rq) 3617 + { 3618 + u64 delta = rq_clock(rq) - rq->idle_stamp; 3619 + u64 max = 2*rq->max_idle_balance_cost; 3620 + 3621 + update_avg(&rq->avg_idle, delta); 3622 + 3623 + if (rq->avg_idle > max) 3624 + rq->avg_idle = max; 3625 + rq->idle_stamp = 0; 3626 + } 3627 + 3616 3628 static void 3617 3629 ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, 3618 3630 struct rq_flags *rf) ··· 3659 3647 rq_unpin_lock(rq, rf); 3660 3648 p->sched_class->task_woken(rq, p); 3661 3649 rq_repin_lock(rq, rf); 3662 - } 3663 - 3664 - if (rq->idle_stamp) { 3665 - u64 delta = rq_clock(rq) - rq->idle_stamp; 3666 - u64 max = 2*rq->max_idle_balance_cost; 3667 - 3668 - update_avg(&rq->avg_idle, delta); 3669 - 3670 - if (rq->avg_idle > max) 3671 - rq->avg_idle = max; 3672 - 3673 - rq->idle_stamp = 0; 3674 3650 } 3675 3651 } 3676 3652
+1
kernel/sched/idle.c
··· 460 460 { 461 461 update_curr_idle(rq); 462 462 scx_update_idle(rq, false, true); 463 + update_rq_avg_idle(rq); 463 464 } 464 465 465 466 static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
+1
kernel/sched/sched.h
··· 1670 1670 1671 1671 #endif /* !CONFIG_FAIR_GROUP_SCHED */ 1672 1672 1673 + extern void update_rq_avg_idle(struct rq *rq); 1673 1674 extern void update_rq_clock(struct rq *rq); 1674 1675 1675 1676 /*