sched_ext: Implement scx_bpf_now()

Returns a high-performance monotonically non-decreasing clock for the current
CPU. The clock returned is in nanoseconds.

It provides the following properties:

1) High performance: Many BPF schedulers call bpf_ktime_get_ns() frequently
to account for execution time and track tasks' runtime properties.
Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which
eventually reads a hardware timestamp counter -- is neither performant nor
scalable. scx_bpf_now() aims to provide a high-performance clock by
using the rq clock in the scheduler core whenever possible.

2) High enough resolution for the BPF scheduler use cases: In most BPF
scheduler use cases, the required clock resolution is lower than the most
accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now() basically
uses the rq clock in the scheduler core whenever it is valid. It considers
that the rq clock is valid from the time the rq clock is updated
(update_rq_clock) until the rq is unlocked (rq_unpin_lock).

3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now()
guarantees the clock never goes backward when comparing them in the same
CPU. On the other hand, when comparing clocks in different CPUs, there
is no such guarantee -- the clock can go backward. It provides a
monotonically *non-decreasing* clock so that it would provide the same
clock values in two different scx_bpf_now() calls in the same CPU
during the same period of when the rq clock is valid.

An rq clock becomes valid when it is updated using update_rq_clock()
and invalidated when the rq is unlocked using rq_unpin_lock().

Let's suppose the following timeline in the scheduler core:

T1. rq_lock(rq)
T2. update_rq_clock(rq)
T3. a sched_ext BPF operation
T4. rq_unlock(rq)
T5. a sched_ext BPF operation
T6. rq_lock(rq)
T7. update_rq_clock(rq)

For [T2, T4), we consider that rq clock is valid (SCX_RQ_CLK_VALID is
set), so scx_bpf_now() calls during [T2, T4) (including T3) will
return the rq clock updated at T2. For duration [T4, T7), when a BPF
scheduler can still call scx_bpf_now() (T5), we consider the rq clock
is invalid (SCX_RQ_CLK_VALID is unset at T4). So when calling
scx_bpf_now() at T5, we will return a fresh clock value by calling
sched_clock_cpu() internally. Also, to prevent getting outdated rq clocks
from a previous scx scheduler, invalidate all the rq clocks when unloading
a BPF scheduler.

One example of calling scx_bpf_now(), when the rq clock is invalid
(like T5), is in scx_central [1]. The scx_central scheduler uses a BPF
timer for preemptive scheduling. In every msec, the timer callback checks
if the currently running tasks exceed their timeslice. At the beginning of
the BPF timer callback (central_timerfn in scx_central.bpf.c), scx_central
gets the current time. When the BPF timer callback runs, the rq clock could
be invalid, the same as T5. In this case, scx_bpf_now() returns a fresh
clock value rather than returning the old one (T2).

[1] https://github.com/sched-ext/scx/blob/main/scheds/c/scx_central.bpf.c

Signed-off-by: Changwoo Min <changwoo@igalia.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

authored by

Changwoo Min and committed by

Tejun Heo 1 year ago 3a9910b5 ea9b2626

+101 -4

3 changed files

expand all

kernel

sched

core.c

ext.c

sched.h

+5 -1

kernel/sched/core.c

··· 789 789 void update_rq_clock(struct rq *rq) 790 790 { 791 791 s64 delta; 792 + u64 clock; 792 793 793 794 lockdep_assert_rq_held(rq); 794 795 ··· 801 800 SCHED_WARN_ON(rq->clock_update_flags & RQCF_UPDATED); 802 801 rq->clock_update_flags |= RQCF_UPDATED; 803 802 #endif 803 + clock = sched_clock_cpu(cpu_of(rq)); 804 + scx_rq_clock_update(rq, clock); 804 805 805 - delta = sched_clock_cpu(cpu_of(rq)) - rq->clock; 806 + delta = clock - rq->clock; 806 807 if (delta < 0) 807 808 return; 808 809 rq->clock += delta; 810 + 809 811 update_rq_clock_task(rq, delta); 810 812 } 811 813

+73 -1

kernel/sched/ext.c

··· 4911 4911 struct task_struct *p; 4912 4912 struct rhashtable_iter rht_iter; 4913 4913 struct scx_dispatch_q *dsq; 4914 - int i, kind; 4914 + int i, kind, cpu; 4915 4915 4916 4916 kind = atomic_read(&scx_exit_kind); 4917 4917 while (true) { ··· 4993 4993 } 4994 4994 scx_task_iter_stop(&sti); 4995 4995 percpu_up_write(&scx_fork_rwsem); 4996 + 4997 + /* 4998 + * Invalidate all the rq clocks to prevent getting outdated 4999 + * rq clocks from a previous scx scheduler. 5000 + */ 5001 + for_each_possible_cpu(cpu) { 5002 + struct rq *rq = cpu_rq(cpu); 5003 + scx_rq_clock_invalidate(rq); 5004 + } 4996 5005 4997 5006 /* no task is on scx, turn off all the switches and flush in-progress calls */ 4998 5007 static_branch_disable(&__scx_ops_enabled); ··· 7608 7599 } 7609 7600 #endif 7610 7601 7602 + /** 7603 + * scx_bpf_now - Returns a high-performance monotonically non-decreasing 7604 + * clock for the current CPU. The clock returned is in nanoseconds. 7605 + * 7606 + * It provides the following properties: 7607 + * 7608 + * 1) High performance: Many BPF schedulers call bpf_ktime_get_ns() frequently 7609 + * to account for execution time and track tasks' runtime properties. 7610 + * Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which 7611 + * eventually reads a hardware timestamp counter -- is neither performant nor 7612 + * scalable. scx_bpf_now() aims to provide a high-performance clock by 7613 + * using the rq clock in the scheduler core whenever possible. 7614 + * 7615 + * 2) High enough resolution for the BPF scheduler use cases: In most BPF 7616 + * scheduler use cases, the required clock resolution is lower than the most 7617 + * accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now() basically 7618 + * uses the rq clock in the scheduler core whenever it is valid. It considers 7619 + * that the rq clock is valid from the time the rq clock is updated 7620 + * (update_rq_clock) until the rq is unlocked (rq_unpin_lock). 7621 + * 7622 + * 3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now() 7623 + * guarantees the clock never goes backward when comparing them in the same 7624 + * CPU. On the other hand, when comparing clocks in different CPUs, there 7625 + * is no such guarantee -- the clock can go backward. It provides a 7626 + * monotonically *non-decreasing* clock so that it would provide the same 7627 + * clock values in two different scx_bpf_now() calls in the same CPU 7628 + * during the same period of when the rq clock is valid. 7629 + */ 7630 + __bpf_kfunc u64 scx_bpf_now(void) 7631 + { 7632 + struct rq *rq; 7633 + u64 clock; 7634 + 7635 + preempt_disable(); 7636 + 7637 + rq = this_rq(); 7638 + if (smp_load_acquire(&rq->scx.flags) & SCX_RQ_CLK_VALID) { 7639 + /* 7640 + * If the rq clock is valid, use the cached rq clock. 7641 + * 7642 + * Note that scx_bpf_now() is re-entrant between a process 7643 + * context and an interrupt context (e.g., timer interrupt). 7644 + * However, we don't need to consider the race between them 7645 + * because such race is not observable from a caller. 7646 + */ 7647 + clock = READ_ONCE(rq->scx.clock); 7648 + } else { 7649 + /* 7650 + * Otherwise, return a fresh rq clock. 7651 + * 7652 + * The rq clock is updated outside of the rq lock. 7653 + * In this case, keep the updated rq clock invalid so the next 7654 + * kfunc call outside the rq lock gets a fresh rq clock. 7655 + */ 7656 + clock = sched_clock_cpu(cpu_of(rq)); 7657 + } 7658 + 7659 + preempt_enable(); 7660 + 7661 + return clock; 7662 + } 7663 + 7611 7664 __bpf_kfunc_end_defs(); 7612 7665 7613 7666 BTF_KFUNCS_START(scx_kfunc_ids_any) ··· 7701 7630 #ifdef CONFIG_CGROUP_SCHED 7702 7631 BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_RCU | KF_ACQUIRE) 7703 7632 #endif 7633 + BTF_ID_FLAGS(func, scx_bpf_now) 7704 7634 BTF_KFUNCS_END(scx_kfunc_ids_any) 7705 7635 7706 7636 static const struct btf_kfunc_id_set scx_kfunc_set_any = {

+23 -2

kernel/sched/sched.h

··· 754 754 SCX_RQ_BAL_PENDING = 1 << 2, /* balance hasn't run yet */ 755 755 SCX_RQ_BAL_KEEP = 1 << 3, /* balance decided to keep current */ 756 756 SCX_RQ_BYPASSING = 1 << 4, 757 + SCX_RQ_CLK_VALID = 1 << 5, /* RQ clock is fresh and valid */ 757 758 758 759 SCX_RQ_IN_WAKEUP = 1 << 16, 759 760 SCX_RQ_IN_BALANCE = 1 << 17, ··· 767 766 unsigned long ops_qseq; 768 767 u64 extra_enq_flags; /* see move_task_to_local_dsq() */ 769 768 u32 nr_running; 770 - u32 flags; 771 769 u32 cpuperf_target; /* [0, SCHED_CAPACITY_SCALE] */ 772 770 bool cpu_released; 771 + u32 flags; 772 + u64 clock; /* current per-rq clock -- see scx_bpf_now() */ 773 773 cpumask_var_t cpus_to_kick; 774 774 cpumask_var_t cpus_to_kick_if_idle; 775 775 cpumask_var_t cpus_to_preempt; ··· 1727 1725 1728 1726 #define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) 1729 1727 #define scx_switched_all() static_branch_unlikely(&__scx_switched_all) 1728 + 1729 + static inline void scx_rq_clock_update(struct rq *rq, u64 clock) 1730 + { 1731 + if (!scx_enabled()) 1732 + return; 1733 + WRITE_ONCE(rq->scx.clock, clock); 1734 + smp_store_release(&rq->scx.flags, rq->scx.flags | SCX_RQ_CLK_VALID); 1735 + } 1736 + 1737 + static inline void scx_rq_clock_invalidate(struct rq *rq) 1738 + { 1739 + if (!scx_enabled()) 1740 + return; 1741 + WRITE_ONCE(rq->scx.flags, rq->scx.flags & ~SCX_RQ_CLK_VALID); 1742 + } 1743 + 1730 1744 #else /* !CONFIG_SCHED_CLASS_EXT */ 1731 1745 #define scx_enabled() false 1732 1746 #define scx_switched_all() false 1747 + 1748 + static inline void scx_rq_clock_update(struct rq *rq, u64 clock) {} 1749 + static inline void scx_rq_clock_invalidate(struct rq *rq) {} 1733 1750 #endif /* !CONFIG_SCHED_CLASS_EXT */ 1734 1751 1735 1752 /* ··· 1780 1759 if (rq->clock_update_flags > RQCF_ACT_SKIP) 1781 1760 rf->clock_update_flags = RQCF_UPDATED; 1782 1761 #endif 1783 - 1762 + scx_rq_clock_invalidate(rq); 1784 1763 lockdep_unpin_lock(__rq_lockp(rq), rf->cookie); 1785 1764 } 1786 1765

Configure Feed

Configure Feed