hrtimer: Avoid more SMP function calls in clock_was_set()

By unconditionally updating the offsets there are more indicators
whether the SMP function calls on clock_was_set() can be avoided:

- When the offset update already happened on the remote CPU then the
remote update attempt will yield the same seqeuence number and no
IPI is required.

- When the remote CPU is currently handling hrtimer_interrupt(). In
that case the remote CPU will reevaluate the timer bases before
reprogramming anyway, so nothing to do.

- After updating it can be checked whether the first expiring timer in
the affected clock bases moves before the first expiring (softirq)
timer of the CPU. If that's not the case then sending the IPI is not
required.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20210713135158.887322464@linutronix.de

Thomas Gleixner 4 years ago 1e7f7fbc 81d741d3

+65 -9

1 changed file

expand all

kernel

time

hrtimer.c

+65 -9

kernel/time/hrtimer.c

··· 866 866 __hrtimer_reprogram(cpu_base, true, timer, expires); 867 867 } 868 868 869 + static bool update_needs_ipi(struct hrtimer_cpu_base *cpu_base, 870 + unsigned int active) 871 + { 872 + struct hrtimer_clock_base *base; 873 + unsigned int seq; 874 + ktime_t expires; 875 + 876 + /* 877 + * Update the base offsets unconditionally so the following 878 + * checks whether the SMP function call is required works. 879 + * 880 + * The update is safe even when the remote CPU is in the hrtimer 881 + * interrupt or the hrtimer soft interrupt and expiring affected 882 + * bases. Either it will see the update before handling a base or 883 + * it will see it when it finishes the processing and reevaluates 884 + * the next expiring timer. 885 + */ 886 + seq = cpu_base->clock_was_set_seq; 887 + hrtimer_update_base(cpu_base); 888 + 889 + /* 890 + * If the sequence did not change over the update then the 891 + * remote CPU already handled it. 892 + */ 893 + if (seq == cpu_base->clock_was_set_seq) 894 + return false; 895 + 896 + /* 897 + * If the remote CPU is currently handling an hrtimer interrupt, it 898 + * will reevaluate the first expiring timer of all clock bases 899 + * before reprogramming. Nothing to do here. 900 + */ 901 + if (cpu_base->in_hrtirq) 902 + return false; 903 + 904 + /* 905 + * Walk the affected clock bases and check whether the first expiring 906 + * timer in a clock base is moving ahead of the first expiring timer of 907 + * @cpu_base. If so, the IPI must be invoked because per CPU clock 908 + * event devices cannot be remotely reprogrammed. 909 + */ 910 + active &= cpu_base->active_bases; 911 + 912 + for_each_active_base(base, cpu_base, active) { 913 + struct timerqueue_node *next; 914 + 915 + next = timerqueue_getnext(&base->active); 916 + expires = ktime_sub(next->expires, base->offset); 917 + if (expires < cpu_base->expires_next) 918 + return true; 919 + 920 + /* Extra check for softirq clock bases */ 921 + if (base->clockid < HRTIMER_BASE_MONOTONIC_SOFT) 922 + continue; 923 + if (cpu_base->softirq_activated) 924 + continue; 925 + if (expires < cpu_base->softirq_expires_next) 926 + return true; 927 + } 928 + return false; 929 + } 930 + 869 931 /* 870 932 * Clock was set. This might affect CLOCK_REALTIME, CLOCK_TAI and 871 933 * CLOCK_BOOTTIME (for late sleep time injection). ··· 962 900 unsigned long flags; 963 901 964 902 raw_spin_lock_irqsave(&cpu_base->lock, flags); 965 - /* 966 - * Only send the IPI when there are timers queued in one of 967 - * the affected clock bases. Otherwise update the base 968 - * remote to ensure that the next enqueue of a timer on 969 - * such a clock base will see the correct offsets. 970 - */ 971 - if (cpu_base->active_bases & bases) 903 + 904 + if (update_needs_ipi(cpu_base, bases)) 972 905 cpumask_set_cpu(cpu, mask); 973 - else 974 - hrtimer_update_base(cpu_base); 906 + 975 907 raw_spin_unlock_irqrestore(&cpu_base->lock, flags); 976 908 } 977 909

Configure Feed

Configure Feed