Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

smp: Improve smp_call_function_single() CSD-lock diagnostics

Both smp_call_function() and smp_call_function_single() use per-CPU
call_single_data_t variable to hold the infamous CSD lock. However,
while smp_call_function() acquires the destination CPU's CSD lock,
smp_call_function_single() instead uses the source CPU's CSD lock.
(These are two separate sets of CSD locks, cfd_data and csd_data,
respectively.)

This otherwise inexplicable pair of choices is explained by their
respective queueing properties. If smp_call_function() where to
use the sending CPU's CSD lock, that would serialize the destination
CPUs' IPI handlers and result in long smp_call_function() latencies,
especially on systems with large numbers of CPUs. For its part, if
smp_call_function_single() were to use the (single) destination CPU's
CSD lock, this would similarly serialize in the case where many CPUs
are sending IPIs to a single "victim" CPU. Plus it would result in
higher levels of memory contention.

Except that if there is no NMI-based stack tracing on a weakly ordered
system where remote unsynchronized stack traces are especially unreliable,
the improved debugging beats the improved queueing. This improved queueing
only matters if a bunch of CPUs are calling smp_call_function_single()
concurrently for a single "victim" CPU, which is not the common case.

Therefore, make smp_call_function_single() use the destination CPU's
csd_data instance in kernels built with CONFIG_CSD_LOCK_WAIT_DEBUG=y
where csdlock_debug_enabled is also set. Otherwise, continue to use
the source CPU's csd_data.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Ulf Hansson <ulf.hansson@linaro.org>
Link: https://patch.msgid.link/25c2eb97-77c8-49a5-80ac-efe78dea272c@paulmck-laptop

authored by

Paul E. McKenney and committed by
Thomas Gleixner
b0473dcd ec39780d

+15 -1
+15 -1
kernel/smp.c
··· 377 377 378 378 static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data); 379 379 380 + #ifdef CONFIG_CSD_LOCK_WAIT_DEBUG 381 + static call_single_data_t *get_single_csd_data(int cpu) 382 + { 383 + if (static_branch_unlikely(&csdlock_debug_enabled)) 384 + return per_cpu_ptr(&csd_data, cpu); 385 + return this_cpu_ptr(&csd_data); 386 + } 387 + #else 388 + static call_single_data_t *get_single_csd_data(int cpu) 389 + { 390 + return this_cpu_ptr(&csd_data); 391 + } 392 + #endif 393 + 380 394 void __smp_call_single_queue(int cpu, struct llist_node *node) 381 395 { 382 396 /* ··· 685 671 686 672 csd = &csd_stack; 687 673 if (!wait) { 688 - csd = this_cpu_ptr(&csd_data); 674 + csd = get_single_csd_data(cpu); 689 675 csd_lock(csd); 690 676 } 691 677