tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

percpu-rw-semaphores: use light/heavy barriers

This patch introduces new barrier pair light_mb() and heavy_mb() for
percpu rw semaphores.

This patch fixes a bug in percpu-rw-semaphores where a barrier was
missing in percpu_up_write.

This patch improves performance on the read path of
percpu-rw-semaphores: on non-x86 cpus, there was a smp_mb() in
percpu_up_read. This patch changes it to a compiler barrier and removes
the "#if defined(X86) ..." condition.

From: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

authored by

Mikulas Patocka and committed by
Linus Torvalds 5c1eabe6 e657e078

+7 -13
+7 -13
include/linux/percpu-rwsem.h
··· 12 12 struct mutex mtx; 13 13 }; 14 14 15 + #define light_mb() barrier() 16 + #define heavy_mb() synchronize_sched() 17 + 15 18 static inline void percpu_down_read(struct percpu_rw_semaphore *p) 16 19 { 17 20 rcu_read_lock(); ··· 27 24 } 28 25 this_cpu_inc(*p->counters); 29 26 rcu_read_unlock(); 27 + light_mb(); /* A, between read of p->locked and read of data, paired with D */ 30 28 } 31 29 32 30 static inline void percpu_up_read(struct percpu_rw_semaphore *p) 33 31 { 34 - /* 35 - * On X86, write operation in this_cpu_dec serves as a memory unlock 36 - * barrier (i.e. memory accesses may be moved before the write, but 37 - * no memory accesses are moved past the write). 38 - * On other architectures this may not be the case, so we need smp_mb() 39 - * there. 40 - */ 41 - #if defined(CONFIG_X86) && (!defined(CONFIG_X86_PPRO_FENCE) && !defined(CONFIG_X86_OOSTORE)) 42 - barrier(); 43 - #else 44 - smp_mb(); 45 - #endif 32 + light_mb(); /* B, between read of the data and write to p->counter, paired with C */ 46 33 this_cpu_dec(*p->counters); 47 34 } 48 35 ··· 54 61 synchronize_rcu(); 55 62 while (__percpu_count(p->counters)) 56 63 msleep(1); 57 - smp_rmb(); /* paired with smp_mb() in percpu_sem_up_read() */ 64 + heavy_mb(); /* C, between read of p->counter and write to data, paired with B */ 58 65 } 59 66 60 67 static inline void percpu_up_write(struct percpu_rw_semaphore *p) 61 68 { 69 + heavy_mb(); /* D, between write to data and write to p->locked, paired with A */ 62 70 p->locked = false; 63 71 mutex_unlock(&p->mtx); 64 72 }