tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

slab: determine barn status racily outside of lock

The possibility of many barn operations is determined by the current
number of full or empty sheaves. Taking the barn->lock just to find out
that e.g. there are no empty sheaves results in unnecessary overhead and
lock contention. Thus perform these checks outside of the lock with a
data_race() annotated variable read and fail quickly without taking the
lock.

Checks for sheaf availability that racily succeed have to be obviously
repeated under the lock for correctness, but we can skip repeating
checks if there are too many sheaves on the given list as the limits
don't need to be strict.

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

Vlastimil Babka 08294229 3c1ea5c5

+20 -7
+20 -7
mm/slub.c
··· 2801 2801 struct slab_sheaf *empty = NULL; 2802 2802 unsigned long flags; 2803 2803 2804 + if (!data_race(barn->nr_empty)) 2805 + return NULL; 2806 + 2804 2807 spin_lock_irqsave(&barn->lock, flags); 2805 2808 2806 - if (barn->nr_empty) { 2809 + if (likely(barn->nr_empty)) { 2807 2810 empty = list_first_entry(&barn->sheaves_empty, 2808 2811 struct slab_sheaf, barn_list); 2809 2812 list_del(&empty->barn_list); ··· 2853 2850 struct slab_sheaf *sheaf = NULL; 2854 2851 unsigned long flags; 2855 2852 2853 + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) 2854 + return NULL; 2855 + 2856 2856 spin_lock_irqsave(&barn->lock, flags); 2857 2857 2858 2858 if (barn->nr_full) { ··· 2886 2880 struct slab_sheaf *full = NULL; 2887 2881 unsigned long flags; 2888 2882 2883 + if (!data_race(barn->nr_full)) 2884 + return NULL; 2885 + 2889 2886 spin_lock_irqsave(&barn->lock, flags); 2890 2887 2891 - if (barn->nr_full) { 2888 + if (likely(barn->nr_full)) { 2892 2889 full = list_first_entry(&barn->sheaves_full, struct slab_sheaf, 2893 2890 barn_list); 2894 2891 list_del(&full->barn_list); ··· 2915 2906 struct slab_sheaf *empty; 2916 2907 unsigned long flags; 2917 2908 2909 + /* we don't repeat this check under barn->lock as it's not critical */ 2910 + if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES) 2911 + return ERR_PTR(-E2BIG); 2912 + if (!data_race(barn->nr_empty)) 2913 + return ERR_PTR(-ENOMEM); 2914 + 2918 2915 spin_lock_irqsave(&barn->lock, flags); 2919 2916 2920 - if (barn->nr_full >= MAX_FULL_SHEAVES) { 2921 - empty = ERR_PTR(-E2BIG); 2922 - } else if (!barn->nr_empty) { 2923 - empty = ERR_PTR(-ENOMEM); 2924 - } else { 2917 + if (likely(barn->nr_empty)) { 2925 2918 empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, 2926 2919 barn_list); 2927 2920 list_del(&empty->barn_list); 2928 2921 list_add(&full->barn_list, &barn->sheaves_full); 2929 2922 barn->nr_empty--; 2930 2923 barn->nr_full++; 2924 + } else { 2925 + empty = ERR_PTR(-ENOMEM); 2931 2926 } 2932 2927 2933 2928 spin_unlock_irqrestore(&barn->lock, flags);