tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

slab: add sheaves to most caches

In the first step to replace cpu (partial) slabs with sheaves, enable
sheaves for almost all caches. Treat args->sheaf_capacity as a minimum,
and calculate sheaf capacity with a formula that roughly follows the
formula for number of objects in cpu partial slabs in set_cpu_partial().

This should achieve roughly similar contention on the barn spin lock as
there's currently for node list_lock without sheaves, to make
benchmarking results comparable. It can be further tuned later.

Don't enable sheaves for bootstrap caches as that wouldn't work. In
order to recognize them by SLAB_NO_OBJ_EXT, make sure the flag exists
even for !CONFIG_SLAB_OBJ_EXT.

This limitation will be lifted for kmalloc caches after the necessary
bootstrapping changes.

Also do not enable sheaves for SLAB_NOLEAKTRACE caches to avoid
recursion with kmemleak tracking (thanks to Breno Leitao).

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Reviewed-by: Hao Li <hao.li@linux.dev>
Tested-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Tested-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

Vlastimil Babka e47c897a 4b038a96

+52 -10
-6
include/linux/slab.h
··· 57 57 #endif 58 58 _SLAB_OBJECT_POISON, 59 59 _SLAB_CMPXCHG_DOUBLE, 60 - #ifdef CONFIG_SLAB_OBJ_EXT 61 60 _SLAB_NO_OBJ_EXT, 62 - #endif 63 61 _SLAB_FLAGS_LAST_BIT 64 62 }; 65 63 ··· 236 238 #define SLAB_TEMPORARY SLAB_RECLAIM_ACCOUNT /* Objects are short-lived */ 237 239 238 240 /* Slab created using create_boot_cache */ 239 - #ifdef CONFIG_SLAB_OBJ_EXT 240 241 #define SLAB_NO_OBJ_EXT __SLAB_FLAG_BIT(_SLAB_NO_OBJ_EXT) 241 - #else 242 - #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED 243 - #endif 244 242 245 243 /* 246 244 * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
+52 -4
mm/slub.c
··· 7893 7893 #endif 7894 7894 } 7895 7895 7896 + static unsigned int calculate_sheaf_capacity(struct kmem_cache *s, 7897 + struct kmem_cache_args *args) 7898 + 7899 + { 7900 + unsigned int capacity; 7901 + size_t size; 7902 + 7903 + 7904 + if (IS_ENABLED(CONFIG_SLUB_TINY) || s->flags & SLAB_DEBUG_FLAGS) 7905 + return 0; 7906 + 7907 + /* 7908 + * Bootstrap caches can't have sheaves for now (SLAB_NO_OBJ_EXT). 7909 + * SLAB_NOLEAKTRACE caches (e.g., kmemleak's object_cache) must not 7910 + * have sheaves to avoid recursion when sheaf allocation triggers 7911 + * kmemleak tracking. 7912 + */ 7913 + if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE)) 7914 + return 0; 7915 + 7916 + /* 7917 + * For now we use roughly similar formula (divided by two as there are 7918 + * two percpu sheaves) as what was used for percpu partial slabs, which 7919 + * should result in similar lock contention (barn or list_lock) 7920 + */ 7921 + if (s->size >= PAGE_SIZE) 7922 + capacity = 4; 7923 + else if (s->size >= 1024) 7924 + capacity = 12; 7925 + else if (s->size >= 256) 7926 + capacity = 26; 7927 + else 7928 + capacity = 60; 7929 + 7930 + /* Increment capacity to make sheaf exactly a kmalloc size bucket */ 7931 + size = struct_size_t(struct slab_sheaf, objects, capacity); 7932 + size = kmalloc_size_roundup(size); 7933 + capacity = (size - struct_size_t(struct slab_sheaf, objects, 0)) / sizeof(void *); 7934 + 7935 + /* 7936 + * Respect an explicit request for capacity that's typically motivated by 7937 + * expected maximum size of kmem_cache_prefill_sheaf() to not end up 7938 + * using low-performance oversize sheaves 7939 + */ 7940 + return max(capacity, args->sheaf_capacity); 7941 + } 7942 + 7896 7943 /* 7897 7944 * calculate_sizes() determines the order and the distribution of data within 7898 7945 * a slab object. ··· 8073 8026 8074 8027 if (s->flags & SLAB_RECLAIM_ACCOUNT) 8075 8028 s->allocflags |= __GFP_RECLAIMABLE; 8029 + 8030 + /* kmalloc caches need extra care to support sheaves */ 8031 + if (!is_kmalloc_cache(s)) 8032 + s->sheaf_capacity = calculate_sheaf_capacity(s, args); 8076 8033 8077 8034 /* 8078 8035 * Determine the number of objects per slab ··· 8682 8631 8683 8632 set_cpu_partial(s); 8684 8633 8685 - if (args->sheaf_capacity && !IS_ENABLED(CONFIG_SLUB_TINY) 8686 - && !(s->flags & SLAB_DEBUG_FLAGS)) { 8634 + if (s->sheaf_capacity) { 8687 8635 s->cpu_sheaves = alloc_percpu(struct slub_percpu_sheaves); 8688 8636 if (!s->cpu_sheaves) { 8689 8637 err = -ENOMEM; 8690 8638 goto out; 8691 8639 } 8692 - // TODO: increase capacity to grow slab_sheaf up to next kmalloc size? 8693 - s->sheaf_capacity = args->sheaf_capacity; 8694 8640 } 8695 8641 8696 8642 #ifdef CONFIG_NUMA