tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

slab: move kfence_alloc() out of internal bulk alloc

SLUB's internal bulk allocation __kmem_cache_alloc_bulk() can currently
allocate some objects from KFENCE, i.e. when refilling a sheaf. It works
but it's conceptually the wrong layer, as KFENCE allocations should only
happen when objects are actually handed out from slab to its users.

Currently for sheaf-enabled caches, slab_alloc_node() can return KFENCE
object via kfence_alloc(), but also via alloc_from_pcs() when a sheaf
was refilled with KFENCE objects. Continuing like this would also
complicate the upcoming sheaf refill changes.

Thus remove KFENCE allocation from __kmem_cache_alloc_bulk() and move it
to the places that return slab objects to users. slab_alloc_node() is
already covered (see above). Add kfence_alloc() to
kmem_cache_alloc_from_sheaf() to handle KFENCE allocations from
prefilled sheafs, with a comment that the caller should not expect the
sheaf size to decrease after every allocation because of this
possibility.

For kmem_cache_alloc_bulk() implement a different strategy to handle
KFENCE upfront and rely on internal batched operations afterwards.
Assume there will be at most once KFENCE allocation per bulk allocation
and then assign its index in the array of objects randomly.

Cc: Alexander Potapenko <glider@google.com>
Cc: Marco Elver <elver@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Link: https://patch.msgid.link/20251105-sheaves-cleanups-v1-2-b8218e1ac7ef@suse.cz
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

Vlastimil Babka ea6b5e57 f6087b92

+36 -8
+36 -8
mm/slub.c
··· 5540 5540 * 5541 5541 * The gfp parameter is meant only to specify __GFP_ZERO or __GFP_ACCOUNT 5542 5542 * memcg charging is forced over limit if necessary, to avoid failure. 5543 + * 5544 + * It is possible that the allocation comes from kfence and then the sheaf 5545 + * size is not decreased. 5543 5546 */ 5544 5547 void * 5545 5548 kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp, ··· 5554 5551 if (sheaf->size == 0) 5555 5552 goto out; 5556 5553 5557 - ret = sheaf->objects[--sheaf->size]; 5554 + ret = kfence_alloc(s, s->object_size, gfp); 5555 + 5556 + if (likely(!ret)) 5557 + ret = sheaf->objects[--sheaf->size]; 5558 5558 5559 5559 init = slab_want_init_on_alloc(gfp, s); 5560 5560 ··· 7405 7399 local_lock_irqsave(&s->cpu_slab->lock, irqflags); 7406 7400 7407 7401 for (i = 0; i < size; i++) { 7408 - void *object = kfence_alloc(s, s->object_size, flags); 7402 + void *object = c->freelist; 7409 7403 7410 - if (unlikely(object)) { 7411 - p[i] = object; 7412 - continue; 7413 - } 7414 - 7415 - object = c->freelist; 7416 7404 if (unlikely(!object)) { 7417 7405 /* 7418 7406 * We may have removed an object from c->freelist using ··· 7487 7487 void **p) 7488 7488 { 7489 7489 unsigned int i = 0; 7490 + void *kfence_obj; 7490 7491 7491 7492 if (!size) 7492 7493 return 0; ··· 7495 7494 s = slab_pre_alloc_hook(s, flags); 7496 7495 if (unlikely(!s)) 7497 7496 return 0; 7497 + 7498 + /* 7499 + * to make things simpler, only assume at most once kfence allocated 7500 + * object per bulk allocation and choose its index randomly 7501 + */ 7502 + kfence_obj = kfence_alloc(s, s->object_size, flags); 7503 + 7504 + if (unlikely(kfence_obj)) { 7505 + if (unlikely(size == 1)) { 7506 + p[0] = kfence_obj; 7507 + goto out; 7508 + } 7509 + size--; 7510 + } 7498 7511 7499 7512 if (s->cpu_sheaves) 7500 7513 i = alloc_from_pcs_bulk(s, size, p); ··· 7521 7506 if (unlikely(__kmem_cache_alloc_bulk(s, flags, size - i, p + i) == 0)) { 7522 7507 if (i > 0) 7523 7508 __kmem_cache_free_bulk(s, i, p); 7509 + if (kfence_obj) 7510 + __kfence_free(kfence_obj); 7524 7511 return 0; 7525 7512 } 7526 7513 } 7527 7514 7515 + if (unlikely(kfence_obj)) { 7516 + int idx = get_random_u32_below(size + 1); 7517 + 7518 + if (idx != size) 7519 + p[size] = p[idx]; 7520 + p[idx] = kfence_obj; 7521 + 7522 + size++; 7523 + } 7524 + 7525 + out: 7528 7526 /* 7529 7527 * memcg and kmem_cache debug support and memory initialization. 7530 7528 * Done outside of the IRQ disabled fastpath loop.