Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'slab-for-6.11' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

Pull slab updates from Vlastimil Babka:
"The most prominent change this time is the kmem_buckets based
hardening of kmalloc() allocations from Kees Cook.

We have also extended the kmalloc() alignment guarantees for
non-power-of-two sizes in a way that benefits rust.

The rest are various cleanups and non-critical fixups.

- Dedicated bucket allocator (Kees Cook)

This series [1] enhances the probabilistic defense against heap
spraying/grooming of CONFIG_RANDOM_KMALLOC_CACHES from last year.

kmalloc() users that are known to be useful for exploits can get
completely separate set of kmalloc caches that can't be shared with
other users. The first converted users are alloc_msg() and
memdup_user().

The hardening is enabled by CONFIG_SLAB_BUCKETS.

- Extended kmalloc() alignment guarantees (Vlastimil Babka)

For years now we have guaranteed natural alignment for power-of-two
allocations, but nothing was defined for other sizes (in practice,
we have two such buckets, kmalloc-96 and kmalloc-192).

To avoid unnecessary padding in the rust layer due to its alignment
rules, extend the guarantee so that the alignment is at least the
largest power-of-two divisor of the requested size.

This fits what rust needs, is a superset of the existing
power-of-two guarantee, and does not in practice change the layout
(and thus does not add overhead due to padding) of the kmalloc-96
and kmalloc-192 caches, unless slab debugging is enabled for them.

- Cleanups and non-critical fixups (Chengming Zhou, Suren
Baghdasaryan, Matthew Willcox, Alex Shi, and Vlastimil Babka)

Various tweaks related to the new alloc profiling code, folio
conversion, debugging and more leftovers after SLAB"

Link: https://lore.kernel.org/all/20240701190152.it.631-kees@kernel.org/ [1]

* tag 'slab-for-6.11' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab:
mm/memcg: alignment memcg_data define condition
mm, slab: move prepare_slab_obj_exts_hook under CONFIG_MEM_ALLOC_PROFILING
mm, slab: move allocation tagging code in the alloc path into a hook
mm/util: Use dedicated slab buckets for memdup_user()
ipc, msg: Use dedicated slab buckets for alloc_msg()
mm/slab: Introduce kmem_buckets_create() and family
mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument
mm/slab: Plumb kmem_buckets into __do_kmalloc_node()
mm/slab: Introduce kmem_buckets typedef
slab, rust: extend kmalloc() alignment guarantees to remove Rust padding
slab: delete useless RED_INACTIVE and RED_ACTIVE
slab: don't put freepointer outside of object if only orig_size
slab: make check_object() more consistent
mm: Reduce the number of slab->folio casts
mm, slab: don't wrap internal functions with alloc_hooks()

+370 -176
+4 -2
Documentation/core-api/memory-allocation.rst
··· 144 144 smaller than page size. 145 145 146 146 The address of a chunk allocated with `kmalloc` is aligned to at least 147 - ARCH_KMALLOC_MINALIGN bytes. For sizes which are a power of two, the 148 - alignment is also guaranteed to be at least the respective size. 147 + ARCH_KMALLOC_MINALIGN bytes. For sizes which are a power of two, the 148 + alignment is also guaranteed to be at least the respective size. For other 149 + sizes, the alignment is guaranteed to be at least the largest power-of-two 150 + divisor of the size. 149 151 150 152 Chunks allocated with kmalloc() can be resized with krealloc(). Similarly 151 153 to kmalloc_array(): a helper for resizing arrays is provided in the form of
+3 -3
include/linux/mm.h
··· 1110 1110 * 1111 1111 * Return: The order of the folio. 1112 1112 */ 1113 - static inline unsigned int folio_order(struct folio *folio) 1113 + static inline unsigned int folio_order(const struct folio *folio) 1114 1114 { 1115 1115 if (!folio_test_large(folio)) 1116 1116 return 0; ··· 2150 2150 * it from being split. It is not necessary for the folio to be locked. 2151 2151 * Return: The base-2 logarithm of the size of this folio. 2152 2152 */ 2153 - static inline unsigned int folio_shift(struct folio *folio) 2153 + static inline unsigned int folio_shift(const struct folio *folio) 2154 2154 { 2155 2155 return PAGE_SHIFT + folio_order(folio); 2156 2156 } ··· 2163 2163 * it from being split. It is not necessary for the folio to be locked. 2164 2164 * Return: The number of bytes in this folio. 2165 2165 */ 2166 - static inline size_t folio_size(struct folio *folio) 2166 + static inline size_t folio_size(const struct folio *folio) 2167 2167 { 2168 2168 return PAGE_SIZE << folio_order(folio); 2169 2169 }
+7 -2
include/linux/mm_types.h
··· 169 169 /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ 170 170 atomic_t _refcount; 171 171 172 - #ifdef CONFIG_SLAB_OBJ_EXT 172 + #ifdef CONFIG_MEMCG 173 173 unsigned long memcg_data; 174 + #elif defined(CONFIG_SLAB_OBJ_EXT) 175 + unsigned long _unused_slab_obj_exts; 174 176 #endif 175 177 176 178 /* ··· 300 298 * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. 301 299 * @_hugetlb_hwpoison: Do not use directly, call raw_hwp_list_head(). 302 300 * @_deferred_list: Folios to be split under memory pressure. 301 + * @_unused_slab_obj_exts: Placeholder to match obj_exts in struct slab. 303 302 * 304 303 * A folio is a physically, virtually and logically contiguous set 305 304 * of bytes. It is a power-of-two in size, and it is aligned to that ··· 335 332 }; 336 333 atomic_t _mapcount; 337 334 atomic_t _refcount; 338 - #ifdef CONFIG_SLAB_OBJ_EXT 335 + #ifdef CONFIG_MEMCG 339 336 unsigned long memcg_data; 337 + #elif defined(CONFIG_SLAB_OBJ_EXT) 338 + unsigned long _unused_slab_obj_exts; 340 339 #endif 341 340 #if defined(WANT_PAGE_VIRTUAL) 342 341 void *virtual;
+2 -5
include/linux/poison.h
··· 38 38 * Magic nums for obj red zoning. 39 39 * Placed in the first word before and the first word after an obj. 40 40 */ 41 - #define RED_INACTIVE 0x09F911029D74E35BULL /* when obj is inactive */ 42 - #define RED_ACTIVE 0xD84156C5635688C0ULL /* when obj is active */ 43 - 44 - #define SLUB_RED_INACTIVE 0xbb 45 - #define SLUB_RED_ACTIVE 0xcc 41 + #define SLUB_RED_INACTIVE 0xbb /* when obj is inactive */ 42 + #define SLUB_RED_ACTIVE 0xcc /* when obj is active */ 46 43 47 44 /* ...and for poisoning */ 48 45 #define POISON_INUSE 0x5a /* for use-uninitialised poisoning */
+65 -32
include/linux/slab.h
··· 426 426 NR_KMALLOC_TYPES 427 427 }; 428 428 429 - extern struct kmem_cache * 430 - kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; 429 + typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; 430 + 431 + extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; 431 432 432 433 /* 433 434 * Define gfp bits that should not be set for KMALLOC_NORMAL. ··· 529 528 530 529 #include <linux/alloc_tag.h> 531 530 532 - void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); 533 - #define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) 534 - 535 531 /** 536 532 * kmem_cache_alloc - Allocate an object 537 533 * @cachep: The cache to allocate from. ··· 549 551 550 552 void kmem_cache_free(struct kmem_cache *s, void *objp); 551 553 554 + kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, 555 + unsigned int useroffset, unsigned int usersize, 556 + void (*ctor)(void *)); 557 + 552 558 /* 553 559 * Bulk allocation and freeing operations. These are accelerated in an 554 560 * allocator specific way to avoid taking locks repeatedly or building ··· 570 568 kmem_cache_free_bulk(NULL, size, p); 571 569 } 572 570 573 - void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment 574 - __alloc_size(1); 575 - #define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) 576 - 577 571 void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, 578 572 int node) __assume_slab_alignment __malloc; 579 573 #define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) 580 574 581 - void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) 582 - __assume_kmalloc_alignment __alloc_size(3); 575 + /* 576 + * These macros allow declaring a kmem_buckets * parameter alongside size, which 577 + * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call 578 + * sites don't have to pass NULL. 579 + */ 580 + #ifdef CONFIG_SLAB_BUCKETS 581 + #define DECL_BUCKET_PARAMS(_size, _b) size_t (_size), kmem_buckets *(_b) 582 + #define PASS_BUCKET_PARAMS(_size, _b) (_size), (_b) 583 + #define PASS_BUCKET_PARAM(_b) (_b) 584 + #else 585 + #define DECL_BUCKET_PARAMS(_size, _b) size_t (_size) 586 + #define PASS_BUCKET_PARAMS(_size, _b) (_size) 587 + #define PASS_BUCKET_PARAM(_b) NULL 588 + #endif 583 589 584 - void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, 585 - int node, size_t size) __assume_kmalloc_alignment 586 - __alloc_size(4); 587 - #define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) 590 + /* 591 + * The following functions are not to be used directly and are intended only 592 + * for internal use from kmalloc() and kmalloc_node() 593 + * with the exception of kunit tests 594 + */ 588 595 589 - #define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) 596 + void *__kmalloc_noprof(size_t size, gfp_t flags) 597 + __assume_kmalloc_alignment __alloc_size(1); 590 598 591 - void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment 592 - __alloc_size(1); 593 - #define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) 599 + void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) 600 + __assume_kmalloc_alignment __alloc_size(1); 594 601 595 - void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment 596 - __alloc_size(1); 597 - #define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) 602 + void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size) 603 + __assume_kmalloc_alignment __alloc_size(3); 604 + 605 + void *__kmalloc_cache_node_noprof(struct kmem_cache *s, gfp_t gfpflags, 606 + int node, size_t size) 607 + __assume_kmalloc_alignment __alloc_size(4); 608 + 609 + void *__kmalloc_large_noprof(size_t size, gfp_t flags) 610 + __assume_page_alignment __alloc_size(1); 611 + 612 + void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) 613 + __assume_page_alignment __alloc_size(1); 598 614 599 615 /** 600 616 * kmalloc - allocate kernel memory ··· 624 604 * 625 605 * The allocated object address is aligned to at least ARCH_KMALLOC_MINALIGN 626 606 * bytes. For @size of power of two bytes, the alignment is also guaranteed 627 - * to be at least to the size. 607 + * to be at least to the size. For other sizes, the alignment is guaranteed to 608 + * be at least the largest power-of-two divisor of @size. 628 609 * 629 610 * The @flags argument may be one of the GFP flags defined at 630 611 * include/linux/gfp_types.h and described at ··· 675 654 unsigned int index; 676 655 677 656 if (size > KMALLOC_MAX_CACHE_SIZE) 678 - return kmalloc_large_noprof(size, flags); 657 + return __kmalloc_large_noprof(size, flags); 679 658 680 659 index = kmalloc_index(size); 681 - return kmalloc_trace_noprof( 660 + return __kmalloc_cache_noprof( 682 661 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 683 662 flags, size); 684 663 } ··· 686 665 } 687 666 #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) 688 667 668 + #define kmem_buckets_alloc(_b, _size, _flags) \ 669 + alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) 670 + 671 + #define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ 672 + alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_)) 673 + 689 674 static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) 690 675 { 691 676 if (__builtin_constant_p(size) && size) { 692 677 unsigned int index; 693 678 694 679 if (size > KMALLOC_MAX_CACHE_SIZE) 695 - return kmalloc_large_node_noprof(size, flags, node); 680 + return __kmalloc_large_node_noprof(size, flags, node); 696 681 697 682 index = kmalloc_index(size); 698 - return kmalloc_node_trace_noprof( 683 + return __kmalloc_cache_node_noprof( 699 684 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 700 685 flags, node, size); 701 686 } 702 - return __kmalloc_node_noprof(size, flags, node); 687 + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); 703 688 } 704 689 #define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) 705 690 ··· 756 729 */ 757 730 #define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO) 758 731 759 - void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, 760 - unsigned long caller) __alloc_size(1); 732 + void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, 733 + unsigned long caller) __alloc_size(1); 734 + #define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ 735 + __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) 761 736 #define kmalloc_node_track_caller(...) \ 762 737 alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) 763 738 ··· 785 756 return NULL; 786 757 if (__builtin_constant_p(n) && __builtin_constant_p(size)) 787 758 return kmalloc_node_noprof(bytes, flags, node); 788 - return __kmalloc_node_noprof(bytes, flags, node); 759 + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node); 789 760 } 790 761 #define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) 791 762 ··· 809 780 #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) 810 781 #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) 811 782 812 - extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); 783 + void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); 784 + #define kvmalloc_node_noprof(size, flags, node) \ 785 + __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) 813 786 #define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) 814 787 815 788 #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) ··· 819 788 #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) 820 789 821 790 #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) 791 + #define kmem_buckets_valloc(_b, _size, _flags) \ 792 + alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) 822 793 823 794 static inline __alloc_size(1, 2) void * 824 795 kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
+12 -1
ipc/msgutil.c
··· 42 42 #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) 43 43 #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) 44 44 45 + static kmem_buckets *msg_buckets __ro_after_init; 46 + 47 + static int __init init_msg_buckets(void) 48 + { 49 + msg_buckets = kmem_buckets_create("msg_msg", SLAB_ACCOUNT, 50 + sizeof(struct msg_msg), 51 + DATALEN_MSG, NULL); 52 + 53 + return 0; 54 + } 55 + subsys_initcall(init_msg_buckets); 45 56 46 57 static struct msg_msg *alloc_msg(size_t len) 47 58 { ··· 61 50 size_t alen; 62 51 63 52 alen = min(len, DATALEN_MSG); 64 - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); 53 + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); 65 54 if (msg == NULL) 66 55 return NULL; 67 56
+1
kernel/configs/hardening.config
··· 20 20 # Randomize allocator freelists, harden metadata. 21 21 CONFIG_SLAB_FREELIST_RANDOM=y 22 22 CONFIG_SLAB_FREELIST_HARDENED=y 23 + CONFIG_SLAB_BUCKETS=y 23 24 CONFIG_SHUFFLE_PAGE_ALLOCATOR=y 24 25 CONFIG_RANDOM_KMALLOC_CACHES=y 25 26
-2
lib/fortify_kunit.c
··· 234 234 checker(expected_size, \ 235 235 kmalloc_array_node(alloc_size, 1, gfp, NUMA_NO_NODE), \ 236 236 kfree(p)); \ 237 - checker(expected_size, __kmalloc(alloc_size, gfp), \ 238 - kfree(p)); \ 239 237 \ 240 238 orig = kmalloc(alloc_size, gfp); \ 241 239 KUNIT_EXPECT_TRUE(test, orig != NULL); \
+1 -1
lib/slub_kunit.c
··· 140 140 { 141 141 struct kmem_cache *s = test_kmem_cache_create("TestSlub_RZ_kmalloc", 32, 142 142 SLAB_KMALLOC|SLAB_STORE_USER|SLAB_RED_ZONE); 143 - u8 *p = kmalloc_trace(s, GFP_KERNEL, 18); 143 + u8 *p = __kmalloc_cache_noprof(s, GFP_KERNEL, 18); 144 144 145 145 kasan_disable_current(); 146 146
+17
mm/Kconfig
··· 273 273 sacrifices to harden the kernel slab allocator against common 274 274 freelist exploit methods. 275 275 276 + config SLAB_BUCKETS 277 + bool "Support allocation from separate kmalloc buckets" 278 + depends on !SLUB_TINY 279 + default SLAB_FREELIST_HARDENED 280 + help 281 + Kernel heap attacks frequently depend on being able to create 282 + specifically-sized allocations with user-controlled contents 283 + that will be allocated into the same kmalloc bucket as a 284 + target object. To avoid sharing these allocation buckets, 285 + provide an explicitly separated set of buckets to be used for 286 + user-controlled allocations. This may very slightly increase 287 + memory fragmentation, though in practice it's only a handful 288 + of extra pages since the bulk of user-controlled allocations 289 + are relatively long-lived. 290 + 291 + If unsure, say Y. 292 + 276 293 config SLUB_STATS 277 294 default n 278 295 bool "Enable performance statistics"
+9 -5
mm/slab.h
··· 97 97 SLAB_MATCH(flags, __page_flags); 98 98 SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ 99 99 SLAB_MATCH(_refcount, __page_refcount); 100 - #ifdef CONFIG_SLAB_OBJ_EXT 100 + #ifdef CONFIG_MEMCG 101 101 SLAB_MATCH(memcg_data, obj_exts); 102 + #elif defined(CONFIG_SLAB_OBJ_EXT) 103 + SLAB_MATCH(_unused_slab_obj_exts, obj_exts); 102 104 #endif 103 105 #undef SLAB_MATCH 104 106 static_assert(sizeof(struct slab) <= sizeof(struct page)); ··· 168 166 */ 169 167 static inline bool slab_test_pfmemalloc(const struct slab *slab) 170 168 { 171 - return folio_test_active((struct folio *)slab_folio(slab)); 169 + return folio_test_active(slab_folio(slab)); 172 170 } 173 171 174 172 static inline void slab_set_pfmemalloc(struct slab *slab) ··· 213 211 214 212 static inline int slab_order(const struct slab *slab) 215 213 { 216 - return folio_order((struct folio *)slab_folio(slab)); 214 + return folio_order(slab_folio(slab)); 217 215 } 218 216 219 217 static inline size_t slab_size(const struct slab *slab) ··· 405 403 * KMALLOC_MAX_CACHE_SIZE and the caller must check that. 406 404 */ 407 405 static inline struct kmem_cache * 408 - kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) 406 + kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) 409 407 { 410 408 unsigned int index; 411 409 410 + if (!b) 411 + b = &kmalloc_caches[kmalloc_type(flags, caller)]; 412 412 if (size <= 192) 413 413 index = kmalloc_size_index[size_index_elem(size)]; 414 414 else 415 415 index = fls(size - 1); 416 416 417 - return kmalloc_caches[kmalloc_type(flags, caller)][index]; 417 + return (*b)[index]; 418 418 } 419 419 420 420 gfp_t kmalloc_fix_flags(gfp_t flags);
+104 -7
mm/slab_common.c
··· 392 392 } 393 393 EXPORT_SYMBOL(kmem_cache_create); 394 394 395 + static struct kmem_cache *kmem_buckets_cache __ro_after_init; 396 + 397 + /** 398 + * kmem_buckets_create - Create a set of caches that handle dynamic sized 399 + * allocations via kmem_buckets_alloc() 400 + * @name: A prefix string which is used in /proc/slabinfo to identify this 401 + * cache. The individual caches with have their sizes as the suffix. 402 + * @flags: SLAB flags (see kmem_cache_create() for details). 403 + * @useroffset: Starting offset within an allocation that may be copied 404 + * to/from userspace. 405 + * @usersize: How many bytes, starting at @useroffset, may be copied 406 + * to/from userspace. 407 + * @ctor: A constructor for the objects, run when new allocations are made. 408 + * 409 + * Cannot be called within an interrupt, but can be interrupted. 410 + * 411 + * Return: a pointer to the cache on success, NULL on failure. When 412 + * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and 413 + * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). 414 + * (i.e. callers only need to check for NULL on failure.) 415 + */ 416 + kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, 417 + unsigned int useroffset, 418 + unsigned int usersize, 419 + void (*ctor)(void *)) 420 + { 421 + kmem_buckets *b; 422 + int idx; 423 + 424 + /* 425 + * When the separate buckets API is not built in, just return 426 + * a non-NULL value for the kmem_buckets pointer, which will be 427 + * unused when performing allocations. 428 + */ 429 + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) 430 + return ZERO_SIZE_PTR; 431 + 432 + if (WARN_ON(!kmem_buckets_cache)) 433 + return NULL; 434 + 435 + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); 436 + if (WARN_ON(!b)) 437 + return NULL; 438 + 439 + flags |= SLAB_NO_MERGE; 440 + 441 + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { 442 + char *short_size, *cache_name; 443 + unsigned int cache_useroffset, cache_usersize; 444 + unsigned int size; 445 + 446 + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) 447 + continue; 448 + 449 + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; 450 + if (!size) 451 + continue; 452 + 453 + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); 454 + if (WARN_ON(!short_size)) 455 + goto fail; 456 + 457 + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); 458 + if (WARN_ON(!cache_name)) 459 + goto fail; 460 + 461 + if (useroffset >= size) { 462 + cache_useroffset = 0; 463 + cache_usersize = 0; 464 + } else { 465 + cache_useroffset = useroffset; 466 + cache_usersize = min(size - cache_useroffset, usersize); 467 + } 468 + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, 469 + 0, flags, cache_useroffset, 470 + cache_usersize, ctor); 471 + kfree(cache_name); 472 + if (WARN_ON(!(*b)[idx])) 473 + goto fail; 474 + } 475 + 476 + return b; 477 + 478 + fail: 479 + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) 480 + kmem_cache_destroy((*b)[idx]); 481 + kfree(b); 482 + 483 + return NULL; 484 + } 485 + EXPORT_SYMBOL(kmem_buckets_create); 486 + 395 487 #ifdef SLAB_SUPPORTS_SYSFS 396 488 /* 397 489 * For a given kmem_cache, kmem_cache_destroy() should only be called ··· 709 617 s->size = s->object_size = size; 710 618 711 619 /* 712 - * For power of two sizes, guarantee natural alignment for kmalloc 713 - * caches, regardless of SL*B debugging options. 620 + * kmalloc caches guarantee alignment of at least the largest 621 + * power-of-two divisor of the size. For power-of-two sizes, 622 + * it is the size itself. 714 623 */ 715 - if (is_power_of_2(size)) 716 - align = max(align, size); 624 + if (flags & SLAB_KMALLOC) 625 + align = max(align, 1U << (ffs(size) - 1)); 717 626 s->align = calculate_alignment(flags, align, size); 718 627 719 628 #ifdef CONFIG_HARDENED_USERCOPY ··· 746 653 return s; 747 654 } 748 655 749 - struct kmem_cache * 750 - kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = 656 + kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = 751 657 { /* initialization for https://llvm.org/pr42570 */ }; 752 658 EXPORT_SYMBOL(kmalloc_caches); 753 659 ··· 795 703 * The flags don't matter since size_index is common to all. 796 704 * Neither does the caller for just getting ->object_size. 797 705 */ 798 - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; 706 + return kmalloc_slab(size, NULL, GFP_KERNEL, 0)->object_size; 799 707 } 800 708 801 709 /* Above the smaller buckets, size is a multiple of page size. */ ··· 1024 932 1025 933 /* Kmalloc array is now usable */ 1026 934 slab_state = UP; 935 + 936 + if (IS_ENABLED(CONFIG_SLAB_BUCKETS)) 937 + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", 938 + sizeof(kmem_buckets), 939 + 0, SLAB_NO_MERGE, NULL); 1027 940 } 1028 941 1029 942 /**
+119 -92
mm/slub.c
··· 788 788 kunit_put_resource(resource); 789 789 return true; 790 790 } 791 + 792 + static bool slab_in_kunit_test(void) 793 + { 794 + struct kunit_resource *resource; 795 + 796 + if (!kunit_get_current_test()) 797 + return false; 798 + 799 + resource = kunit_find_named_resource(current->kunit_test, "slab_errors"); 800 + if (!resource) 801 + return false; 802 + 803 + kunit_put_resource(resource); 804 + return true; 805 + } 791 806 #else 792 807 static inline bool slab_add_kunit_errors(void) { return false; } 808 + static inline bool slab_in_kunit_test(void) { return false; } 793 809 #endif 794 810 795 811 static inline unsigned int size_from_object(struct kmem_cache *s) ··· 978 962 979 963 static void print_slab_info(const struct slab *slab) 980 964 { 981 - struct folio *folio = (struct folio *)slab_folio(slab); 982 - 983 965 pr_err("Slab 0x%p objects=%u used=%u fp=0x%p flags=%pGp\n", 984 966 slab, slab->objects, slab->inuse, slab->freelist, 985 - folio_flags(folio, 0)); 967 + &slab->__page_flags); 986 968 } 987 969 988 970 /* ··· 1206 1192 pr_err("0x%p-0x%p @offset=%tu. First byte 0x%x instead of 0x%x\n", 1207 1193 fault, end - 1, fault - addr, 1208 1194 fault[0], value); 1209 - print_trailer(s, slab, object); 1210 - add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); 1211 1195 1212 1196 skip_bug_print: 1213 1197 restore_bytes(s, what, value, fault, end); ··· 1228 1216 * Padding is extended by another word if Redzoning is enabled and 1229 1217 * object_size == inuse. 1230 1218 * 1231 - * We fill with 0xbb (RED_INACTIVE) for inactive objects and with 1232 - * 0xcc (RED_ACTIVE) for objects in use. 1219 + * We fill with 0xbb (SLUB_RED_INACTIVE) for inactive objects and with 1220 + * 0xcc (SLUB_RED_ACTIVE) for objects in use. 1233 1221 * 1234 1222 * object + s->inuse 1235 1223 * Meta data starts here. ··· 1314 1302 u8 *p = object; 1315 1303 u8 *endobject = object + s->object_size; 1316 1304 unsigned int orig_size, kasan_meta_size; 1305 + int ret = 1; 1317 1306 1318 1307 if (s->flags & SLAB_RED_ZONE) { 1319 1308 if (!check_bytes_and_report(s, slab, object, "Left Redzone", 1320 1309 object - s->red_left_pad, val, s->red_left_pad)) 1321 - return 0; 1310 + ret = 0; 1322 1311 1323 1312 if (!check_bytes_and_report(s, slab, object, "Right Redzone", 1324 1313 endobject, val, s->inuse - s->object_size)) 1325 - return 0; 1314 + ret = 0; 1326 1315 1327 1316 if (slub_debug_orig_size(s) && val == SLUB_RED_ACTIVE) { 1328 1317 orig_size = get_orig_size(s, object); ··· 1332 1319 !check_bytes_and_report(s, slab, object, 1333 1320 "kmalloc Redzone", p + orig_size, 1334 1321 val, s->object_size - orig_size)) { 1335 - return 0; 1322 + ret = 0; 1336 1323 } 1337 1324 } 1338 1325 } else { 1339 1326 if ((s->flags & SLAB_POISON) && s->object_size < s->inuse) { 1340 - check_bytes_and_report(s, slab, p, "Alignment padding", 1327 + if (!check_bytes_and_report(s, slab, p, "Alignment padding", 1341 1328 endobject, POISON_INUSE, 1342 - s->inuse - s->object_size); 1329 + s->inuse - s->object_size)) 1330 + ret = 0; 1343 1331 } 1344 1332 } 1345 1333 ··· 1356 1342 !check_bytes_and_report(s, slab, p, "Poison", 1357 1343 p + kasan_meta_size, POISON_FREE, 1358 1344 s->object_size - kasan_meta_size - 1)) 1359 - return 0; 1345 + ret = 0; 1360 1346 if (kasan_meta_size < s->object_size && 1361 1347 !check_bytes_and_report(s, slab, p, "End Poison", 1362 1348 p + s->object_size - 1, POISON_END, 1)) 1363 - return 0; 1349 + ret = 0; 1364 1350 } 1365 1351 /* 1366 1352 * check_pad_bytes cleans up on its own. 1367 1353 */ 1368 - check_pad_bytes(s, slab, p); 1354 + if (!check_pad_bytes(s, slab, p)) 1355 + ret = 0; 1369 1356 } 1370 1357 1371 - if (!freeptr_outside_object(s) && val == SLUB_RED_ACTIVE) 1372 - /* 1373 - * Object and freepointer overlap. Cannot check 1374 - * freepointer while object is allocated. 1375 - */ 1376 - return 1; 1377 - 1378 - /* Check free pointer validity */ 1379 - if (!check_valid_pointer(s, slab, get_freepointer(s, p))) { 1358 + /* 1359 + * Cannot check freepointer while object is allocated if 1360 + * object and freepointer overlap. 1361 + */ 1362 + if ((freeptr_outside_object(s) || val != SLUB_RED_ACTIVE) && 1363 + !check_valid_pointer(s, slab, get_freepointer(s, p))) { 1380 1364 object_err(s, slab, p, "Freepointer corrupt"); 1381 1365 /* 1382 1366 * No choice but to zap it and thus lose the remainder ··· 1382 1370 * another error because the object count is now wrong. 1383 1371 */ 1384 1372 set_freepointer(s, p, NULL); 1385 - return 0; 1373 + ret = 0; 1386 1374 } 1387 - return 1; 1375 + 1376 + if (!ret && !slab_in_kunit_test()) { 1377 + print_trailer(s, slab, object); 1378 + add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); 1379 + } 1380 + 1381 + return ret; 1388 1382 } 1389 1383 1390 1384 static int check_slab(struct kmem_cache *s, struct slab *slab) ··· 2027 2009 return false; 2028 2010 } 2029 2011 2012 + #else /* CONFIG_SLAB_OBJ_EXT */ 2013 + 2014 + static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, 2015 + gfp_t gfp, bool new_slab) 2016 + { 2017 + return 0; 2018 + } 2019 + 2020 + static inline void free_slab_obj_exts(struct slab *slab) 2021 + { 2022 + } 2023 + 2024 + static inline bool need_slab_obj_ext(void) 2025 + { 2026 + return false; 2027 + } 2028 + 2029 + #endif /* CONFIG_SLAB_OBJ_EXT */ 2030 + 2031 + #ifdef CONFIG_MEM_ALLOC_PROFILING 2032 + 2030 2033 static inline struct slabobj_ext * 2031 2034 prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) 2032 2035 { ··· 2073 2034 } 2074 2035 2075 2036 static inline void 2037 + alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) 2038 + { 2039 + if (need_slab_obj_ext()) { 2040 + struct slabobj_ext *obj_exts; 2041 + 2042 + obj_exts = prepare_slab_obj_exts_hook(s, flags, object); 2043 + /* 2044 + * Currently obj_exts is used only for allocation profiling. 2045 + * If other users appear then mem_alloc_profiling_enabled() 2046 + * check should be added before alloc_tag_add(). 2047 + */ 2048 + if (likely(obj_exts)) 2049 + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); 2050 + } 2051 + } 2052 + 2053 + static inline void 2076 2054 alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, 2077 2055 int objects) 2078 2056 { 2079 - #ifdef CONFIG_MEM_ALLOC_PROFILING 2080 2057 struct slabobj_ext *obj_exts; 2081 2058 int i; 2082 2059 ··· 2108 2053 2109 2054 alloc_tag_sub(&obj_exts[off].ref, s->size); 2110 2055 } 2111 - #endif 2112 2056 } 2113 2057 2114 - #else /* CONFIG_SLAB_OBJ_EXT */ 2058 + #else /* CONFIG_MEM_ALLOC_PROFILING */ 2115 2059 2116 - static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, 2117 - gfp_t gfp, bool new_slab) 2060 + static inline void 2061 + alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) 2118 2062 { 2119 - return 0; 2120 - } 2121 - 2122 - static inline void free_slab_obj_exts(struct slab *slab) 2123 - { 2124 - } 2125 - 2126 - static inline bool need_slab_obj_ext(void) 2127 - { 2128 - return false; 2129 - } 2130 - 2131 - static inline struct slabobj_ext * 2132 - prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) 2133 - { 2134 - return NULL; 2135 2063 } 2136 2064 2137 2065 static inline void ··· 2123 2085 { 2124 2086 } 2125 2087 2126 - #endif /* CONFIG_SLAB_OBJ_EXT */ 2088 + #endif /* CONFIG_MEM_ALLOC_PROFILING */ 2089 + 2127 2090 2128 2091 #ifdef CONFIG_MEMCG_KMEM 2129 2092 ··· 2572 2533 */ 2573 2534 static inline bool slab_test_node_partial(const struct slab *slab) 2574 2535 { 2575 - return folio_test_workingset((struct folio *)slab_folio(slab)); 2536 + return folio_test_workingset(slab_folio(slab)); 2576 2537 } 2577 2538 2578 2539 static inline void slab_set_node_partial(struct slab *slab) ··· 3983 3944 kmemleak_alloc_recursive(p[i], s->object_size, 1, 3984 3945 s->flags, init_flags); 3985 3946 kmsan_slab_alloc(s, p[i], init_flags); 3986 - #ifdef CONFIG_MEM_ALLOC_PROFILING 3987 - if (need_slab_obj_ext()) { 3988 - struct slabobj_ext *obj_exts; 3989 - 3990 - obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); 3991 - /* 3992 - * Currently obj_exts is used only for allocation profiling. 3993 - * If other users appear then mem_alloc_profiling_enabled() 3994 - * check should be added before alloc_tag_add(). 3995 - */ 3996 - if (likely(obj_exts)) 3997 - alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); 3998 - } 3999 - #endif 3947 + alloc_tagging_slab_alloc_hook(s, p[i], flags); 4000 3948 } 4001 3949 4002 3950 return memcg_slab_post_alloc_hook(s, lru, flags, size, p); ··· 4081 4055 * directly to the page allocator. We use __GFP_COMP, because we will need to 4082 4056 * know the allocation order to free the pages properly in kfree. 4083 4057 */ 4084 - static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) 4058 + static void *___kmalloc_large_node(size_t size, gfp_t flags, int node) 4085 4059 { 4086 4060 struct folio *folio; 4087 4061 void *ptr = NULL; ··· 4106 4080 return ptr; 4107 4081 } 4108 4082 4109 - void *kmalloc_large_noprof(size_t size, gfp_t flags) 4083 + void *__kmalloc_large_noprof(size_t size, gfp_t flags) 4110 4084 { 4111 - void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); 4085 + void *ret = ___kmalloc_large_node(size, flags, NUMA_NO_NODE); 4112 4086 4113 4087 trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << get_order(size), 4114 4088 flags, NUMA_NO_NODE); 4115 4089 return ret; 4116 4090 } 4117 - EXPORT_SYMBOL(kmalloc_large_noprof); 4091 + EXPORT_SYMBOL(__kmalloc_large_noprof); 4118 4092 4119 - void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) 4093 + void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) 4120 4094 { 4121 - void *ret = __kmalloc_large_node(size, flags, node); 4095 + void *ret = ___kmalloc_large_node(size, flags, node); 4122 4096 4123 4097 trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << get_order(size), 4124 4098 flags, node); 4125 4099 return ret; 4126 4100 } 4127 - EXPORT_SYMBOL(kmalloc_large_node_noprof); 4101 + EXPORT_SYMBOL(__kmalloc_large_node_noprof); 4128 4102 4129 4103 static __always_inline 4130 - void *__do_kmalloc_node(size_t size, gfp_t flags, int node, 4104 + void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, 4131 4105 unsigned long caller) 4132 4106 { 4133 4107 struct kmem_cache *s; 4134 4108 void *ret; 4135 4109 4136 4110 if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { 4137 - ret = __kmalloc_large_node(size, flags, node); 4111 + ret = __kmalloc_large_node_noprof(size, flags, node); 4138 4112 trace_kmalloc(caller, ret, size, 4139 4113 PAGE_SIZE << get_order(size), flags, node); 4140 4114 return ret; ··· 4143 4117 if (unlikely(!size)) 4144 4118 return ZERO_SIZE_PTR; 4145 4119 4146 - s = kmalloc_slab(size, flags, caller); 4120 + s = kmalloc_slab(size, b, flags, caller); 4147 4121 4148 4122 ret = slab_alloc_node(s, NULL, flags, node, caller, size); 4149 4123 ret = kasan_kmalloc(s, ret, size, flags); 4150 4124 trace_kmalloc(caller, ret, size, s->size, flags, node); 4151 4125 return ret; 4152 4126 } 4153 - 4154 - void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) 4127 + void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) 4155 4128 { 4156 - return __do_kmalloc_node(size, flags, node, _RET_IP_); 4129 + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_); 4157 4130 } 4158 4131 EXPORT_SYMBOL(__kmalloc_node_noprof); 4159 4132 4160 4133 void *__kmalloc_noprof(size_t size, gfp_t flags) 4161 4134 { 4162 - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); 4135 + return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_); 4163 4136 } 4164 4137 EXPORT_SYMBOL(__kmalloc_noprof); 4165 4138 4166 - void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, 4167 - int node, unsigned long caller) 4139 + void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, 4140 + int node, unsigned long caller) 4168 4141 { 4169 - return __do_kmalloc_node(size, flags, node, caller); 4170 - } 4171 - EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); 4142 + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller); 4172 4143 4173 - void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) 4144 + } 4145 + EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof); 4146 + 4147 + void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) 4174 4148 { 4175 4149 void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, 4176 4150 _RET_IP_, size); ··· 4180 4154 ret = kasan_kmalloc(s, ret, size, gfpflags); 4181 4155 return ret; 4182 4156 } 4183 - EXPORT_SYMBOL(kmalloc_trace_noprof); 4157 + EXPORT_SYMBOL(__kmalloc_cache_noprof); 4184 4158 4185 - void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, 4186 - int node, size_t size) 4159 + void *__kmalloc_cache_node_noprof(struct kmem_cache *s, gfp_t gfpflags, 4160 + int node, size_t size) 4187 4161 { 4188 4162 void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); 4189 4163 ··· 4192 4166 ret = kasan_kmalloc(s, ret, size, gfpflags); 4193 4167 return ret; 4194 4168 } 4195 - EXPORT_SYMBOL(kmalloc_node_trace_noprof); 4169 + EXPORT_SYMBOL(__kmalloc_cache_node_noprof); 4196 4170 4197 4171 static noinline void free_to_partial_list( 4198 4172 struct kmem_cache *s, struct slab *slab, ··· 5177 5151 */ 5178 5152 s->inuse = size; 5179 5153 5180 - if (slub_debug_orig_size(s) || 5181 - (flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) || 5182 - ((flags & SLAB_RED_ZONE) && s->object_size < sizeof(void *)) || 5183 - s->ctor) { 5154 + if ((flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) || s->ctor || 5155 + ((flags & SLAB_RED_ZONE) && 5156 + (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { 5184 5157 /* 5185 5158 * Relocate free pointer after the object if it is not 5186 5159 * permitted to overwrite the first word of the object on ··· 5187 5162 * 5188 5163 * This is the case if we do RCU, have a constructor or 5189 5164 * destructor, are poisoning the objects, or are 5190 - * redzoning an object smaller than sizeof(void *). 5165 + * redzoning an object smaller than sizeof(void *) or are 5166 + * redzoning an object with slub_debug_orig_size() enabled, 5167 + * in which case the right redzone may be extended. 5191 5168 * 5192 5169 * The assumption that s->offset >= s->inuse means free 5193 5170 * pointer is outside of the object is used in the
+17 -6
mm/util.c
··· 200 200 } 201 201 EXPORT_SYMBOL(kmemdup_nul); 202 202 203 + static kmem_buckets *user_buckets __ro_after_init; 204 + 205 + static int __init init_user_buckets(void) 206 + { 207 + user_buckets = kmem_buckets_create("memdup_user", 0, 0, INT_MAX, NULL); 208 + 209 + return 0; 210 + } 211 + subsys_initcall(init_user_buckets); 212 + 203 213 /** 204 214 * memdup_user - duplicate memory region from user space 205 215 * ··· 223 213 { 224 214 void *p; 225 215 226 - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); 216 + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); 227 217 if (!p) 228 218 return ERR_PTR(-ENOMEM); 229 219 ··· 249 239 { 250 240 void *p; 251 241 252 - p = kvmalloc(len, GFP_USER); 242 + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); 253 243 if (!p) 254 244 return ERR_PTR(-ENOMEM); 255 245 ··· 609 599 EXPORT_SYMBOL(vm_mmap); 610 600 611 601 /** 612 - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon 602 + * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon 613 603 * failure, fall back to non-contiguous (vmalloc) allocation. 614 604 * @size: size of the request. 605 + * @b: which set of kmalloc buckets to allocate from. 615 606 * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. 616 607 * @node: numa node to allocate from 617 608 * ··· 625 614 * 626 615 * Return: pointer to the allocated memory of %NULL in case of failure 627 616 */ 628 - void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) 617 + void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) 629 618 { 630 619 gfp_t kmalloc_flags = flags; 631 620 void *ret; ··· 647 636 kmalloc_flags &= ~__GFP_NOFAIL; 648 637 } 649 638 650 - ret = kmalloc_node_noprof(size, kmalloc_flags, node); 639 + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), kmalloc_flags, node); 651 640 652 641 /* 653 642 * It doesn't really make sense to fallback to vmalloc for sub page ··· 676 665 flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, 677 666 node, __builtin_return_address(0)); 678 667 } 679 - EXPORT_SYMBOL(kvmalloc_node_noprof); 668 + EXPORT_SYMBOL(__kvmalloc_node_noprof); 680 669 681 670 /** 682 671 * kvfree() - Free memory.
+6 -13
rust/kernel/alloc/allocator.rs
··· 18 18 // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first. 19 19 let layout = new_layout.pad_to_align(); 20 20 21 - let mut size = layout.size(); 22 - 23 - if layout.align() > bindings::ARCH_SLAB_MINALIGN { 24 - // The alignment requirement exceeds the slab guarantee, thus try to enlarge the size 25 - // to use the "power-of-two" size/alignment guarantee (see comments in `kmalloc()` for 26 - // more information). 27 - // 28 - // Note that `layout.size()` (after padding) is guaranteed to be a multiple of 29 - // `layout.align()`, so `next_power_of_two` gives enough alignment guarantee. 30 - size = size.next_power_of_two(); 31 - } 21 + // Note that `layout.size()` (after padding) is guaranteed to be a multiple of `layout.align()` 22 + // which together with the slab guarantees means the `krealloc` will return a properly aligned 23 + // object (see comments in `kmalloc()` for more information). 24 + let size = layout.size(); 32 25 33 26 // SAFETY: 34 27 // - `ptr` is either null or a pointer returned from a previous `k{re}alloc()` by the 35 28 // function safety requirement. 36 - // - `size` is greater than 0 since it's either a `layout.size()` (which cannot be zero 37 - // according to the function safety requirement) or a result from `next_power_of_two()`. 29 + // - `size` is greater than 0 since it's from `layout.size()` (which cannot be zero according 30 + // to the function safety requirement) 38 31 unsafe { bindings::krealloc(ptr as *const core::ffi::c_void, size, flags.0) as *mut u8 } 39 32 } 40 33
+1
scripts/kernel-doc
··· 1729 1729 $prototype =~ s/__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +//; 1730 1730 $prototype =~ s/__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +//; 1731 1731 $prototype =~ s/__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +//; 1732 + $prototype =~ s/DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)/$1, $2/; 1732 1733 my $define = $prototype =~ s/^#\s*define\s+//; #ak added 1733 1734 $prototype =~ s/__attribute_const__ +//; 1734 1735 $prototype =~ s/__attribute__\s*\(\(
+2 -5
tools/include/linux/poison.h
··· 47 47 * Magic nums for obj red zoning. 48 48 * Placed in the first word before and the first word after an obj. 49 49 */ 50 - #define RED_INACTIVE 0x09F911029D74E35BULL /* when obj is inactive */ 51 - #define RED_ACTIVE 0xD84156C5635688C0ULL /* when obj is active */ 52 - 53 - #define SLUB_RED_INACTIVE 0xbb 54 - #define SLUB_RED_ACTIVE 0xcc 50 + #define SLUB_RED_INACTIVE 0xbb /* when obj is inactive */ 51 + #define SLUB_RED_ACTIVE 0xcc /* when obj is active */ 55 52 56 53 /* ...and for poisoning */ 57 54 #define POISON_INUSE 0x5a /* for use-uninitialised poisoning */