Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
debugobjects: Refill per CPU pool more agressively
Right now the per CPU pools are only refilled when they become
empty. That's suboptimal especially when there are still non-freed objects
in the to free list.
Check whether an allocation from the per CPU pool emptied a batch and try
to allocate from the free pool if that still has objects available.
kmem_cache_alloc() kmem_cache_free()
Baseline: 295k 245k
Refill: 225k 173k
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Zhen Lei <thunder.leizhen@huawei.com>
Link: https://lore.kernel.org/all/20241007164914.439053085@linutronix.de
+18
+18
lib/debugobjects.c
+18
lib/debugobjects.c
···
255
255
256
256
if (likely(obj)) {
257
257
pcp->cnt--;
258
+
/*
259
+
* If this emptied a batch try to refill from the
260
+
* free pool. Don't do that if this was the top-most
261
+
* batch as pcpu_free() expects the per CPU pool
262
+
* to be less than ODEBUG_POOL_PERCPU_SIZE.
263
+
*/
264
+
if (unlikely(pcp->cnt < (ODEBUG_POOL_PERCPU_SIZE - ODEBUG_BATCH_SIZE) &&
265
+
!(pcp->cnt % ODEBUG_BATCH_SIZE))) {
266
+
/*
267
+
* Don't try to allocate from the regular pool here
268
+
* to not exhaust it prematurely.
269
+
*/
270
+
if (pool_count(&pool_to_free)) {
271
+
guard(raw_spinlock)(&pool_lock);
272
+
pool_move_batch(pcp, &pool_to_free);
273
+
pcpu_refill_stats();
274
+
}
275
+
}
258
276
return obj;
259
277
}
260
278