Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

memcg: consolidate private id refcount get/put helpers

We currently have two different sets of helpers for getting or putting the
private IDs' refcount for order 0 and large folios. This is redundant.
Just use one and always acquire the refcount of the swapout folio size
unless it's zero, and put the refcount using the folio size if the charge
failed, since the folio size can't change. Then there is no need to
update the refcount for tail pages.

Same for freeing, then only one pair of get/put helper is needed now.

The performance might be slightly better, too: both "inc unless zero" and
"add unless zero" use the same cmpxchg implementation. For large folios,
we saved an atomic operation. And for both order 0 and large folios, we
saved a branch.

Link: https://lkml.kernel.org/r/20260213-memcg-privid-v1-1-d8cb7afcf831@tencent.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Chen Ridong <chenridong@huaweicloud.com>
Acked-by: Shakeel Butt <shakeel.butt@gmail.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kairui Song and committed by
Andrew Morton
37cb8cd0 c9cb94c6

+10 -28
+1 -4
mm/memcontrol-v1.c
··· 635 635 * have an ID allocated to it anymore, charge the closest online 636 636 * ancestor for the swap instead and transfer the memory+swap charge. 637 637 */ 638 - swap_memcg = mem_cgroup_private_id_get_online(memcg); 639 638 nr_entries = folio_nr_pages(folio); 640 - /* Get references for the tail pages, too */ 641 - if (nr_entries > 1) 642 - mem_cgroup_private_id_get_many(swap_memcg, nr_entries - 1); 639 + swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries); 643 640 mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); 644 641 645 642 swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry);
+2 -2
mm/memcontrol-v1.h
··· 27 27 unsigned long memcg_events(struct mem_cgroup *memcg, int event); 28 28 int memory_stat_show(struct seq_file *m, void *v); 29 29 30 - void mem_cgroup_private_id_get_many(struct mem_cgroup *memcg, unsigned int n); 31 - struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg); 30 + struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, 31 + unsigned int n); 32 32 33 33 /* Cgroup v1-specific declarations */ 34 34 #ifdef CONFIG_MEMCG_V1
+7 -22
mm/memcontrol.c
··· 3634 3634 } 3635 3635 } 3636 3636 3637 - void __maybe_unused mem_cgroup_private_id_get_many(struct mem_cgroup *memcg, 3638 - unsigned int n) 3639 - { 3640 - refcount_add(n, &memcg->id.ref); 3641 - } 3642 - 3643 - static void mem_cgroup_private_id_put_many(struct mem_cgroup *memcg, unsigned int n) 3637 + static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned int n) 3644 3638 { 3645 3639 if (refcount_sub_and_test(n, &memcg->id.ref)) { 3646 3640 mem_cgroup_private_id_remove(memcg); ··· 3644 3650 } 3645 3651 } 3646 3652 3647 - static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg) 3653 + struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, unsigned int n) 3648 3654 { 3649 - mem_cgroup_private_id_put_many(memcg, 1); 3650 - } 3651 - 3652 - struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg) 3653 - { 3654 - while (!refcount_inc_not_zero(&memcg->id.ref)) { 3655 + while (!refcount_add_not_zero(n, &memcg->id.ref)) { 3655 3656 /* 3656 3657 * The root cgroup cannot be destroyed, so it's refcount must 3657 3658 * always be >= 1. ··· 3946 3957 3947 3958 drain_all_stock(memcg); 3948 3959 3949 - mem_cgroup_private_id_put(memcg); 3960 + mem_cgroup_private_id_put(memcg, 1); 3950 3961 } 3951 3962 3952 3963 static void mem_cgroup_css_released(struct cgroup_subsys_state *css) ··· 5236 5247 return 0; 5237 5248 } 5238 5249 5239 - memcg = mem_cgroup_private_id_get_online(memcg); 5250 + memcg = mem_cgroup_private_id_get_online(memcg, nr_pages); 5240 5251 5241 5252 if (!mem_cgroup_is_root(memcg) && 5242 5253 !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { 5243 5254 memcg_memory_event(memcg, MEMCG_SWAP_MAX); 5244 5255 memcg_memory_event(memcg, MEMCG_SWAP_FAIL); 5245 - mem_cgroup_private_id_put(memcg); 5256 + mem_cgroup_private_id_put(memcg, nr_pages); 5246 5257 return -ENOMEM; 5247 5258 } 5248 - 5249 - /* Get references for the tail pages, too */ 5250 - if (nr_pages > 1) 5251 - mem_cgroup_private_id_get_many(memcg, nr_pages - 1); 5252 5259 mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); 5253 5260 5254 5261 swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry); ··· 5273 5288 page_counter_uncharge(&memcg->swap, nr_pages); 5274 5289 } 5275 5290 mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); 5276 - mem_cgroup_private_id_put_many(memcg, nr_pages); 5291 + mem_cgroup_private_id_put(memcg, nr_pages); 5277 5292 } 5278 5293 rcu_read_unlock(); 5279 5294 }