Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'cgroup-for-6.19-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup fix from Tejun Heo:
- Fix a race condition in css_rstat_updated() where CMPXCHG without
LOCK prefix could cause lnode corruption when the flusher runs
concurrently on another CPU. The issue was introduced in 6.17 and
causes memcg stats to become corrupted in production.
* tag 'cgroup-for-6.19-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup:
cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated
+8
-5
+8
-5
kernel/cgroup/rstat.c
+8
-5
kernel/cgroup/rstat.c
···
71
71
{
72
72
struct llist_head *lhead;
73
73
struct css_rstat_cpu *rstatc;
74
-
struct css_rstat_cpu __percpu *rstatc_pcpu;
75
74
struct llist_node *self;
76
75
77
76
/*
···
103
104
/*
104
105
* This function can be renentered by irqs and nmis for the same cgroup
105
106
* and may try to insert the same per-cpu lnode into the llist. Note
106
-
* that llist_add() does not protect against such scenarios.
107
+
* that llist_add() does not protect against such scenarios. In addition
108
+
* this same per-cpu lnode can be modified through init_llist_node()
109
+
* from css_rstat_flush() running on a different CPU.
107
110
*
108
111
* To protect against such stacked contexts of irqs/nmis, we use the
109
112
* fact that lnode points to itself when not on a list and then use
110
-
* this_cpu_cmpxchg() to atomically set to NULL to select the winner
113
+
* try_cmpxchg() to atomically set to NULL to select the winner
111
114
* which will call llist_add(). The losers can assume the insertion is
112
115
* successful and the winner will eventually add the per-cpu lnode to
113
116
* the llist.
117
+
*
118
+
* Please note that we can not use this_cpu_cmpxchg() here as on some
119
+
* archs it is not safe against modifications from multiple CPUs.
114
120
*/
115
121
self = &rstatc->lnode;
116
-
rstatc_pcpu = css->rstat_cpu;
117
-
if (this_cpu_cmpxchg(rstatc_pcpu->lnode.next, self, NULL) != self)
122
+
if (!try_cmpxchg(&rstatc->lnode.next, &self, NULL))
118
123
return;
119
124
120
125
lhead = ss_lhead_cpu(css->ss, cpu);