Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

sched/mmcid: Introduce per task/CPU ownership infrastructure

The MM CID management has two fundamental requirements:

1) It has to guarantee that at no given point in time the same CID is
used by concurrent tasks in userspace.

2) The CID space must not exceed the number of possible CPUs in a
system. While most allocators (glibc, tcmalloc, jemalloc) do not care
about that, there seems to be at least librseq depending on it.

The CID space compaction itself is not a functional correctness
requirement, it is only a useful optimization mechanism to reduce the
memory foot print in unused user space pools.

The optimal CID space is:

min(nr_tasks, nr_cpus_allowed);

Where @nr_tasks is the number of actual user space threads associated to
the mm and @nr_cpus_allowed is the superset of all task affinities. It is
growth only as it would be insane to take a racy snapshot of all task
affinities when the affinity of one task changes just do redo it 2
milliseconds later when the next task changes its affinity.

That means that as long as the number of tasks is lower or equal than the
number of CPUs allowed, each task owns a CID. If the number of tasks
exceeds the number of CPUs allowed it switches to per CPU mode, where the
CPUs own the CIDs and the tasks borrow them as long as they are scheduled
in.

For transition periods CIDs can go beyond the optimal space as long as they
don't go beyond the number of possible CPUs.

The current upstream implementation adds overhead into task migration to
keep the CID with the task. It also has to do the CID space consolidation
work from a task work in the exit to user space path. As that work is
assigned to a random task related to a MM this can inflict unwanted exit
latencies.

This can be done differently by implementing a strict CID ownership
mechanism. Either the CIDs are owned by the tasks or by the CPUs. The
latter provides less locality when tasks are heavily migrating, but there
is no justification to optimize for overcommit scenarios and thereby
penalizing everyone else.

Provide the basic infrastructure to implement this:

- Change the UNSET marker to BIT(31) from ~0U
- Add the ONCPU marker as BIT(30)
- Add the TRANSIT marker as BIT(29)

That allows to check for ownership trivially and provides a simple check for
UNSET as well. The TRANSIT marker is required to prevent CID space
exhaustion when switching from per CPU to per task mode.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://patch.msgid.link/20251119172549.960252358@linutronix.de

+75 -4
+3 -1
include/linux/rseq_types.h
··· 92 92 93 93 #ifdef CONFIG_SCHED_MM_CID 94 94 95 - #define MM_CID_UNSET (~0U) 95 + #define MM_CID_UNSET BIT(31) 96 + #define MM_CID_ONCPU BIT(30) 97 + #define MM_CID_TRANSIT BIT(29) 96 98 97 99 /** 98 100 * struct sched_mm_cid - Storage for per task MM CID data
+3 -3
include/linux/sched.h
··· 2299 2299 void sched_mm_cid_after_execve(struct task_struct *t); 2300 2300 void sched_mm_cid_fork(struct task_struct *t); 2301 2301 void sched_mm_cid_exit(struct task_struct *t); 2302 - static inline int task_mm_cid(struct task_struct *t) 2302 + static __always_inline int task_mm_cid(struct task_struct *t) 2303 2303 { 2304 - return t->mm_cid.cid; 2304 + return t->mm_cid.cid & ~(MM_CID_ONCPU | MM_CID_TRANSIT); 2305 2305 } 2306 2306 #else 2307 2307 static inline void sched_mm_cid_before_execve(struct task_struct *t) { } 2308 2308 static inline void sched_mm_cid_after_execve(struct task_struct *t) { } 2309 2309 static inline void sched_mm_cid_fork(struct task_struct *t) { } 2310 2310 static inline void sched_mm_cid_exit(struct task_struct *t) { } 2311 - static inline int task_mm_cid(struct task_struct *t) 2311 + static __always_inline int task_mm_cid(struct task_struct *t) 2312 2312 { 2313 2313 /* 2314 2314 * Use the processor id as a fall-back when the mm cid feature is
+10
kernel/sched/core.c
··· 10386 10386 * 10387 10387 * The mm::mm_cid:pcpu per CPU storage is protected by the CPUs runqueue 10388 10388 * lock. 10389 + * 10390 + * CID ownership: 10391 + * 10392 + * A CID is either owned by a task (stored in task_struct::mm_cid.cid) or 10393 + * by a CPU (stored in mm::mm_cid.pcpu::cid). CIDs owned by CPUs have the 10394 + * MM_CID_ONCPU bit set. During transition from CPU to task ownership mode, 10395 + * MM_CID_TRANSIT is set on the per task CIDs. When this bit is set the 10396 + * task needs to drop the CID into the pool when scheduling out. Both bits 10397 + * (ONCPU and TRANSIT) are filtered out by task_cid() when the CID is 10398 + * actually handed over to user space in the RSEQ memory. 10389 10399 */ 10390 10400 10391 10401 /*
+59
kernel/sched/sched.h
··· 3540 3540 extern const char *preempt_modes[]; 3541 3541 3542 3542 #ifdef CONFIG_SCHED_MM_CID 3543 + 3544 + static __always_inline bool cid_on_cpu(unsigned int cid) 3545 + { 3546 + return cid & MM_CID_ONCPU; 3547 + } 3548 + 3549 + static __always_inline bool cid_in_transit(unsigned int cid) 3550 + { 3551 + return cid & MM_CID_TRANSIT; 3552 + } 3553 + 3554 + static __always_inline unsigned int cpu_cid_to_cid(unsigned int cid) 3555 + { 3556 + return cid & ~MM_CID_ONCPU; 3557 + } 3558 + 3559 + static __always_inline unsigned int cid_to_cpu_cid(unsigned int cid) 3560 + { 3561 + return cid | MM_CID_ONCPU; 3562 + } 3563 + 3564 + static __always_inline unsigned int cid_to_transit_cid(unsigned int cid) 3565 + { 3566 + return cid | MM_CID_TRANSIT; 3567 + } 3568 + 3569 + static __always_inline unsigned int cid_from_transit_cid(unsigned int cid) 3570 + { 3571 + return cid & ~MM_CID_TRANSIT; 3572 + } 3573 + 3574 + static __always_inline bool cid_on_task(unsigned int cid) 3575 + { 3576 + /* True if none of the MM_CID_ONCPU, MM_CID_TRANSIT, MM_CID_UNSET bits is set */ 3577 + return cid < MM_CID_TRANSIT; 3578 + } 3579 + 3580 + static __always_inline void mm_drop_cid(struct mm_struct *mm, unsigned int cid) 3581 + { 3582 + clear_bit(cid, mm_cidmask(mm)); 3583 + } 3584 + 3585 + static __always_inline void mm_unset_cid_on_task(struct task_struct *t) 3586 + { 3587 + unsigned int cid = t->mm_cid.cid; 3588 + 3589 + t->mm_cid.cid = MM_CID_UNSET; 3590 + if (cid_on_task(cid)) 3591 + mm_drop_cid(t->mm, cid); 3592 + } 3593 + 3594 + static __always_inline void mm_drop_cid_on_cpu(struct mm_struct *mm, struct mm_cid_pcpu *pcp) 3595 + { 3596 + /* Clear the ONCPU bit, but do not set UNSET in the per CPU storage */ 3597 + pcp->cid = cpu_cid_to_cid(pcp->cid); 3598 + mm_drop_cid(mm, pcp->cid); 3599 + } 3600 + 3601 + /* Active implementation */ 3543 3602 static inline void init_sched_mm_cid(struct task_struct *t) 3544 3603 { 3545 3604 struct mm_struct *mm = t->mm;