mm/mremap: use an explicit uffd failure path for mremap

Right now it appears that the code is relying upon the returned
destination address having bits outside PAGE_MASK to indicate whether an
error value is specified, and decrementing the increased refcount on the
uffd ctx if so.

This is not a safe means of determining an error value, so instead, be
specific. It makes far more sense to do so in a dedicated error path, so
add mremap_userfaultfd_fail() for this purpose and use this when an error
arises.

A vm_userfaultfd_ctx is not established until we are at the point where
mremap_userfaultfd_prep() is invoked in copy_vma_and_data(), so this is a
no-op until this happens.

That is - uffd remap notification only occurs if the VMA is actually moved
- at which point a UFFD_EVENT_REMAP event is raised.

No errors can occur after this point currently, though it's certainly not
guaranteed this will always remain the case, and we mustn't rely on this.

However, the reason for needing to handle this case is that, when an error
arises on a VMA move at the point of adjusting page tables, we revert this
operation, and propagate the error.

At this point, it is not correct to raise a uffd remap event, and we must
handle it.

This refactoring makes it abundantly clear what we are doing.

We assume vrm->new_addr is always valid, which a prior change made the
case even for mremap() invocations which don't move the VMA, however given
no uffd context would be set up in this case it's immaterial to this
change anyway.

No functional change intended.

Link: https://lkml.kernel.org/r/a70e8a1f7bce9f43d1431065b414e0f212297297.1752770784.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes and committed by

Andrew Morton 10 months ago f9f11398 e49e76c2

+27 -9

3 changed files

expand all

userfaultfd.c

include

linux

userfaultfd_k.h

mremap.c

+10 -5

fs/userfaultfd.c

··· 750 750 if (!ctx) 751 751 return; 752 752 753 - if (to & ~PAGE_MASK) { 754 - userfaultfd_ctx_put(ctx); 755 - return; 756 - } 757 - 758 753 msg_init(&ewq.msg); 759 754 760 755 ewq.msg.event = UFFD_EVENT_REMAP; ··· 758 763 ewq.msg.arg.remap.len = len; 759 764 760 765 userfaultfd_event_wait_completion(ctx, &ewq); 766 + } 767 + 768 + void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx) 769 + { 770 + struct userfaultfd_ctx *ctx = vm_ctx->ctx; 771 + 772 + if (!ctx) 773 + return; 774 + 775 + userfaultfd_ctx_put(ctx); 761 776 } 762 777 763 778 bool userfaultfd_remove(struct vm_area_struct *vma,

include/linux/userfaultfd_k.h

··· 259 259 extern void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *, 260 260 unsigned long from, unsigned long to, 261 261 unsigned long len); 262 + void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *); 262 263 263 264 extern bool userfaultfd_remove(struct vm_area_struct *vma, 264 265 unsigned long start, ··· 369 368 unsigned long from, 370 369 unsigned long to, 371 370 unsigned long len) 371 + { 372 + } 373 + 374 + static inline void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *ctx) 372 375 { 373 376 } 374 377

+12 -4

mm/mremap.c

··· 1729 1729 return 0; 1730 1730 } 1731 1731 1732 - static void notify_uffd(struct vma_remap_struct *vrm, unsigned long to) 1732 + static void notify_uffd(struct vma_remap_struct *vrm, bool failed) 1733 1733 { 1734 1734 struct mm_struct *mm = current->mm; 1735 1735 1736 + /* Regardless of success/failure, we always notify of any unmaps. */ 1736 1737 userfaultfd_unmap_complete(mm, vrm->uf_unmap_early); 1737 - mremap_userfaultfd_complete(vrm->uf, vrm->addr, to, vrm->old_len); 1738 + if (failed) 1739 + mremap_userfaultfd_fail(vrm->uf); 1740 + else 1741 + mremap_userfaultfd_complete(vrm->uf, vrm->addr, 1742 + vrm->new_addr, vrm->old_len); 1738 1743 userfaultfd_unmap_complete(mm, vrm->uf_unmap); 1739 1744 } 1740 1745 ··· 1747 1742 { 1748 1743 struct mm_struct *mm = current->mm; 1749 1744 unsigned long res; 1745 + bool failed; 1750 1746 1751 1747 vrm->old_len = PAGE_ALIGN(vrm->old_len); 1752 1748 vrm->new_len = PAGE_ALIGN(vrm->new_len); ··· 1769 1763 res = vrm_implies_new_addr(vrm) ? mremap_to(vrm) : mremap_at(vrm); 1770 1764 1771 1765 out: 1766 + failed = IS_ERR_VALUE(res); 1767 + 1772 1768 if (vrm->mmap_locked) 1773 1769 mmap_write_unlock(mm); 1774 1770 1775 - if (!IS_ERR_VALUE(res) && vrm->mlocked && vrm->new_len > vrm->old_len) 1771 + if (!failed && vrm->mlocked && vrm->new_len > vrm->old_len) 1776 1772 mm_populate(vrm->new_addr + vrm->old_len, vrm->delta); 1777 1773 1778 - notify_uffd(vrm, res); 1774 + notify_uffd(vrm, failed); 1779 1775 return res; 1780 1776 } 1781 1777

Configure Feed

Configure Feed