page_pool: fragment API support for 32-bit arch with 64-bit DMA

Currently page_pool_alloc_frag() is not supported in 32-bit
arch with 64-bit DMA because of the overlap issue between
pp_frag_count and dma_addr_upper in 'struct page' for those
arches, which seems to be quite common, see [1], which means
driver may need to handle it when using fragment API.

It is assumed that the combination of the above arch with an
address space >16TB does not exist, as all those arches have
64b equivalent, it seems logical to use the 64b version for a
system with a large address space. It is also assumed that dma
address is page aligned when we are dma mapping a page aligned
buffer, see [2].

That means we're storing 12 bits of 0 at the lower end for a
dma address, we can reuse those bits for the above arches to
support 32b+12b, which is 16TB of memory.

If we make a wrong assumption, a warning is emitted so that
user can report to us.

1. https://lore.kernel.org/all/20211117075652.58299-1-linyunsheng@huawei.com/
2. https://lore.kernel.org/all/20230818145145.4b357c89@kernel.org/

Tested-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
CC: Lorenzo Bianconi <lorenzo@kernel.org>
CC: Alexander Duyck <alexander.duyck@gmail.com>
CC: Liang Chen <liangchen.linux@gmail.com>
CC: Guillaume Tucker <guillaume.tucker@collabora.com>
CC: Matthew Wilcox <willy@infradead.org>
CC: Linux-MM <linux-mm@kvack.org>
Link: https://lore.kernel.org/r/20231013064827.61135-2-linyunsheng@huawei.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

authored by

Yunsheng Lin and committed by

Jakub Kicinski 2 years ago 90de47f0 e411a8e3

+24 -23

3 changed files

expand all

include

linux

mm_types.h

net

page_pool

helpers.h

net

core

page_pool.c

+1 -12

include/linux/mm_types.h

··· 125 125 struct page_pool *pp; 126 126 unsigned long _pp_mapping_pad; 127 127 unsigned long dma_addr; 128 - union { 129 - /** 130 - * dma_addr_upper: might require a 64-bit 131 - * value on 32-bit architectures. 132 - */ 133 - unsigned long dma_addr_upper; 134 - /** 135 - * For frag page support, not supported in 136 - * 32-bit architectures with 64-bit DMA. 137 - */ 138 - atomic_long_t pp_frag_count; 139 - }; 128 + atomic_long_t pp_frag_count; 140 129 }; 141 130 struct { /* Tail pages of compound page */ 142 131 unsigned long compound_head; /* Bit zero is set */

+14 -6

include/net/page_pool/helpers.h

··· 197 197 page_pool_put_full_page(pool, page, true); 198 198 } 199 199 200 - #define PAGE_POOL_DMA_USE_PP_FRAG_COUNT \ 200 + #define PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA \ 201 201 (sizeof(dma_addr_t) > sizeof(unsigned long)) 202 202 203 203 /** ··· 211 211 { 212 212 dma_addr_t ret = page->dma_addr; 213 213 214 - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) 215 - ret |= (dma_addr_t)page->dma_addr_upper << 16 << 16; 214 + if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) 215 + ret <<= PAGE_SHIFT; 216 216 217 217 return ret; 218 218 } 219 219 220 - static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr) 220 + static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr) 221 221 { 222 + if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { 223 + page->dma_addr = addr >> PAGE_SHIFT; 224 + 225 + /* We assume page alignment to shave off bottom bits, 226 + * if this "compression" doesn't work we need to drop. 227 + */ 228 + return addr != (dma_addr_t)page->dma_addr << PAGE_SHIFT; 229 + } 230 + 222 231 page->dma_addr = addr; 223 - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) 224 - page->dma_addr_upper = upper_32_bits(addr); 232 + return false; 225 233 } 226 234 227 235 static inline bool page_pool_put(struct page_pool *pool)

+9 -5

net/core/page_pool.c

··· 211 211 */ 212 212 } 213 213 214 - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT && 215 - pool->p.flags & PP_FLAG_PAGE_FRAG) 216 - return -EINVAL; 217 - 218 214 #ifdef CONFIG_PAGE_POOL_STATS 219 215 pool->recycle_stats = alloc_percpu(struct page_pool_recycle_stats); 220 216 if (!pool->recycle_stats) ··· 355 359 if (dma_mapping_error(pool->p.dev, dma)) 356 360 return false; 357 361 358 - page_pool_set_dma_addr(page, dma); 362 + if (page_pool_set_dma_addr(page, dma)) 363 + goto unmap_failed; 359 364 360 365 if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) 361 366 page_pool_dma_sync_for_device(pool, page, pool->p.max_len); 362 367 363 368 return true; 369 + 370 + unmap_failed: 371 + WARN_ON_ONCE("unexpected DMA address, please report to netdev@"); 372 + dma_unmap_page_attrs(pool->p.dev, dma, 373 + PAGE_SIZE << pool->p.order, pool->p.dma_dir, 374 + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); 375 + return false; 364 376 } 365 377 366 378 static void page_pool_set_pp_info(struct page_pool *pool,

Configure Feed

Configure Feed