Merge tag 'xarray-4.20-rc4' of git://git.infradead.org/users/willy/linux-dax

+41 -11

Documentation/core-api/xarray.rst

··· 74 74 new entry and return the previous entry stored at that index. You can 75 75 use :c:func:`xa_erase` instead of calling :c:func:`xa_store` with a 76 76 ``NULL`` entry. There is no difference between an entry that has never 77 - been stored to and one that has most recently had ``NULL`` stored to it. 77 + been stored to, one that has been erased and one that has most recently 78 + had ``NULL`` stored to it. 78 79 79 80 You can conditionally replace an entry at an index by using 80 81 :c:func:`xa_cmpxchg`. Like :c:func:`cmpxchg`, it will only succeed if ··· 106 105 indices. Storing into one index may result in the entry retrieved by 107 106 some, but not all of the other indices changing. 108 107 108 + Sometimes you need to ensure that a subsequent call to :c:func:`xa_store` 109 + will not need to allocate memory. The :c:func:`xa_reserve` function 110 + will store a reserved entry at the indicated index. Users of the normal 111 + API will see this entry as containing ``NULL``. If you do not need to 112 + use the reserved entry, you can call :c:func:`xa_release` to remove the 113 + unused entry. If another user has stored to the entry in the meantime, 114 + :c:func:`xa_release` will do nothing; if instead you want the entry to 115 + become ``NULL``, you should use :c:func:`xa_erase`. 116 + 117 + If all entries in the array are ``NULL``, the :c:func:`xa_empty` function 118 + will return ``true``. 119 + 109 120 Finally, you can remove all entries from an XArray by calling 110 121 :c:func:`xa_destroy`. If the XArray entries are pointers, you may wish 111 122 to free the entries first. You can do this by iterating over all present 112 123 entries in the XArray using the :c:func:`xa_for_each` iterator. 113 124 114 - ID assignment 115 - ------------- 125 + Allocating XArrays 126 + ------------------ 127 + 128 + If you use :c:func:`DEFINE_XARRAY_ALLOC` to define the XArray, or 129 + initialise it by passing ``XA_FLAGS_ALLOC`` to :c:func:`xa_init_flags`, 130 + the XArray changes to track whether entries are in use or not. 116 131 117 132 You can call :c:func:`xa_alloc` to store the entry at any unused index 118 133 in the XArray. If you need to modify the array from interrupt context, 119 134 you can use :c:func:`xa_alloc_bh` or :c:func:`xa_alloc_irq` to disable 120 - interrupts while allocating the ID. Unlike :c:func:`xa_store`, allocating 121 - a ``NULL`` pointer does not delete an entry. Instead it reserves an 122 - entry like :c:func:`xa_reserve` and you can release it using either 123 - :c:func:`xa_erase` or :c:func:`xa_release`. To use ID assignment, the 124 - XArray must be defined with :c:func:`DEFINE_XARRAY_ALLOC`, or initialised 125 - by passing ``XA_FLAGS_ALLOC`` to :c:func:`xa_init_flags`, 135 + interrupts while allocating the ID. 136 + 137 + Using :c:func:`xa_store`, :c:func:`xa_cmpxchg` or :c:func:`xa_insert` 138 + will mark the entry as being allocated. Unlike a normal XArray, storing 139 + ``NULL`` will mark the entry as being in use, like :c:func:`xa_reserve`. 140 + To free an entry, use :c:func:`xa_erase` (or :c:func:`xa_release` if 141 + you only want to free the entry if it's ``NULL``). 142 + 143 + You cannot use ``XA_MARK_0`` with an allocating XArray as this mark 144 + is used to track whether an entry is free or not. The other marks are 145 + available for your use. 126 146 127 147 Memory allocation 128 148 ----------------- ··· 180 158 181 159 Takes xa_lock internally: 182 160 * :c:func:`xa_store` 161 + * :c:func:`xa_store_bh` 162 + * :c:func:`xa_store_irq` 183 163 * :c:func:`xa_insert` 184 164 * :c:func:`xa_erase` 185 165 * :c:func:`xa_erase_bh` ··· 191 167 * :c:func:`xa_alloc` 192 168 * :c:func:`xa_alloc_bh` 193 169 * :c:func:`xa_alloc_irq` 170 + * :c:func:`xa_reserve` 171 + * :c:func:`xa_reserve_bh` 172 + * :c:func:`xa_reserve_irq` 194 173 * :c:func:`xa_destroy` 195 174 * :c:func:`xa_set_mark` 196 175 * :c:func:`xa_clear_mark` ··· 204 177 * :c:func:`__xa_erase` 205 178 * :c:func:`__xa_cmpxchg` 206 179 * :c:func:`__xa_alloc` 180 + * :c:func:`__xa_reserve` 207 181 * :c:func:`__xa_set_mark` 208 182 * :c:func:`__xa_clear_mark` 209 183 ··· 262 234 using :c:func:`xa_lock_irqsave` in both the interrupt handler and process 263 235 context, or :c:func:`xa_lock_irq` in process context and :c:func:`xa_lock` 264 236 in the interrupt handler. Some of the more common patterns have helper 265 - functions such as :c:func:`xa_erase_bh` and :c:func:`xa_erase_irq`. 237 + functions such as :c:func:`xa_store_bh`, :c:func:`xa_store_irq`, 238 + :c:func:`xa_erase_bh` and :c:func:`xa_erase_irq`. 266 239 267 240 Sometimes you need to protect access to the XArray with a mutex because 268 241 that lock sits above another mutex in the locking hierarchy. That does ··· 351 322 - :c:func:`xa_is_zero` 352 323 - Zero entries appear as ``NULL`` through the Normal API, but occupy 353 324 an entry in the XArray which can be used to reserve the index for 354 - future use. 325 + future use. This is used by allocating XArrays for allocated entries 326 + which are ``NULL``. 355 327 356 328 Other internal entries may be added in the future. As far as possible, they 357 329 will be handled by :c:func:`xas_retry`.

+35 -25

fs/dax.c

··· 98 98 return xa_mk_value(flags | (pfn_t_to_pfn(pfn) << DAX_SHIFT)); 99 99 } 100 100 101 - static void *dax_make_page_entry(struct page *page) 102 - { 103 - pfn_t pfn = page_to_pfn_t(page); 104 - return dax_make_entry(pfn, PageHead(page) ? DAX_PMD : 0); 105 - } 106 - 107 101 static bool dax_is_locked(void *entry) 108 102 { 109 103 return xa_to_value(entry) & DAX_LOCKED; ··· 110 116 return 0; 111 117 } 112 118 113 - static int dax_is_pmd_entry(void *entry) 119 + static unsigned long dax_is_pmd_entry(void *entry) 114 120 { 115 121 return xa_to_value(entry) & DAX_PMD; 116 122 } 117 123 118 - static int dax_is_pte_entry(void *entry) 124 + static bool dax_is_pte_entry(void *entry) 119 125 { 120 126 return !(xa_to_value(entry) & DAX_PMD); 121 127 } ··· 216 222 ewait.wait.func = wake_exceptional_entry_func; 217 223 218 224 for (;;) { 219 - entry = xas_load(xas); 220 - if (!entry || xa_is_internal(entry) || 221 - WARN_ON_ONCE(!xa_is_value(entry)) || 225 + entry = xas_find_conflict(xas); 226 + if (!entry || WARN_ON_ONCE(!xa_is_value(entry)) || 222 227 !dax_is_locked(entry)) 223 228 return entry; 224 229 ··· 248 255 { 249 256 void *old; 250 257 258 + BUG_ON(dax_is_locked(entry)); 251 259 xas_reset(xas); 252 260 xas_lock_irq(xas); 253 261 old = xas_store(xas, entry); ··· 346 352 return NULL; 347 353 } 348 354 355 + /* 356 + * dax_lock_mapping_entry - Lock the DAX entry corresponding to a page 357 + * @page: The page whose entry we want to lock 358 + * 359 + * Context: Process context. 360 + * Return: %true if the entry was locked or does not need to be locked. 361 + */ 349 362 bool dax_lock_mapping_entry(struct page *page) 350 363 { 351 364 XA_STATE(xas, NULL, 0); 352 365 void *entry; 366 + bool locked; 353 367 368 + /* Ensure page->mapping isn't freed while we look at it */ 369 + rcu_read_lock(); 354 370 for (;;) { 355 371 struct address_space *mapping = READ_ONCE(page->mapping); 356 372 373 + locked = false; 357 374 if (!dax_mapping(mapping)) 358 - return false; 375 + break; 359 376 360 377 /* 361 378 * In the device-dax case there's no need to lock, a ··· 375 370 * otherwise we would not have a valid pfn_to_page() 376 371 * translation. 377 372 */ 373 + locked = true; 378 374 if (S_ISCHR(mapping->host->i_mode)) 379 - return true; 375 + break; 380 376 381 377 xas.xa = &mapping->i_pages; 382 378 xas_lock_irq(&xas); ··· 388 382 xas_set(&xas, page->index); 389 383 entry = xas_load(&xas); 390 384 if (dax_is_locked(entry)) { 385 + rcu_read_unlock(); 391 386 entry = get_unlocked_entry(&xas); 392 - /* Did the page move while we slept? */ 393 - if (dax_to_pfn(entry) != page_to_pfn(page)) { 394 - xas_unlock_irq(&xas); 395 - continue; 396 - } 387 + xas_unlock_irq(&xas); 388 + put_unlocked_entry(&xas, entry); 389 + rcu_read_lock(); 390 + continue; 397 391 } 398 392 dax_lock_entry(&xas, entry); 399 393 xas_unlock_irq(&xas); 400 - return true; 394 + break; 401 395 } 396 + rcu_read_unlock(); 397 + return locked; 402 398 } 403 399 404 400 void dax_unlock_mapping_entry(struct page *page) 405 401 { 406 402 struct address_space *mapping = page->mapping; 407 403 XA_STATE(xas, &mapping->i_pages, page->index); 404 + void *entry; 408 405 409 406 if (S_ISCHR(mapping->host->i_mode)) 410 407 return; 411 408 412 - dax_unlock_entry(&xas, dax_make_page_entry(page)); 409 + rcu_read_lock(); 410 + entry = xas_load(&xas); 411 + rcu_read_unlock(); 412 + entry = dax_make_entry(page_to_pfn_t(page), dax_is_pmd_entry(entry)); 413 + dax_unlock_entry(&xas, entry); 413 414 } 414 415 415 416 /* ··· 458 445 retry: 459 446 xas_lock_irq(xas); 460 447 entry = get_unlocked_entry(xas); 461 - if (xa_is_internal(entry)) 462 - goto fallback; 463 448 464 449 if (entry) { 465 - if (WARN_ON_ONCE(!xa_is_value(entry))) { 450 + if (!xa_is_value(entry)) { 466 451 xas_set_err(xas, EIO); 467 452 goto out_unlock; 468 453 } ··· 1639 1628 /* Did we race with someone splitting entry or so? */ 1640 1629 if (!entry || 1641 1630 (order == 0 && !dax_is_pte_entry(entry)) || 1642 - (order == PMD_ORDER && (xa_is_internal(entry) || 1643 - !dax_is_pmd_entry(entry)))) { 1631 + (order == PMD_ORDER && !dax_is_pmd_entry(entry))) { 1644 1632 put_unlocked_entry(&xas, entry); 1645 1633 xas_unlock_irq(&xas); 1646 1634 trace_dax_insert_pfn_mkwrite_no_entry(mapping->host, vmf,

+1 -3

fs/nilfs2/btnode.c

··· 266 266 return; 267 267 268 268 if (nbh == NULL) { /* blocksize == pagesize */ 269 - xa_lock_irq(&btnc->i_pages); 270 - __xa_erase(&btnc->i_pages, newkey); 271 - xa_unlock_irq(&btnc->i_pages); 269 + xa_erase_irq(&btnc->i_pages, newkey); 272 270 unlock_page(ctxt->bh->b_page); 273 271 } else 274 272 brelse(nbh);

+203 -64

include/linux/xarray.h

··· 289 289 void xa_init_flags(struct xarray *, gfp_t flags); 290 290 void *xa_load(struct xarray *, unsigned long index); 291 291 void *xa_store(struct xarray *, unsigned long index, void *entry, gfp_t); 292 - void *xa_cmpxchg(struct xarray *, unsigned long index, 293 - void *old, void *entry, gfp_t); 294 - int xa_reserve(struct xarray *, unsigned long index, gfp_t); 292 + void *xa_erase(struct xarray *, unsigned long index); 295 293 void *xa_store_range(struct xarray *, unsigned long first, unsigned long last, 296 294 void *entry, gfp_t); 297 295 bool xa_get_mark(struct xarray *, unsigned long index, xa_mark_t); ··· 339 341 static inline bool xa_marked(const struct xarray *xa, xa_mark_t mark) 340 342 { 341 343 return xa->xa_flags & XA_FLAGS_MARK(mark); 342 - } 343 - 344 - /** 345 - * xa_erase() - Erase this entry from the XArray. 346 - * @xa: XArray. 347 - * @index: Index of entry. 348 - * 349 - * This function is the equivalent of calling xa_store() with %NULL as 350 - * the third argument. The XArray does not need to allocate memory, so 351 - * the user does not need to provide GFP flags. 352 - * 353 - * Context: Process context. Takes and releases the xa_lock. 354 - * Return: The entry which used to be at this index. 355 - */ 356 - static inline void *xa_erase(struct xarray *xa, unsigned long index) 357 - { 358 - return xa_store(xa, index, NULL, 0); 359 - } 360 - 361 - /** 362 - * xa_insert() - Store this entry in the XArray unless another entry is 363 - * already present. 364 - * @xa: XArray. 365 - * @index: Index into array. 366 - * @entry: New entry. 367 - * @gfp: Memory allocation flags. 368 - * 369 - * If you would rather see the existing entry in the array, use xa_cmpxchg(). 370 - * This function is for users who don't care what the entry is, only that 371 - * one is present. 372 - * 373 - * Context: Process context. Takes and releases the xa_lock. 374 - * May sleep if the @gfp flags permit. 375 - * Return: 0 if the store succeeded. -EEXIST if another entry was present. 376 - * -ENOMEM if memory could not be allocated. 377 - */ 378 - static inline int xa_insert(struct xarray *xa, unsigned long index, 379 - void *entry, gfp_t gfp) 380 - { 381 - void *curr = xa_cmpxchg(xa, index, NULL, entry, gfp); 382 - if (!curr) 383 - return 0; 384 - if (xa_is_err(curr)) 385 - return xa_err(curr); 386 - return -EEXIST; 387 - } 388 - 389 - /** 390 - * xa_release() - Release a reserved entry. 391 - * @xa: XArray. 392 - * @index: Index of entry. 393 - * 394 - * After calling xa_reserve(), you can call this function to release the 395 - * reservation. If the entry at @index has been stored to, this function 396 - * will do nothing. 397 - */ 398 - static inline void xa_release(struct xarray *xa, unsigned long index) 399 - { 400 - xa_cmpxchg(xa, index, NULL, NULL, 0); 401 344 } 402 345 403 346 /** ··· 394 455 void *__xa_cmpxchg(struct xarray *, unsigned long index, void *old, 395 456 void *entry, gfp_t); 396 457 int __xa_alloc(struct xarray *, u32 *id, u32 max, void *entry, gfp_t); 458 + int __xa_reserve(struct xarray *, unsigned long index, gfp_t); 397 459 void __xa_set_mark(struct xarray *, unsigned long index, xa_mark_t); 398 460 void __xa_clear_mark(struct xarray *, unsigned long index, xa_mark_t); 399 461 ··· 427 487 } 428 488 429 489 /** 490 + * xa_store_bh() - Store this entry in the XArray. 491 + * @xa: XArray. 492 + * @index: Index into array. 493 + * @entry: New entry. 494 + * @gfp: Memory allocation flags. 495 + * 496 + * This function is like calling xa_store() except it disables softirqs 497 + * while holding the array lock. 498 + * 499 + * Context: Any context. Takes and releases the xa_lock while 500 + * disabling softirqs. 501 + * Return: The entry which used to be at this index. 502 + */ 503 + static inline void *xa_store_bh(struct xarray *xa, unsigned long index, 504 + void *entry, gfp_t gfp) 505 + { 506 + void *curr; 507 + 508 + xa_lock_bh(xa); 509 + curr = __xa_store(xa, index, entry, gfp); 510 + xa_unlock_bh(xa); 511 + 512 + return curr; 513 + } 514 + 515 + /** 516 + * xa_store_irq() - Erase this entry from the XArray. 517 + * @xa: XArray. 518 + * @index: Index into array. 519 + * @entry: New entry. 520 + * @gfp: Memory allocation flags. 521 + * 522 + * This function is like calling xa_store() except it disables interrupts 523 + * while holding the array lock. 524 + * 525 + * Context: Process context. Takes and releases the xa_lock while 526 + * disabling interrupts. 527 + * Return: The entry which used to be at this index. 528 + */ 529 + static inline void *xa_store_irq(struct xarray *xa, unsigned long index, 530 + void *entry, gfp_t gfp) 531 + { 532 + void *curr; 533 + 534 + xa_lock_irq(xa); 535 + curr = __xa_store(xa, index, entry, gfp); 536 + xa_unlock_irq(xa); 537 + 538 + return curr; 539 + } 540 + 541 + /** 430 542 * xa_erase_bh() - Erase this entry from the XArray. 431 543 * @xa: XArray. 432 544 * @index: Index of entry. ··· 487 495 * the third argument. The XArray does not need to allocate memory, so 488 496 * the user does not need to provide GFP flags. 489 497 * 490 - * Context: Process context. Takes and releases the xa_lock while 498 + * Context: Any context. Takes and releases the xa_lock while 491 499 * disabling softirqs. 492 500 * Return: The entry which used to be at this index. 493 501 */ ··· 524 532 xa_unlock_irq(xa); 525 533 526 534 return entry; 535 + } 536 + 537 + /** 538 + * xa_cmpxchg() - Conditionally replace an entry in the XArray. 539 + * @xa: XArray. 540 + * @index: Index into array. 541 + * @old: Old value to test against. 542 + * @entry: New value to place in array. 543 + * @gfp: Memory allocation flags. 544 + * 545 + * If the entry at @index is the same as @old, replace it with @entry. 546 + * If the return value is equal to @old, then the exchange was successful. 547 + * 548 + * Context: Any context. Takes and releases the xa_lock. May sleep 549 + * if the @gfp flags permit. 550 + * Return: The old value at this index or xa_err() if an error happened. 551 + */ 552 + static inline void *xa_cmpxchg(struct xarray *xa, unsigned long index, 553 + void *old, void *entry, gfp_t gfp) 554 + { 555 + void *curr; 556 + 557 + xa_lock(xa); 558 + curr = __xa_cmpxchg(xa, index, old, entry, gfp); 559 + xa_unlock(xa); 560 + 561 + return curr; 562 + } 563 + 564 + /** 565 + * xa_insert() - Store this entry in the XArray unless another entry is 566 + * already present. 567 + * @xa: XArray. 568 + * @index: Index into array. 569 + * @entry: New entry. 570 + * @gfp: Memory allocation flags. 571 + * 572 + * If you would rather see the existing entry in the array, use xa_cmpxchg(). 573 + * This function is for users who don't care what the entry is, only that 574 + * one is present. 575 + * 576 + * Context: Process context. Takes and releases the xa_lock. 577 + * May sleep if the @gfp flags permit. 578 + * Return: 0 if the store succeeded. -EEXIST if another entry was present. 579 + * -ENOMEM if memory could not be allocated. 580 + */ 581 + static inline int xa_insert(struct xarray *xa, unsigned long index, 582 + void *entry, gfp_t gfp) 583 + { 584 + void *curr = xa_cmpxchg(xa, index, NULL, entry, gfp); 585 + if (!curr) 586 + return 0; 587 + if (xa_is_err(curr)) 588 + return xa_err(curr); 589 + return -EEXIST; 527 590 } 528 591 529 592 /** ··· 622 575 * Updates the @id pointer with the index, then stores the entry at that 623 576 * index. A concurrent lookup will not see an uninitialised @id. 624 577 * 625 - * Context: Process context. Takes and releases the xa_lock while 578 + * Context: Any context. Takes and releases the xa_lock while 626 579 * disabling softirqs. May sleep if the @gfp flags permit. 627 580 * Return: 0 on success, -ENOMEM if memory allocation fails or -ENOSPC if 628 581 * there is no more space in the XArray. ··· 666 619 xa_unlock_irq(xa); 667 620 668 621 return err; 622 + } 623 + 624 + /** 625 + * xa_reserve() - Reserve this index in the XArray. 626 + * @xa: XArray. 627 + * @index: Index into array. 628 + * @gfp: Memory allocation flags. 629 + * 630 + * Ensures there is somewhere to store an entry at @index in the array. 631 + * If there is already something stored at @index, this function does 632 + * nothing. If there was nothing there, the entry is marked as reserved. 633 + * Loading from a reserved entry returns a %NULL pointer. 634 + * 635 + * If you do not use the entry that you have reserved, call xa_release() 636 + * or xa_erase() to free any unnecessary memory. 637 + * 638 + * Context: Any context. Takes and releases the xa_lock. 639 + * May sleep if the @gfp flags permit. 640 + * Return: 0 if the reservation succeeded or -ENOMEM if it failed. 641 + */ 642 + static inline 643 + int xa_reserve(struct xarray *xa, unsigned long index, gfp_t gfp) 644 + { 645 + int ret; 646 + 647 + xa_lock(xa); 648 + ret = __xa_reserve(xa, index, gfp); 649 + xa_unlock(xa); 650 + 651 + return ret; 652 + } 653 + 654 + /** 655 + * xa_reserve_bh() - Reserve this index in the XArray. 656 + * @xa: XArray. 657 + * @index: Index into array. 658 + * @gfp: Memory allocation flags. 659 + * 660 + * A softirq-disabling version of xa_reserve(). 661 + * 662 + * Context: Any context. Takes and releases the xa_lock while 663 + * disabling softirqs. 664 + * Return: 0 if the reservation succeeded or -ENOMEM if it failed. 665 + */ 666 + static inline 667 + int xa_reserve_bh(struct xarray *xa, unsigned long index, gfp_t gfp) 668 + { 669 + int ret; 670 + 671 + xa_lock_bh(xa); 672 + ret = __xa_reserve(xa, index, gfp); 673 + xa_unlock_bh(xa); 674 + 675 + return ret; 676 + } 677 + 678 + /** 679 + * xa_reserve_irq() - Reserve this index in the XArray. 680 + * @xa: XArray. 681 + * @index: Index into array. 682 + * @gfp: Memory allocation flags. 683 + * 684 + * An interrupt-disabling version of xa_reserve(). 685 + * 686 + * Context: Process context. Takes and releases the xa_lock while 687 + * disabling interrupts. 688 + * Return: 0 if the reservation succeeded or -ENOMEM if it failed. 689 + */ 690 + static inline 691 + int xa_reserve_irq(struct xarray *xa, unsigned long index, gfp_t gfp) 692 + { 693 + int ret; 694 + 695 + xa_lock_irq(xa); 696 + ret = __xa_reserve(xa, index, gfp); 697 + xa_unlock_irq(xa); 698 + 699 + return ret; 700 + } 701 + 702 + /** 703 + * xa_release() - Release a reserved entry. 704 + * @xa: XArray. 705 + * @index: Index of entry. 706 + * 707 + * After calling xa_reserve(), you can call this function to release the 708 + * reservation. If the entry at @index has been stored to, this function 709 + * will do nothing. 710 + */ 711 + static inline void xa_release(struct xarray *xa, unsigned long index) 712 + { 713 + xa_cmpxchg(xa, index, NULL, NULL, 0); 669 714 } 670 715 671 716 /* Everything below here is the Advanced API. Proceed with caution. */

+47 -3

lib/test_xarray.c

··· 208 208 XA_BUG_ON(xa, xa_get_mark(xa, i, XA_MARK_2)); 209 209 210 210 /* We should see two elements in the array */ 211 + rcu_read_lock(); 211 212 xas_for_each(&xas, entry, ULONG_MAX) 212 213 seen++; 214 + rcu_read_unlock(); 213 215 XA_BUG_ON(xa, seen != 2); 214 216 215 217 /* One of which is marked */ 216 218 xas_set(&xas, 0); 217 219 seen = 0; 220 + rcu_read_lock(); 218 221 xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_0) 219 222 seen++; 223 + rcu_read_unlock(); 220 224 XA_BUG_ON(xa, seen != 1); 221 225 } 222 226 XA_BUG_ON(xa, xa_get_mark(xa, next, XA_MARK_0)); ··· 377 373 xa_erase_index(xa, 12345678); 378 374 XA_BUG_ON(xa, !xa_empty(xa)); 379 375 376 + /* And so does xa_insert */ 377 + xa_reserve(xa, 12345678, GFP_KERNEL); 378 + XA_BUG_ON(xa, xa_insert(xa, 12345678, xa_mk_value(12345678), 0) != 0); 379 + xa_erase_index(xa, 12345678); 380 + XA_BUG_ON(xa, !xa_empty(xa)); 381 + 380 382 /* Can iterate through a reserved entry */ 381 383 xa_store_index(xa, 5, GFP_KERNEL); 382 384 xa_reserve(xa, 6, GFP_KERNEL); ··· 446 436 XA_BUG_ON(xa, xa_load(xa, max) != NULL); 447 437 XA_BUG_ON(xa, xa_load(xa, min - 1) != NULL); 448 438 439 + xas_lock(&xas); 449 440 XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(min)) != xa_mk_value(index)); 441 + xas_unlock(&xas); 450 442 XA_BUG_ON(xa, xa_load(xa, min) != xa_mk_value(min)); 451 443 XA_BUG_ON(xa, xa_load(xa, max - 1) != xa_mk_value(min)); 452 444 XA_BUG_ON(xa, xa_load(xa, max) != NULL); ··· 464 452 XA_STATE(xas, xa, index); 465 453 xa_store_order(xa, index, order, xa_mk_value(0), GFP_KERNEL); 466 454 455 + xas_lock(&xas); 467 456 XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(1)) != xa_mk_value(0)); 468 457 XA_BUG_ON(xa, xas.xa_index != index); 469 458 XA_BUG_ON(xa, xas_store(&xas, NULL) != xa_mk_value(1)); 459 + xas_unlock(&xas); 470 460 XA_BUG_ON(xa, !xa_empty(xa)); 471 461 } 472 462 #endif ··· 512 498 rcu_read_unlock(); 513 499 514 500 /* We can erase multiple values with a single store */ 515 - xa_store_order(xa, 0, 63, NULL, GFP_KERNEL); 501 + xa_store_order(xa, 0, BITS_PER_LONG - 1, NULL, GFP_KERNEL); 516 502 XA_BUG_ON(xa, !xa_empty(xa)); 517 503 518 504 /* Even when the first slot is empty but the others aren't */ ··· 716 702 } 717 703 } 718 704 719 - static noinline void check_find(struct xarray *xa) 705 + static noinline void check_find_1(struct xarray *xa) 720 706 { 721 707 unsigned long i, j, k; 722 708 ··· 762 748 XA_BUG_ON(xa, xa_get_mark(xa, i, XA_MARK_0)); 763 749 } 764 750 XA_BUG_ON(xa, !xa_empty(xa)); 751 + } 752 + 753 + static noinline void check_find_2(struct xarray *xa) 754 + { 755 + void *entry; 756 + unsigned long i, j, index = 0; 757 + 758 + xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { 759 + XA_BUG_ON(xa, true); 760 + } 761 + 762 + for (i = 0; i < 1024; i++) { 763 + xa_store_index(xa, index, GFP_KERNEL); 764 + j = 0; 765 + index = 0; 766 + xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { 767 + XA_BUG_ON(xa, xa_mk_value(index) != entry); 768 + XA_BUG_ON(xa, index != j++); 769 + } 770 + } 771 + 772 + xa_destroy(xa); 773 + } 774 + 775 + static noinline void check_find(struct xarray *xa) 776 + { 777 + check_find_1(xa); 778 + check_find_2(xa); 765 779 check_multi_find(xa); 766 780 check_multi_find_2(xa); 767 781 } ··· 1109 1067 __check_store_range(xa, 4095 + i, 4095 + j); 1110 1068 __check_store_range(xa, 4096 + i, 4096 + j); 1111 1069 __check_store_range(xa, 123456 + i, 123456 + j); 1112 - __check_store_range(xa, UINT_MAX + i, UINT_MAX + j); 1070 + __check_store_range(xa, (1 << 24) + i, (1 << 24) + j); 1113 1071 } 1114 1072 } 1115 1073 } ··· 1188 1146 XA_STATE(xas, xa, 1 << order); 1189 1147 1190 1148 xa_store_order(xa, 0, order, xa, GFP_KERNEL); 1149 + rcu_read_lock(); 1191 1150 xas_load(&xas); 1192 1151 XA_BUG_ON(xa, xas.xa_node->count == 0); 1193 1152 XA_BUG_ON(xa, xas.xa_node->count > (1 << order)); 1194 1153 XA_BUG_ON(xa, xas.xa_node->nr_values != 0); 1154 + rcu_read_unlock(); 1195 1155 1196 1156 xa_store_order(xa, 1 << order, order, xa_mk_value(1 << order), 1197 1157 GFP_KERNEL);

+60 -79

lib/xarray.c

··· 610 610 * (see the xa_cmpxchg() implementation for an example). 611 611 * 612 612 * Return: If the slot already existed, returns the contents of this slot. 613 - * If the slot was newly created, returns NULL. If it failed to create the 614 - * slot, returns NULL and indicates the error in @xas. 613 + * If the slot was newly created, returns %NULL. If it failed to create the 614 + * slot, returns %NULL and indicates the error in @xas. 615 615 */ 616 616 static void *xas_create(struct xa_state *xas) 617 617 { ··· 1334 1334 XA_STATE(xas, xa, index); 1335 1335 return xas_result(&xas, xas_store(&xas, NULL)); 1336 1336 } 1337 - EXPORT_SYMBOL_GPL(__xa_erase); 1337 + EXPORT_SYMBOL(__xa_erase); 1338 1338 1339 1339 /** 1340 - * xa_store() - Store this entry in the XArray. 1340 + * xa_erase() - Erase this entry from the XArray. 1341 1341 * @xa: XArray. 1342 - * @index: Index into array. 1343 - * @entry: New entry. 1344 - * @gfp: Memory allocation flags. 1342 + * @index: Index of entry. 1345 1343 * 1346 - * After this function returns, loads from this index will return @entry. 1347 - * Storing into an existing multislot entry updates the entry of every index. 1348 - * The marks associated with @index are unaffected unless @entry is %NULL. 1344 + * This function is the equivalent of calling xa_store() with %NULL as 1345 + * the third argument. The XArray does not need to allocate memory, so 1346 + * the user does not need to provide GFP flags. 1349 1347 * 1350 - * Context: Process context. Takes and releases the xa_lock. May sleep 1351 - * if the @gfp flags permit. 1352 - * Return: The old entry at this index on success, xa_err(-EINVAL) if @entry 1353 - * cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation 1354 - * failed. 1348 + * Context: Any context. Takes and releases the xa_lock. 1349 + * Return: The entry which used to be at this index. 1355 1350 */ 1356 - void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) 1351 + void *xa_erase(struct xarray *xa, unsigned long index) 1357 1352 { 1358 - XA_STATE(xas, xa, index); 1359 - void *curr; 1353 + void *entry; 1360 1354 1361 - if (WARN_ON_ONCE(xa_is_internal(entry))) 1362 - return XA_ERROR(-EINVAL); 1355 + xa_lock(xa); 1356 + entry = __xa_erase(xa, index); 1357 + xa_unlock(xa); 1363 1358 1364 - do { 1365 - xas_lock(&xas); 1366 - curr = xas_store(&xas, entry); 1367 - if (xa_track_free(xa) && entry) 1368 - xas_clear_mark(&xas, XA_FREE_MARK); 1369 - xas_unlock(&xas); 1370 - } while (xas_nomem(&xas, gfp)); 1371 - 1372 - return xas_result(&xas, curr); 1359 + return entry; 1373 1360 } 1374 - EXPORT_SYMBOL(xa_store); 1361 + EXPORT_SYMBOL(xa_erase); 1375 1362 1376 1363 /** 1377 1364 * __xa_store() - Store this entry in the XArray. ··· 1382 1395 1383 1396 if (WARN_ON_ONCE(xa_is_internal(entry))) 1384 1397 return XA_ERROR(-EINVAL); 1398 + if (xa_track_free(xa) && !entry) 1399 + entry = XA_ZERO_ENTRY; 1385 1400 1386 1401 do { 1387 1402 curr = xas_store(&xas, entry); 1388 - if (xa_track_free(xa) && entry) 1403 + if (xa_track_free(xa)) 1389 1404 xas_clear_mark(&xas, XA_FREE_MARK); 1390 1405 } while (__xas_nomem(&xas, gfp)); 1391 1406 ··· 1396 1407 EXPORT_SYMBOL(__xa_store); 1397 1408 1398 1409 /** 1399 - * xa_cmpxchg() - Conditionally replace an entry in the XArray. 1410 + * xa_store() - Store this entry in the XArray. 1400 1411 * @xa: XArray. 1401 1412 * @index: Index into array. 1402 - * @old: Old value to test against. 1403 - * @entry: New value to place in array. 1413 + * @entry: New entry. 1404 1414 * @gfp: Memory allocation flags. 1405 1415 * 1406 - * If the entry at @index is the same as @old, replace it with @entry. 1407 - * If the return value is equal to @old, then the exchange was successful. 1416 + * After this function returns, loads from this index will return @entry. 1417 + * Storing into an existing multislot entry updates the entry of every index. 1418 + * The marks associated with @index are unaffected unless @entry is %NULL. 1408 1419 * 1409 - * Context: Process context. Takes and releases the xa_lock. May sleep 1410 - * if the @gfp flags permit. 1411 - * Return: The old value at this index or xa_err() if an error happened. 1420 + * Context: Any context. Takes and releases the xa_lock. 1421 + * May sleep if the @gfp flags permit. 1422 + * Return: The old entry at this index on success, xa_err(-EINVAL) if @entry 1423 + * cannot be stored in an XArray, or xa_err(-ENOMEM) if memory allocation 1424 + * failed. 1412 1425 */ 1413 - void *xa_cmpxchg(struct xarray *xa, unsigned long index, 1414 - void *old, void *entry, gfp_t gfp) 1426 + void *xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) 1415 1427 { 1416 - XA_STATE(xas, xa, index); 1417 1428 void *curr; 1418 1429 1419 - if (WARN_ON_ONCE(xa_is_internal(entry))) 1420 - return XA_ERROR(-EINVAL); 1430 + xa_lock(xa); 1431 + curr = __xa_store(xa, index, entry, gfp); 1432 + xa_unlock(xa); 1421 1433 1422 - do { 1423 - xas_lock(&xas); 1424 - curr = xas_load(&xas); 1425 - if (curr == XA_ZERO_ENTRY) 1426 - curr = NULL; 1427 - if (curr == old) { 1428 - xas_store(&xas, entry); 1429 - if (xa_track_free(xa) && entry) 1430 - xas_clear_mark(&xas, XA_FREE_MARK); 1431 - } 1432 - xas_unlock(&xas); 1433 - } while (xas_nomem(&xas, gfp)); 1434 - 1435 - return xas_result(&xas, curr); 1434 + return curr; 1436 1435 } 1437 - EXPORT_SYMBOL(xa_cmpxchg); 1436 + EXPORT_SYMBOL(xa_store); 1438 1437 1439 1438 /** 1440 1439 * __xa_cmpxchg() - Store this entry in the XArray. ··· 1448 1471 1449 1472 if (WARN_ON_ONCE(xa_is_internal(entry))) 1450 1473 return XA_ERROR(-EINVAL); 1474 + if (xa_track_free(xa) && !entry) 1475 + entry = XA_ZERO_ENTRY; 1451 1476 1452 1477 do { 1453 1478 curr = xas_load(&xas); ··· 1457 1478 curr = NULL; 1458 1479 if (curr == old) { 1459 1480 xas_store(&xas, entry); 1460 - if (xa_track_free(xa) && entry) 1481 + if (xa_track_free(xa)) 1461 1482 xas_clear_mark(&xas, XA_FREE_MARK); 1462 1483 } 1463 1484 } while (__xas_nomem(&xas, gfp)); ··· 1467 1488 EXPORT_SYMBOL(__xa_cmpxchg); 1468 1489 1469 1490 /** 1470 - * xa_reserve() - Reserve this index in the XArray. 1491 + * __xa_reserve() - Reserve this index in the XArray. 1471 1492 * @xa: XArray. 1472 1493 * @index: Index into array. 1473 1494 * @gfp: Memory allocation flags. ··· 1475 1496 * Ensures there is somewhere to store an entry at @index in the array. 1476 1497 * If there is already something stored at @index, this function does 1477 1498 * nothing. If there was nothing there, the entry is marked as reserved. 1478 - * Loads from @index will continue to see a %NULL pointer until a 1479 - * subsequent store to @index. 1499 + * Loading from a reserved entry returns a %NULL pointer. 1480 1500 * 1481 1501 * If you do not use the entry that you have reserved, call xa_release() 1482 1502 * or xa_erase() to free any unnecessary memory. 1483 1503 * 1484 - * Context: Process context. Takes and releases the xa_lock, IRQ or BH safe 1485 - * if specified in XArray flags. May sleep if the @gfp flags permit. 1504 + * Context: Any context. Expects the xa_lock to be held on entry. May 1505 + * release the lock, sleep and reacquire the lock if the @gfp flags permit. 1486 1506 * Return: 0 if the reservation succeeded or -ENOMEM if it failed. 1487 1507 */ 1488 - int xa_reserve(struct xarray *xa, unsigned long index, gfp_t gfp) 1508 + int __xa_reserve(struct xarray *xa, unsigned long index, gfp_t gfp) 1489 1509 { 1490 1510 XA_STATE(xas, xa, index); 1491 - unsigned int lock_type = xa_lock_type(xa); 1492 1511 void *curr; 1493 1512 1494 1513 do { 1495 - xas_lock_type(&xas, lock_type); 1496 1514 curr = xas_load(&xas); 1497 - if (!curr) 1515 + if (!curr) { 1498 1516 xas_store(&xas, XA_ZERO_ENTRY); 1499 - xas_unlock_type(&xas, lock_type); 1500 - } while (xas_nomem(&xas, gfp)); 1517 + if (xa_track_free(xa)) 1518 + xas_clear_mark(&xas, XA_FREE_MARK); 1519 + } 1520 + } while (__xas_nomem(&xas, gfp)); 1501 1521 1502 1522 return xas_error(&xas); 1503 1523 } 1504 - EXPORT_SYMBOL(xa_reserve); 1524 + EXPORT_SYMBOL(__xa_reserve); 1505 1525 1506 1526 #ifdef CONFIG_XARRAY_MULTI 1507 1527 static void xas_set_range(struct xa_state *xas, unsigned long first, ··· 1565 1587 do { 1566 1588 xas_lock(&xas); 1567 1589 if (entry) { 1568 - unsigned int order = (last == ~0UL) ? 64 : 1569 - ilog2(last + 1); 1590 + unsigned int order = BITS_PER_LONG; 1591 + if (last + 1) 1592 + order = __ffs(last + 1); 1570 1593 xas_set_order(&xas, last, order); 1571 1594 xas_create(&xas); 1572 1595 if (xas_error(&xas)) ··· 1641 1662 * @index: Index of entry. 1642 1663 * @mark: Mark number. 1643 1664 * 1644 - * Attempting to set a mark on a NULL entry does not succeed. 1665 + * Attempting to set a mark on a %NULL entry does not succeed. 1645 1666 * 1646 1667 * Context: Any context. Expects xa_lock to be held on entry. 1647 1668 */ ··· 1653 1674 if (entry) 1654 1675 xas_set_mark(&xas, mark); 1655 1676 } 1656 - EXPORT_SYMBOL_GPL(__xa_set_mark); 1677 + EXPORT_SYMBOL(__xa_set_mark); 1657 1678 1658 1679 /** 1659 1680 * __xa_clear_mark() - Clear this mark on this entry while locked. ··· 1671 1692 if (entry) 1672 1693 xas_clear_mark(&xas, mark); 1673 1694 } 1674 - EXPORT_SYMBOL_GPL(__xa_clear_mark); 1695 + EXPORT_SYMBOL(__xa_clear_mark); 1675 1696 1676 1697 /** 1677 1698 * xa_get_mark() - Inquire whether this mark is set on this entry. ··· 1711 1732 * @index: Index of entry. 1712 1733 * @mark: Mark number. 1713 1734 * 1714 - * Attempting to set a mark on a NULL entry does not succeed. 1735 + * Attempting to set a mark on a %NULL entry does not succeed. 1715 1736 * 1716 1737 * Context: Process context. Takes and releases the xa_lock. 1717 1738 */ ··· 1808 1829 entry = xas_find_marked(&xas, max, filter); 1809 1830 else 1810 1831 entry = xas_find(&xas, max); 1832 + if (xas.xa_node == XAS_BOUNDS) 1833 + break; 1811 1834 if (xas.xa_shift) { 1812 1835 if (xas.xa_index & ((1UL << xas.xa_shift) - 1)) 1813 1836 continue; ··· 1880 1899 * 1881 1900 * The @filter may be an XArray mark value, in which case entries which are 1882 1901 * marked with that mark will be copied. It may also be %XA_PRESENT, in 1883 - * which case all entries which are not NULL will be copied. 1902 + * which case all entries which are not %NULL will be copied. 1884 1903 * 1885 1904 * The entries returned may not represent a snapshot of the XArray at a 1886 1905 * moment in time. For example, if another thread stores to index 5, then

Configure Feed

Configure Feed