docs: handling-regressions: add, trim, and sort quotes from Linus

+464 -233

1 changed file

expand all

Documentation

process

+464 -233

Documentation/process/handling-regressions.rst

··· 461 461 Quotes from Linus about regression 462 462 ---------------------------------- 463 463 464 - Find below a few real life examples of how Linus Torvalds expects regressions to 465 - be handled: 464 + The following statements from Linus Torvalds provide some insight into Linux 465 + "no regressions" rule and how he expects regressions to be handled: 466 466 467 - * From `2017-10-26 (1/2) 468 - <https://lore.kernel.org/lkml/CA+55aFwiiQYJ+YoLKCXjN_beDVfu38mg=Ggg5LFOcqHE8Qi7Zw@mail.gmail.com/>`_:: 467 + On how quickly regressions should be fixed 468 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 469 469 470 - If you break existing user space setups THAT IS A REGRESSION. 470 + * From `2026-01-22 <https://lore.kernel.org/all/CAHk-=wheQNiW_WtHGO7bKkT7Uib-p+ai2JP9M+z+FYcZ6CAxYA@mail.gmail.com/>`_:: 471 471 472 - It's not ok to say "but we'll fix the user space setup". 472 + But a user complaining should basically result in an immediate fix - 473 + possibly a "revert and rethink". 473 474 474 - Really. NOT OK. 475 + With a later clarification on `2026-01-28 <https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/>`_:: 475 476 476 - [...] 477 + It's also worth noting that "immediate" obviously doesn't mean "right 478 + this *second* when the problem has been reported". 477 479 478 - The first rule is: 480 + But if it's a regression with a known commit that caused it, I think 481 + the rule of thumb should generally be "within a week", preferably 482 + before the next rc. 479 483 480 - - we don't cause regressions 484 + * From `2023-04-21 <https://lore.kernel.org/all/CAHk-=wgD98pmSK3ZyHk_d9kZ2bhgN6DuNZMAJaV0WTtbkf=RDw@mail.gmail.com/>`_:: 481 485 482 - and the corollary is that when regressions *do* occur, we admit to 483 - them and fix them, instead of blaming user space. 486 + Known-broken commits either 487 + (a) get a timely fix that doesn't have other questions 488 + or 489 + (b) get reverted 484 490 485 - The fact that you have apparently been denying the regression now for 486 - three weeks means that I will revert, and I will stop pulling apparmor 487 - requests until the people involved understand how kernel development 488 - is done. 489 - 490 - * From `2017-10-26 (2/2) 491 - <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 491 + * From `2021-09-20(2) <https://lore.kernel.org/all/CAHk-=wgOvmtRw1TNbMC1rn5YqyTKyn0hz+sc4k0DGNn++u9aYw@mail.gmail.com/>`_:: 492 492 493 - People should basically always feel like they can update their kernel 494 - and simply not have to worry about it. 493 + [...] review shouldn't hold up reported regressions of existing code. That's 494 + just basic _testing_ - either the fix should be applied, or - if the fix is 495 + too invasive or too ugly - the problematic source of the regression should 496 + be reverted. 495 497 496 - I refuse to introduce "you can only update the kernel if you also 497 - update that other program" kind of limitations. If the kernel used to 498 - work for you, the rule is that it continues to work for you. 498 + Review should be about new code, it shouldn't be holding up "there's a 499 + bug report, here's the obvious fix". 499 500 500 - There have been exceptions, but they are few and far between, and they 501 - generally have some major and fundamental reasons for having happened, 502 - that were basically entirely unavoidable, and people _tried_hard_ to 503 - avoid them. Maybe we can't practically support the hardware any more 504 - after it is decades old and nobody uses it with modern kernels any 505 - more. Maybe there's a serious security issue with how we did things, 506 - and people actually depended on that fundamentally broken model. Maybe 507 - there was some fundamental other breakage that just _had_ to have a 508 - flag day for very core and fundamental reasons. 501 + * From `2023-05-08 <https://lore.kernel.org/all/CAHk-=wgzU8_dGn0Yg+DyX7ammTkDUCyEJ4C=NvnHRhxKWC7Wpw@mail.gmail.com/>`_:: 509 502 510 - And notice that this is very much about *breaking* peoples environments. 503 + If something doesn't even build, it should damn well be fixed ASAP. 511 504 512 - Behavioral changes happen, and maybe we don't even support some 513 - feature any more. There's a number of fields in /proc/<pid>/stat that 514 - are printed out as zeroes, simply because they don't even *exist* in 515 - the kernel any more, or because showing them was a mistake (typically 516 - an information leak). But the numbers got replaced by zeroes, so that 517 - the code that used to parse the fields still works. The user might not 518 - see everything they used to see, and so behavior is clearly different, 519 - but things still _work_, even if they might no longer show sensitive 520 - (or no longer relevant) information. 505 + On how fixing regressions with reverts can help prevent maintainer burnout 506 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 521 507 522 - But if something actually breaks, then the change must get fixed or 523 - reverted. And it gets fixed in the *kernel*. Not by saying "well, fix 524 - your user space then". It was a kernel change that exposed the 525 - problem, it needs to be the kernel that corrects for it, because we 526 - have a "upgrade in place" model. We don't have a "upgrade with new 527 - user space". 508 + * From `2026-01-28 <https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/>`_:: 528 509 529 - And I seriously will refuse to take code from people who do not 530 - understand and honor this very simple rule. 510 + > So how can I/we make "immediate fixes" happen more often without 511 + > contributing to maintainer burnout? 531 512 532 - This rule is also not going to change. 513 + [...] the "revert and rethink" model [...] often a good idea in general [...] 533 514 534 - And yes, I realize that the kernel is "special" in this respect. I'm 535 - proud of it. 515 + Exactly so that maintainers don't get stressed out over having a pending 516 + problem report that people keep pestering them about. 536 517 537 - I have seen, and can point to, lots of projects that go "We need to 538 - break that use case in order to make progress" or "you relied on 539 - undocumented behavior, it sucks to be you" or "there's a better way to 540 - do what you want to do, and you have to change to that new better 541 - way", and I simply don't think that's acceptable outside of very early 542 - alpha releases that have experimental users that know what they signed 543 - up for. The kernel hasn't been in that situation for the last two 544 - decades. 518 + I think people are sometimes a bit too bought into whatever changes 519 + they made, and reverting is seen as "too drastic", but I think it's 520 + often the quick and easy solution for when there isn't some obvious 521 + response to a regression report. 545 522 546 - We do API breakage _inside_ the kernel all the time. We will fix 547 - internal problems by saying "you now need to do XYZ", but then it's 548 - about internal kernel API's, and the people who do that then also 549 - obviously have to fix up all the in-kernel users of that API. Nobody 550 - can say "I now broke the API you used, and now _you_ need to fix it 551 - up". Whoever broke something gets to fix it too. 523 + On mainlining fixes when the last -rc or a new release is close 524 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 552 525 553 - And we simply do not break user space. 526 + * From `2026-02-01 <https://lore.kernel.org/all/CAHk-%3DwhXTw1oPsa%2BTLuY1Rc9D1OAiPVOdR_-R2xG45kwDObKdA@mail.gmail.com/>`_:: 554 527 555 - * From `2020-05-21 556 - <https://lore.kernel.org/all/CAHk-=wiVi7mSrsMP=fLXQrXK_UimybW=ziLOwSzFTtoXUacWVQ@mail.gmail.com/>`_:: 528 + So I think I'd rather see them hit rc8 (later today) and have a week 529 + of testing in my tree and be reverted if they cause problems, than 530 + have them go in after rc8 and then cause problems in the 6.19 release 531 + instead. 557 532 558 - The rules about regressions have never been about any kind of 559 - documented behavior, or where the code lives. 533 + * From `2023-04-20 <https://lore.kernel.org/all/CAHk-=wis_qQy4oDNynNKi5b7Qhosmxtoj1jxo5wmB6SRUwQUBQ@mail.gmail.com/>`_:: 560 534 561 - The rules about regressions are always about "breaks user workflow". 535 + But something like this, where the regression was in the previous release 536 + and it's just a clear fix with no semantic subtlety, I consider to be just a 537 + regular regression that should be expedited - partly to make it into stable, 538 + and partly to avoid having to put the fix into _another_ stable kernel. 562 539 563 - Users are literally the _only_ thing that matters. 540 + On sending merge requests with just one fix 541 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 564 542 565 - No amount of "you shouldn't have used this" or "that behavior was 566 - undefined, it's your own fault your app broke" or "that used to work 567 - simply because of a kernel bug" is at all relevant. 543 + * From `2024-04-24 <https://lore.kernel.org/all/CAHk-=wjy_ph9URuFt-pq+2AJ__p7gFDx=yzVSCsx16xAYvNw9g@mail.gmail.com/>`_:: 568 544 569 - Now, reality is never entirely black-and-white. So we've had things 570 - like "serious security issue" etc that just forces us to make changes 571 - that may break user space. But even then the rule is that we don't 572 - really have other options that would allow things to continue. 545 + If the issue is just that there's nothing else happening, I think people 546 + should just point me to the patch and say "can you apply this single fix?" 573 547 574 - And obviously, if users take years to even notice that something 575 - broke, or if we have sane ways to work around the breakage that 576 - doesn't make for too much trouble for users (ie "ok, there are a 577 - handful of users, and they can use a kernel command line to work 578 - around it" kind of things) we've also been a bit less strict. 548 + * From `2023-04-20 <https://lore.kernel.org/all/CAHk-=wis_qQy4oDNynNKi5b7Qhosmxtoj1jxo5wmB6SRUwQUBQ@mail.gmail.com/>`_:: 579 549 580 - But no, "that was documented to be broken" (whether it's because the 581 - code was in staging or because the man-page said something else) is 582 - irrelevant. If staging code is so useful that people end up using it, 583 - that means that it's basically regular kernel code with a flag saying 584 - "please clean this up". 550 + I'm always open to direct fixes when there is no controversy about the fix. 551 + No problem. I still happily deal with individual patches. 585 552 586 - The other side of the coin is that people who talk about "API 587 - stability" are entirely wrong. API's don't matter either. You can make 588 - any changes to an API you like - as long as nobody notices. 553 + On the importance of pointing to bug reports using Link:/Closes: tags 554 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 589 555 590 - Again, the regression rule is not about documentation, not about 591 - API's, and not about the phase of the moon. 556 + * From `2025-07-29(1) <https://lore.kernel.org/all/CAHk-=wj2kJRPWx8B09AAtzj+_g+T6UBX11TP0ebs1WJdTtv=WQ@mail.gmail.com/>`_:: 592 557 593 - It's entirely about "we caused problems for user space that used to work". 558 + [...] revert like this, it really would be good to link to the problems, so 559 + that when people try to re-enable it, they have the history for why it 560 + didn't work the first time. 594 561 595 - * From `2017-11-05 596 - <https://lore.kernel.org/all/CA+55aFzUvbGjD8nQ-+3oiMBx14c_6zOj2n7KLN3UsJ-qsd4Dcw@mail.gmail.com/>`_:: 562 + * From `2022-05-08 <https://lore.kernel.org/all/CAHk-=wjMmSZzMJ3Xnskdg4+GGz=5p5p+GSYyFBTh0f-DgvdBWg@mail.gmail.com/>`_:: 597 563 598 - And our regression rule has never been "behavior doesn't change". 599 - That would mean that we could never make any changes at all. 564 + So I have to once more complain [...] 600 565 601 - For example, we do things like add new error handling etc all the 602 - time, which we then sometimes even add tests for in our kselftest 603 - directory. 566 + [...] There's no link to the actual problem the patch fixes. 604 567 605 - So clearly behavior changes all the time and we don't consider that a 606 - regression per se. 568 + * From `2022-06-22 <https://lore.kernel.org/all/CAHk-=wjxzafG-=J8oT30s7upn4RhBs6TX-uVFZ5rME+L5_DoJA@mail.gmail.com/>`_:: 607 569 608 - The rule for a regression for the kernel is that some real user 609 - workflow breaks. Not some test. Not a "look, I used to be able to do 610 - X, now I can't". 570 + See, *that* link [to the report] would have been useful in the commit. 611 571 612 - * From `2018-08-03 613 - <https://lore.kernel.org/all/CA+55aFwWZX=CXmWDTkDGb36kf12XmTehmQjbiMPCqCRG2hi9kw@mail.gmail.com/>`_:: 572 + On why the "no regressions" rule exists 573 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 614 574 615 - YOU ARE MISSING THE #1 KERNEL RULE. 575 + * From `2026-01-22 <https://lore.kernel.org/all/CAHk-=wheQNiW_WtHGO7bKkT7Uib-p+ai2JP9M+z+FYcZ6CAxYA@mail.gmail.com/>`_:: 616 576 617 - We do not regress, and we do not regress exactly because your are 100% wrong. 577 + But the basic rule is: be so good about backwards compatibility that 578 + users never have to worry about upgrading. They should absolutely feel 579 + confident that any kernel-reported problem will either be solved, or 580 + have an easy solution that is appropriate for *them* (ie a 581 + non-technical user shouldn't be expected to be able to do a lot). 618 582 619 - And the reason you state for your opinion is in fact exactly *WHY* you 620 - are wrong. 583 + Because the last thing we want is people holding back from trying new 584 + kernels. 621 585 622 - Your "good reasons" are pure and utter garbage. 586 + * From `2024-05-28 <https://lore.kernel.org/all/CAHk-=wgtb7y-bEh7tPDvDWru7ZKQ8-KMjZ53Tsk37zsPPdwXbA@mail.gmail.com/>`_:: 623 587 624 - The whole point of "we do not regress" is so that people can upgrade 625 - the kernel and never have to worry about it. 588 + I introduced that "no regressions" rule something like two decades 589 + ago, because people need to be able to update their kernel without 590 + fear of something they relied on suddenly stopping to work. 626 591 627 - > Kernel had a bug which has been fixed 592 + * From `2018-08-03 <https://lore.kernel.org/all/CA+55aFwWZX=CXmWDTkDGb36kf12XmTehmQjbiMPCqCRG2hi9kw@mail.gmail.com/>`_:: 628 593 629 - That is *ENTIRELY* immaterial. 594 + The whole point of "we do not regress" is so that people can upgrade 595 + the kernel and never have to worry about it. 630 596 631 - Guys, whether something was buggy or not DOES NOT MATTER. 597 + [...] 632 598 633 - Why? 599 + Because the only thing that matters IS THE USER. 634 600 635 - Bugs happen. That's a fact of life. Arguing that "we had to break 636 - something because we were fixing a bug" is completely insane. We fix 637 - tens of bugs every single day, thinking that "fixing a bug" means that 638 - we can break something is simply NOT TRUE. 601 + * From `2017-10-26(1) <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 639 602 640 - So bugs simply aren't even relevant to the discussion. They happen, 641 - they get found, they get fixed, and it has nothing to do with "we 642 - break users". 603 + If the kernel used to work for you, the rule is that it continues to work 604 + for you. 643 605 644 - Because the only thing that matters IS THE USER. 606 + [...] 645 607 646 - How hard is that to understand? 608 + People should basically always feel like they can update their kernel 609 + and simply not have to worry about it. 647 610 648 - Anybody who uses "but it was buggy" as an argument is entirely missing 649 - the point. As far as the USER was concerned, it wasn't buggy - it 650 - worked for him/her. 611 + I refuse to introduce "you can only update the kernel if you also 612 + update that other program" kind of limitations. If the kernel used to 613 + work for you, the rule is that it continues to work for you. 651 614 652 - Maybe it worked *because* the user had taken the bug into account, 653 - maybe it worked because the user didn't notice - again, it doesn't 654 - matter. It worked for the user. 615 + On exceptions to the "no regressions" rule 616 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 655 617 656 - Breaking a user workflow for a "bug" is absolutely the WORST reason 657 - for breakage you can imagine. 618 + * From `2026-01-22 <https://lore.kernel.org/all/CAHk-=wheQNiW_WtHGO7bKkT7Uib-p+ai2JP9M+z+FYcZ6CAxYA@mail.gmail.com/>`_:: 658 619 659 - It's basically saying "I took something that worked, and I broke it, 660 - but now it's better". Do you not see how f*cking insane that statement 661 - is? 620 + There are _very_ few exceptions to that rule, the main one being "the 621 + problem was a fundamental huge and gaping security issue and we *had* to 622 + make that change, and we couldn't even make your limited use-case just 623 + continue to work". 662 624 663 - And without users, your program is not a program, it's a pointless 664 - piece of code that you might as well throw away. 625 + The other exception is "the problem was reported years after it was 626 + introduced, and now most people rely on the new behavior". 665 627 666 - Seriously. This is *why* the #1 rule for kernel development is "we 667 - don't break users". Because "I fixed a bug" is absolutely NOT AN 668 - ARGUMENT if that bug fix broke a user setup. You actually introduced a 669 - MUCH BIGGER bug by "fixing" something that the user clearly didn't 670 - even care about. 628 + [...] 671 629 672 - And dammit, we upgrade the kernel ALL THE TIME without upgrading any 673 - other programs at all. It is absolutely required, because flag-days 674 - and dependencies are horribly bad. 630 + Now, if it's one or two users and you can just get them to recompile, 631 + that's one thing. Niche hardware and odd use-cases can sometimes be 632 + solved that way, and regressions can sometimes be fixed by handholding 633 + every single reporter if the reporter is willing and able to change 634 + his or her workflow. 675 635 676 - And it is also required simply because I as a kernel developer do not 677 - upgrade random other tools that I don't even care about as I develop 678 - the kernel, and I want any of my users to feel safe doing the same 679 - time. 636 + * From `2023-04-20 <https://lore.kernel.org/all/CAHk-=wis_qQy4oDNynNKi5b7Qhosmxtoj1jxo5wmB6SRUwQUBQ@mail.gmail.com/>`_:: 680 637 681 - So no. Your rule is COMPLETELY wrong. If you cannot upgrade a kernel 682 - without upgrading some other random binary, then we have a problem. 638 + And yes, I do consider "regression in an earlier release" to be a 639 + regression that needs fixing. 683 640 684 - * From `2021-06-05 685 - <https://lore.kernel.org/all/CAHk-=wiUVqHN76YUwhkjZzwTdjMMJf_zN4+u7vEJjmEGh3recw@mail.gmail.com/>`_:: 641 + There's obviously a time limit: if that "regression in an earlier 642 + release" was a year or more ago, and just took forever for people to 643 + notice, and it had semantic changes that now mean that fixing the 644 + regression could cause a _new_ regression, then that can cause me to 645 + go "Oh, now the new semantics are what we have to live with". 686 646 687 - THERE ARE NO VALID ARGUMENTS FOR REGRESSIONS. 647 + * From `2017-10-26(2) <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 688 648 689 - Honestly, security people need to understand that "not working" is not 690 - a success case of security. It's a failure case. 649 + There have been exceptions, but they are few and far between, and they 650 + generally have some major and fundamental reasons for having happened, 651 + that were basically entirely unavoidable, and people _tried_hard_ to 652 + avoid them. Maybe we can't practically support the hardware any more 653 + after it is decades old and nobody uses it with modern kernels any 654 + more. Maybe there's a serious security issue with how we did things, 655 + and people actually depended on that fundamentally broken model. Maybe 656 + there was some fundamental other breakage that just _had_ to have a 657 + flag day for very core and fundamental reasons. 691 658 692 - Yes, "not working" may be secure. But security in that case is *pointless*. 659 + On situations where updating something in userspace can resolve regressions 660 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 693 661 694 - * From `2011-05-06 (1/3) 695 - <https://lore.kernel.org/all/BANLkTim9YvResB+PwRp7QTK-a5VNg2PvmQ@mail.gmail.com/>`_:: 662 + * From `2018-08-03 <https://lore.kernel.org/all/CA+55aFwWZX=CXmWDTkDGb36kf12XmTehmQjbiMPCqCRG2hi9kw@mail.gmail.com/>`_:: 696 663 697 - Binary compatibility is more important. 664 + And dammit, we upgrade the kernel ALL THE TIME without upgrading any 665 + other programs at all. It is absolutely required, because flag-days 666 + and dependencies are horribly bad. 698 667 699 - And if binaries don't use the interface to parse the format (or just 700 - parse it wrongly - see the fairly recent example of adding uuid's to 701 - /proc/self/mountinfo), then it's a regression. 668 + And it is also required simply because I as a kernel developer do not 669 + upgrade random other tools that I don't even care about as I develop the 670 + kernel, and I want any of my users to feel safe doing the same time. 702 671 703 - And regressions get reverted, unless there are security issues or 704 - similar that makes us go "Oh Gods, we really have to break things". 672 + * From `2017-10-26(3) <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 705 673 706 - I don't understand why this simple logic is so hard for some kernel 707 - developers to understand. Reality matters. Your personal wishes matter 708 - NOT AT ALL. 674 + But if something actually breaks, then the change must get fixed or 675 + reverted. And it gets fixed in the *kernel*. Not by saying "well, fix your 676 + user space then". It was a kernel change that exposed the problem, it needs 677 + to be the kernel that corrects for it, because we have a "upgrade in place" 678 + model. We don't have a "upgrade with new user space". 709 679 710 - If you made an interface that can be used without parsing the 711 - interface description, then we're stuck with the interface. Theory 712 - simply doesn't matter. 680 + And I seriously will refuse to take code from people who do not understand 681 + and honor this very simple rule. 713 682 714 - You could help fix the tools, and try to avoid the compatibility 715 - issues that way. There aren't that many of them. 683 + This rule is also not going to change. 716 684 717 - From `2011-05-06 (2/3) 718 - <https://lore.kernel.org/all/BANLkTi=KVXjKR82sqsz4gwjr+E0vtqCmvA@mail.gmail.com/>`_:: 685 + And yes, I realize that the kernel is "special" in this respect. I'm proud 686 + of it. 719 687 720 - it's clearly NOT an internal tracepoint. By definition. It's being 721 - used by powertop. 688 + * From `2017-10-26(4) <https://lore.kernel.org/all/CA+55aFwiiQYJ+YoLKCXjN_beDVfu38mg=Ggg5LFOcqHE8Qi7Zw@mail.gmail.com/>`_:: 722 689 723 - From `2011-05-06 (3/3) 724 - <https://lore.kernel.org/all/BANLkTinazaXRdGovYL7rRVp+j6HbJ7pzhg@mail.gmail.com/>`_:: 690 + If you break existing user space setups THAT IS A REGRESSION. 725 691 726 - We have programs that use that ABI and thus it's a regression if they break. 692 + It's not ok to say "but we'll fix the user space setup". 727 693 728 - * From `2012-07-06 <https://lore.kernel.org/all/CA+55aFwnLJ+0sjx92EGREGTWOx84wwKaraSzpTNJwPVV8edw8g@mail.gmail.com/>`_:: 694 + Really. NOT OK. 729 695 730 - > Now this got me wondering if Debian _unstable_ actually qualifies as a 731 - > standard distro userspace. 696 + On what qualifies as userspace interface, ABI, API, documented interfaces, etc. 697 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 732 698 733 - Oh, if the kernel breaks some standard user space, that counts. Tons 734 - of people run Debian unstable 699 + * From `2026-01-20 <https://lore.kernel.org/all/CAHk-=wga8Qu0-OSE9VZbviq9GuqwhPhLUXeAt-S7_9+fMCLkKg@mail.gmail.com/>`_:: 735 700 736 - * From `2019-09-15 737 - <https://lore.kernel.org/lkml/CAHk-=wiP4K8DRJWsCo=20hn_6054xBamGKF2kPgUzpB5aMaofA@mail.gmail.com/>`_:: 701 + So I absolutely detest the whole notion of "ABI changes". It's a 702 + meaningless concept, and I hate it with a passion, [...] 738 703 739 - One _particularly_ last-minute revert is the top-most commit (ignoring 740 - the version change itself) done just before the release, and while 741 - it's very annoying, it's perhaps also instructive. 704 + The Linux rule for regressions is basically based on the philosophical 705 + question of "If a tree falls in the forest, and nobody is around to 706 + hear it, does it make a sound?". 742 707 743 - What's instructive about it is that I reverted a commit that wasn't 744 - actually buggy. In fact, it was doing exactly what it set out to do, 745 - and did it very well. In fact it did it _so_ well that the much 746 - improved IO patterns it caused then ended up revealing a user-visible 747 - regression due to a real bug in a completely unrelated area. 708 + So the only thing that matters is if something breaks user-*conscious* 709 + behavior. 748 710 749 - The actual details of that regression are not the reason I point that 750 - revert out as instructive, though. It's more that it's an instructive 751 - example of what counts as a regression, and what the whole "no 752 - regressions" kernel rule means. The reverted commit didn't change any 753 - API's, and it didn't introduce any new bugs. But it ended up exposing 754 - another problem, and as such caused a kernel upgrade to fail for a 755 - user. So it got reverted. 711 + And when that happens, the distinction between "bug fix" and "new 712 + feature" and "ABI change" matters not one whit, and the change needs 713 + to be done differently. 756 714 757 - The point here being that we revert based on user-reported _behavior_, 758 - not based on some "it changes the ABI" or "it caused a bug" concept. 759 - The problem was really pre-existing, and it just didn't happen to 760 - trigger before. The better IO patterns introduced by the change just 761 - happened to expose an old bug, and people had grown to depend on the 762 - previously benign behavior of that old issue. 715 + [...] 763 716 764 - And never fear, we'll re-introduce the fix that improved on the IO 765 - patterns once we've decided just how to handle the fact that we had a 766 - bad interaction with an interface that people had then just happened 767 - to rely on incidental behavior for before. It's just that we'll have 768 - to hash through how to do that (there are no less than three different 769 - patches by three different developers being discussed, and there might 770 - be more coming...). In the meantime, I reverted the thing that exposed 771 - the problem to users for this release, even if I hope it will be 772 - re-introduced (perhaps even backported as a stable patch) once we have 773 - consensus about the issue it exposed. 717 + I just wanted to point out that the argument about whether it's an ABI 718 + change or not is irrelevant. If it turns out that some program - not a test 719 + script, but something with relevance to conscious user expectations ~ 720 + depended on the old broken behavior, then it needs to be done some other 721 + way. 774 722 775 - Take-away from the whole thing: it's not about whether you change the 776 - kernel-userspace ABI, or fix a bug, or about whether the old code 777 - "should never have worked in the first place". It's about whether 778 - something breaks existing users' workflow. 723 + * From `2026-02-13 <https://lore.kernel.org/all/CAHk-=whY-N8kjm8kiFUV5Ei-8AuYw--EPGD-AR3Pd+5GTx2sAQ@mail.gmail.com/>`_:: 779 724 780 - Anyway, that was my little aside on the whole regression thing. Since 781 - it's that "first rule of kernel programming", I felt it is perhaps 782 - worth just bringing it up every once in a while 725 + > [...] this should not fall under the don't break user space rule [...] 726 + 727 + Note that the rule is about breaking *users*, not breaking user space per 728 + se. [...] 729 + 730 + If some user setup breaks, things need fixing. 731 + 732 + [...] but I want to make it very clear that there are no excuses about "user 733 + space applications". 734 + 735 + * From `2021-09-20(4) <https://lore.kernel.org/all/CAHk-=wi7DB2SJ-wngVvsJ7Ak2cM556Q8437sOXo4EJt2BWPdEg@mail.gmail.com/>`_:: 736 + 737 + [...] a regression is a bit like Schrödinger's cat - if nobody is around 738 + to notice it and it doesn't actually affect any real workload, then you 739 + can treat the regression as if it doesn't exist. 740 + 741 + * From `2020-05-21 <https://lore.kernel.org/all/CAHk-=wiVi7mSrsMP=fLXQrXK_UimybW=ziLOwSzFTtoXUacWVQ@mail.gmail.com/>`_:: 742 + 743 + The rules about regressions have never been about any kind of documented 744 + behavior, or where the code lives. 745 + 746 + The rules about regressions are always about "breaks user workflow". 747 + 748 + Users are literally the _only_ thing that matters. 749 + 750 + * From `2019-09-15 <https://lore.kernel.org/lkml/CAHk-=wiP4K8DRJWsCo=20hn_6054xBamGKF2kPgUzpB5aMaofA@mail.gmail.com/>`_:: 751 + 752 + One _particularly_ last-minute revert is the top-most commit (ignoring 753 + the version change itself) done just before the release, and while 754 + it's very annoying, it's perhaps also instructive. 755 + 756 + What's instructive about it is that I reverted a commit that wasn't 757 + actually buggy. In fact, it was doing exactly what it set out to do, 758 + and did it very well. In fact it did it _so_ well that the much 759 + improved IO patterns it caused then ended up revealing a user-visible 760 + regression due to a real bug in a completely unrelated area. 761 + 762 + The actual details of that regression are not the reason I point that 763 + revert out as instructive, though. It's more that it's an instructive 764 + example of what counts as a regression, and what the whole "no 765 + regressions" kernel rule means. 766 + 767 + [...] The reverted commit didn't change any API's, and it didn't introduce 768 + any new bugs. But it ended up exposing another problem, and as such caused 769 + a kernel upgrade to fail for a user. So it got reverted. 770 + 771 + The point here being that we revert based on user-reported _behavior_, not 772 + based on some "it changes the ABI" or "it caused a bug" concept. The problem 773 + was really pre-existing, and it just didn't happen to trigger before. [...] 774 + 775 + Take-away from the whole thing: it's not about whether you change the 776 + kernel-userspace ABI, or fix a bug, or about whether the old code 777 + "should never have worked in the first place". It's about whether 778 + something breaks existing users' workflow. 779 + 780 + * From `2017-11-05 <https://lore.kernel.org/all/CA+55aFzUvbGjD8nQ-+3oiMBx14c_6zOj2n7KLN3UsJ-qsd4Dcw@mail.gmail.com/>`_:: 781 + 782 + And our regression rule has never been "behavior doesn't change". 783 + That would mean that we could never make any changes at all. 784 + 785 + * From `2020-05-21 <https://lore.kernel.org/all/CAHk-=wiVi7mSrsMP=fLXQrXK_UimybW=ziLOwSzFTtoXUacWVQ@mail.gmail.com/>`_:: 786 + 787 + No amount of "you shouldn't have used this" or "that behavior was 788 + undefined, it's your own fault your app broke" or "that used to work 789 + simply because of a kernel bug" is at all relevant. 790 + 791 + * From `2021-05-21 <https://lore.kernel.org/all/CAHk-=wiVi7mSrsMP=fLXQrXK_UimybW=ziLOwSzFTtoXUacWVQ@mail.gmail.com/>`_:: 792 + 793 + But no, "that was documented to be broken" (whether it's because the code 794 + was in staging or because the man-page said something else) is irrelevant. 795 + If staging code is so useful that people end up using it, that means that 796 + it's basically regular kernel code with a flag saying "please clean this 797 + up". 798 + 799 + [...] 800 + 801 + The other side of the coin is that people who talk about "API stability" are 802 + entirely wrong. API's don't matter either. You can make any changes to an 803 + API you like - as long as nobody notices. 804 + 805 + Again, the regression rule is not about documentation, not about API's, and 806 + not about the phase of the moon. 807 + 808 + * From `2012-07-06 <https://lore.kernel.org/all/CA+55aFwnLJ+0sjx92EGREGTWOx84wwKaraSzpTNJwPVV8edw8g@mail.gmail.com/>`_:: 809 + 810 + > Now this got me wondering if Debian _unstable_ actually qualifies as a 811 + > standard distro userspace. 812 + 813 + Oh, if the kernel breaks some standard user space, that counts. Tons 814 + of people run Debian unstable 815 + 816 + * From `2011-05-06 <https://lore.kernel.org/all/BANLkTi=KVXjKR82sqsz4gwjr+E0vtqCmvA@mail.gmail.com/>`_:: 817 + 818 + It's clearly NOT an internal tracepoint. By definition. It's being 819 + used by powertop. 820 + 821 + On regressions noticed by users or test-suites/CIs 822 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 823 + 824 + * From `2026-01-22 <https://lore.kernel.org/all/CAHk-=wheQNiW_WtHGO7bKkT7Uib-p+ai2JP9M+z+FYcZ6CAxYA@mail.gmail.com/>`_:: 825 + 826 + Users complaining is the only real line in the end. 827 + 828 + [...] a test-suite complaining is then often a *very* good indication that 829 + maybe users will hit some problem, and test suite issues should be taken 830 + very seriously [...] 831 + 832 + But a test-suite error isn't necessarily where you have to draw the 833 + line - it's a big red flag [...] 834 + 835 + * From `2024-29-01 <https://lore.kernel.org/all/CAHk-=wg8BrZEzjJ5kUyZzHPZmFqH6ooMN1gRBCofxxCfucgjaw@mail.gmail.com/>`_:: 836 + 837 + The "no regressions" rule is not about made-up "if I do this, behavior 838 + changes". 839 + 840 + The "no regressions" rule is about *users*. 841 + 842 + If you have an actual user that has been doing insane things, and we 843 + change something, and now the insane thing no longer works, at that 844 + point it's a regression, and we'll sigh, and go "Users are insane" and 845 + have to fix it. 846 + 847 + But if you have some random test that now behaves differently, it's 848 + not a regression. It's a *warning* sign, sure: tests are useful. 849 + 850 + On accepting when a regression occurred 851 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 852 + 853 + * From `2026-01-22 <https://lore.kernel.org/all/CAHk-=wheQNiW_WtHGO7bKkT7Uib-p+ai2JP9M+z+FYcZ6CAxYA@mail.gmail.com/>`_:: 854 + 855 + But starting to argue about users reporting breaking changes is 856 + basically the final line for me. I have a couple of people that I have 857 + in my spam block-list and refuse to have anything to do with, and they 858 + have generally been about exactly that. 859 + 860 + Note how it's not about making mistakes and _causing_ the regression. 861 + That's normal. That's development. But then arguing about it is a 862 + no-no. 863 + 864 + * From `2024-06-23 <https://lore.kernel.org/all/CAHk-=wi_KMO_rJ6OCr8mAWBRg-irziM=T9wxGC+J1VVoQb39gw@mail.gmail.com/>`_:: 865 + 866 + We don't introduce regressions and then blame others. 867 + 868 + There's a very clear rule in kernel development: things that break 869 + other things ARE NOT FIXES. 870 + 871 + EVER. 872 + 873 + They get reverted, or the thing they broke gets fixed. 874 + 875 + * From `2021-06-05 <https://lore.kernel.org/all/CAHk-=wiUVqHN76YUwhkjZzwTdjMMJf_zN4+u7vEJjmEGh3recw@mail.gmail.com/>`_:: 876 + 877 + THERE ARE NO VALID ARGUMENTS FOR REGRESSIONS. 878 + 879 + Honestly, security people need to understand that "not working" is not 880 + a success case of security. It's a failure case. 881 + 882 + Yes, "not working" may be secure. But security in that case is *pointless*. 883 + 884 + * From `2017-10-26(5) <https://lore.kernel.org/lkml/CA+55aFwiiQYJ+YoLKCXjN_beDVfu38mg=Ggg5LFOcqHE8Qi7Zw@mail.gmail.com/>`_:: 885 + 886 + [...] when regressions *do* occur, we admit to them and fix them, instead of 887 + blaming user space. 888 + 889 + The fact that you have apparently been denying the regression now for 890 + three weeks means that I will revert, and I will stop pulling apparmor 891 + requests until the people involved understand how kernel development 892 + is done. 893 + 894 + On back-and-forth 895 + ~~~~~~~~~~~~~~~~~ 896 + 897 + * From `2024-05-28 <https://lore.kernel.org/all/CAHk-=wgtb7y-bEh7tPDvDWru7ZKQ8-KMjZ53Tsk37zsPPdwXbA@mail.gmail.com/>`_:: 898 + 899 + The "no regressions" rule is that we do not introduce NEW bugs. 900 + 901 + It *literally* came about because we had an endless dance of "fix two 902 + bugs, introduce one new one", and that then resulted in a system that 903 + you cannot TRUST. 904 + 905 + * From `2021-09-20(1) <https://lore.kernel.org/all/CAHk-=wi7DB2SJ-wngVvsJ7Ak2cM556Q8437sOXo4EJt2BWPdEg@mail.gmail.com/>`_:: 906 + 907 + And the thing that makes regressions special is that back when I 908 + wasn't so strict about these things, we'd end up in endless "seesaw 909 + situations" where somebody would fix something, it would break 910 + something else, then that something else would break, and it would 911 + never actually converge on anything reliable at all. 912 + 913 + * From `2015-08-13 <https://lore.kernel.org/all/CA+55aFxk8-BsiKwr_S-c+4G6wihKPQVMLE34H9wOZpeua6W9+Q@mail.gmail.com/>`_:: 914 + 915 + The strict policy of no regressions actually originally started mainly wrt 916 + suspend/resume issues, where the "fix one machine, break another" kind of 917 + back-and-forth caused endless problems, and meant that we didn't actually 918 + necessarily make any forward progress, just moving a problem around. 919 + 920 + On changes with a risk of causing regressions 921 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 922 + 923 + * From `2023-06-02 <https://lore.kernel.org/all/CAHk-=wgyAGUMHmQM-5Eb556z5xiHZB7cF05qjrtUH4F7P-1rSA@mail.gmail.com/>`_:: 924 + 925 + So what I think you should do is to fix the bug right, with a clean 926 + patch, and no crazy hacks. That is something we can then apply and 927 + test. All the while knowing full well that "uhhuh, this is a visible 928 + change, we may have to revert it". 929 + 930 + If then some *real* load ends up showing a regression, we may just be 931 + screwed. Our current behavior may be buggy, but we have the rule that 932 + once user space depends on kernel bugs, they become features pretty 933 + much by definition, however much we might dislike it. 934 + 935 + On in-kernel workarounds to avoid regressions 936 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 937 + 938 + * From `2017-10-26(6) <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 939 + 940 + Behavioral changes happen, and maybe we don't even support some 941 + feature any more. There's a number of fields in /proc/<pid>/stat that 942 + are printed out as zeroes, simply because they don't even *exist* in 943 + the kernel any more, or because showing them was a mistake (typically 944 + an information leak). But the numbers got replaced by zeroes, so that 945 + the code that used to parse the fields still works. The user might not 946 + see everything they used to see, and so behavior is clearly different, 947 + but things still _work_, even if they might no longer show sensitive 948 + (or no longer relevant) information. 949 + 950 + On regressions caused by bugfixes 951 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 952 + 953 + * From `2018-08-03 <https://lore.kernel.org/all/CA+55aFwWZX=CXmWDTkDGb36kf12XmTehmQjbiMPCqCRG2hi9kw@mail.gmail.com/>`_:: 954 + 955 + > Kernel had a bug which has been fixed 956 + 957 + That is *ENTIRELY* immaterial. 958 + 959 + Guys, whether something was buggy or not DOES NOT MATTER. 960 + 961 + [...] 962 + 963 + It's basically saying "I took something that worked, and I broke it, 964 + but now it's better". Do you not see how f*cking insane that statement 965 + is? 966 + 967 + On internal API changes 968 + ~~~~~~~~~~~~~~~~~~~~~~~ 969 + 970 + * From `2017-10-26(7) <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_:: 971 + 972 + We do API breakage _inside_ the kernel all the time. We will fix 973 + internal problems by saying "you now need to do XYZ", but then it's 974 + about internal kernel API's, and the people who do that then also 975 + obviously have to fix up all the in-kernel users of that API. Nobody 976 + can say "I now broke the API you used, and now _you_ need to fix it 977 + up". Whoever broke something gets to fix it too. 978 + 979 + On regressions only found after a long time 980 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 981 + 982 + * From `2024-03-28 <https://lore.kernel.org/all/CAHk-=wgFuoHpMk_Z_R3qMXVDgq0N1592+bABkyGjwwSL4zBtHA@mail.gmail.com/>`_:: 983 + 984 + I'm definitely not reverting a patch from almost a decade ago as a 985 + regression. 986 + 987 + If it took that long to find, it can't be that critical of a regression. 988 + 989 + So yes, let's treat it as a regular bug. 990 + 991 + On testing regressions fixes in linux-next 992 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 993 + 994 + * On `maintainers summit 2024 <https://lwn.net/Articles/990599/>`_:: 995 + 996 + So running fixes though linux-next is just a waste of time. 997 + 998 + On a few other aspects related to regressions 999 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1000 + 1001 + * From `2025-07-29(2) <https://lore.kernel.org/all/CAHk-=wjj9DvOZtmTkoLtyfHmy5mNKy6q_96d9=4FUEDXre=cww@mail.gmail.com/>`_ 1002 + [which `is not quite a regression, but a huge inconvenience <https://lore.kernel.org/all/CAHk-=wgO0Rx2LcYT4f75Xs46orbJ4JxO2jbAFQnVKDYAjV5HeQ@mail.gmail.com/>`_]:: 1003 + 1004 + I no longer have sound. 1005 + 1006 + I also suspect that it's purely because "make oldconfig" doesn't work, 1007 + and probably turned off my old Intel HDA settings. Or something. 1008 + 1009 + Renaming config parameters is *bad*. I've harped on the Kconfig phase 1010 + of the kernel build probably being our nastiest point, and a real pain 1011 + point to people getting involved with development simply because 1012 + building your own kernel can be so daunting with hundreds of fairly 1013 + esoteric questions. 783 1014 784 1015 .. 785 1016 end-of-content

Configure Feed

Configure Feed