tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'irq-core-2025-09-29' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull irq core updates from Thomas Gleixner:
"A set of updates for the interrupt core subsystem:

- Introduce irq_chip_[startup|shutdown]_parent() to prepare for
addressing a few short comings in the PCI/MSI interrupt subsystem.

It allows to utilize the interrupt chip startup/shutdown callbacks
for initializing the interrupt chip hierarchy properly on certain
RISCV implementations and provides a mechanism to reduce the
overhead of masking and unmasking PCI/MSI interrupts during
operation when the underlying MSI provider can mask the interrupt.

The actual usage comes with the interrupt driver pull request.

- Add generic error handling for devm_request_*_irq()

This allows to remove the zoo of random error printk's all over the
usage sites.

- Add a mechanism to warn about long-running interrupt handlers

Long running interrupt handlers can introduce latencies and
tracking them down is a tedious task. The tracking has to be
enabled with a threshold on the kernel command line and utilizes a
static branch to remove the overhead when disabled.

- Update and extend the selftests which validate the CPU hotplug
interrupt migration logic

- Allow dropping the per CPU softirq lock on PREEMPT_RT kernels,
which causes contention and latencies all over the place.

The serialization requirements have been pushed down into the
actual affected usage sites already.

- The usual small cleanups and improvements"

* tag 'irq-core-2025-09-29' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT
softirq: Provide a handshake for canceling tasklets via polling
genirq/test: Ensure CPU 1 is online for hotplug test
genirq/test: Drop CONFIG_GENERIC_IRQ_MIGRATION assumptions
genirq/test: Depend on SPARSE_IRQ
genirq/test: Fail early if interrupt request fails
genirq/test: Factor out fake-virq setup
genirq/test: Select IRQ_DOMAIN
genirq/test: Fix depth tests on architectures with NOREQUEST by default.
genirq: Add support for warning on long-running interrupt handlers
genirq/devres: Add error handling in devm_request_*_irq()
genirq: Add irq_chip_(startup/shutdown)_parent()
genirq: Remove GENERIC_IRQ_LEGACY

Linus Torvalds 3b2074c7 1d17e808

+344 -106
+5
Documentation/admin-guide/kernel-parameters.txt
··· 2606 2606 for it. Intended to get systems with badly broken 2607 2607 firmware running. 2608 2608 2609 + irqhandler.duration_warn_us= [KNL] 2610 + Warn if an IRQ handler exceeds the specified duration 2611 + threshold in microseconds. Useful for identifying 2612 + long-running IRQs in the system. 2613 + 2609 2614 irqpoll [HW] 2610 2615 When an interrupt is not handled search all handlers 2611 2616 for it. Also check all handlers each timer
+2 -4
include/linux/irq.h
··· 669 669 extern int irq_chip_get_parent_state(struct irq_data *data, 670 670 enum irqchip_irq_state which, 671 671 bool *state); 672 + extern void irq_chip_shutdown_parent(struct irq_data *data); 673 + extern unsigned int irq_chip_startup_parent(struct irq_data *data); 672 674 extern void irq_chip_enable_parent(struct irq_data *data); 673 675 extern void irq_chip_disable_parent(struct irq_data *data); 674 676 extern void irq_chip_ack_parent(struct irq_data *data); ··· 977 975 { 978 976 irq_free_descs(irq, 1); 979 977 } 980 - 981 - #ifdef CONFIG_GENERIC_IRQ_LEGACY 982 - void irq_init_desc(unsigned int irq); 983 - #endif 984 978 985 979 /** 986 980 * struct irq_chip_regs - register offsets for struct irq_gci
+13
kernel/Kconfig.preempt
··· 103 103 Select this if you are building a kernel for systems which 104 104 require real-time guarantees. 105 105 106 + config PREEMPT_RT_NEEDS_BH_LOCK 107 + bool "Enforce softirq synchronisation on PREEMPT_RT" 108 + depends on PREEMPT_RT 109 + help 110 + Enforce synchronisation across the softirqs context. On PREEMPT_RT 111 + the softirq is preemptible. This enforces the same per-CPU BLK 112 + semantic non-PREEMPT_RT builds have. This should not be needed 113 + because per-CPU locks were added to avoid the per-CPU BKL. 114 + 115 + This switch provides the old behaviour for testing reasons. Select 116 + this if you suspect an error with preemptible softirq and want test 117 + the old synchronized behaviour. 118 + 106 119 config PREEMPT_COUNT 107 120 bool 108 121
+2 -4
kernel/irq/Kconfig
··· 6 6 config MAY_HAVE_SPARSE_IRQ 7 7 bool 8 8 9 - # Legacy support, required for itanic 10 - config GENERIC_IRQ_LEGACY 11 - bool 12 - 13 9 # Enable the generic irq autoprobe mechanism 14 10 config GENERIC_IRQ_PROBE 15 11 bool ··· 143 147 config IRQ_KUNIT_TEST 144 148 bool "KUnit tests for IRQ management APIs" if !KUNIT_ALL_TESTS 145 149 depends on KUNIT=y 150 + depends on SPARSE_IRQ 146 151 default KUNIT_ALL_TESTS 152 + select IRQ_DOMAIN 147 153 imply SMP 148 154 help 149 155 This option enables KUnit tests for the IRQ subsystem API. These are
+37
kernel/irq/chip.c
··· 1260 1260 EXPORT_SYMBOL_GPL(irq_chip_get_parent_state); 1261 1261 1262 1262 /** 1263 + * irq_chip_shutdown_parent - Shutdown the parent interrupt 1264 + * @data: Pointer to interrupt specific data 1265 + * 1266 + * Invokes the irq_shutdown() callback of the parent if available or falls 1267 + * back to irq_chip_disable_parent(). 1268 + */ 1269 + void irq_chip_shutdown_parent(struct irq_data *data) 1270 + { 1271 + struct irq_data *parent = data->parent_data; 1272 + 1273 + if (parent->chip->irq_shutdown) 1274 + parent->chip->irq_shutdown(parent); 1275 + else 1276 + irq_chip_disable_parent(data); 1277 + } 1278 + EXPORT_SYMBOL_GPL(irq_chip_shutdown_parent); 1279 + 1280 + /** 1281 + * irq_chip_startup_parent - Startup the parent interrupt 1282 + * @data: Pointer to interrupt specific data 1283 + * 1284 + * Invokes the irq_startup() callback of the parent if available or falls 1285 + * back to irq_chip_enable_parent(). 1286 + */ 1287 + unsigned int irq_chip_startup_parent(struct irq_data *data) 1288 + { 1289 + struct irq_data *parent = data->parent_data; 1290 + 1291 + if (parent->chip->irq_startup) 1292 + return parent->chip->irq_startup(parent); 1293 + 1294 + irq_chip_enable_parent(data); 1295 + return 0; 1296 + } 1297 + EXPORT_SYMBOL_GPL(irq_chip_startup_parent); 1298 + 1299 + /** 1263 1300 * irq_chip_enable_parent - Enable the parent interrupt (defaults to unmask if 1264 1301 * NULL) 1265 1302 * @data: Pointer to interrupt specific data
+87 -40
kernel/irq/devres.c
··· 30 30 return this->irq == match->irq && this->dev_id == match->dev_id; 31 31 } 32 32 33 - /** 34 - * devm_request_threaded_irq - allocate an interrupt line for a managed device 35 - * @dev: device to request interrupt for 36 - * @irq: Interrupt line to allocate 37 - * @handler: Function to be called when the IRQ occurs 38 - * @thread_fn: function to be called in a threaded interrupt context. NULL 39 - * for devices which handle everything in @handler 40 - * @irqflags: Interrupt type flags 41 - * @devname: An ascii name for the claiming device, dev_name(dev) if NULL 42 - * @dev_id: A cookie passed back to the handler function 43 - * 44 - * Except for the extra @dev argument, this function takes the 45 - * same arguments and performs the same function as 46 - * request_threaded_irq(). IRQs requested with this function will be 47 - * automatically freed on driver detach. 48 - * 49 - * If an IRQ allocated with this function needs to be freed 50 - * separately, devm_free_irq() must be used. 51 - */ 52 - int devm_request_threaded_irq(struct device *dev, unsigned int irq, 53 - irq_handler_t handler, irq_handler_t thread_fn, 54 - unsigned long irqflags, const char *devname, 55 - void *dev_id) 33 + static int devm_request_result(struct device *dev, int rc, unsigned int irq, 34 + irq_handler_t handler, irq_handler_t thread_fn, 35 + const char *devname) 36 + { 37 + if (rc >= 0) 38 + return rc; 39 + 40 + return dev_err_probe(dev, rc, "request_irq(%u) %ps %ps %s\n", 41 + irq, handler, thread_fn, devname ? : ""); 42 + } 43 + 44 + static int __devm_request_threaded_irq(struct device *dev, unsigned int irq, 45 + irq_handler_t handler, 46 + irq_handler_t thread_fn, 47 + unsigned long irqflags, 48 + const char *devname, void *dev_id) 56 49 { 57 50 struct irq_devres *dr; 58 51 int rc; ··· 71 78 72 79 return 0; 73 80 } 74 - EXPORT_SYMBOL(devm_request_threaded_irq); 75 81 76 82 /** 77 - * devm_request_any_context_irq - allocate an interrupt line for a managed device 78 - * @dev: device to request interrupt for 79 - * @irq: Interrupt line to allocate 80 - * @handler: Function to be called when the IRQ occurs 81 - * @irqflags: Interrupt type flags 82 - * @devname: An ascii name for the claiming device, dev_name(dev) if NULL 83 - * @dev_id: A cookie passed back to the handler function 83 + * devm_request_threaded_irq - allocate an interrupt line for a managed device with error logging 84 + * @dev: Device to request interrupt for 85 + * @irq: Interrupt line to allocate 86 + * @handler: Function to be called when the interrupt occurs 87 + * @thread_fn: Function to be called in a threaded interrupt context. NULL 88 + * for devices which handle everything in @handler 89 + * @irqflags: Interrupt type flags 90 + * @devname: An ascii name for the claiming device, dev_name(dev) if NULL 91 + * @dev_id: A cookie passed back to the handler function 84 92 * 85 - * Except for the extra @dev argument, this function takes the 86 - * same arguments and performs the same function as 87 - * request_any_context_irq(). IRQs requested with this function will be 88 - * automatically freed on driver detach. 93 + * Except for the extra @dev argument, this function takes the same 94 + * arguments and performs the same function as request_threaded_irq(). 95 + * Interrupts requested with this function will be automatically freed on 96 + * driver detach. 89 97 * 90 - * If an IRQ allocated with this function needs to be freed 91 - * separately, devm_free_irq() must be used. 98 + * If an interrupt allocated with this function needs to be freed 99 + * separately, devm_free_irq() must be used. 100 + * 101 + * When the request fails, an error message is printed with contextual 102 + * information (device name, interrupt number, handler functions and 103 + * error code). Don't add extra error messages at the call sites. 104 + * 105 + * Return: 0 on success or a negative error number. 92 106 */ 93 - int devm_request_any_context_irq(struct device *dev, unsigned int irq, 94 - irq_handler_t handler, unsigned long irqflags, 95 - const char *devname, void *dev_id) 107 + int devm_request_threaded_irq(struct device *dev, unsigned int irq, 108 + irq_handler_t handler, irq_handler_t thread_fn, 109 + unsigned long irqflags, const char *devname, 110 + void *dev_id) 111 + { 112 + int rc = __devm_request_threaded_irq(dev, irq, handler, thread_fn, 113 + irqflags, devname, dev_id); 114 + 115 + return devm_request_result(dev, rc, irq, handler, thread_fn, devname); 116 + } 117 + EXPORT_SYMBOL(devm_request_threaded_irq); 118 + 119 + static int __devm_request_any_context_irq(struct device *dev, unsigned int irq, 120 + irq_handler_t handler, 121 + unsigned long irqflags, 122 + const char *devname, void *dev_id) 96 123 { 97 124 struct irq_devres *dr; 98 125 int rc; ··· 136 123 devres_add(dev, dr); 137 124 138 125 return rc; 126 + } 127 + 128 + /** 129 + * devm_request_any_context_irq - allocate an interrupt line for a managed device with error logging 130 + * @dev: Device to request interrupt for 131 + * @irq: Interrupt line to allocate 132 + * @handler: Function to be called when the interrupt occurs 133 + * @irqflags: Interrupt type flags 134 + * @devname: An ascii name for the claiming device, dev_name(dev) if NULL 135 + * @dev_id: A cookie passed back to the handler function 136 + * 137 + * Except for the extra @dev argument, this function takes the same 138 + * arguments and performs the same function as request_any_context_irq(). 139 + * Interrupts requested with this function will be automatically freed on 140 + * driver detach. 141 + * 142 + * If an interrupt allocated with this function needs to be freed 143 + * separately, devm_free_irq() must be used. 144 + * 145 + * When the request fails, an error message is printed with contextual 146 + * information (device name, interrupt number, handler functions and 147 + * error code). Don't add extra error messages at the call sites. 148 + * 149 + * Return: IRQC_IS_HARDIRQ or IRQC_IS_NESTED on success, or a negative error 150 + * number. 151 + */ 152 + int devm_request_any_context_irq(struct device *dev, unsigned int irq, 153 + irq_handler_t handler, unsigned long irqflags, 154 + const char *devname, void *dev_id) 155 + { 156 + int rc = __devm_request_any_context_irq(dev, irq, handler, irqflags, 157 + devname, dev_id); 158 + 159 + return devm_request_result(dev, rc, irq, handler, NULL, devname); 139 160 } 140 161 EXPORT_SYMBOL(devm_request_any_context_irq); 141 162
+48 -1
kernel/irq/handle.c
··· 136 136 wake_up_process(action->thread); 137 137 } 138 138 139 + static DEFINE_STATIC_KEY_FALSE(irqhandler_duration_check_enabled); 140 + static u64 irqhandler_duration_threshold_ns __ro_after_init; 141 + 142 + static int __init irqhandler_duration_check_setup(char *arg) 143 + { 144 + unsigned long val; 145 + int ret; 146 + 147 + ret = kstrtoul(arg, 0, &val); 148 + if (ret) { 149 + pr_err("Unable to parse irqhandler.duration_warn_us setting: ret=%d\n", ret); 150 + return 0; 151 + } 152 + 153 + if (!val) { 154 + pr_err("Invalid irqhandler.duration_warn_us setting, must be > 0\n"); 155 + return 0; 156 + } 157 + 158 + irqhandler_duration_threshold_ns = val * 1000; 159 + static_branch_enable(&irqhandler_duration_check_enabled); 160 + 161 + return 1; 162 + } 163 + __setup("irqhandler.duration_warn_us=", irqhandler_duration_check_setup); 164 + 165 + static inline void irqhandler_duration_check(u64 ts_start, unsigned int irq, 166 + const struct irqaction *action) 167 + { 168 + u64 delta_ns = local_clock() - ts_start; 169 + 170 + if (unlikely(delta_ns > irqhandler_duration_threshold_ns)) { 171 + pr_warn_ratelimited("[CPU%u] long duration of IRQ[%u:%ps], took: %llu us\n", 172 + smp_processor_id(), irq, action->handler, 173 + div_u64(delta_ns, NSEC_PER_USEC)); 174 + } 175 + } 176 + 139 177 irqreturn_t __handle_irq_event_percpu(struct irq_desc *desc) 140 178 { 141 179 irqreturn_t retval = IRQ_NONE; ··· 193 155 lockdep_hardirq_threaded(); 194 156 195 157 trace_irq_handler_entry(irq, action); 196 - res = action->handler(irq, action->dev_id); 158 + 159 + if (static_branch_unlikely(&irqhandler_duration_check_enabled)) { 160 + u64 ts_start = local_clock(); 161 + 162 + res = action->handler(irq, action->dev_id); 163 + irqhandler_duration_check(ts_start, irq, action); 164 + } else { 165 + res = action->handler(irq, action->dev_id); 166 + } 167 + 197 168 trace_irq_handler_exit(irq, action, res); 198 169 199 170 if (WARN_ONCE(!irqs_disabled(),"irq %u handler %pS enabled interrupts\n",
+30 -25
kernel/irq/irq_test.c
··· 41 41 .flags = IRQCHIP_SKIP_SET_WAKE, 42 42 }; 43 43 44 + static int irq_test_setup_fake_irq(struct kunit *test, struct irq_affinity_desc *affd) 45 + { 46 + struct irq_desc *desc; 47 + int virq; 48 + 49 + virq = irq_domain_alloc_descs(-1, 1, 0, NUMA_NO_NODE, affd); 50 + KUNIT_ASSERT_GE(test, virq, 0); 51 + 52 + irq_set_chip_and_handler(virq, &fake_irq_chip, handle_simple_irq); 53 + 54 + desc = irq_to_desc(virq); 55 + KUNIT_ASSERT_PTR_NE(test, desc, NULL); 56 + 57 + /* On some architectures, IRQs are NOREQUEST | NOPROBE by default. */ 58 + irq_settings_clr_norequest(desc); 59 + 60 + return virq; 61 + } 62 + 44 63 static void irq_disable_depth_test(struct kunit *test) 45 64 { 46 65 struct irq_desc *desc; 47 66 int virq, ret; 48 67 49 - virq = irq_domain_alloc_descs(-1, 1, 0, NUMA_NO_NODE, NULL); 50 - KUNIT_ASSERT_GE(test, virq, 0); 51 - 52 - irq_set_chip_and_handler(virq, &dummy_irq_chip, handle_simple_irq); 68 + virq = irq_test_setup_fake_irq(test, NULL); 53 69 54 70 desc = irq_to_desc(virq); 55 71 KUNIT_ASSERT_PTR_NE(test, desc, NULL); 56 72 57 73 ret = request_irq(virq, noop_handler, 0, "test_irq", NULL); 58 - KUNIT_EXPECT_EQ(test, ret, 0); 74 + KUNIT_ASSERT_EQ(test, ret, 0); 59 75 60 76 KUNIT_EXPECT_EQ(test, desc->depth, 0); 61 77 ··· 89 73 struct irq_desc *desc; 90 74 int virq, ret; 91 75 92 - virq = irq_domain_alloc_descs(-1, 1, 0, NUMA_NO_NODE, NULL); 93 - KUNIT_ASSERT_GE(test, virq, 0); 94 - 95 - irq_set_chip_and_handler(virq, &dummy_irq_chip, handle_simple_irq); 76 + virq = irq_test_setup_fake_irq(test, NULL); 96 77 97 78 desc = irq_to_desc(virq); 98 79 KUNIT_ASSERT_PTR_NE(test, desc, NULL); 99 80 100 81 ret = request_irq(virq, noop_handler, 0, "test_irq", NULL); 101 - KUNIT_EXPECT_EQ(test, ret, 0); 82 + KUNIT_ASSERT_EQ(test, ret, 0); 102 83 103 84 KUNIT_EXPECT_EQ(test, desc->depth, 0); 104 85 ··· 106 93 KUNIT_EXPECT_GE(test, desc->depth, 1); 107 94 108 95 ret = request_irq(virq, noop_handler, 0, "test_irq", NULL); 109 - KUNIT_EXPECT_EQ(test, ret, 0); 96 + KUNIT_ASSERT_EQ(test, ret, 0); 110 97 KUNIT_EXPECT_EQ(test, desc->depth, 0); 111 98 112 99 free_irq(virq, NULL); ··· 125 112 if (!IS_ENABLED(CONFIG_SMP)) 126 113 kunit_skip(test, "requires CONFIG_SMP for managed shutdown"); 127 114 128 - virq = irq_domain_alloc_descs(-1, 1, 0, NUMA_NO_NODE, &affinity); 129 - KUNIT_ASSERT_GE(test, virq, 0); 130 - 131 - irq_set_chip_and_handler(virq, &dummy_irq_chip, handle_simple_irq); 115 + virq = irq_test_setup_fake_irq(test, &affinity); 132 116 133 117 desc = irq_to_desc(virq); 134 118 KUNIT_ASSERT_PTR_NE(test, desc, NULL); ··· 134 124 KUNIT_ASSERT_PTR_NE(test, data, NULL); 135 125 136 126 ret = request_irq(virq, noop_handler, 0, "test_irq", NULL); 137 - KUNIT_EXPECT_EQ(test, ret, 0); 127 + KUNIT_ASSERT_EQ(test, ret, 0); 138 128 139 129 KUNIT_EXPECT_TRUE(test, irqd_is_activated(data)); 140 130 KUNIT_EXPECT_TRUE(test, irqd_is_started(data)); ··· 179 169 kunit_skip(test, "requires more than 1 CPU for CPU hotplug"); 180 170 if (!cpu_is_hotpluggable(1)) 181 171 kunit_skip(test, "CPU 1 must be hotpluggable"); 172 + if (!cpu_online(1)) 173 + kunit_skip(test, "CPU 1 must be online"); 182 174 183 175 cpumask_copy(&affinity.mask, cpumask_of(1)); 184 176 185 - virq = irq_domain_alloc_descs(-1, 1, 0, NUMA_NO_NODE, &affinity); 186 - KUNIT_ASSERT_GE(test, virq, 0); 187 - 188 - irq_set_chip_and_handler(virq, &fake_irq_chip, handle_simple_irq); 177 + virq = irq_test_setup_fake_irq(test, &affinity); 189 178 190 179 desc = irq_to_desc(virq); 191 180 KUNIT_ASSERT_PTR_NE(test, desc, NULL); ··· 193 184 KUNIT_ASSERT_PTR_NE(test, data, NULL); 194 185 195 186 ret = request_irq(virq, noop_handler, 0, "test_irq", NULL); 196 - KUNIT_EXPECT_EQ(test, ret, 0); 187 + KUNIT_ASSERT_EQ(test, ret, 0); 197 188 198 189 KUNIT_EXPECT_TRUE(test, irqd_is_activated(data)); 199 190 KUNIT_EXPECT_TRUE(test, irqd_is_started(data)); ··· 205 196 KUNIT_EXPECT_EQ(test, desc->depth, 1); 206 197 207 198 KUNIT_EXPECT_EQ(test, remove_cpu(1), 0); 208 - KUNIT_EXPECT_FALSE(test, irqd_is_activated(data)); 209 - KUNIT_EXPECT_FALSE(test, irqd_is_started(data)); 210 199 KUNIT_EXPECT_GE(test, desc->depth, 1); 211 200 KUNIT_EXPECT_EQ(test, add_cpu(1), 0); 212 201 213 - KUNIT_EXPECT_FALSE(test, irqd_is_activated(data)); 214 - KUNIT_EXPECT_FALSE(test, irqd_is_started(data)); 215 202 KUNIT_EXPECT_EQ(test, desc->depth, 1); 216 203 217 204 enable_irq(virq);
-7
kernel/irq/irqdesc.c
··· 653 653 irq_insert_desc(irq, irq_desc + irq); 654 654 } 655 655 656 - #ifdef CONFIG_GENERIC_IRQ_LEGACY 657 - void irq_init_desc(unsigned int irq) 658 - { 659 - free_desc(irq); 660 - } 661 - #endif 662 - 663 656 #endif /* !CONFIG_SPARSE_IRQ */ 664 657 665 658 int handle_irq_desc(struct irq_desc *desc)
+120 -25
kernel/softirq.c
··· 165 165 /* First entry of a task into a BH disabled section? */ 166 166 if (!current->softirq_disable_cnt) { 167 167 if (preemptible()) { 168 - local_lock(&softirq_ctrl.lock); 168 + if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK)) 169 + local_lock(&softirq_ctrl.lock); 170 + else 171 + migrate_disable(); 172 + 169 173 /* Required to meet the RCU bottomhalf requirements. */ 170 174 rcu_read_lock(); 171 175 } else { ··· 181 177 * Track the per CPU softirq disabled state. On RT this is per CPU 182 178 * state to allow preemption of bottom half disabled sections. 183 179 */ 184 - newcnt = __this_cpu_add_return(softirq_ctrl.cnt, cnt); 185 - /* 186 - * Reflect the result in the task state to prevent recursion on the 187 - * local lock and to make softirq_count() & al work. 188 - */ 189 - current->softirq_disable_cnt = newcnt; 180 + if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK)) { 181 + newcnt = this_cpu_add_return(softirq_ctrl.cnt, cnt); 182 + /* 183 + * Reflect the result in the task state to prevent recursion on the 184 + * local lock and to make softirq_count() & al work. 185 + */ 186 + current->softirq_disable_cnt = newcnt; 190 187 191 - if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && newcnt == cnt) { 192 - raw_local_irq_save(flags); 193 - lockdep_softirqs_off(ip); 194 - raw_local_irq_restore(flags); 188 + if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && newcnt == cnt) { 189 + raw_local_irq_save(flags); 190 + lockdep_softirqs_off(ip); 191 + raw_local_irq_restore(flags); 192 + } 193 + } else { 194 + bool sirq_dis = false; 195 + 196 + if (!current->softirq_disable_cnt) 197 + sirq_dis = true; 198 + 199 + this_cpu_add(softirq_ctrl.cnt, cnt); 200 + current->softirq_disable_cnt += cnt; 201 + WARN_ON_ONCE(current->softirq_disable_cnt < 0); 202 + 203 + if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && sirq_dis) { 204 + raw_local_irq_save(flags); 205 + lockdep_softirqs_off(ip); 206 + raw_local_irq_restore(flags); 207 + } 195 208 } 196 209 } 197 210 EXPORT_SYMBOL(__local_bh_disable_ip); ··· 216 195 static void __local_bh_enable(unsigned int cnt, bool unlock) 217 196 { 218 197 unsigned long flags; 198 + bool sirq_en = false; 219 199 int newcnt; 220 200 221 - DEBUG_LOCKS_WARN_ON(current->softirq_disable_cnt != 222 - this_cpu_read(softirq_ctrl.cnt)); 201 + if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK)) { 202 + DEBUG_LOCKS_WARN_ON(current->softirq_disable_cnt != 203 + this_cpu_read(softirq_ctrl.cnt)); 204 + if (softirq_count() == cnt) 205 + sirq_en = true; 206 + } else { 207 + if (current->softirq_disable_cnt == cnt) 208 + sirq_en = true; 209 + } 223 210 224 - if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && softirq_count() == cnt) { 211 + if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && sirq_en) { 225 212 raw_local_irq_save(flags); 226 213 lockdep_softirqs_on(_RET_IP_); 227 214 raw_local_irq_restore(flags); 228 215 } 229 216 230 - newcnt = __this_cpu_sub_return(softirq_ctrl.cnt, cnt); 231 - current->softirq_disable_cnt = newcnt; 217 + if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK)) { 218 + newcnt = this_cpu_sub_return(softirq_ctrl.cnt, cnt); 219 + current->softirq_disable_cnt = newcnt; 232 220 233 - if (!newcnt && unlock) { 234 - rcu_read_unlock(); 235 - local_unlock(&softirq_ctrl.lock); 221 + if (!newcnt && unlock) { 222 + rcu_read_unlock(); 223 + local_unlock(&softirq_ctrl.lock); 224 + } 225 + } else { 226 + current->softirq_disable_cnt -= cnt; 227 + this_cpu_sub(softirq_ctrl.cnt, cnt); 228 + if (unlock && !current->softirq_disable_cnt) { 229 + migrate_enable(); 230 + rcu_read_unlock(); 231 + } else { 232 + WARN_ON_ONCE(current->softirq_disable_cnt < 0); 233 + } 236 234 } 237 235 } 238 236 ··· 268 228 lock_map_release(&bh_lock_map); 269 229 270 230 local_irq_save(flags); 271 - curcnt = __this_cpu_read(softirq_ctrl.cnt); 231 + if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK)) 232 + curcnt = this_cpu_read(softirq_ctrl.cnt); 233 + else 234 + curcnt = current->softirq_disable_cnt; 272 235 273 236 /* 274 237 * If this is not reenabling soft interrupts, no point in trying to ··· 848 805 return false; 849 806 } 850 807 808 + #ifdef CONFIG_PREEMPT_RT 809 + struct tasklet_sync_callback { 810 + spinlock_t cb_lock; 811 + atomic_t cb_waiters; 812 + }; 813 + 814 + static DEFINE_PER_CPU(struct tasklet_sync_callback, tasklet_sync_callback) = { 815 + .cb_lock = __SPIN_LOCK_UNLOCKED(tasklet_sync_callback.cb_lock), 816 + .cb_waiters = ATOMIC_INIT(0), 817 + }; 818 + 819 + static void tasklet_lock_callback(void) 820 + { 821 + spin_lock(this_cpu_ptr(&tasklet_sync_callback.cb_lock)); 822 + } 823 + 824 + static void tasklet_unlock_callback(void) 825 + { 826 + spin_unlock(this_cpu_ptr(&tasklet_sync_callback.cb_lock)); 827 + } 828 + 829 + static void tasklet_callback_cancel_wait_running(void) 830 + { 831 + struct tasklet_sync_callback *sync_cb = this_cpu_ptr(&tasklet_sync_callback); 832 + 833 + atomic_inc(&sync_cb->cb_waiters); 834 + spin_lock(&sync_cb->cb_lock); 835 + atomic_dec(&sync_cb->cb_waiters); 836 + spin_unlock(&sync_cb->cb_lock); 837 + } 838 + 839 + static void tasklet_callback_sync_wait_running(void) 840 + { 841 + struct tasklet_sync_callback *sync_cb = this_cpu_ptr(&tasklet_sync_callback); 842 + 843 + if (atomic_read(&sync_cb->cb_waiters)) { 844 + spin_unlock(&sync_cb->cb_lock); 845 + spin_lock(&sync_cb->cb_lock); 846 + } 847 + } 848 + 849 + #else /* !CONFIG_PREEMPT_RT: */ 850 + 851 + static void tasklet_lock_callback(void) { } 852 + static void tasklet_unlock_callback(void) { } 853 + static void tasklet_callback_sync_wait_running(void) { } 854 + 855 + #ifdef CONFIG_SMP 856 + static void tasklet_callback_cancel_wait_running(void) { } 857 + #endif 858 + #endif /* !CONFIG_PREEMPT_RT */ 859 + 851 860 static void tasklet_action_common(struct tasklet_head *tl_head, 852 861 unsigned int softirq_nr) 853 862 { ··· 911 816 tl_head->tail = &tl_head->head; 912 817 local_irq_enable(); 913 818 819 + tasklet_lock_callback(); 914 820 while (list) { 915 821 struct tasklet_struct *t = list; 916 822 ··· 931 835 } 932 836 } 933 837 tasklet_unlock(t); 838 + tasklet_callback_sync_wait_running(); 934 839 continue; 935 840 } 936 841 tasklet_unlock(t); ··· 944 847 __raise_softirq_irqoff(softirq_nr); 945 848 local_irq_enable(); 946 849 } 850 + tasklet_unlock_callback(); 947 851 } 948 852 949 853 static __latent_entropy void tasklet_action(void) ··· 995 897 /* 996 898 * Prevent a live lock when current preempted soft 997 899 * interrupt processing or prevents ksoftirqd from 998 - * running. If the tasklet runs on a different CPU 999 - * then this has no effect other than doing the BH 1000 - * disable/enable dance for nothing. 900 + * running. 1001 901 */ 1002 - local_bh_disable(); 1003 - local_bh_enable(); 902 + tasklet_callback_cancel_wait_running(); 1004 903 } else { 1005 904 cpu_relax(); 1006 905 }