Merge tag 'smp-urgent-2021-09-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

+459 -78

Documentation/core-api/cpu_hotplug.rst

··· 2 2 CPU hotplug in the Kernel 3 3 ========================= 4 4 5 - :Date: December, 2016 5 + :Date: September, 2021 6 6 :Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>, 7 - Rusty Russell <rusty@rustcorp.com.au>, 8 - Srivatsa Vaddagiri <vatsa@in.ibm.com>, 9 - Ashok Raj <ashok.raj@intel.com>, 10 - Joel Schopp <jschopp@austin.ibm.com> 7 + Rusty Russell <rusty@rustcorp.com.au>, 8 + Srivatsa Vaddagiri <vatsa@in.ibm.com>, 9 + Ashok Raj <ashok.raj@intel.com>, 10 + Joel Schopp <jschopp@austin.ibm.com>, 11 + Thomas Gleixner <tglx@linutronix.de> 11 12 12 13 Introduction 13 14 ============ ··· 159 158 * Once all services are migrated, kernel calls an arch specific routine 160 159 ``__cpu_disable()`` to perform arch specific cleanup. 161 160 162 - Using the hotplug API 163 - --------------------- 164 161 165 - It is possible to receive notifications once a CPU is offline or onlined. This 166 - might be important to certain drivers which need to perform some kind of setup 167 - or clean up functions based on the number of available CPUs:: 162 + The CPU hotplug API 163 + =================== 168 164 169 - #include <linux/cpuhotplug.h> 165 + CPU hotplug state machine 166 + ------------------------- 170 167 171 - ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "X/Y:online", 172 - Y_online, Y_prepare_down); 168 + CPU hotplug uses a trivial state machine with a linear state space from 169 + CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown 170 + callback. 173 171 174 - *X* is the subsystem and *Y* the particular driver. The *Y_online* callback 175 - will be invoked during registration on all online CPUs. If an error 176 - occurs during the online callback the *Y_prepare_down* callback will be 177 - invoked on all CPUs on which the online callback was previously invoked. 178 - After registration completed, the *Y_online* callback will be invoked 179 - once a CPU is brought online and *Y_prepare_down* will be invoked when a 180 - CPU is shutdown. All resources which were previously allocated in 181 - *Y_online* should be released in *Y_prepare_down*. 182 - The return value *ret* is negative if an error occurred during the 183 - registration process. Otherwise a positive value is returned which 184 - contains the allocated hotplug for dynamically allocated states 185 - (*CPUHP_AP_ONLINE_DYN*). It will return zero for predefined states. 172 + When a CPU is onlined, the startup callbacks are invoked sequentially until 173 + the state CPUHP_ONLINE is reached. They can also be invoked when the 174 + callbacks of a state are set up or an instance is added to a multi-instance 175 + state. 186 176 187 - The callback can be remove by invoking ``cpuhp_remove_state()``. In case of a 188 - dynamically allocated state (*CPUHP_AP_ONLINE_DYN*) use the returned state. 189 - During the removal of a hotplug state the teardown callback will be invoked. 177 + When a CPU is offlined the teardown callbacks are invoked in the reverse 178 + order sequentially until the state CPUHP_OFFLINE is reached. They can also 179 + be invoked when the callbacks of a state are removed or an instance is 180 + removed from a multi-instance state. 190 181 191 - Multiple instances 192 - ~~~~~~~~~~~~~~~~~~ 182 + If a usage site requires only a callback in one direction of the hotplug 183 + operations (CPU online or CPU offline) then the other not-required callback 184 + can be set to NULL when the state is set up. 193 185 194 - If a driver has multiple instances and each instance needs to perform the 195 - callback independently then it is likely that a ''multi-state'' should be used. 196 - First a multi-state state needs to be registered:: 186 + The state space is divided into three sections: 197 187 198 - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "X/Y:online, 199 - Y_online, Y_prepare_down); 200 - Y_hp_online = ret; 188 + * The PREPARE section 201 189 202 - The ``cpuhp_setup_state_multi()`` behaves similar to ``cpuhp_setup_state()`` 203 - except it prepares the callbacks for a multi state and does not invoke 204 - the callbacks. This is a one time setup. 205 - Once a new instance is allocated, you need to register this new instance:: 190 + The PREPARE section covers the state space from CPUHP_OFFLINE to 191 + CPUHP_BRINGUP_CPU. 206 192 207 - ret = cpuhp_state_add_instance(Y_hp_online, &d->node); 193 + The startup callbacks in this section are invoked before the CPU is 194 + started during a CPU online operation. The teardown callbacks are invoked 195 + after the CPU has become dysfunctional during a CPU offline operation. 208 196 209 - This function will add this instance to your previously allocated 210 - *Y_hp_online* state and invoke the previously registered callback 211 - (*Y_online*) on all online CPUs. The *node* element is a ``struct 212 - hlist_node`` member of your per-instance data structure. 197 + The callbacks are invoked on a control CPU as they can't obviously run on 198 + the hotplugged CPU which is either not yet started or has become 199 + dysfunctional already. 213 200 214 - On removal of the instance:: 201 + The startup callbacks are used to setup resources which are required to 202 + bring a CPU successfully online. The teardown callbacks are used to free 203 + resources or to move pending work to an online CPU after the hotplugged 204 + CPU became dysfunctional. 215 205 216 - cpuhp_state_remove_instance(Y_hp_online, &d->node) 206 + The startup callbacks are allowed to fail. If a callback fails, the CPU 207 + online operation is aborted and the CPU is brought down to the previous 208 + state (usually CPUHP_OFFLINE) again. 217 209 218 - should be invoked which will invoke the teardown callback on all online 219 - CPUs. 210 + The teardown callbacks in this section are not allowed to fail. 220 211 221 - Manual setup 222 - ~~~~~~~~~~~~ 212 + * The STARTING section 223 213 224 - Usually it is handy to invoke setup and teardown callbacks on registration or 225 - removal of a state because usually the operation needs to performed once a CPU 226 - goes online (offline) and during initial setup (shutdown) of the driver. However 227 - each registration and removal function is also available with a ``_nocalls`` 228 - suffix which does not invoke the provided callbacks if the invocation of the 229 - callbacks is not desired. During the manual setup (or teardown) the functions 230 - ``cpus_read_lock()`` and ``cpus_read_unlock()`` should be used to inhibit CPU 231 - hotplug operations. 214 + The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1 215 + and CPUHP_AP_ONLINE. 232 216 217 + The startup callbacks in this section are invoked on the hotplugged CPU 218 + with interrupts disabled during a CPU online operation in the early CPU 219 + setup code. The teardown callbacks are invoked with interrupts disabled 220 + on the hotplugged CPU during a CPU offline operation shortly before the 221 + CPU is completely shut down. 233 222 234 - The ordering of the events 235 - -------------------------- 223 + The callbacks in this section are not allowed to fail. 236 224 237 - The hotplug states are defined in ``include/linux/cpuhotplug.h``: 225 + The callbacks are used for low level hardware initialization/shutdown and 226 + for core subsystems. 238 227 239 - * The states *CPUHP_OFFLINE* … *CPUHP_AP_OFFLINE* are invoked before the 240 - CPU is up. 241 - * The states *CPUHP_AP_OFFLINE* … *CPUHP_AP_ONLINE* are invoked 242 - just the after the CPU has been brought up. The interrupts are off and 243 - the scheduler is not yet active on this CPU. Starting with *CPUHP_AP_OFFLINE* 244 - the callbacks are invoked on the target CPU. 245 - * The states between *CPUHP_AP_ONLINE_DYN* and *CPUHP_AP_ONLINE_DYN_END* are 246 - reserved for the dynamic allocation. 247 - * The states are invoked in the reverse order on CPU shutdown starting with 248 - *CPUHP_ONLINE* and stopping at *CPUHP_OFFLINE*. Here the callbacks are 249 - invoked on the CPU that will be shutdown until *CPUHP_AP_OFFLINE*. 228 + * The ONLINE section 250 229 251 - A dynamically allocated state via *CPUHP_AP_ONLINE_DYN* is often enough. 252 - However if an earlier invocation during the bring up or shutdown is required 253 - then an explicit state should be acquired. An explicit state might also be 254 - required if the hotplug event requires specific ordering in respect to 255 - another hotplug event. 230 + The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and 231 + CPUHP_ONLINE. 232 + 233 + The startup callbacks in this section are invoked on the hotplugged CPU 234 + during a CPU online operation. The teardown callbacks are invoked on the 235 + hotplugged CPU during a CPU offline operation. 236 + 237 + The callbacks are invoked in the context of the per CPU hotplug thread, 238 + which is pinned on the hotplugged CPU. The callbacks are invoked with 239 + interrupts and preemption enabled. 240 + 241 + The callbacks are allowed to fail. When a callback fails the hotplug 242 + operation is aborted and the CPU is brought back to the previous state. 243 + 244 + CPU online/offline operations 245 + ----------------------------- 246 + 247 + A successful online operation looks like this:: 248 + 249 + [CPUHP_OFFLINE] 250 + [CPUHP_OFFLINE + 1]->startup() -> success 251 + [CPUHP_OFFLINE + 2]->startup() -> success 252 + [CPUHP_OFFLINE + 3] -> skipped because startup == NULL 253 + ... 254 + [CPUHP_BRINGUP_CPU]->startup() -> success 255 + === End of PREPARE section 256 + [CPUHP_BRINGUP_CPU + 1]->startup() -> success 257 + ... 258 + [CPUHP_AP_ONLINE]->startup() -> success 259 + === End of STARTUP section 260 + [CPUHP_AP_ONLINE + 1]->startup() -> success 261 + ... 262 + [CPUHP_ONLINE - 1]->startup() -> success 263 + [CPUHP_ONLINE] 264 + 265 + A successful offline operation looks like this:: 266 + 267 + [CPUHP_ONLINE] 268 + [CPUHP_ONLINE - 1]->teardown() -> success 269 + ... 270 + [CPUHP_AP_ONLINE + 1]->teardown() -> success 271 + === Start of STARTUP section 272 + [CPUHP_AP_ONLINE]->teardown() -> success 273 + ... 274 + [CPUHP_BRINGUP_ONLINE - 1]->teardown() 275 + ... 276 + === Start of PREPARE section 277 + [CPUHP_BRINGUP_CPU]->teardown() 278 + [CPUHP_OFFLINE + 3]->teardown() 279 + [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL 280 + [CPUHP_OFFLINE + 1]->teardown() 281 + [CPUHP_OFFLINE] 282 + 283 + A failed online operation looks like this:: 284 + 285 + [CPUHP_OFFLINE] 286 + [CPUHP_OFFLINE + 1]->startup() -> success 287 + [CPUHP_OFFLINE + 2]->startup() -> success 288 + [CPUHP_OFFLINE + 3] -> skipped because startup == NULL 289 + ... 290 + [CPUHP_BRINGUP_CPU]->startup() -> success 291 + === End of PREPARE section 292 + [CPUHP_BRINGUP_CPU + 1]->startup() -> success 293 + ... 294 + [CPUHP_AP_ONLINE]->startup() -> success 295 + === End of STARTUP section 296 + [CPUHP_AP_ONLINE + 1]->startup() -> success 297 + --- 298 + [CPUHP_AP_ONLINE + N]->startup() -> fail 299 + [CPUHP_AP_ONLINE + (N - 1)]->teardown() 300 + ... 301 + [CPUHP_AP_ONLINE + 1]->teardown() 302 + === Start of STARTUP section 303 + [CPUHP_AP_ONLINE]->teardown() 304 + ... 305 + [CPUHP_BRINGUP_ONLINE - 1]->teardown() 306 + ... 307 + === Start of PREPARE section 308 + [CPUHP_BRINGUP_CPU]->teardown() 309 + [CPUHP_OFFLINE + 3]->teardown() 310 + [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL 311 + [CPUHP_OFFLINE + 1]->teardown() 312 + [CPUHP_OFFLINE] 313 + 314 + A failed offline operation looks like this:: 315 + 316 + [CPUHP_ONLINE] 317 + [CPUHP_ONLINE - 1]->teardown() -> success 318 + ... 319 + [CPUHP_ONLINE - N]->teardown() -> fail 320 + [CPUHP_ONLINE - (N - 1)]->startup() 321 + ... 322 + [CPUHP_ONLINE - 1]->startup() 323 + [CPUHP_ONLINE] 324 + 325 + Recursive failures cannot be handled sensibly. Look at the following 326 + example of a recursive fail due to a failed offline operation: :: 327 + 328 + [CPUHP_ONLINE] 329 + [CPUHP_ONLINE - 1]->teardown() -> success 330 + ... 331 + [CPUHP_ONLINE - N]->teardown() -> fail 332 + [CPUHP_ONLINE - (N - 1)]->startup() -> success 333 + [CPUHP_ONLINE - (N - 2)]->startup() -> fail 334 + 335 + The CPU hotplug state machine stops right here and does not try to go back 336 + down again because that would likely result in an endless loop:: 337 + 338 + [CPUHP_ONLINE - (N - 1)]->teardown() -> success 339 + [CPUHP_ONLINE - N]->teardown() -> fail 340 + [CPUHP_ONLINE - (N - 1)]->startup() -> success 341 + [CPUHP_ONLINE - (N - 2)]->startup() -> fail 342 + [CPUHP_ONLINE - (N - 1)]->teardown() -> success 343 + [CPUHP_ONLINE - N]->teardown() -> fail 344 + 345 + Lather, rinse and repeat. In this case the CPU left in state:: 346 + 347 + [CPUHP_ONLINE - (N - 1)] 348 + 349 + which at least lets the system make progress and gives the user a chance to 350 + debug or even resolve the situation. 351 + 352 + Allocating a state 353 + ------------------ 354 + 355 + There are two ways to allocate a CPU hotplug state: 356 + 357 + * Static allocation 358 + 359 + Static allocation has to be used when the subsystem or driver has 360 + ordering requirements versus other CPU hotplug states. E.g. the PERF core 361 + startup callback has to be invoked before the PERF driver startup 362 + callbacks during a CPU online operation. During a CPU offline operation 363 + the driver teardown callbacks have to be invoked before the core teardown 364 + callback. The statically allocated states are described by constants in 365 + the cpuhp_state enum which can be found in include/linux/cpuhotplug.h. 366 + 367 + Insert the state into the enum at the proper place so the ordering 368 + requirements are fulfilled. The state constant has to be used for state 369 + setup and removal. 370 + 371 + Static allocation is also required when the state callbacks are not set 372 + up at runtime and are part of the initializer of the CPU hotplug state 373 + array in kernel/cpu.c. 374 + 375 + * Dynamic allocation 376 + 377 + When there are no ordering requirements for the state callbacks then 378 + dynamic allocation is the preferred method. The state number is allocated 379 + by the setup function and returned to the caller on success. 380 + 381 + Only the PREPARE and ONLINE sections provide a dynamic allocation 382 + range. The STARTING section does not as most of the callbacks in that 383 + section have explicit ordering requirements. 384 + 385 + Setup of a CPU hotplug state 386 + ---------------------------- 387 + 388 + The core code provides the following functions to setup a state: 389 + 390 + * cpuhp_setup_state(state, name, startup, teardown) 391 + * cpuhp_setup_state_nocalls(state, name, startup, teardown) 392 + * cpuhp_setup_state_cpuslocked(state, name, startup, teardown) 393 + * cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown) 394 + 395 + For cases where a driver or a subsystem has multiple instances and the same 396 + CPU hotplug state callbacks need to be invoked for each instance, the CPU 397 + hotplug core provides multi-instance support. The advantage over driver 398 + specific instance lists is that the instance related functions are fully 399 + serialized against CPU hotplug operations and provide the automatic 400 + invocations of the state callbacks on add and removal. To set up such a 401 + multi-instance state the following function is available: 402 + 403 + * cpuhp_setup_state_multi(state, name, startup, teardown) 404 + 405 + The @state argument is either a statically allocated state or one of the 406 + constants for dynamically allocated states - CPUHP_PREPARE_DYN, 407 + CPUHP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for 408 + which a dynamic state should be allocated. 409 + 410 + The @name argument is used for sysfs output and for instrumentation. The 411 + naming convention is "subsys:mode" or "subsys/driver:mode", 412 + e.g. "perf:mode" or "perf/x86:mode". The common mode names are: 413 + 414 + ======== ======================================================= 415 + prepare For states in the PREPARE section 416 + 417 + dead For states in the PREPARE section which do not provide 418 + a startup callback 419 + 420 + starting For states in the STARTING section 421 + 422 + dying For states in the STARTING section which do not provide 423 + a startup callback 424 + 425 + online For states in the ONLINE section 426 + 427 + offline For states in the ONLINE section which do not provide 428 + a startup callback 429 + ======== ======================================================= 430 + 431 + As the @name argument is only used for sysfs and instrumentation other mode 432 + descriptors can be used as well if they describe the nature of the state 433 + better than the common ones. 434 + 435 + Examples for @name arguments: "perf/online", "perf/x86:prepare", 436 + "RCU/tree:dying", "sched/waitempty" 437 + 438 + The @startup argument is a function pointer to the callback which should be 439 + invoked during a CPU online operation. If the usage site does not require a 440 + startup callback set the pointer to NULL. 441 + 442 + The @teardown argument is a function pointer to the callback which should 443 + be invoked during a CPU offline operation. If the usage site does not 444 + require a teardown callback set the pointer to NULL. 445 + 446 + The functions differ in the way how the installed callbacks are treated: 447 + 448 + * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked() 449 + and cpuhp_setup_state_multi() only install the callbacks 450 + 451 + * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the 452 + callbacks and invoke the @startup callback (if not NULL) for all online 453 + CPUs which have currently a state greater than the newly installed 454 + state. Depending on the state section the callback is either invoked on 455 + the current CPU (PREPARE section) or on each online CPU (ONLINE 456 + section) in the context of the CPU's hotplug thread. 457 + 458 + If a callback fails for CPU N then the teardown callback for CPU 459 + 0 .. N-1 is invoked to rollback the operation. The state setup fails, 460 + the callbacks for the state are not installed and in case of dynamic 461 + allocation the allocated state is freed. 462 + 463 + The state setup and the callback invocations are serialized against CPU 464 + hotplug operations. If the setup function has to be called from a CPU 465 + hotplug read locked region, then the _cpuslocked() variants have to be 466 + used. These functions cannot be used from within CPU hotplug callbacks. 467 + 468 + The function return values: 469 + ======== =================================================================== 470 + 0 Statically allocated state was successfully set up 471 + 472 + >0 Dynamically allocated state was successfully set up. 473 + 474 + The returned number is the state number which was allocated. If 475 + the state callbacks have to be removed later, e.g. module 476 + removal, then this number has to be saved by the caller and used 477 + as @state argument for the state remove function. For 478 + multi-instance states the dynamically allocated state number is 479 + also required as @state argument for the instance add/remove 480 + operations. 481 + 482 + <0 Operation failed 483 + ======== =================================================================== 484 + 485 + Removal of a CPU hotplug state 486 + ------------------------------ 487 + 488 + To remove a previously set up state, the following functions are provided: 489 + 490 + * cpuhp_remove_state(state) 491 + * cpuhp_remove_state_nocalls(state) 492 + * cpuhp_remove_state_nocalls_cpuslocked(state) 493 + * cpuhp_remove_multi_state(state) 494 + 495 + The @state argument is either a statically allocated state or the state 496 + number which was allocated in the dynamic range by cpuhp_setup_state*(). If 497 + the state is in the dynamic range, then the state number is freed and 498 + available for dynamic allocation again. 499 + 500 + The functions differ in the way how the installed callbacks are treated: 501 + 502 + * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked() 503 + and cpuhp_remove_multi_state() only remove the callbacks. 504 + 505 + * cpuhp_remove_state() removes the callbacks and invokes the teardown 506 + callback (if not NULL) for all online CPUs which have currently a state 507 + greater than the removed state. Depending on the state section the 508 + callback is either invoked on the current CPU (PREPARE section) or on 509 + each online CPU (ONLINE section) in the context of the CPU's hotplug 510 + thread. 511 + 512 + In order to complete the removal, the teardown callback should not fail. 513 + 514 + The state removal and the callback invocations are serialized against CPU 515 + hotplug operations. If the remove function has to be called from a CPU 516 + hotplug read locked region, then the _cpuslocked() variants have to be 517 + used. These functions cannot be used from within CPU hotplug callbacks. 518 + 519 + If a multi-instance state is removed then the caller has to remove all 520 + instances first. 521 + 522 + Multi-Instance state instance management 523 + ---------------------------------------- 524 + 525 + Once the multi-instance state is set up, instances can be added to the 526 + state: 527 + 528 + * cpuhp_state_add_instance(state, node) 529 + * cpuhp_state_add_instance_nocalls(state, node) 530 + 531 + The @state argument is either a statically allocated state or the state 532 + number which was allocated in the dynamic range by cpuhp_setup_state_multi(). 533 + 534 + The @node argument is a pointer to an hlist_node which is embedded in the 535 + instance's data structure. The pointer is handed to the multi-instance 536 + state callbacks and can be used by the callback to retrieve the instance 537 + via container_of(). 538 + 539 + The functions differ in the way how the installed callbacks are treated: 540 + 541 + * cpuhp_state_add_instance_nocalls() and only adds the instance to the 542 + multi-instance state's node list. 543 + 544 + * cpuhp_state_add_instance() adds the instance and invokes the startup 545 + callback (if not NULL) associated with @state for all online CPUs which 546 + have currently a state greater than @state. The callback is only 547 + invoked for the to be added instance. Depending on the state section 548 + the callback is either invoked on the current CPU (PREPARE section) or 549 + on each online CPU (ONLINE section) in the context of the CPU's hotplug 550 + thread. 551 + 552 + If a callback fails for CPU N then the teardown callback for CPU 553 + 0 .. N-1 is invoked to rollback the operation, the function fails and 554 + the instance is not added to the node list of the multi-instance state. 555 + 556 + To remove an instance from the state's node list these functions are 557 + available: 558 + 559 + * cpuhp_state_remove_instance(state, node) 560 + * cpuhp_state_remove_instance_nocalls(state, node) 561 + 562 + The arguments are the same as for the the cpuhp_state_add_instance*() 563 + variants above. 564 + 565 + The functions differ in the way how the installed callbacks are treated: 566 + 567 + * cpuhp_state_remove_instance_nocalls() only removes the instance from the 568 + state's node list. 569 + 570 + * cpuhp_state_remove_instance() removes the instance and invokes the 571 + teardown callback (if not NULL) associated with @state for all online 572 + CPUs which have currently a state greater than @state. The callback is 573 + only invoked for the to be removed instance. Depending on the state 574 + section the callback is either invoked on the current CPU (PREPARE 575 + section) or on each online CPU (ONLINE section) in the context of the 576 + CPU's hotplug thread. 577 + 578 + In order to complete the removal, the teardown callback should not fail. 579 + 580 + The node list add/remove operations and the callback invocations are 581 + serialized against CPU hotplug operations. These functions cannot be used 582 + from within CPU hotplug callbacks and CPU hotplug read locked regions. 583 + 584 + Examples 585 + -------- 586 + 587 + Setup and teardown a statically allocated state in the STARTING section for 588 + notifications on online and offline operations:: 589 + 590 + ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying); 591 + if (ret < 0) 592 + return ret; 593 + .... 594 + cpuhp_remove_state(CPUHP_SUBSYS_STARTING); 595 + 596 + Setup and teardown a dynamically allocated state in the ONLINE section 597 + for notifications on offline operations:: 598 + 599 + state = cpuhp_setup_state(CPUHP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline); 600 + if (state < 0) 601 + return state; 602 + .... 603 + cpuhp_remove_state(state); 604 + 605 + Setup and teardown a dynamically allocated state in the ONLINE section 606 + for notifications on online operations without invoking the callbacks:: 607 + 608 + state = cpuhp_setup_state_nocalls(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL); 609 + if (state < 0) 610 + return state; 611 + .... 612 + cpuhp_remove_state_nocalls(state); 613 + 614 + Setup, use and teardown a dynamically allocated multi-instance state in the 615 + ONLINE section for notifications on online and offline operation:: 616 + 617 + state = cpuhp_setup_state_multi(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline); 618 + if (state < 0) 619 + return state; 620 + .... 621 + ret = cpuhp_state_add_instance(state, &inst1->node); 622 + if (ret) 623 + return ret; 624 + .... 625 + ret = cpuhp_state_add_instance(state, &inst2->node); 626 + if (ret) 627 + return ret; 628 + .... 629 + cpuhp_remove_instance(state, &inst1->node); 630 + .... 631 + cpuhp_remove_instance(state, &inst2->node); 632 + .... 633 + remove_multi_state(state); 634 + 256 635 257 636 Testing of hotplug states 258 637 =========================

+2 -5

arch/arm64/kernel/cacheinfo.c

··· 43 43 this_leaf->type = type; 44 44 } 45 45 46 - static int __init_cache_level(unsigned int cpu) 46 + int init_cache_level(unsigned int cpu) 47 47 { 48 48 unsigned int ctype, level, leaves, fw_level; 49 49 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); ··· 78 78 return 0; 79 79 } 80 80 81 - static int __populate_cache_leaves(unsigned int cpu) 81 + int populate_cache_leaves(unsigned int cpu) 82 82 { 83 83 unsigned int level, idx; 84 84 enum cache_type type; ··· 97 97 } 98 98 return 0; 99 99 } 100 - 101 - DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) 102 - DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves)

+2 -5

arch/mips/kernel/cacheinfo.c

··· 17 17 leaf++; \ 18 18 } while (0) 19 19 20 - static int __init_cache_level(unsigned int cpu) 20 + int init_cache_level(unsigned int cpu) 21 21 { 22 22 struct cpuinfo_mips *c = &current_cpu_data; 23 23 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); ··· 74 74 cpumask_set_cpu(cpu1, cpu_map); 75 75 } 76 76 77 - static int __populate_cache_leaves(unsigned int cpu) 77 + int populate_cache_leaves(unsigned int cpu) 78 78 { 79 79 struct cpuinfo_mips *c = &current_cpu_data; 80 80 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); ··· 114 114 115 115 return 0; 116 116 } 117 - 118 - DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) 119 - DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves)

+2 -5

arch/riscv/kernel/cacheinfo.c

··· 113 113 } 114 114 } 115 115 116 - static int __init_cache_level(unsigned int cpu) 116 + int init_cache_level(unsigned int cpu) 117 117 { 118 118 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); 119 119 struct device_node *np = of_cpu_device_node_get(cpu); ··· 155 155 return 0; 156 156 } 157 157 158 - static int __populate_cache_leaves(unsigned int cpu) 158 + int populate_cache_leaves(unsigned int cpu) 159 159 { 160 160 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); 161 161 struct cacheinfo *this_leaf = this_cpu_ci->info_list; ··· 187 187 188 188 return 0; 189 189 } 190 - 191 - DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) 192 - DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves)

+2 -5

arch/x86/kernel/cpu/cacheinfo.c

··· 985 985 this_leaf->priv = base->nb; 986 986 } 987 987 988 - static int __init_cache_level(unsigned int cpu) 988 + int init_cache_level(unsigned int cpu) 989 989 { 990 990 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); 991 991 ··· 1014 1014 id4_regs->id = c->apicid >> index_msb; 1015 1015 } 1016 1016 1017 - static int __populate_cache_leaves(unsigned int cpu) 1017 + int populate_cache_leaves(unsigned int cpu) 1018 1018 { 1019 1019 unsigned int idx, ret; 1020 1020 struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); ··· 1033 1033 1034 1034 return 0; 1035 1035 } 1036 - 1037 - DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) 1038 - DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves)

-18

include/linux/cacheinfo.h

··· 79 79 bool cpu_map_populated; 80 80 }; 81 81 82 - /* 83 - * Helpers to make sure "func" is executed on the cpu whose cache 84 - * attributes are being detected 85 - */ 86 - #define DEFINE_SMP_CALL_CACHE_FUNCTION(func) \ 87 - static inline void _##func(void *ret) \ 88 - { \ 89 - int cpu = smp_processor_id(); \ 90 - *(int *)ret = __##func(cpu); \ 91 - } \ 92 - \ 93 - int func(unsigned int cpu) \ 94 - { \ 95 - int ret; \ 96 - smp_call_function_single(cpu, _##func, &ret, true); \ 97 - return ret; \ 98 - } 99 - 100 82 struct cpu_cacheinfo *get_cpu_cacheinfo(unsigned int cpu); 101 83 int init_cache_level(unsigned int cpu); 102 84 int populate_cache_leaves(unsigned int cpu);

-6

include/linux/cpu.h

··· 143 143 static inline void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) { } 144 144 #endif /* !CONFIG_HOTPLUG_CPU */ 145 145 146 - /* Wrappers which go away once all code is converted */ 147 - static inline void cpu_hotplug_begin(void) { cpus_write_lock(); } 148 - static inline void cpu_hotplug_done(void) { cpus_write_unlock(); } 149 - static inline void get_online_cpus(void) { cpus_read_lock(); } 150 - static inline void put_online_cpus(void) { cpus_read_unlock(); } 151 - 152 146 #ifdef CONFIG_PM_SLEEP_SMP 153 147 extern int freeze_secondary_cpus(int primary); 154 148 extern void thaw_secondary_cpus(void);

+110 -22

include/linux/cpuhotplug.h

··· 22 22 * AP_ACTIVE AP_ACTIVE 23 23 */ 24 24 25 + /* 26 + * CPU hotplug states. The state machine invokes the installed state 27 + * startup callbacks sequentially from CPUHP_OFFLINE + 1 to CPUHP_ONLINE 28 + * during a CPU online operation. During a CPU offline operation the 29 + * installed teardown callbacks are invoked in the reverse order from 30 + * CPU_ONLINE - 1 down to CPUHP_OFFLINE. 31 + * 32 + * The state space has three sections: PREPARE, STARTING and ONLINE. 33 + * 34 + * PREPARE: The callbacks are invoked on a control CPU before the 35 + * hotplugged CPU is started up or after the hotplugged CPU has died. 36 + * 37 + * STARTING: The callbacks are invoked on the hotplugged CPU from the low level 38 + * hotplug startup/teardown code with interrupts disabled. 39 + * 40 + * ONLINE: The callbacks are invoked on the hotplugged CPU from the per CPU 41 + * hotplug thread with interrupts and preemption enabled. 42 + * 43 + * Adding explicit states to this enum is only necessary when: 44 + * 45 + * 1) The state is within the STARTING section 46 + * 47 + * 2) The state has ordering constraints vs. other states in the 48 + * same section. 49 + * 50 + * If neither #1 nor #2 apply, please use the dynamic state space when 51 + * setting up a state by using CPUHP_PREPARE_DYN or CPUHP_PREPARE_ONLINE 52 + * for the @state argument of the setup function. 53 + * 54 + * See Documentation/core-api/cpu_hotplug.rst for further information and 55 + * examples. 56 + */ 25 57 enum cpuhp_state { 26 58 CPUHP_INVALID = -1, 59 + 60 + /* PREPARE section invoked on a control CPU */ 27 61 CPUHP_OFFLINE = 0, 28 62 CPUHP_CREATE_THREADS, 29 63 CPUHP_PERF_PREPARE, ··· 129 95 CPUHP_BP_PREPARE_DYN, 130 96 CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, 131 97 CPUHP_BRINGUP_CPU, 98 + 99 + /* 100 + * STARTING section invoked on the hotplugged CPU in low level 101 + * bringup and teardown code. 102 + */ 132 103 CPUHP_AP_IDLE_DEAD, 133 104 CPUHP_AP_OFFLINE, 134 105 CPUHP_AP_SCHED_STARTING, ··· 194 155 CPUHP_AP_ARM_CACHE_B15_RAC_DYING, 195 156 CPUHP_AP_ONLINE, 196 157 CPUHP_TEARDOWN_CPU, 158 + 159 + /* Online section invoked on the hotplugged CPU from the hotplug thread */ 197 160 CPUHP_AP_ONLINE_IDLE, 198 161 CPUHP_AP_SCHED_WAIT_EMPTY, 199 162 CPUHP_AP_SMPBOOT_THREADS, ··· 257 216 int (*teardown)(unsigned int cpu), 258 217 bool multi_instance); 259 218 /** 260 - * cpuhp_setup_state - Setup hotplug state callbacks with calling the callbacks 219 + * cpuhp_setup_state - Setup hotplug state callbacks with calling the @startup 220 + * callback 261 221 * @state: The state for which the calls are installed 262 222 * @name: Name of the callback (will be used in debug output) 263 - * @startup: startup callback function 264 - * @teardown: teardown callback function 223 + * @startup: startup callback function or NULL if not required 224 + * @teardown: teardown callback function or NULL if not required 265 225 * 266 - * Installs the callback functions and invokes the startup callback on 267 - * the present cpus which have already reached the @state. 226 + * Installs the callback functions and invokes the @startup callback on 227 + * the online cpus which have already reached the @state. 268 228 */ 269 229 static inline int cpuhp_setup_state(enum cpuhp_state state, 270 230 const char *name, ··· 275 233 return __cpuhp_setup_state(state, name, true, startup, teardown, false); 276 234 } 277 235 236 + /** 237 + * cpuhp_setup_state_cpuslocked - Setup hotplug state callbacks with calling 238 + * @startup callback from a cpus_read_lock() 239 + * held region 240 + * @state: The state for which the calls are installed 241 + * @name: Name of the callback (will be used in debug output) 242 + * @startup: startup callback function or NULL if not required 243 + * @teardown: teardown callback function or NULL if not required 244 + * 245 + * Same as cpuhp_setup_state() except that it must be invoked from within a 246 + * cpus_read_lock() held region. 247 + */ 278 248 static inline int cpuhp_setup_state_cpuslocked(enum cpuhp_state state, 279 249 const char *name, 280 250 int (*startup)(unsigned int cpu), ··· 298 244 299 245 /** 300 246 * cpuhp_setup_state_nocalls - Setup hotplug state callbacks without calling the 301 - * callbacks 247 + * @startup callback 302 248 * @state: The state for which the calls are installed 303 249 * @name: Name of the callback. 304 - * @startup: startup callback function 305 - * @teardown: teardown callback function 250 + * @startup: startup callback function or NULL if not required 251 + * @teardown: teardown callback function or NULL if not required 306 252 * 307 - * Same as @cpuhp_setup_state except that no calls are executed are invoked 308 - * during installation of this callback. NOP if SMP=n or HOTPLUG_CPU=n. 253 + * Same as cpuhp_setup_state() except that the @startup callback is not 254 + * invoked during installation. NOP if SMP=n or HOTPLUG_CPU=n. 309 255 */ 310 256 static inline int cpuhp_setup_state_nocalls(enum cpuhp_state state, 311 257 const char *name, ··· 316 262 false); 317 263 } 318 264 265 + /** 266 + * cpuhp_setup_state_nocalls_cpuslocked - Setup hotplug state callbacks without 267 + * invoking the @startup callback from 268 + * a cpus_read_lock() held region 269 + * callbacks 270 + * @state: The state for which the calls are installed 271 + * @name: Name of the callback. 272 + * @startup: startup callback function or NULL if not required 273 + * @teardown: teardown callback function or NULL if not required 274 + * 275 + * Same as cpuhp_setup_state_nocalls() except that it must be invoked from 276 + * within a cpus_read_lock() held region. 277 + */ 319 278 static inline int cpuhp_setup_state_nocalls_cpuslocked(enum cpuhp_state state, 320 279 const char *name, 321 280 int (*startup)(unsigned int cpu), ··· 342 275 * cpuhp_setup_state_multi - Add callbacks for multi state 343 276 * @state: The state for which the calls are installed 344 277 * @name: Name of the callback. 345 - * @startup: startup callback function 346 - * @teardown: teardown callback function 278 + * @startup: startup callback function or NULL if not required 279 + * @teardown: teardown callback function or NULL if not required 347 280 * 348 281 * Sets the internal multi_instance flag and prepares a state to work as a multi 349 282 * instance callback. No callbacks are invoked at this point. The callbacks are 350 283 * invoked once an instance for this state are registered via 351 - * @cpuhp_state_add_instance or @cpuhp_state_add_instance_nocalls. 284 + * cpuhp_state_add_instance() or cpuhp_state_add_instance_nocalls() 352 285 */ 353 286 static inline int cpuhp_setup_state_multi(enum cpuhp_state state, 354 287 const char *name, ··· 373 306 * @state: The state for which the instance is installed 374 307 * @node: The node for this individual state. 375 308 * 376 - * Installs the instance for the @state and invokes the startup callback on 377 - * the present cpus which have already reached the @state. The @state must have 378 - * been earlier marked as multi-instance by @cpuhp_setup_state_multi. 309 + * Installs the instance for the @state and invokes the registered startup 310 + * callback on the online cpus which have already reached the @state. The 311 + * @state must have been earlier marked as multi-instance by 312 + * cpuhp_setup_state_multi(). 379 313 */ 380 314 static inline int cpuhp_state_add_instance(enum cpuhp_state state, 381 315 struct hlist_node *node) ··· 390 322 * @state: The state for which the instance is installed 391 323 * @node: The node for this individual state. 392 324 * 393 - * Installs the instance for the @state The @state must have been earlier 394 - * marked as multi-instance by @cpuhp_setup_state_multi. 325 + * Installs the instance for the @state. The @state must have been earlier 326 + * marked as multi-instance by cpuhp_setup_state_multi. NOP if SMP=n or 327 + * HOTPLUG_CPU=n. 395 328 */ 396 329 static inline int cpuhp_state_add_instance_nocalls(enum cpuhp_state state, 397 330 struct hlist_node *node) ··· 400 331 return __cpuhp_state_add_instance(state, node, false); 401 332 } 402 333 334 + /** 335 + * cpuhp_state_add_instance_nocalls_cpuslocked - Add an instance for a state 336 + * without invoking the startup 337 + * callback from a cpus_read_lock() 338 + * held region. 339 + * @state: The state for which the instance is installed 340 + * @node: The node for this individual state. 341 + * 342 + * Same as cpuhp_state_add_instance_nocalls() except that it must be 343 + * invoked from within a cpus_read_lock() held region. 344 + */ 403 345 static inline int 404 346 cpuhp_state_add_instance_nocalls_cpuslocked(enum cpuhp_state state, 405 347 struct hlist_node *node) ··· 426 346 * @state: The state for which the calls are removed 427 347 * 428 348 * Removes the callback functions and invokes the teardown callback on 429 - * the present cpus which have already reached the @state. 349 + * the online cpus which have already reached the @state. 430 350 */ 431 351 static inline void cpuhp_remove_state(enum cpuhp_state state) 432 352 { ··· 435 355 436 356 /** 437 357 * cpuhp_remove_state_nocalls - Remove hotplug state callbacks without invoking 438 - * teardown 358 + * the teardown callback 439 359 * @state: The state for which the calls are removed 440 360 */ 441 361 static inline void cpuhp_remove_state_nocalls(enum cpuhp_state state) ··· 443 363 __cpuhp_remove_state(state, false); 444 364 } 445 365 366 + /** 367 + * cpuhp_remove_state_nocalls_cpuslocked - Remove hotplug state callbacks without invoking 368 + * teardown from a cpus_read_lock() held region. 369 + * @state: The state for which the calls are removed 370 + * 371 + * Same as cpuhp_remove_state nocalls() except that it must be invoked 372 + * from within a cpus_read_lock() held region. 373 + */ 446 374 static inline void cpuhp_remove_state_nocalls_cpuslocked(enum cpuhp_state state) 447 375 { 448 376 __cpuhp_remove_state_cpuslocked(state, false); ··· 478 390 * @state: The state from which the instance is removed 479 391 * @node: The node for this individual state. 480 392 * 481 - * Removes the instance and invokes the teardown callback on the present cpus 482 - * which have already reached the @state. 393 + * Removes the instance and invokes the teardown callback on the online cpus 394 + * which have already reached @state. 483 395 */ 484 396 static inline int cpuhp_state_remove_instance(enum cpuhp_state state, 485 397 struct hlist_node *node)

Configure Feed

Configure Feed