+64 -80

Documentation/block/bfq-iosched.txt

reviewed

··· 16 16 schedules that may lead to a lower throughput. If your main or only 17 17 goal, for a given device, is to achieve the maximum-possible 18 18 throughput at all times, then do switch off all low-latency heuristics 19 19 - for that device, by setting low_latency to 0. Full details in Section 3. 19 19 + for that device, by setting low_latency to 0. See Section 3 for 20 20 + details on how to configure BFQ for the desired tradeoff between 21 21 + latency and throughput, or on how to maximize throughput. 20 22 21 23 On average CPUs, the current version of BFQ can handle devices 22 24 performing at most ~30K IOPS; at most ~50 KIOPS on faster CPUs. As a 23 25 reference, 30-50 KIOPS correspond to very high bandwidths with 24 26 sequential I/O (e.g., 8-12 GB/s if I/O requests are 256 KB large), and 25 25 - to 120-200 MB/s with 4KB random I/O. BFQ has not yet been tested on 26 26 - multi-queue devices. 27 27 + to 120-200 MB/s with 4KB random I/O. BFQ is currently being tested on 28 28 + multi-queue devices too. 27 29 28 30 The table of contents follow. Impatients can just jump to Section 3. 29 31 ··· 35 33 1-1 Personal systems 36 34 1-2 Server systems 37 35 2. How does BFQ work? 38 38 - 3. What are BFQ's tunable? 36 36 + 3. What are BFQ's tunables and how to properly configure BFQ? 39 37 4. BFQ group scheduling 40 38 4-1 Service guarantees provided 41 39 4-2 Interface ··· 147 145 contrast, BFQ may idle the device for a short time interval, 148 146 giving the process the chance to go on being served if it issues 149 147 a new request in time. Device idling typically boosts the 150 150 - throughput on rotational devices, if processes do synchronous 151 151 - and sequential I/O. In addition, under BFQ, device idling is 152 152 - also instrumental in guaranteeing the desired throughput 153 153 - fraction to processes issuing sync requests (see the description 154 154 - of the slice_idle tunable in this document, or [1, 2], for more 155 155 - details). 148 148 + throughput on rotational devices and on non-queueing flash-based 149 149 + devices, if processes do synchronous and sequential I/O. In 150 150 + addition, under BFQ, device idling is also instrumental in 151 151 + guaranteeing the desired throughput fraction to processes 152 152 + issuing sync requests (see the description of the slice_idle 153 153 + tunable in this document, or [1, 2], for more details). 156 154 157 155 - With respect to idling for service guarantees, if several 158 156 processes are competing for the device at the same time, but 159 159 - all processes (and groups, after the following commit) have 160 160 - the same weight, then BFQ guarantees the expected throughput 161 161 - distribution without ever idling the device. Throughput is 162 162 - thus as high as possible in this common scenario. 157 157 + all processes and groups have the same weight, then BFQ 158 158 + guarantees the expected throughput distribution without ever 159 159 + idling the device. Throughput is thus as high as possible in 160 160 + this common scenario. 161 161 + 162 162 + - On flash-based storage with internal queueing of commands 163 163 + (typically NCQ), device idling happens to be always detrimental 164 164 + for throughput. So, with these devices, BFQ performs idling 165 165 + only when strictly needed for service guarantees, i.e., for 166 166 + guaranteeing low latency or fairness. In these cases, overall 167 167 + throughput may be sub-optimal. No solution currently exists to 168 168 + provide both strong service guarantees and optimal throughput 169 169 + on devices with internal queueing. 163 170 164 171 - If low-latency mode is enabled (default configuration), BFQ 165 172 executes some special heuristics to detect interactive and soft ··· 202 191 - Queues are scheduled according to a variant of WF2Q+, named 203 192 B-WF2Q+, and implemented using an augmented rb-tree to preserve an 204 193 O(log N) overall complexity. See [2] for more details. B-WF2Q+ is 205 205 - also ready for hierarchical scheduling. However, for a cleaner 206 206 - logical breakdown, the code that enables and completes 207 207 - hierarchical support is provided in the next commit, which focuses 208 208 - exactly on this feature. 194 194 + also ready for hierarchical scheduling, details in Section 4. 209 195 210 196 - B-WF2Q+ guarantees a tight deviation with respect to an ideal, 211 197 perfectly fair, and smooth service. In particular, B-WF2Q+ ··· 257 249 the Idle class, to prevent it from starving. 258 250 259 251 260 260 - 3. What are BFQ's tunable? 261 261 - ========================== 252 252 + 3. What are BFQ's tunables and how to properly configure BFQ? 253 253 + ============================================================= 262 254 263 263 - The tunables back_seek-max, back_seek_penalty, fifo_expire_async and 264 264 - fifo_expire_sync below are the same as in CFQ. Their description is 265 265 - just copied from that for CFQ. Some considerations in the description 266 266 - of slice_idle are copied from CFQ too. 255 255 + Most BFQ tunables affect service guarantees (basically latency and 256 256 + fairness) and throughput. For full details on how to choose the 257 257 + desired tradeoff between service guarantees and throughput, see the 258 258 + parameters slice_idle, strict_guarantees and low_latency. For details 259 259 + on how to maximise throughput, see slice_idle, timeout_sync and 260 260 + max_budget. The other performance-related parameters have been 261 261 + inherited from, and have been preserved mostly for compatibility with 262 262 + CFQ. So far, no performance improvement has been reported after 263 263 + changing the latter parameters in BFQ. 264 264 + 265 265 + In particular, the tunables back_seek-max, back_seek_penalty, 266 266 + fifo_expire_async and fifo_expire_sync below are the same as in 267 267 + CFQ. Their description is just copied from that for CFQ. Some 268 268 + considerations in the description of slice_idle are copied from CFQ 269 269 + too. 267 270 268 271 per-process ioprio and weight 269 272 ----------------------------- ··· 304 285 305 286 Setting slice_idle to 0 will remove all the idling on queues and one 306 287 should see an overall improved throughput on faster storage devices 307 307 - like multiple SATA/SAS disks in hardware RAID configuration. 288 288 + like multiple SATA/SAS disks in hardware RAID configuration, as well 289 289 + as flash-based storage with internal command queueing (and 290 290 + parallelism). 308 291 309 292 So depending on storage and workload, it might be useful to set 310 293 slice_idle=0. In general for SATA/SAS disks and software RAID of 311 294 SATA/SAS disks keeping slice_idle enabled should be useful. For any 312 295 configurations where there are multiple spindles behind single LUN 313 313 - (Host based hardware RAID controller or for storage arrays), setting 314 314 - slice_idle=0 might end up in better throughput and acceptable 315 315 - latencies. 296 296 + (Host based hardware RAID controller or for storage arrays), or with 297 297 + flash-based fast storage, setting slice_idle=0 might end up in better 298 298 + throughput and acceptable latencies. 316 299 317 300 Idling is however necessary to have service guarantees enforced in 318 301 case of differentiated weights or differentiated I/O-request lengths. ··· 333 312 where it is beneficial also for throughput, idling can severely impact 334 313 throughput. One important case is random workload. Because of this 335 314 issue, BFQ tends to avoid idling as much as possible, when it is not 336 336 - beneficial also for throughput. As a consequence of this behavior, and 337 337 - of further issues described for the strict_guarantees tunable, 338 338 - short-term service guarantees may be occasionally violated. And, in 339 339 - some cases, these guarantees may be more important than guaranteeing 340 340 - maximum throughput. For example, in video playing/streaming, a very 341 341 - low drop rate may be more important than maximum throughput. In these 342 342 - cases, consider setting the strict_guarantees parameter. 315 315 + beneficial also for throughput (as detailed in Section 2). As a 316 316 + consequence of this behavior, and of further issues described for the 317 317 + strict_guarantees tunable, short-term service guarantees may be 318 318 + occasionally violated. And, in some cases, these guarantees may be 319 319 + more important than guaranteeing maximum throughput. For example, in 320 320 + video playing/streaming, a very low drop rate may be more important 321 321 + than maximum throughput. In these cases, consider setting the 322 322 + strict_guarantees parameter. 343 323 344 324 strict_guarantees 345 325 ----------------- ··· 442 420 to the maximum number of sectors that can be served during 443 421 timeout_sync, according to the estimated peak rate. 444 422 423 423 + For specific devices, some users have occasionally reported to have 424 424 + reached a higher throughput by setting max_budget explicitly, i.e., by 425 425 + setting max_budget to a higher value than 0. In particular, they have 426 426 + set max_budget to higher values than those to which BFQ would have set 427 427 + it with auto-tuning. An alternative way to achieve this goal is to 428 428 + just increase the value of timeout_sync, leaving max_budget equal to 0. 429 429 + 445 430 weights 446 431 ------- 447 432 448 433 Read-only parameter, used to show the weights of the currently active 449 434 BFQ queues. 450 450 - 451 451 - 452 452 - wr_ tunables 453 453 - ------------ 454 454 - 455 455 - BFQ exports a few parameters to control/tune the behavior of 456 456 - low-latency heuristics. 457 457 - 458 458 - wr_coeff 459 459 - 460 460 - Factor by which the weight of a weight-raised queue is multiplied. If 461 461 - the queue is deemed soft real-time, then the weight is further 462 462 - multiplied by an additional, constant factor. 463 463 - 464 464 - wr_max_time 465 465 - 466 466 - Maximum duration of a weight-raising period for an interactive task 467 467 - (ms). If set to zero (default value), then this value is computed 468 468 - automatically, as a function of the peak rate of the device. In any 469 469 - case, when the value of this parameter is read, it always reports the 470 470 - current duration, regardless of whether it has been set manually or 471 471 - computed automatically. 472 472 - 473 473 - wr_max_softrt_rate 474 474 - 475 475 - Maximum service rate below which a queue is deemed to be associated 476 476 - with a soft real-time application, and is then weight-raised 477 477 - accordingly (sectors/sec). 478 478 - 479 479 - wr_min_idle_time 480 480 - 481 481 - Minimum idle period after which interactive weight-raising may be 482 482 - reactivated for a queue (in ms). 483 483 - 484 484 - wr_rt_max_time 485 485 - 486 486 - Maximum weight-raising duration for soft real-time queues (in ms). The 487 487 - start time from which this duration is considered is automatically 488 488 - moved forward if the queue is detected to be still soft real-time 489 489 - before the current soft real-time weight-raising period finishes. 490 490 - 491 491 - wr_min_inter_arr_async 492 492 - 493 493 - Minimum period between I/O request arrivals after which weight-raising 494 494 - may be reactivated for an already busy async queue (in ms). 495 435 496 436 497 437 4. Group scheduling with BFQ

+9 -9

block/bfq-cgroup.c

reviewed

··· 206 206 bfqg->ref++; 207 207 } 208 208 209 209 - void bfqg_put(struct bfq_group *bfqg) 209 209 + static void bfqg_put(struct bfq_group *bfqg) 210 210 { 211 211 bfqg->ref--; 212 212 ··· 385 385 return cpd_to_bfqgd(blkcg_to_cpd(blkcg, &blkcg_policy_bfq)); 386 386 } 387 387 388 388 - struct blkcg_policy_data *bfq_cpd_alloc(gfp_t gfp) 388 388 + static struct blkcg_policy_data *bfq_cpd_alloc(gfp_t gfp) 389 389 { 390 390 struct bfq_group_data *bgd; 391 391 ··· 395 395 return &bgd->pd; 396 396 } 397 397 398 398 - void bfq_cpd_init(struct blkcg_policy_data *cpd) 398 398 + static void bfq_cpd_init(struct blkcg_policy_data *cpd) 399 399 { 400 400 struct bfq_group_data *d = cpd_to_bfqgd(cpd); 401 401 ··· 403 403 CGROUP_WEIGHT_DFL : BFQ_WEIGHT_LEGACY_DFL; 404 404 } 405 405 406 406 - void bfq_cpd_free(struct blkcg_policy_data *cpd) 406 406 + static void bfq_cpd_free(struct blkcg_policy_data *cpd) 407 407 { 408 408 kfree(cpd_to_bfqgd(cpd)); 409 409 } 410 410 411 411 - struct blkg_policy_data *bfq_pd_alloc(gfp_t gfp, int node) 411 411 + static struct blkg_policy_data *bfq_pd_alloc(gfp_t gfp, int node) 412 412 { 413 413 struct bfq_group *bfqg; 414 414 ··· 426 426 return &bfqg->pd; 427 427 } 428 428 429 429 - void bfq_pd_init(struct blkg_policy_data *pd) 429 429 + static void bfq_pd_init(struct blkg_policy_data *pd) 430 430 { 431 431 struct blkcg_gq *blkg = pd_to_blkg(pd); 432 432 struct bfq_group *bfqg = blkg_to_bfqg(blkg); ··· 445 445 bfqg->rq_pos_tree = RB_ROOT; 446 446 } 447 447 448 448 - void bfq_pd_free(struct blkg_policy_data *pd) 448 448 + static void bfq_pd_free(struct blkg_policy_data *pd) 449 449 { 450 450 struct bfq_group *bfqg = pd_to_bfqg(pd); 451 451 ··· 453 453 bfqg_put(bfqg); 454 454 } 455 455 456 456 - void bfq_pd_reset_stats(struct blkg_policy_data *pd) 456 456 + static void bfq_pd_reset_stats(struct blkg_policy_data *pd) 457 457 { 458 458 struct bfq_group *bfqg = pd_to_bfqg(pd); 459 459 ··· 740 740 * blkio already grabs the queue_lock for us, so no need to use 741 741 * RCU-based magic 742 742 */ 743 743 - void bfq_pd_offline(struct blkg_policy_data *pd) 743 743 + static void bfq_pd_offline(struct blkg_policy_data *pd) 744 744 { 745 745 struct bfq_service_tree *st; 746 746 struct bfq_group *bfqg = pd_to_bfqg(pd);

+49 -26

block/bfq-iosched.c

reviewed

··· 239 239 static int T_fast[2]; 240 240 static int device_speed_thresh[2]; 241 241 242 242 - #define RQ_BIC(rq) ((struct bfq_io_cq *) (rq)->elv.priv[0]) 242 242 + #define RQ_BIC(rq) icq_to_bic((rq)->elv.priv[0]) 243 243 #define RQ_BFQQ(rq) ((rq)->elv.priv[1]) 244 244 245 245 struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync) ··· 720 720 entity->budget = new_budget; 721 721 bfq_log_bfqq(bfqd, bfqq, "updated next rq: new budget %lu", 722 722 new_budget); 723 723 - bfq_requeue_bfqq(bfqd, bfqq); 723 723 + bfq_requeue_bfqq(bfqd, bfqq, false); 724 724 } 725 725 } 726 726 ··· 2563 2563 2564 2564 bfq_del_bfqq_busy(bfqd, bfqq, true); 2565 2565 } else { 2566 2566 - bfq_requeue_bfqq(bfqd, bfqq); 2566 2566 + bfq_requeue_bfqq(bfqd, bfqq, true); 2567 2567 /* 2568 2568 * Resort priority tree of potential close cooperators. 2569 2569 */ ··· 3780 3780 default: 3781 3781 dev_err(bfqq->bfqd->queue->backing_dev_info->dev, 3782 3782 "bfq: bad prio class %d\n", ioprio_class); 3783 3783 + /* fall through */ 3783 3784 case IOPRIO_CLASS_NONE: 3784 3785 /* 3785 3786 * No prio set, inherit CPU scheduling settings. ··· 4802 4801 return sprintf(page, "%u\n", var); 4803 4802 } 4804 4803 4805 4805 - static void bfq_var_store(unsigned long *var, const char *page) 4804 4804 + static int bfq_var_store(unsigned long *var, const char *page) 4806 4805 { 4807 4806 unsigned long new_val; 4808 4807 int ret = kstrtoul(page, 10, &new_val); 4809 4808 4810 4810 - if (ret == 0) 4811 4811 - *var = new_val; 4809 4809 + if (ret) 4810 4810 + return ret; 4811 4811 + *var = new_val; 4812 4812 + return 0; 4812 4813 } 4813 4814 4814 4815 #define SHOW_FUNCTION(__FUNC, __VAR, __CONV) \ ··· 4851 4848 __FUNC(struct elevator_queue *e, const char *page, size_t count) \ 4852 4849 { \ 4853 4850 struct bfq_data *bfqd = e->elevator_data; \ 4854 4854 - unsigned long uninitialized_var(__data); \ 4855 4855 - bfq_var_store(&__data, (page)); \ 4856 4856 - if (__data < (MIN)) \ 4857 4857 - __data = (MIN); \ 4858 4858 - else if (__data > (MAX)) \ 4859 4859 - __data = (MAX); \ 4851 4851 + unsigned long __data, __min = (MIN), __max = (MAX); \ 4852 4852 + int ret; \ 4853 4853 + \ 4854 4854 + ret = bfq_var_store(&__data, (page)); \ 4855 4855 + if (ret) \ 4856 4856 + return ret; \ 4857 4857 + if (__data < __min) \ 4858 4858 + __data = __min; \ 4859 4859 + else if (__data > __max) \ 4860 4860 + __data = __max; \ 4860 4861 if (__CONV == 1) \ 4861 4862 *(__PTR) = msecs_to_jiffies(__data); \ 4862 4863 else if (__CONV == 2) \ ··· 4883 4876 static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\ 4884 4877 { \ 4885 4878 struct bfq_data *bfqd = e->elevator_data; \ 4886 4886 - unsigned long uninitialized_var(__data); \ 4887 4887 - bfq_var_store(&__data, (page)); \ 4888 4888 - if (__data < (MIN)) \ 4889 4889 - __data = (MIN); \ 4890 4890 - else if (__data > (MAX)) \ 4891 4891 - __data = (MAX); \ 4879 4879 + unsigned long __data, __min = (MIN), __max = (MAX); \ 4880 4880 + int ret; \ 4881 4881 + \ 4882 4882 + ret = bfq_var_store(&__data, (page)); \ 4883 4883 + if (ret) \ 4884 4884 + return ret; \ 4885 4885 + if (__data < __min) \ 4886 4886 + __data = __min; \ 4887 4887 + else if (__data > __max) \ 4888 4888 + __data = __max; \ 4892 4889 *(__PTR) = (u64)__data * NSEC_PER_USEC; \ 4893 4890 return count; \ 4894 4891 } ··· 4904 4893 const char *page, size_t count) 4905 4894 { 4906 4895 struct bfq_data *bfqd = e->elevator_data; 4907 4907 - unsigned long uninitialized_var(__data); 4896 4896 + unsigned long __data; 4897 4897 + int ret; 4908 4898 4909 4909 - bfq_var_store(&__data, (page)); 4899 4899 + ret = bfq_var_store(&__data, (page)); 4900 4900 + if (ret) 4901 4901 + return ret; 4910 4902 4911 4903 if (__data == 0) 4912 4904 bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd); ··· 4932 4918 const char *page, size_t count) 4933 4919 { 4934 4920 struct bfq_data *bfqd = e->elevator_data; 4935 4935 - unsigned long uninitialized_var(__data); 4921 4921 + unsigned long __data; 4922 4922 + int ret; 4936 4923 4937 4937 - bfq_var_store(&__data, (page)); 4924 4924 + ret = bfq_var_store(&__data, (page)); 4925 4925 + if (ret) 4926 4926 + return ret; 4938 4927 4939 4928 if (__data < 1) 4940 4929 __data = 1; ··· 4955 4938 const char *page, size_t count) 4956 4939 { 4957 4940 struct bfq_data *bfqd = e->elevator_data; 4958 4958 - unsigned long uninitialized_var(__data); 4941 4941 + unsigned long __data; 4942 4942 + int ret; 4959 4943 4960 4960 - bfq_var_store(&__data, (page)); 4944 4944 + ret = bfq_var_store(&__data, (page)); 4945 4945 + if (ret) 4946 4946 + return ret; 4961 4947 4962 4948 if (__data > 1) 4963 4949 __data = 1; ··· 4977 4957 const char *page, size_t count) 4978 4958 { 4979 4959 struct bfq_data *bfqd = e->elevator_data; 4980 4980 - unsigned long uninitialized_var(__data); 4960 4960 + unsigned long __data; 4961 4961 + int ret; 4981 4962 4982 4982 - bfq_var_store(&__data, (page)); 4963 4963 + ret = bfq_var_store(&__data, (page)); 4964 4964 + if (ret) 4965 4965 + return ret; 4983 4966 4984 4967 if (__data > 1) 4985 4968 __data = 1;

+2 -2

block/bfq-iosched.h

reviewed

··· 817 817 struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync); 818 818 void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync); 819 819 struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic); 820 820 - void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq); 821 820 void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq); 822 821 void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_entity *entity, 823 822 struct rb_root *root); ··· 916 917 void bfq_deactivate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, 917 918 bool ins_into_idle_tree, bool expiration); 918 919 void bfq_activate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq); 919 919 - void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq); 920 920 + void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, 921 921 + bool expiration); 920 922 void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq, 921 923 bool expiration); 922 924 void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq);

+56 -35

block/bfq-wf2q.c

reviewed

··· 44 44 BFQ_DEFAULT_GRP_CLASS - 1; 45 45 } 46 46 47 47 - static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd); 47 47 + static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd, 48 48 + bool expiration); 48 49 49 50 static bool bfq_update_parent_budget(struct bfq_entity *next_in_service); 50 51 ··· 55 54 * @new_entity: if not NULL, pointer to the entity whose activation, 56 55 * requeueing or repositionig triggered the invocation of 57 56 * this function. 57 57 + * @expiration: id true, this function is being invoked after the 58 58 + * expiration of the in-service entity 58 59 * 59 60 * This function is called to update sd->next_in_service, which, in 60 61 * its turn, may change as a consequence of the insertion or ··· 75 72 * entity. 76 73 */ 77 74 static bool bfq_update_next_in_service(struct bfq_sched_data *sd, 78 78 - struct bfq_entity *new_entity) 75 75 + struct bfq_entity *new_entity, 76 76 + bool expiration) 79 77 { 80 78 struct bfq_entity *next_in_service = sd->next_in_service; 81 79 bool parent_sched_may_change = false; 80 80 + bool change_without_lookup = false; 82 81 83 82 /* 84 83 * If this update is triggered by the activation, requeueing 85 84 * or repositiong of an entity that does not coincide with 86 85 * sd->next_in_service, then a full lookup in the active tree 87 86 * can be avoided. In fact, it is enough to check whether the 88 88 - * just-modified entity has a higher priority than 89 89 - * sd->next_in_service, or, even if it has the same priority 90 90 - * as sd->next_in_service, is eligible and has a lower virtual 87 87 + * just-modified entity has the same priority as 88 88 + * sd->next_in_service, is eligible and has a lower virtual 91 89 * finish time than sd->next_in_service. If this compound 92 90 * condition holds, then the new entity becomes the new 93 91 * next_in_service. Otherwise no change is needed. ··· 100 96 * set to true, and left as true if 101 97 * sd->next_in_service is NULL. 102 98 */ 103 103 - bool replace_next = true; 99 99 + change_without_lookup = true; 104 100 105 101 /* 106 102 * If there is already a next_in_service candidate 107 107 - * entity, then compare class priorities or timestamps 108 108 - * to decide whether to replace sd->service_tree with 109 109 - * new_entity. 103 103 + * entity, then compare timestamps to decide whether 104 104 + * to replace sd->service_tree with new_entity. 110 105 */ 111 106 if (next_in_service) { 112 107 unsigned int new_entity_class_idx = ··· 113 110 struct bfq_service_tree *st = 114 111 sd->service_tree + new_entity_class_idx; 115 112 116 116 - /* 117 117 - * For efficiency, evaluate the most likely 118 118 - * sub-condition first. 119 119 - */ 120 120 - replace_next = 113 113 + change_without_lookup = 121 114 (new_entity_class_idx == 122 115 bfq_class_idx(next_in_service) 123 116 && 124 117 !bfq_gt(new_entity->start, st->vtime) 125 118 && 126 119 bfq_gt(next_in_service->finish, 127 127 - new_entity->finish)) 128 128 - || 129 129 - new_entity_class_idx < 130 130 - bfq_class_idx(next_in_service); 120 120 + new_entity->finish)); 131 121 } 132 122 133 133 - if (replace_next) 123 123 + if (change_without_lookup) 134 124 next_in_service = new_entity; 135 135 - } else /* invoked because of a deactivation: lookup needed */ 136 136 - next_in_service = bfq_lookup_next_entity(sd); 125 125 + } 137 126 138 138 - if (next_in_service) { 127 127 + if (!change_without_lookup) /* lookup needed */ 128 128 + next_in_service = bfq_lookup_next_entity(sd, expiration); 129 129 + 130 130 + if (next_in_service) 139 131 parent_sched_may_change = !sd->next_in_service || 140 132 bfq_update_parent_budget(next_in_service); 141 141 - } 142 133 143 134 sd->next_in_service = next_in_service; 144 135 ··· 1124 1127 * @requeue: true if this is a requeue, which implies that bfqq is 1125 1128 * being expired; thus ALL its ancestors stop being served and must 1126 1129 * therefore be requeued 1130 1130 + * @expiration: true if this function is being invoked in the expiration path 1131 1131 + * of the in-service queue 1127 1132 */ 1128 1133 static void bfq_activate_requeue_entity(struct bfq_entity *entity, 1129 1134 bool non_blocking_wait_rq, 1130 1130 - bool requeue) 1135 1135 + bool requeue, bool expiration) 1131 1136 { 1132 1137 struct bfq_sched_data *sd; 1133 1138 ··· 1137 1138 sd = entity->sched_data; 1138 1139 __bfq_activate_requeue_entity(entity, sd, non_blocking_wait_rq); 1139 1140 1140 1140 - if (!bfq_update_next_in_service(sd, entity) && !requeue) 1141 1141 + if (!bfq_update_next_in_service(sd, entity, expiration) && 1142 1142 + !requeue) 1141 1143 break; 1142 1144 } 1143 1145 } ··· 1194 1194 * bfq_deactivate_entity - deactivate an entity representing a bfq_queue. 1195 1195 * @entity: the entity to deactivate. 1196 1196 * @ins_into_idle_tree: true if the entity can be put into the idle tree 1197 1197 + * @expiration: true if this function is being invoked in the expiration path 1198 1198 + * of the in-service queue 1197 1199 */ 1198 1200 static void bfq_deactivate_entity(struct bfq_entity *entity, 1199 1201 bool ins_into_idle_tree, ··· 1224 1222 * then, since entity has just been 1225 1223 * deactivated, a new one must be found. 1226 1224 */ 1227 1227 - bfq_update_next_in_service(sd, NULL); 1225 1225 + bfq_update_next_in_service(sd, NULL, expiration); 1228 1226 1229 1227 if (sd->next_in_service || sd->in_service_entity) { 1230 1228 /* ··· 1283 1281 __bfq_requeue_entity(entity); 1284 1282 1285 1283 sd = entity->sched_data; 1286 1286 - if (!bfq_update_next_in_service(sd, entity) && 1284 1284 + if (!bfq_update_next_in_service(sd, entity, expiration) && 1287 1285 !expiration) 1288 1286 /* 1289 1287 * next_in_service unchanged or not causing ··· 1418 1416 /** 1419 1417 * bfq_lookup_next_entity - return the first eligible entity in @sd. 1420 1418 * @sd: the sched_data. 1419 1419 + * @expiration: true if we are on the expiration path of the in-service queue 1421 1420 * 1422 1421 * This function is invoked when there has been a change in the trees 1423 1423 - * for sd, and we need know what is the new next entity after this 1424 1424 - * change. 1422 1422 + * for sd, and we need to know what is the new next entity to serve 1423 1423 + * after this change. 1425 1424 */ 1426 1426 - static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd) 1425 1425 + static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd, 1426 1426 + bool expiration) 1427 1427 { 1428 1428 struct bfq_service_tree *st = sd->service_tree; 1429 1429 struct bfq_service_tree *idle_class_st = st + (BFQ_IOPRIO_CLASSES - 1); ··· 1452 1448 * class, unless the idle class needs to be served. 1453 1449 */ 1454 1450 for (; class_idx < BFQ_IOPRIO_CLASSES; class_idx++) { 1451 1451 + /* 1452 1452 + * If expiration is true, then bfq_lookup_next_entity 1453 1453 + * is being invoked as a part of the expiration path 1454 1454 + * of the in-service queue. In this case, even if 1455 1455 + * sd->in_service_entity is not NULL, 1456 1456 + * sd->in_service_entiy at this point is actually not 1457 1457 + * in service any more, and, if needed, has already 1458 1458 + * been properly queued or requeued into the right 1459 1459 + * tree. The reason why sd->in_service_entity is still 1460 1460 + * not NULL here, even if expiration is true, is that 1461 1461 + * sd->in_service_entiy is reset as a last step in the 1462 1462 + * expiration path. So, if expiration is true, tell 1463 1463 + * __bfq_lookup_next_entity that there is no 1464 1464 + * sd->in_service_entity. 1465 1465 + */ 1455 1466 entity = __bfq_lookup_next_entity(st + class_idx, 1456 1456 - sd->in_service_entity); 1467 1467 + sd->in_service_entity && 1468 1468 + !expiration); 1457 1469 1458 1470 if (entity) 1459 1471 break; ··· 1582 1562 for_each_entity(entity) { 1583 1563 struct bfq_sched_data *sd = entity->sched_data; 1584 1564 1585 1585 - if (!bfq_update_next_in_service(sd, NULL)) 1565 1565 + if (!bfq_update_next_in_service(sd, NULL, false)) 1586 1566 break; 1587 1567 } 1588 1568 ··· 1630 1610 struct bfq_entity *entity = &bfqq->entity; 1631 1611 1632 1612 bfq_activate_requeue_entity(entity, bfq_bfqq_non_blocking_wait_rq(bfqq), 1633 1633 - false); 1613 1613 + false, false); 1634 1614 bfq_clear_bfqq_non_blocking_wait_rq(bfqq); 1635 1615 } 1636 1616 1637 1637 - void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1617 1617 + void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, 1618 1618 + bool expiration) 1638 1619 { 1639 1620 struct bfq_entity *entity = &bfqq->entity; 1640 1621 1641 1622 bfq_activate_requeue_entity(entity, false, 1642 1642 - bfqq == bfqd->in_service_queue); 1623 1623 + bfqq == bfqd->in_service_queue, expiration); 1643 1624 } 1644 1625 1645 1626 /*

+104 -89

drivers/block/loop.c

reviewed

··· 213 213 */ 214 214 blk_mq_freeze_queue(lo->lo_queue); 215 215 lo->use_dio = use_dio; 216 216 - if (use_dio) 216 216 + if (use_dio) { 217 217 + queue_flag_clear_unlocked(QUEUE_FLAG_NOMERGES, lo->lo_queue); 217 218 lo->lo_flags |= LO_FLAGS_DIRECT_IO; 218 218 - else 219 219 + } else { 220 220 + queue_flag_set_unlocked(QUEUE_FLAG_NOMERGES, lo->lo_queue); 219 221 lo->lo_flags &= ~LO_FLAGS_DIRECT_IO; 222 222 + } 220 223 blk_mq_unfreeze_queue(lo->lo_queue); 221 224 } 222 225 ··· 463 460 blk_mq_end_request(rq, cmd->ret < 0 ? BLK_STS_IOERR : BLK_STS_OK); 464 461 } 465 462 463 463 + static void lo_rw_aio_do_completion(struct loop_cmd *cmd) 464 464 + { 465 465 + if (!atomic_dec_and_test(&cmd->ref)) 466 466 + return; 467 467 + kfree(cmd->bvec); 468 468 + cmd->bvec = NULL; 469 469 + blk_mq_complete_request(cmd->rq); 470 470 + } 471 471 + 466 472 static void lo_rw_aio_complete(struct kiocb *iocb, long ret, long ret2) 467 473 { 468 474 struct loop_cmd *cmd = container_of(iocb, struct loop_cmd, iocb); 469 475 470 476 cmd->ret = ret; 471 471 - blk_mq_complete_request(cmd->rq); 477 477 + lo_rw_aio_do_completion(cmd); 472 478 } 473 479 474 480 static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, ··· 485 473 { 486 474 struct iov_iter iter; 487 475 struct bio_vec *bvec; 488 488 - struct bio *bio = cmd->rq->bio; 476 476 + struct request *rq = cmd->rq; 477 477 + struct bio *bio = rq->bio; 489 478 struct file *file = lo->lo_backing_file; 479 479 + unsigned int offset; 480 480 + int segments = 0; 490 481 int ret; 491 482 492 492 - /* nomerge for loop request queue */ 493 493 - WARN_ON(cmd->rq->bio != cmd->rq->biotail); 483 483 + if (rq->bio != rq->biotail) { 484 484 + struct req_iterator iter; 485 485 + struct bio_vec tmp; 494 486 495 495 - bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); 487 487 + __rq_for_each_bio(bio, rq) 488 488 + segments += bio_segments(bio); 489 489 + bvec = kmalloc(sizeof(struct bio_vec) * segments, GFP_NOIO); 490 490 + if (!bvec) 491 491 + return -EIO; 492 492 + cmd->bvec = bvec; 493 493 + 494 494 + /* 495 495 + * The bios of the request may be started from the middle of 496 496 + * the 'bvec' because of bio splitting, so we can't directly 497 497 + * copy bio->bi_iov_vec to new bvec. The rq_for_each_segment 498 498 + * API will take care of all details for us. 499 499 + */ 500 500 + rq_for_each_segment(tmp, rq, iter) { 501 501 + *bvec = tmp; 502 502 + bvec++; 503 503 + } 504 504 + bvec = cmd->bvec; 505 505 + offset = 0; 506 506 + } else { 507 507 + /* 508 508 + * Same here, this bio may be started from the middle of the 509 509 + * 'bvec' because of bio splitting, so offset from the bvec 510 510 + * must be passed to iov iterator 511 511 + */ 512 512 + offset = bio->bi_iter.bi_bvec_done; 513 513 + bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); 514 514 + segments = bio_segments(bio); 515 515 + } 516 516 + atomic_set(&cmd->ref, 2); 517 517 + 496 518 iov_iter_bvec(&iter, ITER_BVEC | rw, bvec, 497 497 - bio_segments(bio), blk_rq_bytes(cmd->rq)); 498 498 - /* 499 499 - * This bio may be started from the middle of the 'bvec' 500 500 - * because of bio splitting, so offset from the bvec must 501 501 - * be passed to iov iterator 502 502 - */ 503 503 - iter.iov_offset = bio->bi_iter.bi_bvec_done; 519 519 + segments, blk_rq_bytes(rq)); 520 520 + iter.iov_offset = offset; 504 521 505 522 cmd->iocb.ki_pos = pos; 506 523 cmd->iocb.ki_filp = file; ··· 540 499 ret = call_write_iter(file, &cmd->iocb, &iter); 541 500 else 542 501 ret = call_read_iter(file, &cmd->iocb, &iter); 502 502 + 503 503 + lo_rw_aio_do_completion(cmd); 543 504 544 505 if (ret != -EIOCBQUEUED) 545 506 cmd->iocb.ki_complete(&cmd->iocb, ret, 0); ··· 589 546 } 590 547 } 591 548 592 592 - struct switch_request { 593 593 - struct file *file; 594 594 - struct completion wait; 595 595 - }; 596 596 - 597 549 static inline void loop_update_dio(struct loop_device *lo) 598 550 { 599 551 __loop_update_dio(lo, io_is_direct(lo->lo_backing_file) | 600 552 lo->use_dio); 601 601 - } 602 602 - 603 603 - /* 604 604 - * Do the actual switch; called from the BIO completion routine 605 605 - */ 606 606 - static void do_loop_switch(struct loop_device *lo, struct switch_request *p) 607 607 - { 608 608 - struct file *file = p->file; 609 609 - struct file *old_file = lo->lo_backing_file; 610 610 - struct address_space *mapping; 611 611 - 612 612 - /* if no new file, only flush of queued bios requested */ 613 613 - if (!file) 614 614 - return; 615 615 - 616 616 - mapping = file->f_mapping; 617 617 - mapping_set_gfp_mask(old_file->f_mapping, lo->old_gfp_mask); 618 618 - lo->lo_backing_file = file; 619 619 - lo->lo_blocksize = S_ISBLK(mapping->host->i_mode) ? 620 620 - mapping->host->i_bdev->bd_block_size : PAGE_SIZE; 621 621 - lo->old_gfp_mask = mapping_gfp_mask(mapping); 622 622 - mapping_set_gfp_mask(mapping, lo->old_gfp_mask & ~(__GFP_IO|__GFP_FS)); 623 623 - loop_update_dio(lo); 624 624 - } 625 625 - 626 626 - /* 627 627 - * loop_switch performs the hard work of switching a backing store. 628 628 - * First it needs to flush existing IO, it does this by sending a magic 629 629 - * BIO down the pipe. The completion of this BIO does the actual switch. 630 630 - */ 631 631 - static int loop_switch(struct loop_device *lo, struct file *file) 632 632 - { 633 633 - struct switch_request w; 634 634 - 635 635 - w.file = file; 636 636 - 637 637 - /* freeze queue and wait for completion of scheduled requests */ 638 638 - blk_mq_freeze_queue(lo->lo_queue); 639 639 - 640 640 - /* do the switch action */ 641 641 - do_loop_switch(lo, &w); 642 642 - 643 643 - /* unfreeze */ 644 644 - blk_mq_unfreeze_queue(lo->lo_queue); 645 645 - 646 646 - return 0; 647 647 - } 648 648 - 649 649 - /* 650 650 - * Helper to flush the IOs in loop, but keeping loop thread running 651 651 - */ 652 652 - static int loop_flush(struct loop_device *lo) 653 653 - { 654 654 - /* loop not yet configured, no running thread, nothing to flush */ 655 655 - if (lo->lo_state != Lo_bound) 656 656 - return 0; 657 657 - return loop_switch(lo, NULL); 658 553 } 659 554 660 555 static void loop_reread_partitions(struct loop_device *lo, ··· 659 678 goto out_putf; 660 679 661 680 /* and ... switch */ 662 662 - error = loop_switch(lo, file); 663 663 - if (error) 664 664 - goto out_putf; 681 681 + blk_mq_freeze_queue(lo->lo_queue); 682 682 + mapping_set_gfp_mask(old_file->f_mapping, lo->old_gfp_mask); 683 683 + lo->lo_backing_file = file; 684 684 + lo->old_gfp_mask = mapping_gfp_mask(file->f_mapping); 685 685 + mapping_set_gfp_mask(file->f_mapping, 686 686 + lo->old_gfp_mask & ~(__GFP_IO|__GFP_FS)); 687 687 + loop_update_dio(lo); 688 688 + blk_mq_unfreeze_queue(lo->lo_queue); 665 689 666 690 fput(old_file); 667 691 if (lo->lo_flags & LO_FLAGS_PARTSCAN) ··· 853 867 struct file *file, *f; 854 868 struct inode *inode; 855 869 struct address_space *mapping; 856 856 - unsigned lo_blocksize; 857 870 int lo_flags = 0; 858 871 int error; 859 872 loff_t size; ··· 896 911 !file->f_op->write_iter) 897 912 lo_flags |= LO_FLAGS_READ_ONLY; 898 913 899 899 - lo_blocksize = S_ISBLK(inode->i_mode) ? 900 900 - inode->i_bdev->bd_block_size : PAGE_SIZE; 901 901 - 902 914 error = -EFBIG; 903 915 size = get_loop_size(lo, file); 904 916 if ((loff_t)(sector_t)size != size) ··· 909 927 set_device_ro(bdev, (lo_flags & LO_FLAGS_READ_ONLY) != 0); 910 928 911 929 lo->use_dio = false; 912 912 - lo->lo_blocksize = lo_blocksize; 913 930 lo->lo_device = bdev; 914 931 lo->lo_flags = lo_flags; 915 932 lo->lo_backing_file = file; ··· 928 947 /* let user-space know about the new size */ 929 948 kobject_uevent(&disk_to_dev(bdev->bd_disk)->kobj, KOBJ_CHANGE); 930 949 931 931 - set_blocksize(bdev, lo_blocksize); 950 950 + set_blocksize(bdev, S_ISBLK(inode->i_mode) ? 951 951 + block_size(inode->i_bdev) : PAGE_SIZE); 932 952 933 953 lo->lo_state = Lo_bound; 934 954 if (part_shift) ··· 1035 1053 memset(lo->lo_encrypt_key, 0, LO_KEY_SIZE); 1036 1054 memset(lo->lo_crypt_name, 0, LO_NAME_SIZE); 1037 1055 memset(lo->lo_file_name, 0, LO_NAME_SIZE); 1056 1056 + blk_queue_logical_block_size(lo->lo_queue, 512); 1057 1057 + blk_queue_physical_block_size(lo->lo_queue, 512); 1058 1058 + blk_queue_io_min(lo->lo_queue, 512); 1038 1059 if (bdev) { 1039 1060 bdput(bdev); 1040 1061 invalidate_bdev(bdev); ··· 1321 1336 return error; 1322 1337 } 1323 1338 1339 1339 + static int loop_set_block_size(struct loop_device *lo, unsigned long arg) 1340 1340 + { 1341 1341 + if (lo->lo_state != Lo_bound) 1342 1342 + return -ENXIO; 1343 1343 + 1344 1344 + if (arg < 512 || arg > PAGE_SIZE || !is_power_of_2(arg)) 1345 1345 + return -EINVAL; 1346 1346 + 1347 1347 + blk_mq_freeze_queue(lo->lo_queue); 1348 1348 + 1349 1349 + blk_queue_logical_block_size(lo->lo_queue, arg); 1350 1350 + blk_queue_physical_block_size(lo->lo_queue, arg); 1351 1351 + blk_queue_io_min(lo->lo_queue, arg); 1352 1352 + loop_update_dio(lo); 1353 1353 + 1354 1354 + blk_mq_unfreeze_queue(lo->lo_queue); 1355 1355 + 1356 1356 + return 0; 1357 1357 + } 1358 1358 + 1324 1359 static int lo_ioctl(struct block_device *bdev, fmode_t mode, 1325 1360 unsigned int cmd, unsigned long arg) 1326 1361 { ··· 1388 1383 err = -EPERM; 1389 1384 if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN)) 1390 1385 err = loop_set_dio(lo, arg); 1386 1386 + break; 1387 1387 + case LOOP_SET_BLOCK_SIZE: 1388 1388 + err = -EPERM; 1389 1389 + if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN)) 1390 1390 + err = loop_set_block_size(lo, arg); 1391 1391 break; 1392 1392 default: 1393 1393 err = lo->ioctl ? lo->ioctl(lo, cmd, arg) : -EINVAL; ··· 1593 1583 err = loop_clr_fd(lo); 1594 1584 if (!err) 1595 1585 return; 1596 1596 - } else { 1586 1586 + } else if (lo->lo_state == Lo_bound) { 1597 1587 /* 1598 1588 * Otherwise keep thread (if running) and config, 1599 1589 * but flush possible ongoing bios in thread. 1600 1590 */ 1601 1601 - loop_flush(lo); 1591 1591 + blk_mq_freeze_queue(lo->lo_queue); 1592 1592 + blk_mq_unfreeze_queue(lo->lo_queue); 1602 1593 } 1603 1594 1604 1595 mutex_unlock(&lo->lo_ctl_mutex); ··· 1781 1770 } 1782 1771 lo->lo_queue->queuedata = lo; 1783 1772 1773 1773 + blk_queue_max_hw_sectors(lo->lo_queue, BLK_DEF_MAX_SECTORS); 1774 1774 + 1784 1775 /* 1785 1785 - * It doesn't make sense to enable merge because the I/O 1786 1786 - * submitted to backing file is handled page by page. 1776 1776 + * By default, we do buffer IO, so it doesn't make sense to enable 1777 1777 + * merge because the I/O submitted to backing file is handled page by 1778 1778 + * page. For directio mode, merge does help to dispatch bigger request 1779 1779 + * to underlayer disk. We will enable merge once directio is enabled. 1787 1780 */ 1788 1781 queue_flag_set_unlocked(QUEUE_FLAG_NOMERGES, lo->lo_queue); 1789 1782

+5 -3

drivers/block/loop.h

reviewed

··· 48 48 49 49 struct file * lo_backing_file; 50 50 struct block_device *lo_device; 51 51 - unsigned lo_blocksize; 52 51 void *key_data; 53 52 54 53 gfp_t old_gfp_mask; ··· 67 68 struct loop_cmd { 68 69 struct kthread_work work; 69 70 struct request *rq; 70 70 - struct list_head list; 71 71 - bool use_aio; /* use AIO interface to handle I/O */ 71 71 + union { 72 72 + bool use_aio; /* use AIO interface to handle I/O */ 73 73 + atomic_t ref; /* only for aio */ 74 74 + }; 72 75 long ret; 73 76 struct kiocb iocb; 77 77 + struct bio_vec *bvec; 74 78 }; 75 79 76 80 /* Support for loadable transfer modules */

+3 -1

drivers/md/bcache/alloc.c

reviewed

··· 68 68 #include <linux/random.h> 69 69 #include <trace/events/bcache.h> 70 70 71 71 + #define MAX_OPEN_BUCKETS 128 72 72 + 71 73 /* Bucket heap / gen */ 72 74 73 75 uint8_t bch_inc_gen(struct cache *ca, struct bucket *b) ··· 673 671 674 672 spin_lock_init(&c->data_bucket_lock); 675 673 676 676 - for (i = 0; i < 6; i++) { 674 674 + for (i = 0; i < MAX_OPEN_BUCKETS; i++) { 677 675 struct open_bucket *b = kzalloc(sizeof(*b), GFP_KERNEL); 678 676 if (!b) 679 677 return -ENOMEM;

+1

drivers/md/bcache/bcache.h

reviewed

··· 333 333 /* Limit number of writeback bios in flight */ 334 334 struct semaphore in_flight; 335 335 struct task_struct *writeback_thread; 336 336 + struct workqueue_struct *writeback_write_wq; 336 337 337 338 struct keybuf writeback_keys; 338 339

+2 -13

drivers/md/bcache/closure.c

reviewed

··· 70 70 list = llist_del_all(&wait_list->list); 71 71 72 72 /* We first reverse the list to preserve FIFO ordering and fairness */ 73 73 - 74 74 - while (list) { 75 75 - struct llist_node *t = list; 76 76 - list = llist_next(list); 77 77 - 78 78 - t->next = reverse; 79 79 - reverse = t; 80 80 - } 73 73 + reverse = llist_reverse_order(list); 81 74 82 75 /* Then do the wakeups */ 83 83 - 84 84 - while (reverse) { 85 85 - cl = container_of(reverse, struct closure, list); 86 86 - reverse = llist_next(reverse); 87 87 - 76 76 + llist_for_each_entry(cl, reverse, list) { 88 77 closure_set_waiting(cl, 0); 89 78 closure_sub(cl, CLOSURE_WAITING + 1); 90 79 }

-4

drivers/md/bcache/closure.h

reviewed

··· 312 312 * been dropped with closure_put()), it will resume execution at @fn running out 313 313 * of @wq (or, if @wq is NULL, @fn will be called by closure_put() directly). 314 314 * 315 315 - * NOTE: This macro expands to a return in the calling function! 316 316 - * 317 315 * This is because after calling continue_at() you no longer have a ref on @cl, 318 316 * and whatever @cl owns may be freed out from under you - a running closure fn 319 317 * has a ref on its own closure which continue_at() drops. ··· 337 339 * 338 340 * Causes @fn to be executed out of @cl, in @wq context (or called directly if 339 341 * @wq is NULL). 340 340 - * 341 341 - * NOTE: like continue_at(), this macro expands to a return in the caller! 342 342 * 343 343 * The ref the caller of continue_at_nobarrier() had on @cl is now owned by @fn, 344 344 * thus it's not safe to touch anything protected by @cl after a

+3 -9

drivers/md/bcache/request.c

reviewed

··· 196 196 struct data_insert_op *op = container_of(cl, struct data_insert_op, cl); 197 197 struct bio *bio = op->bio, *n; 198 198 199 199 - if (atomic_sub_return(bio_sectors(bio), &op->c->sectors_to_gc) < 0) 200 200 - wake_up_gc(op->c); 201 201 - 202 199 if (op->bypass) 203 200 return bch_data_invalidate(cl); 201 201 + 202 202 + if (atomic_sub_return(bio_sectors(bio), &op->c->sectors_to_gc) < 0) 203 203 + wake_up_gc(op->c); 204 204 205 205 /* 206 206 * Journal writes are marked REQ_PREFLUSH; if the original write was a ··· 398 398 } 399 399 400 400 if (!congested && !dc->sequential_cutoff) 401 401 - goto rescale; 402 402 - 403 403 - if (!congested && 404 404 - mode == CACHE_MODE_WRITEBACK && 405 405 - op_is_write(bio->bi_opf) && 406 406 - op_is_sync(bio->bi_opf)) 407 401 goto rescale; 408 402 409 403 spin_lock(&dc->io_lock);

+6 -4

drivers/md/bcache/super.c

reviewed

··· 1026 1026 } 1027 1027 1028 1028 if (BDEV_STATE(&dc->sb) == BDEV_STATE_DIRTY) { 1029 1029 - bch_sectors_dirty_init(dc); 1029 1029 + bch_sectors_dirty_init(&dc->disk); 1030 1030 atomic_set(&dc->has_dirty, 1); 1031 1031 atomic_inc(&dc->count); 1032 1032 bch_writeback_queue(dc); ··· 1059 1059 cancel_delayed_work_sync(&dc->writeback_rate_update); 1060 1060 if (!IS_ERR_OR_NULL(dc->writeback_thread)) 1061 1061 kthread_stop(dc->writeback_thread); 1062 1062 + if (dc->writeback_write_wq) 1063 1063 + destroy_workqueue(dc->writeback_write_wq); 1062 1064 1063 1065 mutex_lock(&bch_register_lock); 1064 1066 ··· 1230 1228 goto err; 1231 1229 1232 1230 bcache_device_attach(d, c, u - c->uuids); 1231 1231 + bch_sectors_dirty_init(d); 1233 1232 bch_flash_dev_request_init(d); 1234 1233 add_disk(d->disk); 1235 1234 ··· 1376 1373 struct cache *ca; 1377 1374 struct btree *b; 1378 1375 unsigned i; 1379 1379 - 1380 1380 - if (!c) 1381 1381 - closure_return(cl); 1382 1376 1383 1377 bch_cache_accounting_destroy(&c->accounting); 1384 1378 ··· 1964 1964 else 1965 1965 err = "device busy"; 1966 1966 mutex_unlock(&bch_register_lock); 1967 1967 + if (!IS_ERR(bdev)) 1968 1968 + bdput(bdev); 1967 1969 if (attr == &ksysfs_register_quiet) 1968 1970 goto out; 1969 1971 }

+16 -3

drivers/md/bcache/sysfs.c

reviewed

··· 192 192 { 193 193 struct cached_dev *dc = container_of(kobj, struct cached_dev, 194 194 disk.kobj); 195 195 - unsigned v = size; 195 195 + ssize_t v = size; 196 196 struct cache_set *c; 197 197 struct kobj_uevent_env *env; 198 198 ··· 227 227 bch_cached_dev_run(dc); 228 228 229 229 if (attr == &sysfs_cache_mode) { 230 230 - ssize_t v = bch_read_string_list(buf, bch_cache_modes + 1); 230 230 + v = bch_read_string_list(buf, bch_cache_modes + 1); 231 231 232 232 if (v < 0) 233 233 return v; ··· 615 615 bch_cache_accounting_clear(&c->accounting); 616 616 } 617 617 618 618 - if (attr == &sysfs_trigger_gc) 618 618 + if (attr == &sysfs_trigger_gc) { 619 619 + /* 620 620 + * Garbage collection thread only works when sectors_to_gc < 0, 621 621 + * when users write to sysfs entry trigger_gc, most of time 622 622 + * they want to forcibly triger gargage collection. Here -1 is 623 623 + * set to c->sectors_to_gc, to make gc_should_run() give a 624 624 + * chance to permit gc thread to run. "give a chance" means 625 625 + * before going into gc_should_run(), there is still chance 626 626 + * that c->sectors_to_gc being set to other positive value. So 627 627 + * writing sysfs entry trigger_gc won't always make sure gc 628 628 + * thread takes effect. 629 629 + */ 630 630 + atomic_set(&c->sectors_to_gc, -1); 619 631 wake_up_gc(c); 632 632 + } 620 633 621 634 if (attr == &sysfs_prune_cache) { 622 635 struct shrink_control sc;

+31 -11

drivers/md/bcache/util.c

reviewed

··· 74 74 STRTO_H(strtoll, long long) 75 75 STRTO_H(strtoull, unsigned long long) 76 76 77 77 + /** 78 78 + * bch_hprint() - formats @v to human readable string for sysfs. 79 79 + * 80 80 + * @v - signed 64 bit integer 81 81 + * @buf - the (at least 8 byte) buffer to format the result into. 82 82 + * 83 83 + * Returns the number of bytes used by format. 84 84 + */ 77 85 ssize_t bch_hprint(char *buf, int64_t v) 78 86 { 79 87 static const char units[] = "?kMGTPEZY"; 80 80 - char dec[4] = ""; 81 81 - int u, t = 0; 88 88 + int u = 0, t; 82 89 83 83 - for (u = 0; v >= 1024 || v <= -1024; u++) { 84 84 - t = v & ~(~0 << 10); 85 85 - v >>= 10; 86 86 - } 90 90 + uint64_t q; 87 91 88 88 - if (!u) 89 89 - return sprintf(buf, "%llu", v); 92 92 + if (v < 0) 93 93 + q = -v; 94 94 + else 95 95 + q = v; 90 96 91 91 - if (v < 100 && v > -100) 92 92 - snprintf(dec, sizeof(dec), ".%i", t / 100); 97 97 + /* For as long as the number is more than 3 digits, but at least 98 98 + * once, shift right / divide by 1024. Keep the remainder for 99 99 + * a digit after the decimal point. 100 100 + */ 101 101 + do { 102 102 + u++; 93 103 94 94 - return sprintf(buf, "%lli%s%c", v, dec, units[u]); 104 104 + t = q & ~(~0 << 10); 105 105 + q >>= 10; 106 106 + } while (q >= 1000); 107 107 + 108 108 + if (v < 0) 109 109 + /* '-', up to 3 digits, '.', 1 digit, 1 character, null; 110 110 + * yields 8 bytes. 111 111 + */ 112 112 + return sprintf(buf, "-%llu.%i%c", q, t * 10 / 1024, units[u]); 113 113 + else 114 114 + return sprintf(buf, "%llu.%i%c", q, t * 10 / 1024, units[u]); 95 115 } 96 116 97 117 ssize_t bch_snprint_string_list(char *buf, size_t size, const char * const list[],

+13 -7

drivers/md/bcache/writeback.c

reviewed

··· 21 21 static void __update_writeback_rate(struct cached_dev *dc) 22 22 { 23 23 struct cache_set *c = dc->disk.c; 24 24 - uint64_t cache_sectors = c->nbuckets * c->sb.bucket_size; 24 24 + uint64_t cache_sectors = c->nbuckets * c->sb.bucket_size - 25 25 + bcache_flash_devs_sectors_dirty(c); 25 26 uint64_t cache_dirty_target = 26 27 div_u64(cache_sectors * dc->writeback_percent, 100); 27 28 ··· 187 186 188 187 closure_bio_submit(&io->bio, cl); 189 188 190 190 - continue_at(cl, write_dirty_finish, system_wq); 189 189 + continue_at(cl, write_dirty_finish, io->dc->writeback_write_wq); 191 190 } 192 191 193 192 static void read_dirty_endio(struct bio *bio) ··· 207 206 208 207 closure_bio_submit(&io->bio, cl); 209 208 210 210 - continue_at(cl, write_dirty, system_wq); 209 209 + continue_at(cl, write_dirty, io->dc->writeback_write_wq); 211 210 } 212 211 213 212 static void read_dirty(struct cached_dev *dc) ··· 482 481 return MAP_CONTINUE; 483 482 } 484 483 485 485 - void bch_sectors_dirty_init(struct cached_dev *dc) 484 484 + void bch_sectors_dirty_init(struct bcache_device *d) 486 485 { 487 486 struct sectors_dirty_init op; 488 487 489 488 bch_btree_op_init(&op.op, -1); 490 490 - op.inode = dc->disk.id; 489 489 + op.inode = d->id; 491 490 492 492 - bch_btree_map_keys(&op.op, dc->disk.c, &KEY(op.inode, 0, 0), 491 491 + bch_btree_map_keys(&op.op, d->c, &KEY(op.inode, 0, 0), 493 492 sectors_dirty_init_fn, 0); 494 493 495 495 - dc->disk.sectors_dirty_last = bcache_dev_sectors_dirty(&dc->disk); 494 494 + d->sectors_dirty_last = bcache_dev_sectors_dirty(d); 496 495 } 497 496 498 497 void bch_cached_dev_writeback_init(struct cached_dev *dc) ··· 516 515 517 516 int bch_cached_dev_writeback_start(struct cached_dev *dc) 518 517 { 518 518 + dc->writeback_write_wq = alloc_workqueue("bcache_writeback_wq", 519 519 + WQ_MEM_RECLAIM, 0); 520 520 + if (!dc->writeback_write_wq) 521 521 + return -ENOMEM; 522 522 + 519 523 dc->writeback_thread = kthread_create(bch_writeback_thread, dc, 520 524 "bcache_writeback"); 521 525 if (IS_ERR(dc->writeback_thread))

+20 -1

drivers/md/bcache/writeback.h

reviewed

··· 14 14 return ret; 15 15 } 16 16 17 17 + static inline uint64_t bcache_flash_devs_sectors_dirty(struct cache_set *c) 18 18 + { 19 19 + uint64_t i, ret = 0; 20 20 + 21 21 + mutex_lock(&bch_register_lock); 22 22 + 23 23 + for (i = 0; i < c->nr_uuids; i++) { 24 24 + struct bcache_device *d = c->devices[i]; 25 25 + 26 26 + if (!d || !UUID_FLASH_ONLY(&c->uuids[i])) 27 27 + continue; 28 28 + ret += bcache_dev_sectors_dirty(d); 29 29 + } 30 30 + 31 31 + mutex_unlock(&bch_register_lock); 32 32 + 33 33 + return ret; 34 34 + } 35 35 + 17 36 static inline unsigned offset_to_stripe(struct bcache_device *d, 18 37 uint64_t offset) 19 38 { ··· 103 84 104 85 void bcache_dev_sectors_dirty_add(struct cache_set *, unsigned, uint64_t, int); 105 86 106 106 - void bch_sectors_dirty_init(struct cached_dev *dc); 87 87 + void bch_sectors_dirty_init(struct bcache_device *); 107 88 void bch_cached_dev_writeback_init(struct cached_dev *); 108 89 int bch_cached_dev_writeback_start(struct cached_dev *); 109 90

+236 -103

drivers/nvme/host/core.c

reviewed

··· 76 76 77 77 static struct class *nvme_class; 78 78 79 79 + static __le32 nvme_get_log_dw10(u8 lid, size_t size) 80 80 + { 81 81 + return cpu_to_le32((((size / 4) - 1) << 16) | lid); 82 82 + } 83 83 + 79 84 int nvme_reset_ctrl(struct nvme_ctrl *ctrl) 80 85 { 81 86 if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ··· 113 108 case NVME_SC_WRITE_FAULT: 114 109 case NVME_SC_READ_ERROR: 115 110 case NVME_SC_UNWRITTEN_BLOCK: 111 111 + case NVME_SC_ACCESS_DENIED: 112 112 + case NVME_SC_READ_ONLY: 116 113 return BLK_STS_MEDIUM; 114 114 + case NVME_SC_GUARD_CHECK: 115 115 + case NVME_SC_APPTAG_CHECK: 116 116 + case NVME_SC_REFTAG_CHECK: 117 117 + case NVME_SC_INVALID_PI: 118 118 + return BLK_STS_PROTECTION; 119 119 + case NVME_SC_RESERVATION_CONFLICT: 120 120 + return BLK_STS_NEXUS; 117 121 default: 118 122 return BLK_STS_IOERR; 119 123 } ··· 176 162 enum nvme_ctrl_state new_state) 177 163 { 178 164 enum nvme_ctrl_state old_state; 165 165 + unsigned long flags; 179 166 bool changed = false; 180 167 181 181 - spin_lock_irq(&ctrl->lock); 168 168 + spin_lock_irqsave(&ctrl->lock, flags); 182 169 183 170 old_state = ctrl->state; 184 171 switch (new_state) { ··· 240 225 if (changed) 241 226 ctrl->state = new_state; 242 227 243 243 - spin_unlock_irq(&ctrl->lock); 228 228 + spin_unlock_irqrestore(&ctrl->lock, flags); 244 229 245 230 return changed; 246 231 } ··· 322 307 memset(&c, 0, sizeof(c)); 323 308 324 309 c.directive.opcode = nvme_admin_directive_send; 325 325 - c.directive.nsid = cpu_to_le32(0xffffffff); 310 310 + c.directive.nsid = cpu_to_le32(NVME_NSID_ALL); 326 311 c.directive.doper = NVME_DIR_SND_ID_OP_ENABLE; 327 312 c.directive.dtype = NVME_DIR_IDENTIFY; 328 313 c.directive.tdtype = NVME_DIR_STREAMS; ··· 372 357 if (ret) 373 358 return ret; 374 359 375 375 - ret = nvme_get_stream_params(ctrl, &s, 0xffffffff); 360 360 + ret = nvme_get_stream_params(ctrl, &s, NVME_NSID_ALL); 376 361 if (ret) 377 362 return ret; 378 363 ··· 600 585 } 601 586 EXPORT_SYMBOL_GPL(nvme_submit_sync_cmd); 602 587 603 603 - int __nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, 604 604 - void __user *ubuffer, unsigned bufflen, 605 605 - void __user *meta_buffer, unsigned meta_len, u32 meta_seed, 606 606 - u32 *result, unsigned timeout) 588 588 + static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf, 589 589 + unsigned len, u32 seed, bool write) 590 590 + { 591 591 + struct bio_integrity_payload *bip; 592 592 + int ret = -ENOMEM; 593 593 + void *buf; 594 594 + 595 595 + buf = kmalloc(len, GFP_KERNEL); 596 596 + if (!buf) 597 597 + goto out; 598 598 + 599 599 + ret = -EFAULT; 600 600 + if (write && copy_from_user(buf, ubuf, len)) 601 601 + goto out_free_meta; 602 602 + 603 603 + bip = bio_integrity_alloc(bio, GFP_KERNEL, 1); 604 604 + if (IS_ERR(bip)) { 605 605 + ret = PTR_ERR(bip); 606 606 + goto out_free_meta; 607 607 + } 608 608 + 609 609 + bip->bip_iter.bi_size = len; 610 610 + bip->bip_iter.bi_sector = seed; 611 611 + ret = bio_integrity_add_page(bio, virt_to_page(buf), len, 612 612 + offset_in_page(buf)); 613 613 + if (ret == len) 614 614 + return buf; 615 615 + ret = -ENOMEM; 616 616 + out_free_meta: 617 617 + kfree(buf); 618 618 + out: 619 619 + return ERR_PTR(ret); 620 620 + } 621 621 + 622 622 + static int nvme_submit_user_cmd(struct request_queue *q, 623 623 + struct nvme_command *cmd, void __user *ubuffer, 624 624 + unsigned bufflen, void __user *meta_buffer, unsigned meta_len, 625 625 + u32 meta_seed, u32 *result, unsigned timeout) 607 626 { 608 627 bool write = nvme_is_write(cmd); 609 628 struct nvme_ns *ns = q->queuedata; ··· 659 610 if (ret) 660 611 goto out; 661 612 bio = req->bio; 662 662 - 663 663 - if (!disk) 664 664 - goto submit; 665 613 bio->bi_disk = disk; 666 666 - 667 667 - if (meta_buffer && meta_len) { 668 668 - struct bio_integrity_payload *bip; 669 669 - 670 670 - meta = kmalloc(meta_len, GFP_KERNEL); 671 671 - if (!meta) { 672 672 - ret = -ENOMEM; 614 614 + if (disk && meta_buffer && meta_len) { 615 615 + meta = nvme_add_user_metadata(bio, meta_buffer, meta_len, 616 616 + meta_seed, write); 617 617 + if (IS_ERR(meta)) { 618 618 + ret = PTR_ERR(meta); 673 619 goto out_unmap; 674 674 - } 675 675 - 676 676 - if (write) { 677 677 - if (copy_from_user(meta, meta_buffer, 678 678 - meta_len)) { 679 679 - ret = -EFAULT; 680 680 - goto out_free_meta; 681 681 - } 682 682 - } 683 683 - 684 684 - bip = bio_integrity_alloc(bio, GFP_KERNEL, 1); 685 685 - if (IS_ERR(bip)) { 686 686 - ret = PTR_ERR(bip); 687 687 - goto out_free_meta; 688 688 - } 689 689 - 690 690 - bip->bip_iter.bi_size = meta_len; 691 691 - bip->bip_iter.bi_sector = meta_seed; 692 692 - 693 693 - ret = bio_integrity_add_page(bio, virt_to_page(meta), 694 694 - meta_len, offset_in_page(meta)); 695 695 - if (ret != meta_len) { 696 696 - ret = -ENOMEM; 697 697 - goto out_free_meta; 698 620 } 699 621 } 700 622 } 701 701 - submit: 623 623 + 702 624 blk_execute_rq(req->q, disk, req, 0); 703 625 if (nvme_req(req)->flags & NVME_REQ_CANCELLED) 704 626 ret = -EINTR; ··· 681 661 if (copy_to_user(meta_buffer, meta, meta_len)) 682 662 ret = -EFAULT; 683 663 } 684 684 - out_free_meta: 685 664 kfree(meta); 686 665 out_unmap: 687 666 if (bio) ··· 688 669 out: 689 670 blk_mq_free_request(req); 690 671 return ret; 691 691 - } 692 692 - 693 693 - int nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, 694 694 - void __user *ubuffer, unsigned bufflen, u32 *result, 695 695 - unsigned timeout) 696 696 - { 697 697 - return __nvme_submit_user_cmd(q, cmd, ubuffer, bufflen, NULL, 0, 0, 698 698 - result, timeout); 699 672 } 700 673 701 674 static void nvme_keep_alive_end_io(struct request *rq, blk_status_t status) ··· 779 768 return error; 780 769 } 781 770 782 782 - static int nvme_identify_ns_descs(struct nvme_ns *ns, unsigned nsid) 771 771 + static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid, 772 772 + u8 *eui64, u8 *nguid, uuid_t *uuid) 783 773 { 784 774 struct nvme_command c = { }; 785 775 int status; ··· 796 784 if (!data) 797 785 return -ENOMEM; 798 786 799 799 - status = nvme_submit_sync_cmd(ns->ctrl->admin_q, &c, data, 787 787 + status = nvme_submit_sync_cmd(ctrl->admin_q, &c, data, 800 788 NVME_IDENTIFY_DATA_SIZE); 801 789 if (status) 802 790 goto free_data; ··· 810 798 switch (cur->nidt) { 811 799 case NVME_NIDT_EUI64: 812 800 if (cur->nidl != NVME_NIDT_EUI64_LEN) { 813 813 - dev_warn(ns->ctrl->device, 801 801 + dev_warn(ctrl->device, 814 802 "ctrl returned bogus length: %d for NVME_NIDT_EUI64\n", 815 803 cur->nidl); 816 804 goto free_data; 817 805 } 818 806 len = NVME_NIDT_EUI64_LEN; 819 819 - memcpy(ns->eui, data + pos + sizeof(*cur), len); 807 807 + memcpy(eui64, data + pos + sizeof(*cur), len); 820 808 break; 821 809 case NVME_NIDT_NGUID: 822 810 if (cur->nidl != NVME_NIDT_NGUID_LEN) { 823 823 - dev_warn(ns->ctrl->device, 811 811 + dev_warn(ctrl->device, 824 812 "ctrl returned bogus length: %d for NVME_NIDT_NGUID\n", 825 813 cur->nidl); 826 814 goto free_data; 827 815 } 828 816 len = NVME_NIDT_NGUID_LEN; 829 829 - memcpy(ns->nguid, data + pos + sizeof(*cur), len); 817 817 + memcpy(nguid, data + pos + sizeof(*cur), len); 830 818 break; 831 819 case NVME_NIDT_UUID: 832 820 if (cur->nidl != NVME_NIDT_UUID_LEN) { 833 833 - dev_warn(ns->ctrl->device, 821 821 + dev_warn(ctrl->device, 834 822 "ctrl returned bogus length: %d for NVME_NIDT_UUID\n", 835 823 cur->nidl); 836 824 goto free_data; 837 825 } 838 826 len = NVME_NIDT_UUID_LEN; 839 839 - uuid_copy(&ns->uuid, data + pos + sizeof(*cur)); 827 827 + uuid_copy(uuid, data + pos + sizeof(*cur)); 840 828 break; 841 829 default: 842 830 /* Skip unnkown types */ ··· 861 849 return nvme_submit_sync_cmd(dev->admin_q, &c, ns_list, 0x1000); 862 850 } 863 851 864 864 - static int nvme_identify_ns(struct nvme_ctrl *dev, unsigned nsid, 865 865 - struct nvme_id_ns **id) 852 852 + static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, 853 853 + unsigned nsid) 866 854 { 855 855 + struct nvme_id_ns *id; 867 856 struct nvme_command c = { }; 868 857 int error; 869 858 ··· 873 860 c.identify.nsid = cpu_to_le32(nsid); 874 861 c.identify.cns = NVME_ID_CNS_NS; 875 862 876 876 - *id = kmalloc(sizeof(struct nvme_id_ns), GFP_KERNEL); 877 877 - if (!*id) 878 878 - return -ENOMEM; 863 863 + id = kmalloc(sizeof(*id), GFP_KERNEL); 864 864 + if (!id) 865 865 + return NULL; 879 866 880 880 - error = nvme_submit_sync_cmd(dev->admin_q, &c, *id, 881 881 - sizeof(struct nvme_id_ns)); 882 882 - if (error) 883 883 - kfree(*id); 884 884 - return error; 867 867 + error = nvme_submit_sync_cmd(ctrl->admin_q, &c, id, sizeof(*id)); 868 868 + if (error) { 869 869 + dev_warn(ctrl->device, "Identify namespace failed\n"); 870 870 + kfree(id); 871 871 + return NULL; 872 872 + } 873 873 + 874 874 + return id; 885 875 } 886 876 887 877 static int nvme_set_features(struct nvme_ctrl *dev, unsigned fid, unsigned dword11, ··· 979 963 c.rw.apptag = cpu_to_le16(io.apptag); 980 964 c.rw.appmask = cpu_to_le16(io.appmask); 981 965 982 982 - return __nvme_submit_user_cmd(ns->queue, &c, 966 966 + return nvme_submit_user_cmd(ns->queue, &c, 983 967 (void __user *)(uintptr_t)io.addr, length, 984 968 metadata, meta_len, io.slba, NULL, 0); 985 969 } ··· 1017 1001 1018 1002 status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, 1019 1003 (void __user *)(uintptr_t)cmd.addr, cmd.data_len, 1020 1020 - &cmd.result, timeout); 1004 1004 + (void __user *)(uintptr_t)cmd.metadata, cmd.metadata, 1005 1005 + 0, &cmd.result, timeout); 1021 1006 if (status >= 0) { 1022 1007 if (put_user(cmd.result, &ucmd->result)) 1023 1008 return -EFAULT; ··· 1176 1159 blk_queue_max_write_zeroes_sectors(ns->queue, UINT_MAX); 1177 1160 } 1178 1161 1179 1179 - static int nvme_revalidate_ns(struct nvme_ns *ns, struct nvme_id_ns **id) 1162 1162 + static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid, 1163 1163 + struct nvme_id_ns *id, u8 *eui64, u8 *nguid, uuid_t *uuid) 1180 1164 { 1181 1181 - if (nvme_identify_ns(ns->ctrl, ns->ns_id, id)) { 1182 1182 - dev_warn(ns->ctrl->dev, "%s: Identify failure\n", __func__); 1183 1183 - return -ENODEV; 1184 1184 - } 1185 1185 - 1186 1186 - if ((*id)->ncap == 0) { 1187 1187 - kfree(*id); 1188 1188 - return -ENODEV; 1189 1189 - } 1190 1190 - 1191 1191 - if (ns->ctrl->vs >= NVME_VS(1, 1, 0)) 1192 1192 - memcpy(ns->eui, (*id)->eui64, sizeof(ns->eui)); 1193 1193 - if (ns->ctrl->vs >= NVME_VS(1, 2, 0)) 1194 1194 - memcpy(ns->nguid, (*id)->nguid, sizeof(ns->nguid)); 1195 1195 - if (ns->ctrl->vs >= NVME_VS(1, 3, 0)) { 1165 1165 + if (ctrl->vs >= NVME_VS(1, 1, 0)) 1166 1166 + memcpy(eui64, id->eui64, sizeof(id->eui64)); 1167 1167 + if (ctrl->vs >= NVME_VS(1, 2, 0)) 1168 1168 + memcpy(nguid, id->nguid, sizeof(id->nguid)); 1169 1169 + if (ctrl->vs >= NVME_VS(1, 3, 0)) { 1196 1170 /* Don't treat error as fatal we potentially 1197 1171 * already have a NGUID or EUI-64 1198 1172 */ 1199 1199 - if (nvme_identify_ns_descs(ns, ns->ns_id)) 1200 1200 - dev_warn(ns->ctrl->device, 1173 1173 + if (nvme_identify_ns_descs(ctrl, nsid, eui64, nguid, uuid)) 1174 1174 + dev_warn(ctrl->device, 1201 1175 "%s: Identify Descriptors failed\n", __func__); 1202 1176 } 1203 1203 - 1204 1204 - return 0; 1205 1177 } 1206 1178 1207 1179 static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id) ··· 1231 1225 static int nvme_revalidate_disk(struct gendisk *disk) 1232 1226 { 1233 1227 struct nvme_ns *ns = disk->private_data; 1234 1234 - struct nvme_id_ns *id = NULL; 1235 1235 - int ret; 1228 1228 + struct nvme_ctrl *ctrl = ns->ctrl; 1229 1229 + struct nvme_id_ns *id; 1230 1230 + u8 eui64[8] = { 0 }, nguid[16] = { 0 }; 1231 1231 + uuid_t uuid = uuid_null; 1232 1232 + int ret = 0; 1236 1233 1237 1234 if (test_bit(NVME_NS_DEAD, &ns->flags)) { 1238 1235 set_capacity(disk, 0); 1239 1236 return -ENODEV; 1240 1237 } 1241 1238 1242 1242 - ret = nvme_revalidate_ns(ns, &id); 1243 1243 - if (ret) 1244 1244 - return ret; 1239 1239 + id = nvme_identify_ns(ctrl, ns->ns_id); 1240 1240 + if (!id) 1241 1241 + return -ENODEV; 1245 1242 1246 1246 - __nvme_revalidate_disk(disk, id); 1243 1243 + if (id->ncap == 0) { 1244 1244 + ret = -ENODEV; 1245 1245 + goto out; 1246 1246 + } 1247 1247 + 1248 1248 + nvme_report_ns_ids(ctrl, ns->ns_id, id, eui64, nguid, &uuid); 1249 1249 + if (!uuid_equal(&ns->uuid, &uuid) || 1250 1250 + memcmp(&ns->nguid, &nguid, sizeof(ns->nguid)) || 1251 1251 + memcmp(&ns->eui, &eui64, sizeof(ns->eui))) { 1252 1252 + dev_err(ctrl->device, 1253 1253 + "identifiers changed for nsid %d\n", ns->ns_id); 1254 1254 + ret = -ENODEV; 1255 1255 + } 1256 1256 + 1257 1257 + out: 1247 1258 kfree(id); 1248 1248 - 1249 1249 - return 0; 1259 1259 + return ret; 1250 1260 } 1251 1261 1252 1262 static char nvme_pr_type(enum pr_type type) ··· 1462 1440 1463 1441 ctrl->ctrl_config = NVME_CC_CSS_NVM; 1464 1442 ctrl->ctrl_config |= (page_shift - 12) << NVME_CC_MPS_SHIFT; 1465 1465 - ctrl->ctrl_config |= NVME_CC_ARB_RR | NVME_CC_SHN_NONE; 1443 1443 + ctrl->ctrl_config |= NVME_CC_AMS_RR | NVME_CC_SHN_NONE; 1466 1444 ctrl->ctrl_config |= NVME_CC_IOSQES | NVME_CC_IOCQES; 1467 1445 ctrl->ctrl_config |= NVME_CC_ENABLE; 1468 1446 ··· 1475 1453 1476 1454 int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl) 1477 1455 { 1478 1478 - unsigned long timeout = jiffies + (shutdown_timeout * HZ); 1456 1456 + unsigned long timeout = jiffies + (ctrl->shutdown_timeout * HZ); 1479 1457 u32 csts; 1480 1458 int ret; 1481 1459 ··· 1522 1500 if (ctrl->vwc & NVME_CTRL_VWC_PRESENT) 1523 1501 vwc = true; 1524 1502 blk_queue_write_cache(q, vwc, vwc); 1503 1503 + } 1504 1504 + 1505 1505 + static int nvme_configure_timestamp(struct nvme_ctrl *ctrl) 1506 1506 + { 1507 1507 + __le64 ts; 1508 1508 + int ret; 1509 1509 + 1510 1510 + if (!(ctrl->oncs & NVME_CTRL_ONCS_TIMESTAMP)) 1511 1511 + return 0; 1512 1512 + 1513 1513 + ts = cpu_to_le64(ktime_to_ms(ktime_get_real())); 1514 1514 + ret = nvme_set_features(ctrl, NVME_FEAT_TIMESTAMP, 0, &ts, sizeof(ts), 1515 1515 + NULL); 1516 1516 + if (ret) 1517 1517 + dev_warn_once(ctrl->device, 1518 1518 + "could not set timestamp (%d)\n", ret); 1519 1519 + return ret; 1525 1520 } 1526 1521 1527 1522 static int nvme_configure_apst(struct nvme_ctrl *ctrl) ··· 1843 1804 ctrl->sgls = le32_to_cpu(id->sgls); 1844 1805 ctrl->kas = le16_to_cpu(id->kas); 1845 1806 1807 1807 + if (id->rtd3e) { 1808 1808 + /* us -> s */ 1809 1809 + u32 transition_time = le32_to_cpu(id->rtd3e) / 1000000; 1810 1810 + 1811 1811 + ctrl->shutdown_timeout = clamp_t(unsigned int, transition_time, 1812 1812 + shutdown_timeout, 60); 1813 1813 + 1814 1814 + if (ctrl->shutdown_timeout != shutdown_timeout) 1815 1815 + dev_warn(ctrl->device, 1816 1816 + "Shutdown timeout set to %u seconds\n", 1817 1817 + ctrl->shutdown_timeout); 1818 1818 + } else 1819 1819 + ctrl->shutdown_timeout = shutdown_timeout; 1820 1820 + 1846 1821 ctrl->npss = id->npss; 1847 1822 ctrl->apsta = id->apsta; 1848 1823 prev_apst_enabled = ctrl->apst_enabled; ··· 1907 1854 dev_pm_qos_hide_latency_tolerance(ctrl->device); 1908 1855 1909 1856 ret = nvme_configure_apst(ctrl); 1857 1857 + if (ret < 0) 1858 1858 + return ret; 1859 1859 + 1860 1860 + ret = nvme_configure_timestamp(ctrl); 1910 1861 if (ret < 0) 1911 1862 return ret; 1912 1863 ··· 2368 2311 2369 2312 sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->instance); 2370 2313 2371 2371 - if (nvme_revalidate_ns(ns, &id)) 2314 2314 + id = nvme_identify_ns(ctrl, nsid); 2315 2315 + if (!id) 2372 2316 goto out_free_queue; 2317 2317 + 2318 2318 + if (id->ncap == 0) 2319 2319 + goto out_free_id; 2320 2320 + 2321 2321 + nvme_report_ns_ids(ctrl, ns->ns_id, id, ns->eui, ns->nguid, &ns->uuid); 2373 2322 2374 2323 if (nvme_nvm_ns_supported(ns, id) && 2375 2324 nvme_nvm_register(ns, disk_name, node)) { ··· 2597 2534 spin_unlock_irq(&ctrl->lock); 2598 2535 } 2599 2536 2537 2537 + static bool nvme_ctrl_pp_status(struct nvme_ctrl *ctrl) 2538 2538 + { 2539 2539 + 2540 2540 + u32 csts; 2541 2541 + 2542 2542 + if (ctrl->ops->reg_read32(ctrl, NVME_REG_CSTS, &csts)) 2543 2543 + return false; 2544 2544 + 2545 2545 + if (csts == ~0) 2546 2546 + return false; 2547 2547 + 2548 2548 + return ((ctrl->ctrl_config & NVME_CC_ENABLE) && (csts & NVME_CSTS_PP)); 2549 2549 + } 2550 2550 + 2551 2551 + static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) 2552 2552 + { 2553 2553 + struct nvme_command c = { }; 2554 2554 + struct nvme_fw_slot_info_log *log; 2555 2555 + 2556 2556 + log = kmalloc(sizeof(*log), GFP_KERNEL); 2557 2557 + if (!log) 2558 2558 + return; 2559 2559 + 2560 2560 + c.common.opcode = nvme_admin_get_log_page; 2561 2561 + c.common.nsid = cpu_to_le32(NVME_NSID_ALL); 2562 2562 + c.common.cdw10[0] = nvme_get_log_dw10(NVME_LOG_FW_SLOT, sizeof(*log)); 2563 2563 + 2564 2564 + if (!nvme_submit_sync_cmd(ctrl->admin_q, &c, log, sizeof(*log))) 2565 2565 + dev_warn(ctrl->device, 2566 2566 + "Get FW SLOT INFO log error\n"); 2567 2567 + kfree(log); 2568 2568 + } 2569 2569 + 2570 2570 + static void nvme_fw_act_work(struct work_struct *work) 2571 2571 + { 2572 2572 + struct nvme_ctrl *ctrl = container_of(work, 2573 2573 + struct nvme_ctrl, fw_act_work); 2574 2574 + unsigned long fw_act_timeout; 2575 2575 + 2576 2576 + if (ctrl->mtfa) 2577 2577 + fw_act_timeout = jiffies + 2578 2578 + msecs_to_jiffies(ctrl->mtfa * 100); 2579 2579 + else 2580 2580 + fw_act_timeout = jiffies + 2581 2581 + msecs_to_jiffies(admin_timeout * 1000); 2582 2582 + 2583 2583 + nvme_stop_queues(ctrl); 2584 2584 + while (nvme_ctrl_pp_status(ctrl)) { 2585 2585 + if (time_after(jiffies, fw_act_timeout)) { 2586 2586 + dev_warn(ctrl->device, 2587 2587 + "Fw activation timeout, reset controller\n"); 2588 2588 + nvme_reset_ctrl(ctrl); 2589 2589 + break; 2590 2590 + } 2591 2591 + msleep(100); 2592 2592 + } 2593 2593 + 2594 2594 + if (ctrl->state != NVME_CTRL_LIVE) 2595 2595 + return; 2596 2596 + 2597 2597 + nvme_start_queues(ctrl); 2598 2598 + /* read FW slot informationi to clear the AER*/ 2599 2599 + nvme_get_fw_slot_info(ctrl); 2600 2600 + } 2601 2601 + 2600 2602 void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status, 2601 2603 union nvme_result *res) 2602 2604 { ··· 2687 2559 case NVME_AER_NOTICE_NS_CHANGED: 2688 2560 dev_info(ctrl->device, "rescanning\n"); 2689 2561 nvme_queue_scan(ctrl); 2562 2562 + break; 2563 2563 + case NVME_AER_NOTICE_FW_ACT_STARTING: 2564 2564 + schedule_work(&ctrl->fw_act_work); 2690 2565 break; 2691 2566 default: 2692 2567 dev_warn(ctrl->device, "async event result %08x\n", result); ··· 2738 2607 nvme_stop_keep_alive(ctrl); 2739 2608 flush_work(&ctrl->async_event_work); 2740 2609 flush_work(&ctrl->scan_work); 2610 2610 + cancel_work_sync(&ctrl->fw_act_work); 2741 2611 } 2742 2612 EXPORT_SYMBOL_GPL(nvme_stop_ctrl); 2743 2613 ··· 2802 2670 ctrl->quirks = quirks; 2803 2671 INIT_WORK(&ctrl->scan_work, nvme_scan_work); 2804 2672 INIT_WORK(&ctrl->async_event_work, nvme_async_event_work); 2673 2673 + INIT_WORK(&ctrl->fw_act_work, nvme_fw_act_work); 2805 2674 2806 2675 ret = nvme_set_instance(ctrl); 2807 2676 if (ret)

+12 -11

drivers/nvme/host/fabrics.c

reviewed

··· 22 22 #include "fabrics.h" 23 23 24 24 static LIST_HEAD(nvmf_transports); 25 25 - static DEFINE_MUTEX(nvmf_transports_mutex); 25 25 + static DECLARE_RWSEM(nvmf_transports_rwsem); 26 26 27 27 static LIST_HEAD(nvmf_hosts); 28 28 static DEFINE_MUTEX(nvmf_hosts_mutex); ··· 75 75 76 76 kref_init(&host->ref); 77 77 snprintf(host->nqn, NVMF_NQN_SIZE, 78 78 - "nqn.2014-08.org.nvmexpress:NVMf:uuid:%pUb", &host->id); 78 78 + "nqn.2014-08.org.nvmexpress:uuid:%pUb", &host->id); 79 79 80 80 mutex_lock(&nvmf_hosts_mutex); 81 81 list_add_tail(&host->list, &nvmf_hosts); ··· 495 495 if (!ops->create_ctrl) 496 496 return -EINVAL; 497 497 498 498 - mutex_lock(&nvmf_transports_mutex); 498 498 + down_write(&nvmf_transports_rwsem); 499 499 list_add_tail(&ops->entry, &nvmf_transports); 500 500 - mutex_unlock(&nvmf_transports_mutex); 500 500 + up_write(&nvmf_transports_rwsem); 501 501 502 502 return 0; 503 503 } ··· 514 514 */ 515 515 void nvmf_unregister_transport(struct nvmf_transport_ops *ops) 516 516 { 517 517 - mutex_lock(&nvmf_transports_mutex); 517 517 + down_write(&nvmf_transports_rwsem); 518 518 list_del(&ops->entry); 519 519 - mutex_unlock(&nvmf_transports_mutex); 519 519 + up_write(&nvmf_transports_rwsem); 520 520 } 521 521 EXPORT_SYMBOL_GPL(nvmf_unregister_transport); 522 522 ··· 525 525 { 526 526 struct nvmf_transport_ops *ops; 527 527 528 528 - lockdep_assert_held(&nvmf_transports_mutex); 528 528 + lockdep_assert_held(&nvmf_transports_rwsem); 529 529 530 530 list_for_each_entry(ops, &nvmf_transports, entry) { 531 531 if (strcmp(ops->name, opts->transport) == 0) ··· 735 735 goto out; 736 736 } 737 737 if (uuid_parse(p, &hostid)) { 738 738 + pr_err("Invalid hostid %s\n", p); 738 739 ret = -EINVAL; 739 740 goto out; 740 741 } ··· 851 850 goto out_free_opts; 852 851 opts->mask &= ~NVMF_REQUIRED_OPTS; 853 852 854 854 - mutex_lock(&nvmf_transports_mutex); 853 853 + down_read(&nvmf_transports_rwsem); 855 854 ops = nvmf_lookup_transport(opts); 856 855 if (!ops) { 857 856 pr_info("no handler found for transport %s.\n", ··· 878 877 dev_warn(ctrl->device, 879 878 "controller returned incorrect NQN: \"%s\".\n", 880 879 ctrl->subnqn); 881 881 - mutex_unlock(&nvmf_transports_mutex); 880 880 + up_read(&nvmf_transports_rwsem); 882 881 ctrl->ops->delete_ctrl(ctrl); 883 882 return ERR_PTR(-EINVAL); 884 883 } 885 884 886 886 - mutex_unlock(&nvmf_transports_mutex); 885 885 + up_read(&nvmf_transports_rwsem); 887 886 return ctrl; 888 887 889 888 out_unlock: 890 890 - mutex_unlock(&nvmf_transports_mutex); 889 889 + up_read(&nvmf_transports_rwsem); 891 890 out_free_opts: 892 891 nvmf_free_options(opts); 893 892 return ERR_PTR(ret);

+107 -38

drivers/nvme/host/fc.c

reviewed

··· 220 220 static void __nvme_fc_delete_hw_queue(struct nvme_fc_ctrl *, 221 221 struct nvme_fc_queue *, unsigned int); 222 222 223 223 + static void 224 224 + nvme_fc_free_lport(struct kref *ref) 225 225 + { 226 226 + struct nvme_fc_lport *lport = 227 227 + container_of(ref, struct nvme_fc_lport, ref); 228 228 + unsigned long flags; 229 229 + 230 230 + WARN_ON(lport->localport.port_state != FC_OBJSTATE_DELETED); 231 231 + WARN_ON(!list_empty(&lport->endp_list)); 232 232 + 233 233 + /* remove from transport list */ 234 234 + spin_lock_irqsave(&nvme_fc_lock, flags); 235 235 + list_del(&lport->port_list); 236 236 + spin_unlock_irqrestore(&nvme_fc_lock, flags); 237 237 + 238 238 + /* let the LLDD know we've finished tearing it down */ 239 239 + lport->ops->localport_delete(&lport->localport); 240 240 + 241 241 + ida_simple_remove(&nvme_fc_local_port_cnt, lport->localport.port_num); 242 242 + ida_destroy(&lport->endp_cnt); 243 243 + 244 244 + put_device(lport->dev); 245 245 + 246 246 + kfree(lport); 247 247 + } 248 248 + 249 249 + static void 250 250 + nvme_fc_lport_put(struct nvme_fc_lport *lport) 251 251 + { 252 252 + kref_put(&lport->ref, nvme_fc_free_lport); 253 253 + } 254 254 + 255 255 + static int 256 256 + nvme_fc_lport_get(struct nvme_fc_lport *lport) 257 257 + { 258 258 + return kref_get_unless_zero(&lport->ref); 259 259 + } 260 260 + 261 261 + 262 262 + static struct nvme_fc_lport * 263 263 + nvme_fc_attach_to_unreg_lport(struct nvme_fc_port_info *pinfo) 264 264 + { 265 265 + struct nvme_fc_lport *lport; 266 266 + unsigned long flags; 267 267 + 268 268 + spin_lock_irqsave(&nvme_fc_lock, flags); 269 269 + 270 270 + list_for_each_entry(lport, &nvme_fc_lport_list, port_list) { 271 271 + if (lport->localport.node_name != pinfo->node_name || 272 272 + lport->localport.port_name != pinfo->port_name) 273 273 + continue; 274 274 + 275 275 + if (lport->localport.port_state != FC_OBJSTATE_DELETED) { 276 276 + lport = ERR_PTR(-EEXIST); 277 277 + goto out_done; 278 278 + } 279 279 + 280 280 + if (!nvme_fc_lport_get(lport)) { 281 281 + /* 282 282 + * fails if ref cnt already 0. If so, 283 283 + * act as if lport already deleted 284 284 + */ 285 285 + lport = NULL; 286 286 + goto out_done; 287 287 + } 288 288 + 289 289 + /* resume the lport */ 290 290 + 291 291 + lport->localport.port_role = pinfo->port_role; 292 292 + lport->localport.port_id = pinfo->port_id; 293 293 + lport->localport.port_state = FC_OBJSTATE_ONLINE; 294 294 + 295 295 + spin_unlock_irqrestore(&nvme_fc_lock, flags); 296 296 + 297 297 + return lport; 298 298 + } 299 299 + 300 300 + lport = NULL; 301 301 + 302 302 + out_done: 303 303 + spin_unlock_irqrestore(&nvme_fc_lock, flags); 304 304 + 305 305 + return lport; 306 306 + } 223 307 224 308 /** 225 309 * nvme_fc_register_localport - transport entry point called by an ··· 340 256 ret = -EINVAL; 341 257 goto out_reghost_failed; 342 258 } 259 259 + 260 260 + /* 261 261 + * look to see if there is already a localport that had been 262 262 + * deregistered and in the process of waiting for all the 263 263 + * references to fully be removed. If the references haven't 264 264 + * expired, we can simply re-enable the localport. Remoteports 265 265 + * and controller reconnections should resume naturally. 266 266 + */ 267 267 + newrec = nvme_fc_attach_to_unreg_lport(pinfo); 268 268 + 269 269 + /* found an lport, but something about its state is bad */ 270 270 + if (IS_ERR(newrec)) { 271 271 + ret = PTR_ERR(newrec); 272 272 + goto out_reghost_failed; 273 273 + 274 274 + /* found existing lport, which was resumed */ 275 275 + } else if (newrec) { 276 276 + *portptr = &newrec->localport; 277 277 + return 0; 278 278 + } 279 279 + 280 280 + /* nothing found - allocate a new localport struct */ 343 281 344 282 newrec = kmalloc((sizeof(*newrec) + template->local_priv_sz), 345 283 GFP_KERNEL); ··· 415 309 return ret; 416 310 } 417 311 EXPORT_SYMBOL_GPL(nvme_fc_register_localport); 418 418 - 419 419 - static void 420 420 - nvme_fc_free_lport(struct kref *ref) 421 421 - { 422 422 - struct nvme_fc_lport *lport = 423 423 - container_of(ref, struct nvme_fc_lport, ref); 424 424 - unsigned long flags; 425 425 - 426 426 - WARN_ON(lport->localport.port_state != FC_OBJSTATE_DELETED); 427 427 - WARN_ON(!list_empty(&lport->endp_list)); 428 428 - 429 429 - /* remove from transport list */ 430 430 - spin_lock_irqsave(&nvme_fc_lock, flags); 431 431 - list_del(&lport->port_list); 432 432 - spin_unlock_irqrestore(&nvme_fc_lock, flags); 433 433 - 434 434 - /* let the LLDD know we've finished tearing it down */ 435 435 - lport->ops->localport_delete(&lport->localport); 436 436 - 437 437 - ida_simple_remove(&nvme_fc_local_port_cnt, lport->localport.port_num); 438 438 - ida_destroy(&lport->endp_cnt); 439 439 - 440 440 - put_device(lport->dev); 441 441 - 442 442 - kfree(lport); 443 443 - } 444 444 - 445 445 - static void 446 446 - nvme_fc_lport_put(struct nvme_fc_lport *lport) 447 447 - { 448 448 - kref_put(&lport->ref, nvme_fc_free_lport); 449 449 - } 450 450 - 451 451 - static int 452 452 - nvme_fc_lport_get(struct nvme_fc_lport *lport) 453 453 - { 454 454 - return kref_get_unless_zero(&lport->ref); 455 455 - } 456 312 457 313 /** 458 314 * nvme_fc_unregister_localport - transport entry point called by an ··· 2799 2731 ret = blk_mq_alloc_tag_set(&ctrl->admin_tag_set); 2800 2732 if (ret) 2801 2733 goto out_free_queues; 2734 2734 + ctrl->ctrl.admin_tagset = &ctrl->admin_tag_set; 2802 2735 2803 2736 ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set); 2804 2737 if (IS_ERR(ctrl->ctrl.admin_q)) {

+5 -12

drivers/nvme/host/nvme.h

reviewed

··· 125 125 struct kref kref; 126 126 int instance; 127 127 struct blk_mq_tag_set *tagset; 128 128 + struct blk_mq_tag_set *admin_tagset; 128 129 struct list_head namespaces; 129 130 struct mutex namespaces_mutex; 130 131 struct device *device; /* char device */ ··· 143 142 u16 cntlid; 144 143 145 144 u32 ctrl_config; 145 145 + u16 mtfa; 146 146 u32 queue_count; 147 147 148 148 u64 cap; ··· 162 160 u16 kas; 163 161 u8 npss; 164 162 u8 apsta; 163 163 + unsigned int shutdown_timeout; 165 164 unsigned int kato; 166 165 bool subsystem; 167 166 unsigned long quirks; ··· 170 167 struct work_struct scan_work; 171 168 struct work_struct async_event_work; 172 169 struct delayed_work ka_work; 170 170 + struct work_struct fw_act_work; 173 171 174 172 /* Power saving configuration */ 175 173 u64 ps_max_latency_us; ··· 211 207 bool ext; 212 208 u8 pi_type; 213 209 unsigned long flags; 214 214 - u16 noiob; 215 215 - 216 210 #define NVME_NS_REMOVING 0 217 211 #define NVME_NS_DEAD 1 218 218 - 219 219 - u64 mode_select_num_blocks; 220 220 - u32 mode_select_block_len; 212 212 + u16 noiob; 221 213 }; 222 214 223 215 struct nvme_ctrl_ops { ··· 314 314 int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, 315 315 union nvme_result *result, void *buffer, unsigned bufflen, 316 316 unsigned timeout, int qid, int at_head, int flags); 317 317 - int nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, 318 318 - void __user *ubuffer, unsigned bufflen, u32 *result, 319 319 - unsigned timeout); 320 320 - int __nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, 321 321 - void __user *ubuffer, unsigned bufflen, 322 322 - void __user *meta_buffer, unsigned meta_len, u32 meta_seed, 323 323 - u32 *result, unsigned timeout); 324 317 int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count); 325 318 void nvme_start_keep_alive(struct nvme_ctrl *ctrl); 326 319 void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);

+6 -3

drivers/nvme/host/pci.c

reviewed

··· 556 556 int nprps, i; 557 557 558 558 length -= (page_size - offset); 559 559 - if (length <= 0) 559 559 + if (length <= 0) { 560 560 + iod->first_dma = 0; 560 561 return BLK_STS_OK; 562 562 + } 561 563 562 564 dma_len -= (page_size - offset); 563 565 if (dma_len) { ··· 669 667 if (blk_rq_map_integrity_sg(q, req->bio, &iod->meta_sg) != 1) 670 668 goto out_unmap; 671 669 672 672 - if (rq_data_dir(req)) 670 670 + if (req_op(req) == REQ_OP_WRITE) 673 671 nvme_dif_remap(req, nvme_dif_prep); 674 672 675 673 if (!dma_map_sg(dev->dev, &iod->meta_sg, 1, dma_dir)) ··· 697 695 if (iod->nents) { 698 696 dma_unmap_sg(dev->dev, iod->sg, iod->nents, dma_dir); 699 697 if (blk_integrity_rq(req)) { 700 700 - if (!rq_data_dir(req)) 698 698 + if (req_op(req) == REQ_OP_READ) 701 699 nvme_dif_remap(req, nvme_dif_complete); 702 700 dma_unmap_sg(dev->dev, &iod->meta_sg, 1, dma_dir); 703 701 } ··· 1379 1377 1380 1378 if (blk_mq_alloc_tag_set(&dev->admin_tagset)) 1381 1379 return -ENOMEM; 1380 1380 + dev->ctrl.admin_tagset = &dev->admin_tagset; 1382 1381 1383 1382 dev->ctrl.admin_q = blk_mq_init_queue(&dev->admin_tagset); 1384 1383 if (IS_ERR(dev->ctrl.admin_q)) {

+290 -278

drivers/nvme/host/rdma.c

reviewed

··· 37 37 38 38 #define NVME_RDMA_CONNECT_TIMEOUT_MS 3000 /* 3 second */ 39 39 40 40 - #define NVME_RDMA_MAX_SEGMENT_SIZE 0xffffff /* 24-bit SGL field */ 41 41 - 42 40 #define NVME_RDMA_MAX_SEGMENTS 256 43 41 44 42 #define NVME_RDMA_MAX_INLINE_SEGMENTS 1 ··· 149 151 static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id, 150 152 struct rdma_cm_event *event); 151 153 static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc); 154 154 + 155 155 + static const struct blk_mq_ops nvme_rdma_mq_ops; 156 156 + static const struct blk_mq_ops nvme_rdma_admin_mq_ops; 152 157 153 158 /* XXX: really should move to a generic header sooner or later.. */ 154 159 static inline void put_unaligned_le24(u32 val, u8 *p) ··· 501 500 return ret; 502 501 } 503 502 504 504 - static int nvme_rdma_init_queue(struct nvme_rdma_ctrl *ctrl, 503 503 + static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, 505 504 int idx, size_t queue_size) 506 505 { 507 506 struct nvme_rdma_queue *queue; ··· 559 558 560 559 static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) 561 560 { 561 561 + if (!test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags)) 562 562 + return; 563 563 + 562 564 rdma_disconnect(queue->cm_id); 563 565 ib_drain_qp(queue->qp); 564 566 } 565 567 566 568 static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue) 567 569 { 568 568 - nvme_rdma_destroy_queue_ib(queue); 569 569 - rdma_destroy_id(queue->cm_id); 570 570 - } 571 571 - 572 572 - static void nvme_rdma_stop_and_free_queue(struct nvme_rdma_queue *queue) 573 573 - { 574 570 if (test_and_set_bit(NVME_RDMA_Q_DELETING, &queue->flags)) 575 571 return; 576 576 - nvme_rdma_stop_queue(queue); 577 577 - nvme_rdma_free_queue(queue); 572 572 + 573 573 + nvme_rdma_destroy_queue_ib(queue); 574 574 + rdma_destroy_id(queue->cm_id); 578 575 } 579 576 580 577 static void nvme_rdma_free_io_queues(struct nvme_rdma_ctrl *ctrl) ··· 580 581 int i; 581 582 582 583 for (i = 1; i < ctrl->ctrl.queue_count; i++) 583 583 - nvme_rdma_stop_and_free_queue(&ctrl->queues[i]); 584 584 + nvme_rdma_free_queue(&ctrl->queues[i]); 584 585 } 585 586 586 586 - static int nvme_rdma_connect_io_queues(struct nvme_rdma_ctrl *ctrl) 587 587 + static void nvme_rdma_stop_io_queues(struct nvme_rdma_ctrl *ctrl) 588 588 + { 589 589 + int i; 590 590 + 591 591 + for (i = 1; i < ctrl->ctrl.queue_count; i++) 592 592 + nvme_rdma_stop_queue(&ctrl->queues[i]); 593 593 + } 594 594 + 595 595 + static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx) 596 596 + { 597 597 + int ret; 598 598 + 599 599 + if (idx) 600 600 + ret = nvmf_connect_io_queue(&ctrl->ctrl, idx); 601 601 + else 602 602 + ret = nvmf_connect_admin_queue(&ctrl->ctrl); 603 603 + 604 604 + if (!ret) 605 605 + set_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[idx].flags); 606 606 + else 607 607 + dev_info(ctrl->ctrl.device, 608 608 + "failed to connect queue: %d ret=%d\n", idx, ret); 609 609 + return ret; 610 610 + } 611 611 + 612 612 + static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl) 587 613 { 588 614 int i, ret = 0; 589 615 590 616 for (i = 1; i < ctrl->ctrl.queue_count; i++) { 591 591 - ret = nvmf_connect_io_queue(&ctrl->ctrl, i); 592 592 - if (ret) { 593 593 - dev_info(ctrl->ctrl.device, 594 594 - "failed to connect i/o queue: %d\n", ret); 595 595 - goto out_free_queues; 596 596 - } 597 597 - set_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[i].flags); 617 617 + ret = nvme_rdma_start_queue(ctrl, i); 618 618 + if (ret) 619 619 + goto out_stop_queues; 598 620 } 599 621 600 622 return 0; 601 623 602 602 - out_free_queues: 603 603 - nvme_rdma_free_io_queues(ctrl); 624 624 + out_stop_queues: 625 625 + for (i--; i >= 1; i--) 626 626 + nvme_rdma_stop_queue(&ctrl->queues[i]); 604 627 return ret; 605 628 } 606 629 607 607 - static int nvme_rdma_init_io_queues(struct nvme_rdma_ctrl *ctrl) 630 630 + static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) 608 631 { 609 632 struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; 610 633 struct ib_device *ibdev = ctrl->device->dev; ··· 655 634 "creating %d I/O queues.\n", nr_io_queues); 656 635 657 636 for (i = 1; i < ctrl->ctrl.queue_count; i++) { 658 658 - ret = nvme_rdma_init_queue(ctrl, i, 659 659 - ctrl->ctrl.opts->queue_size); 660 660 - if (ret) { 661 661 - dev_info(ctrl->ctrl.device, 662 662 - "failed to initialize i/o queue: %d\n", ret); 637 637 + ret = nvme_rdma_alloc_queue(ctrl, i, 638 638 + ctrl->ctrl.sqsize + 1); 639 639 + if (ret) 663 640 goto out_free_queues; 664 664 - } 665 641 } 666 642 667 643 return 0; 668 644 669 645 out_free_queues: 670 646 for (i--; i >= 1; i--) 671 671 - nvme_rdma_stop_and_free_queue(&ctrl->queues[i]); 647 647 + nvme_rdma_free_queue(&ctrl->queues[i]); 672 648 673 649 return ret; 674 650 } 675 651 676 676 - static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl) 652 652 + static void nvme_rdma_free_tagset(struct nvme_ctrl *nctrl, bool admin) 653 653 + { 654 654 + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); 655 655 + struct blk_mq_tag_set *set = admin ? 656 656 + &ctrl->admin_tag_set : &ctrl->tag_set; 657 657 + 658 658 + blk_mq_free_tag_set(set); 659 659 + nvme_rdma_dev_put(ctrl->device); 660 660 + } 661 661 + 662 662 + static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl, 663 663 + bool admin) 664 664 + { 665 665 + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); 666 666 + struct blk_mq_tag_set *set; 667 667 + int ret; 668 668 + 669 669 + if (admin) { 670 670 + set = &ctrl->admin_tag_set; 671 671 + memset(set, 0, sizeof(*set)); 672 672 + set->ops = &nvme_rdma_admin_mq_ops; 673 673 + set->queue_depth = NVME_RDMA_AQ_BLKMQ_DEPTH; 674 674 + set->reserved_tags = 2; /* connect + keep-alive */ 675 675 + set->numa_node = NUMA_NO_NODE; 676 676 + set->cmd_size = sizeof(struct nvme_rdma_request) + 677 677 + SG_CHUNK_SIZE * sizeof(struct scatterlist); 678 678 + set->driver_data = ctrl; 679 679 + set->nr_hw_queues = 1; 680 680 + set->timeout = ADMIN_TIMEOUT; 681 681 + } else { 682 682 + set = &ctrl->tag_set; 683 683 + memset(set, 0, sizeof(*set)); 684 684 + set->ops = &nvme_rdma_mq_ops; 685 685 + set->queue_depth = nctrl->opts->queue_size; 686 686 + set->reserved_tags = 1; /* fabric connect */ 687 687 + set->numa_node = NUMA_NO_NODE; 688 688 + set->flags = BLK_MQ_F_SHOULD_MERGE; 689 689 + set->cmd_size = sizeof(struct nvme_rdma_request) + 690 690 + SG_CHUNK_SIZE * sizeof(struct scatterlist); 691 691 + set->driver_data = ctrl; 692 692 + set->nr_hw_queues = nctrl->queue_count - 1; 693 693 + set->timeout = NVME_IO_TIMEOUT; 694 694 + } 695 695 + 696 696 + ret = blk_mq_alloc_tag_set(set); 697 697 + if (ret) 698 698 + goto out; 699 699 + 700 700 + /* 701 701 + * We need a reference on the device as long as the tag_set is alive, 702 702 + * as the MRs in the request structures need a valid ib_device. 703 703 + */ 704 704 + ret = nvme_rdma_dev_get(ctrl->device); 705 705 + if (!ret) { 706 706 + ret = -EINVAL; 707 707 + goto out_free_tagset; 708 708 + } 709 709 + 710 710 + return set; 711 711 + 712 712 + out_free_tagset: 713 713 + blk_mq_free_tag_set(set); 714 714 + out: 715 715 + return ERR_PTR(ret); 716 716 + } 717 717 + 718 718 + static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl, 719 719 + bool remove) 677 720 { 678 721 nvme_rdma_free_qe(ctrl->queues[0].device->dev, &ctrl->async_event_sqe, 679 722 sizeof(struct nvme_command), DMA_TO_DEVICE); 680 680 - nvme_rdma_stop_and_free_queue(&ctrl->queues[0]); 681 681 - blk_cleanup_queue(ctrl->ctrl.admin_q); 682 682 - blk_mq_free_tag_set(&ctrl->admin_tag_set); 683 683 - nvme_rdma_dev_put(ctrl->device); 723 723 + nvme_rdma_stop_queue(&ctrl->queues[0]); 724 724 + if (remove) { 725 725 + blk_cleanup_queue(ctrl->ctrl.admin_q); 726 726 + nvme_rdma_free_tagset(&ctrl->ctrl, true); 727 727 + } 728 728 + nvme_rdma_free_queue(&ctrl->queues[0]); 729 729 + } 730 730 + 731 731 + static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, 732 732 + bool new) 733 733 + { 734 734 + int error; 735 735 + 736 736 + error = nvme_rdma_alloc_queue(ctrl, 0, NVME_AQ_DEPTH); 737 737 + if (error) 738 738 + return error; 739 739 + 740 740 + ctrl->device = ctrl->queues[0].device; 741 741 + 742 742 + ctrl->max_fr_pages = min_t(u32, NVME_RDMA_MAX_SEGMENTS, 743 743 + ctrl->device->dev->attrs.max_fast_reg_page_list_len); 744 744 + 745 745 + if (new) { 746 746 + ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true); 747 747 + if (IS_ERR(ctrl->ctrl.admin_tagset)) 748 748 + goto out_free_queue; 749 749 + 750 750 + ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set); 751 751 + if (IS_ERR(ctrl->ctrl.admin_q)) { 752 752 + error = PTR_ERR(ctrl->ctrl.admin_q); 753 753 + goto out_free_tagset; 754 754 + } 755 755 + } else { 756 756 + error = blk_mq_reinit_tagset(&ctrl->admin_tag_set, 757 757 + nvme_rdma_reinit_request); 758 758 + if (error) 759 759 + goto out_free_queue; 760 760 + } 761 761 + 762 762 + error = nvme_rdma_start_queue(ctrl, 0); 763 763 + if (error) 764 764 + goto out_cleanup_queue; 765 765 + 766 766 + error = ctrl->ctrl.ops->reg_read64(&ctrl->ctrl, NVME_REG_CAP, 767 767 + &ctrl->ctrl.cap); 768 768 + if (error) { 769 769 + dev_err(ctrl->ctrl.device, 770 770 + "prop_get NVME_REG_CAP failed\n"); 771 771 + goto out_cleanup_queue; 772 772 + } 773 773 + 774 774 + ctrl->ctrl.sqsize = 775 775 + min_t(int, NVME_CAP_MQES(ctrl->ctrl.cap), ctrl->ctrl.sqsize); 776 776 + 777 777 + error = nvme_enable_ctrl(&ctrl->ctrl, ctrl->ctrl.cap); 778 778 + if (error) 779 779 + goto out_cleanup_queue; 780 780 + 781 781 + ctrl->ctrl.max_hw_sectors = 782 782 + (ctrl->max_fr_pages - 1) << (ilog2(SZ_4K) - 9); 783 783 + 784 784 + error = nvme_init_identify(&ctrl->ctrl); 785 785 + if (error) 786 786 + goto out_cleanup_queue; 787 787 + 788 788 + error = nvme_rdma_alloc_qe(ctrl->queues[0].device->dev, 789 789 + &ctrl->async_event_sqe, sizeof(struct nvme_command), 790 790 + DMA_TO_DEVICE); 791 791 + if (error) 792 792 + goto out_cleanup_queue; 793 793 + 794 794 + return 0; 795 795 + 796 796 + out_cleanup_queue: 797 797 + if (new) 798 798 + blk_cleanup_queue(ctrl->ctrl.admin_q); 799 799 + out_free_tagset: 800 800 + if (new) 801 801 + nvme_rdma_free_tagset(&ctrl->ctrl, true); 802 802 + out_free_queue: 803 803 + nvme_rdma_free_queue(&ctrl->queues[0]); 804 804 + return error; 805 805 + } 806 806 + 807 807 + static void nvme_rdma_destroy_io_queues(struct nvme_rdma_ctrl *ctrl, 808 808 + bool remove) 809 809 + { 810 810 + nvme_rdma_stop_io_queues(ctrl); 811 811 + if (remove) { 812 812 + blk_cleanup_queue(ctrl->ctrl.connect_q); 813 813 + nvme_rdma_free_tagset(&ctrl->ctrl, false); 814 814 + } 815 815 + nvme_rdma_free_io_queues(ctrl); 816 816 + } 817 817 + 818 818 + static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) 819 819 + { 820 820 + int ret; 821 821 + 822 822 + ret = nvme_rdma_alloc_io_queues(ctrl); 823 823 + if (ret) 824 824 + return ret; 825 825 + 826 826 + if (new) { 827 827 + ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, false); 828 828 + if (IS_ERR(ctrl->ctrl.tagset)) 829 829 + goto out_free_io_queues; 830 830 + 831 831 + ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set); 832 832 + if (IS_ERR(ctrl->ctrl.connect_q)) { 833 833 + ret = PTR_ERR(ctrl->ctrl.connect_q); 834 834 + goto out_free_tag_set; 835 835 + } 836 836 + } else { 837 837 + ret = blk_mq_reinit_tagset(&ctrl->tag_set, 838 838 + nvme_rdma_reinit_request); 839 839 + if (ret) 840 840 + goto out_free_io_queues; 841 841 + 842 842 + blk_mq_update_nr_hw_queues(&ctrl->tag_set, 843 843 + ctrl->ctrl.queue_count - 1); 844 844 + } 845 845 + 846 846 + ret = nvme_rdma_start_io_queues(ctrl); 847 847 + if (ret) 848 848 + goto out_cleanup_connect_q; 849 849 + 850 850 + return 0; 851 851 + 852 852 + out_cleanup_connect_q: 853 853 + if (new) 854 854 + blk_cleanup_queue(ctrl->ctrl.connect_q); 855 855 + out_free_tag_set: 856 856 + if (new) 857 857 + nvme_rdma_free_tagset(&ctrl->ctrl, false); 858 858 + out_free_io_queues: 859 859 + nvme_rdma_free_io_queues(ctrl); 860 860 + return ret; 684 861 } 685 862 686 863 static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) ··· 927 708 928 709 ++ctrl->ctrl.nr_reconnects; 929 710 930 930 - if (ctrl->ctrl.queue_count > 1) { 931 931 - nvme_rdma_free_io_queues(ctrl); 711 711 + if (ctrl->ctrl.queue_count > 1) 712 712 + nvme_rdma_destroy_io_queues(ctrl, false); 932 713 933 933 - ret = blk_mq_reinit_tagset(&ctrl->tag_set, 934 934 - nvme_rdma_reinit_request); 935 935 - if (ret) 936 936 - goto requeue; 937 937 - } 938 938 - 939 939 - nvme_rdma_stop_and_free_queue(&ctrl->queues[0]); 940 940 - 941 941 - ret = blk_mq_reinit_tagset(&ctrl->admin_tag_set, 942 942 - nvme_rdma_reinit_request); 943 943 - if (ret) 944 944 - goto requeue; 945 945 - 946 946 - ret = nvme_rdma_init_queue(ctrl, 0, NVME_AQ_DEPTH); 947 947 - if (ret) 948 948 - goto requeue; 949 949 - 950 950 - ret = nvmf_connect_admin_queue(&ctrl->ctrl); 951 951 - if (ret) 952 952 - goto requeue; 953 953 - 954 954 - set_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[0].flags); 955 955 - 956 956 - ret = nvme_enable_ctrl(&ctrl->ctrl, ctrl->ctrl.cap); 714 714 + nvme_rdma_destroy_admin_queue(ctrl, false); 715 715 + ret = nvme_rdma_configure_admin_queue(ctrl, false); 957 716 if (ret) 958 717 goto requeue; 959 718 960 719 if (ctrl->ctrl.queue_count > 1) { 961 961 - ret = nvme_rdma_init_io_queues(ctrl); 720 720 + ret = nvme_rdma_configure_io_queues(ctrl, false); 962 721 if (ret) 963 722 goto requeue; 964 964 - 965 965 - ret = nvme_rdma_connect_io_queues(ctrl); 966 966 - if (ret) 967 967 - goto requeue; 968 968 - 969 969 - blk_mq_update_nr_hw_queues(&ctrl->tag_set, 970 970 - ctrl->ctrl.queue_count - 1); 971 723 } 972 724 973 725 changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); ··· 961 771 { 962 772 struct nvme_rdma_ctrl *ctrl = container_of(work, 963 773 struct nvme_rdma_ctrl, err_work); 964 964 - int i; 965 774 966 775 nvme_stop_ctrl(&ctrl->ctrl); 967 776 968 968 - for (i = 0; i < ctrl->ctrl.queue_count; i++) 969 969 - clear_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[i].flags); 970 970 - 971 971 - if (ctrl->ctrl.queue_count > 1) 777 777 + if (ctrl->ctrl.queue_count > 1) { 972 778 nvme_stop_queues(&ctrl->ctrl); 779 779 + nvme_rdma_stop_io_queues(ctrl); 780 780 + } 973 781 blk_mq_quiesce_queue(ctrl->ctrl.admin_q); 782 782 + nvme_rdma_stop_queue(&ctrl->queues[0]); 974 783 975 784 /* We must take care of fastfail/requeue all our inflight requests */ 976 785 if (ctrl->ctrl.queue_count > 1) ··· 1054 865 1055 866 if (req->mr->need_inval) { 1056 867 res = nvme_rdma_inv_rkey(queue, req); 1057 1057 - if (res < 0) { 868 868 + if (unlikely(res < 0)) { 1058 869 dev_err(ctrl->ctrl.device, 1059 870 "Queueing INV WR for rkey %#x failed (%d)\n", 1060 871 req->mr->rkey, res); ··· 1123 934 * the block virtual boundary. 1124 935 */ 1125 936 nr = ib_map_mr_sg(req->mr, req->sg_table.sgl, count, NULL, SZ_4K); 1126 1126 - if (nr < count) { 937 937 + if (unlikely(nr < count)) { 1127 938 if (nr < 0) 1128 939 return nr; 1129 940 return -EINVAL; ··· 1259 1070 first = &wr; 1260 1071 1261 1072 ret = ib_post_send(queue->qp, first, &bad_wr); 1262 1262 - if (ret) { 1073 1073 + if (unlikely(ret)) { 1263 1074 dev_err(queue->ctrl->ctrl.device, 1264 1075 "%s failed with error code %d\n", __func__, ret); 1265 1076 } ··· 1285 1096 wr.num_sge = 1; 1286 1097 1287 1098 ret = ib_post_recv(queue->qp, &wr, &bad_wr); 1288 1288 - if (ret) { 1099 1099 + if (unlikely(ret)) { 1289 1100 dev_err(queue->ctrl->ctrl.device, 1290 1101 "%s failed with error code %d\n", __func__, ret); 1291 1102 } ··· 1645 1456 blk_mq_start_request(rq); 1646 1457 1647 1458 err = nvme_rdma_map_data(queue, rq, c); 1648 1648 - if (err < 0) { 1459 1459 + if (unlikely(err < 0)) { 1649 1460 dev_err(queue->ctrl->ctrl.device, 1650 1461 "Failed to map data (%d)\n", err); 1651 1462 nvme_cleanup_cmd(rq); ··· 1659 1470 flush = true; 1660 1471 err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge, 1661 1472 req->mr->need_inval ? &req->reg_wr.wr : NULL, flush); 1662 1662 - if (err) { 1473 1473 + if (unlikely(err)) { 1663 1474 nvme_rdma_unmap_data(queue, rq); 1664 1475 goto err; 1665 1476 } ··· 1727 1538 .timeout = nvme_rdma_timeout, 1728 1539 }; 1729 1540 1730 1730 - static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl) 1731 1731 - { 1732 1732 - int error; 1733 1733 - 1734 1734 - error = nvme_rdma_init_queue(ctrl, 0, NVME_AQ_DEPTH); 1735 1735 - if (error) 1736 1736 - return error; 1737 1737 - 1738 1738 - ctrl->device = ctrl->queues[0].device; 1739 1739 - 1740 1740 - /* 1741 1741 - * We need a reference on the device as long as the tag_set is alive, 1742 1742 - * as the MRs in the request structures need a valid ib_device. 1743 1743 - */ 1744 1744 - error = -EINVAL; 1745 1745 - if (!nvme_rdma_dev_get(ctrl->device)) 1746 1746 - goto out_free_queue; 1747 1747 - 1748 1748 - ctrl->max_fr_pages = min_t(u32, NVME_RDMA_MAX_SEGMENTS, 1749 1749 - ctrl->device->dev->attrs.max_fast_reg_page_list_len); 1750 1750 - 1751 1751 - memset(&ctrl->admin_tag_set, 0, sizeof(ctrl->admin_tag_set)); 1752 1752 - ctrl->admin_tag_set.ops = &nvme_rdma_admin_mq_ops; 1753 1753 - ctrl->admin_tag_set.queue_depth = NVME_RDMA_AQ_BLKMQ_DEPTH; 1754 1754 - ctrl->admin_tag_set.reserved_tags = 2; /* connect + keep-alive */ 1755 1755 - ctrl->admin_tag_set.numa_node = NUMA_NO_NODE; 1756 1756 - ctrl->admin_tag_set.cmd_size = sizeof(struct nvme_rdma_request) + 1757 1757 - SG_CHUNK_SIZE * sizeof(struct scatterlist); 1758 1758 - ctrl->admin_tag_set.driver_data = ctrl; 1759 1759 - ctrl->admin_tag_set.nr_hw_queues = 1; 1760 1760 - ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT; 1761 1761 - 1762 1762 - error = blk_mq_alloc_tag_set(&ctrl->admin_tag_set); 1763 1763 - if (error) 1764 1764 - goto out_put_dev; 1765 1765 - 1766 1766 - ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set); 1767 1767 - if (IS_ERR(ctrl->ctrl.admin_q)) { 1768 1768 - error = PTR_ERR(ctrl->ctrl.admin_q); 1769 1769 - goto out_free_tagset; 1770 1770 - } 1771 1771 - 1772 1772 - error = nvmf_connect_admin_queue(&ctrl->ctrl); 1773 1773 - if (error) 1774 1774 - goto out_cleanup_queue; 1775 1775 - 1776 1776 - set_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[0].flags); 1777 1777 - 1778 1778 - error = nvmf_reg_read64(&ctrl->ctrl, NVME_REG_CAP, 1779 1779 - &ctrl->ctrl.cap); 1780 1780 - if (error) { 1781 1781 - dev_err(ctrl->ctrl.device, 1782 1782 - "prop_get NVME_REG_CAP failed\n"); 1783 1783 - goto out_cleanup_queue; 1784 1784 - } 1785 1785 - 1786 1786 - ctrl->ctrl.sqsize = 1787 1787 - min_t(int, NVME_CAP_MQES(ctrl->ctrl.cap), ctrl->ctrl.sqsize); 1788 1788 - 1789 1789 - error = nvme_enable_ctrl(&ctrl->ctrl, ctrl->ctrl.cap); 1790 1790 - if (error) 1791 1791 - goto out_cleanup_queue; 1792 1792 - 1793 1793 - ctrl->ctrl.max_hw_sectors = 1794 1794 - (ctrl->max_fr_pages - 1) << (ilog2(SZ_4K) - 9); 1795 1795 - 1796 1796 - error = nvme_init_identify(&ctrl->ctrl); 1797 1797 - if (error) 1798 1798 - goto out_cleanup_queue; 1799 1799 - 1800 1800 - error = nvme_rdma_alloc_qe(ctrl->queues[0].device->dev, 1801 1801 - &ctrl->async_event_sqe, sizeof(struct nvme_command), 1802 1802 - DMA_TO_DEVICE); 1803 1803 - if (error) 1804 1804 - goto out_cleanup_queue; 1805 1805 - 1806 1806 - return 0; 1807 1807 - 1808 1808 - out_cleanup_queue: 1809 1809 - blk_cleanup_queue(ctrl->ctrl.admin_q); 1810 1810 - out_free_tagset: 1811 1811 - /* disconnect and drain the queue before freeing the tagset */ 1812 1812 - nvme_rdma_stop_queue(&ctrl->queues[0]); 1813 1813 - blk_mq_free_tag_set(&ctrl->admin_tag_set); 1814 1814 - out_put_dev: 1815 1815 - nvme_rdma_dev_put(ctrl->device); 1816 1816 - out_free_queue: 1817 1817 - nvme_rdma_free_queue(&ctrl->queues[0]); 1818 1818 - return error; 1819 1819 - } 1820 1820 - 1821 1821 - static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl) 1541 1541 + static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown) 1822 1542 { 1823 1543 cancel_work_sync(&ctrl->err_work); 1824 1544 cancel_delayed_work_sync(&ctrl->reconnect_work); ··· 1736 1638 nvme_stop_queues(&ctrl->ctrl); 1737 1639 blk_mq_tagset_busy_iter(&ctrl->tag_set, 1738 1640 nvme_cancel_request, &ctrl->ctrl); 1739 1739 - nvme_rdma_free_io_queues(ctrl); 1641 1641 + nvme_rdma_destroy_io_queues(ctrl, shutdown); 1740 1642 } 1741 1643 1742 1742 - if (test_bit(NVME_RDMA_Q_LIVE, &ctrl->queues[0].flags)) 1644 1644 + if (shutdown) 1743 1645 nvme_shutdown_ctrl(&ctrl->ctrl); 1646 1646 + else 1647 1647 + nvme_disable_ctrl(&ctrl->ctrl, ctrl->ctrl.cap); 1744 1648 1745 1649 blk_mq_quiesce_queue(ctrl->ctrl.admin_q); 1746 1650 blk_mq_tagset_busy_iter(&ctrl->admin_tag_set, 1747 1651 nvme_cancel_request, &ctrl->ctrl); 1748 1652 blk_mq_unquiesce_queue(ctrl->ctrl.admin_q); 1749 1749 - nvme_rdma_destroy_admin_queue(ctrl); 1653 1653 + nvme_rdma_destroy_admin_queue(ctrl, shutdown); 1750 1654 } 1751 1655 1752 1752 - static void __nvme_rdma_remove_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown) 1656 1656 + static void nvme_rdma_remove_ctrl(struct nvme_rdma_ctrl *ctrl) 1753 1657 { 1754 1754 - nvme_stop_ctrl(&ctrl->ctrl); 1755 1658 nvme_remove_namespaces(&ctrl->ctrl); 1756 1756 - if (shutdown) 1757 1757 - nvme_rdma_shutdown_ctrl(ctrl); 1758 1758 - 1659 1659 + nvme_rdma_shutdown_ctrl(ctrl, true); 1759 1660 nvme_uninit_ctrl(&ctrl->ctrl); 1760 1760 - if (ctrl->ctrl.tagset) { 1761 1761 - blk_cleanup_queue(ctrl->ctrl.connect_q); 1762 1762 - blk_mq_free_tag_set(&ctrl->tag_set); 1763 1763 - nvme_rdma_dev_put(ctrl->device); 1764 1764 - } 1765 1765 - 1766 1661 nvme_put_ctrl(&ctrl->ctrl); 1767 1662 } 1768 1663 ··· 1764 1673 struct nvme_rdma_ctrl *ctrl = container_of(work, 1765 1674 struct nvme_rdma_ctrl, delete_work); 1766 1675 1767 1767 - __nvme_rdma_remove_ctrl(ctrl, true); 1676 1676 + nvme_stop_ctrl(&ctrl->ctrl); 1677 1677 + nvme_rdma_remove_ctrl(ctrl); 1768 1678 } 1769 1679 1770 1680 static int __nvme_rdma_del_ctrl(struct nvme_rdma_ctrl *ctrl) ··· 1797 1705 return ret; 1798 1706 } 1799 1707 1800 1800 - static void nvme_rdma_remove_ctrl_work(struct work_struct *work) 1801 1801 - { 1802 1802 - struct nvme_rdma_ctrl *ctrl = container_of(work, 1803 1803 - struct nvme_rdma_ctrl, delete_work); 1804 1804 - 1805 1805 - __nvme_rdma_remove_ctrl(ctrl, false); 1806 1806 - } 1807 1807 - 1808 1708 static void nvme_rdma_reset_ctrl_work(struct work_struct *work) 1809 1709 { 1810 1710 struct nvme_rdma_ctrl *ctrl = ··· 1805 1721 bool changed; 1806 1722 1807 1723 nvme_stop_ctrl(&ctrl->ctrl); 1808 1808 - nvme_rdma_shutdown_ctrl(ctrl); 1724 1724 + nvme_rdma_shutdown_ctrl(ctrl, false); 1809 1725 1810 1810 - ret = nvme_rdma_configure_admin_queue(ctrl); 1811 1811 - if (ret) { 1812 1812 - /* ctrl is already shutdown, just remove the ctrl */ 1813 1813 - INIT_WORK(&ctrl->delete_work, nvme_rdma_remove_ctrl_work); 1814 1814 - goto del_dead_ctrl; 1815 1815 - } 1726 1726 + ret = nvme_rdma_configure_admin_queue(ctrl, false); 1727 1727 + if (ret) 1728 1728 + goto out_fail; 1816 1729 1817 1730 if (ctrl->ctrl.queue_count > 1) { 1818 1818 - ret = blk_mq_reinit_tagset(&ctrl->tag_set, 1819 1819 - nvme_rdma_reinit_request); 1731 1731 + ret = nvme_rdma_configure_io_queues(ctrl, false); 1820 1732 if (ret) 1821 1821 - goto del_dead_ctrl; 1822 1822 - 1823 1823 - ret = nvme_rdma_init_io_queues(ctrl); 1824 1824 - if (ret) 1825 1825 - goto del_dead_ctrl; 1826 1826 - 1827 1827 - ret = nvme_rdma_connect_io_queues(ctrl); 1828 1828 - if (ret) 1829 1829 - goto del_dead_ctrl; 1830 1830 - 1831 1831 - blk_mq_update_nr_hw_queues(&ctrl->tag_set, 1832 1832 - ctrl->ctrl.queue_count - 1); 1733 1733 + goto out_fail; 1833 1734 } 1834 1735 1835 1736 changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); ··· 1824 1755 1825 1756 return; 1826 1757 1827 1827 - del_dead_ctrl: 1828 1828 - /* Deleting this dead controller... */ 1758 1758 + out_fail: 1829 1759 dev_warn(ctrl->ctrl.device, "Removing after reset failure\n"); 1830 1830 - WARN_ON(!queue_work(nvme_wq, &ctrl->delete_work)); 1760 1760 + nvme_rdma_remove_ctrl(ctrl); 1831 1761 } 1832 1762 1833 1763 static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = { ··· 1841 1773 .delete_ctrl = nvme_rdma_del_ctrl, 1842 1774 .get_address = nvmf_get_address, 1843 1775 }; 1844 1844 - 1845 1845 - static int nvme_rdma_create_io_queues(struct nvme_rdma_ctrl *ctrl) 1846 1846 - { 1847 1847 - int ret; 1848 1848 - 1849 1849 - ret = nvme_rdma_init_io_queues(ctrl); 1850 1850 - if (ret) 1851 1851 - return ret; 1852 1852 - 1853 1853 - /* 1854 1854 - * We need a reference on the device as long as the tag_set is alive, 1855 1855 - * as the MRs in the request structures need a valid ib_device. 1856 1856 - */ 1857 1857 - ret = -EINVAL; 1858 1858 - if (!nvme_rdma_dev_get(ctrl->device)) 1859 1859 - goto out_free_io_queues; 1860 1860 - 1861 1861 - memset(&ctrl->tag_set, 0, sizeof(ctrl->tag_set)); 1862 1862 - ctrl->tag_set.ops = &nvme_rdma_mq_ops; 1863 1863 - ctrl->tag_set.queue_depth = ctrl->ctrl.opts->queue_size; 1864 1864 - ctrl->tag_set.reserved_tags = 1; /* fabric connect */ 1865 1865 - ctrl->tag_set.numa_node = NUMA_NO_NODE; 1866 1866 - ctrl->tag_set.flags = BLK_MQ_F_SHOULD_MERGE; 1867 1867 - ctrl->tag_set.cmd_size = sizeof(struct nvme_rdma_request) + 1868 1868 - SG_CHUNK_SIZE * sizeof(struct scatterlist); 1869 1869 - ctrl->tag_set.driver_data = ctrl; 1870 1870 - ctrl->tag_set.nr_hw_queues = ctrl->ctrl.queue_count - 1; 1871 1871 - ctrl->tag_set.timeout = NVME_IO_TIMEOUT; 1872 1872 - 1873 1873 - ret = blk_mq_alloc_tag_set(&ctrl->tag_set); 1874 1874 - if (ret) 1875 1875 - goto out_put_dev; 1876 1876 - ctrl->ctrl.tagset = &ctrl->tag_set; 1877 1877 - 1878 1878 - ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set); 1879 1879 - if (IS_ERR(ctrl->ctrl.connect_q)) { 1880 1880 - ret = PTR_ERR(ctrl->ctrl.connect_q); 1881 1881 - goto out_free_tag_set; 1882 1882 - } 1883 1883 - 1884 1884 - ret = nvme_rdma_connect_io_queues(ctrl); 1885 1885 - if (ret) 1886 1886 - goto out_cleanup_connect_q; 1887 1887 - 1888 1888 - return 0; 1889 1889 - 1890 1890 - out_cleanup_connect_q: 1891 1891 - blk_cleanup_queue(ctrl->ctrl.connect_q); 1892 1892 - out_free_tag_set: 1893 1893 - blk_mq_free_tag_set(&ctrl->tag_set); 1894 1894 - out_put_dev: 1895 1895 - nvme_rdma_dev_put(ctrl->device); 1896 1896 - out_free_io_queues: 1897 1897 - nvme_rdma_free_io_queues(ctrl); 1898 1898 - return ret; 1899 1899 - } 1900 1776 1901 1777 static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, 1902 1778 struct nvmf_ctrl_options *opts) ··· 1899 1887 if (!ctrl->queues) 1900 1888 goto out_uninit_ctrl; 1901 1889 1902 1902 - ret = nvme_rdma_configure_admin_queue(ctrl); 1890 1890 + ret = nvme_rdma_configure_admin_queue(ctrl, true); 1903 1891 if (ret) 1904 1892 goto out_kfree_queues; 1905 1893 ··· 1934 1922 } 1935 1923 1936 1924 if (opts->nr_io_queues) { 1937 1937 - ret = nvme_rdma_create_io_queues(ctrl); 1925 1925 + ret = nvme_rdma_configure_io_queues(ctrl, true); 1938 1926 if (ret) 1939 1927 goto out_remove_admin_queue; 1940 1928 } ··· 1956 1944 return &ctrl->ctrl; 1957 1945 1958 1946 out_remove_admin_queue: 1959 1959 - nvme_rdma_destroy_admin_queue(ctrl); 1947 1947 + nvme_rdma_destroy_admin_queue(ctrl, true); 1960 1948 out_kfree_queues: 1961 1949 kfree(ctrl->queues); 1962 1950 out_uninit_ctrl:

+19 -14

drivers/nvme/target/admin-cmd.c

reviewed

··· 100 100 u16 status; 101 101 102 102 WARN_ON(req == NULL || slog == NULL); 103 103 - if (req->cmd->get_log_page.nsid == cpu_to_le32(0xFFFFFFFF)) 103 103 + if (req->cmd->get_log_page.nsid == cpu_to_le32(NVME_NSID_ALL)) 104 104 status = nvmet_get_smart_log_all(req, slog); 105 105 else 106 106 status = nvmet_get_smart_log_nsid(req, slog); ··· 168 168 nvmet_req_complete(req, status); 169 169 } 170 170 171 171 - static void copy_and_pad(char *dst, int dst_len, const char *src, int src_len) 172 172 - { 173 173 - int len = min(src_len, dst_len); 174 174 - 175 175 - memcpy(dst, src, len); 176 176 - if (dst_len > len) 177 177 - memset(dst + len, ' ', dst_len - len); 178 178 - } 179 179 - 180 171 static void nvmet_execute_identify_ctrl(struct nvmet_req *req) 181 172 { 182 173 struct nvmet_ctrl *ctrl = req->sq->ctrl; ··· 187 196 188 197 bin2hex(id->sn, &ctrl->subsys->serial, 189 198 min(sizeof(ctrl->subsys->serial), sizeof(id->sn) / 2)); 190 190 - copy_and_pad(id->mn, sizeof(id->mn), model, sizeof(model) - 1); 191 191 - copy_and_pad(id->fr, sizeof(id->fr), UTS_RELEASE, strlen(UTS_RELEASE)); 199 199 + memcpy_and_pad(id->mn, sizeof(id->mn), model, sizeof(model) - 1, ' '); 200 200 + memcpy_and_pad(id->fr, sizeof(id->fr), 201 201 + UTS_RELEASE, strlen(UTS_RELEASE), ' '); 192 202 193 203 id->rab = 6; 194 204 ··· 443 451 u32 val32; 444 452 u16 status = 0; 445 453 446 446 - switch (cdw10 & 0xf) { 454 454 + switch (cdw10 & 0xff) { 447 455 case NVME_FEAT_NUM_QUEUES: 448 456 nvmet_set_result(req, 449 457 (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16)); ··· 452 460 val32 = le32_to_cpu(req->cmd->common.cdw10[1]); 453 461 req->sq->ctrl->kato = DIV_ROUND_UP(val32, 1000); 454 462 nvmet_set_result(req, req->sq->ctrl->kato); 463 463 + break; 464 464 + case NVME_FEAT_HOST_ID: 465 465 + status = NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR; 455 466 break; 456 467 default: 457 468 status = NVME_SC_INVALID_FIELD | NVME_SC_DNR; ··· 470 475 u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10[0]); 471 476 u16 status = 0; 472 477 473 473 - switch (cdw10 & 0xf) { 478 478 + switch (cdw10 & 0xff) { 474 479 /* 475 480 * These features are mandatory in the spec, but we don't 476 481 * have a useful way to implement them. We'll eventually ··· 503 508 break; 504 509 case NVME_FEAT_KATO: 505 510 nvmet_set_result(req, req->sq->ctrl->kato * 1000); 511 511 + break; 512 512 + case NVME_FEAT_HOST_ID: 513 513 + /* need 128-bit host identifier flag */ 514 514 + if (!(req->cmd->common.cdw10[1] & cpu_to_le32(1 << 0))) { 515 515 + status = NVME_SC_INVALID_FIELD | NVME_SC_DNR; 516 516 + break; 517 517 + } 518 518 + 519 519 + status = nvmet_copy_to_sgl(req, 0, &req->sq->ctrl->hostid, 520 520 + sizeof(req->sq->ctrl->hostid)); 506 521 break; 507 522 default: 508 523 status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;

+1 -1

drivers/nvme/target/configfs.c

reviewed

··· 444 444 goto out; 445 445 446 446 ret = -EINVAL; 447 447 - if (nsid == 0 || nsid == 0xffffffff) 447 447 + if (nsid == 0 || nsid == NVME_NSID_ALL) 448 448 goto out; 449 449 450 450 ret = -ENOMEM;

+8 -7

drivers/nvme/target/core.c

reviewed

··· 538 538 539 539 static inline bool nvmet_cc_en(u32 cc) 540 540 { 541 541 - return cc & 0x1; 541 541 + return (cc >> NVME_CC_EN_SHIFT) & 0x1; 542 542 } 543 543 544 544 static inline u8 nvmet_cc_css(u32 cc) 545 545 { 546 546 - return (cc >> 4) & 0x7; 546 546 + return (cc >> NVME_CC_CSS_SHIFT) & 0x7; 547 547 } 548 548 549 549 static inline u8 nvmet_cc_mps(u32 cc) 550 550 { 551 551 - return (cc >> 7) & 0xf; 551 551 + return (cc >> NVME_CC_MPS_SHIFT) & 0xf; 552 552 } 553 553 554 554 static inline u8 nvmet_cc_ams(u32 cc) 555 555 { 556 556 - return (cc >> 11) & 0x7; 556 556 + return (cc >> NVME_CC_AMS_SHIFT) & 0x7; 557 557 } 558 558 559 559 static inline u8 nvmet_cc_shn(u32 cc) 560 560 { 561 561 - return (cc >> 14) & 0x3; 561 561 + return (cc >> NVME_CC_SHN_SHIFT) & 0x3; 562 562 } 563 563 564 564 static inline u8 nvmet_cc_iosqes(u32 cc) 565 565 { 566 566 - return (cc >> 16) & 0xf; 566 566 + return (cc >> NVME_CC_IOSQES_SHIFT) & 0xf; 567 567 } 568 568 569 569 static inline u8 nvmet_cc_iocqes(u32 cc) 570 570 { 571 571 - return (cc >> 20) & 0xf; 571 571 + return (cc >> NVME_CC_IOCQES_SHIFT) & 0xf; 572 572 } 573 573 574 574 static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl) ··· 749 749 hostnqn, subsysnqn); 750 750 req->rsp->result.u32 = IPO_IATTR_CONNECT_DATA(hostnqn); 751 751 up_read(&nvmet_config_sem); 752 752 + status = NVME_SC_CONNECT_INVALID_HOST | NVME_SC_DNR; 752 753 goto out_put_subsystem; 753 754 } 754 755 up_read(&nvmet_config_sem);

+1

drivers/nvme/target/fabrics-cmd.c

reviewed

··· 154 154 le32_to_cpu(c->kato), &ctrl); 155 155 if (status) 156 156 goto out; 157 157 + uuid_copy(&ctrl->hostid, &d->hostid); 157 158 158 159 status = nvmet_install_queue(ctrl, req); 159 160 if (status) {

+10 -38

drivers/nvme/target/fc.c

reviewed

··· 58 58 struct work_struct work; 59 59 } __aligned(sizeof(unsigned long long)); 60 60 61 61 - #define NVMET_FC_MAX_KB_PER_XFR 256 61 61 + #define NVMET_FC_MAX_SEQ_LENGTH (256 * 1024) 62 62 + #define NVMET_FC_MAX_XFR_SGENTS (NVMET_FC_MAX_SEQ_LENGTH / PAGE_SIZE) 62 63 63 64 enum nvmet_fcp_datadir { 64 65 NVMET_FCP_NODATA, ··· 75 74 struct nvme_fc_ersp_iu rspiubuf; 76 75 dma_addr_t rspdma; 77 76 struct scatterlist *data_sg; 78 78 - struct scatterlist *next_sg; 79 77 int data_sg_cnt; 80 80 - u32 next_sg_offset; 81 78 u32 total_length; 82 79 u32 offset; 83 80 enum nvmet_fcp_datadir io_dir; ··· 111 112 struct ida assoc_cnt; 112 113 struct nvmet_port *port; 113 114 struct kref ref; 115 115 + u32 max_sg_cnt; 114 116 }; 115 117 116 118 struct nvmet_fc_defer_fcp_req { ··· 994 994 INIT_LIST_HEAD(&newrec->assoc_list); 995 995 kref_init(&newrec->ref); 996 996 ida_init(&newrec->assoc_cnt); 997 997 + newrec->max_sg_cnt = min_t(u32, NVMET_FC_MAX_XFR_SGENTS, 998 998 + template->max_sgl_segments); 997 999 998 1000 ret = nvmet_fc_alloc_ls_iodlist(newrec); 999 1001 if (ret) { ··· 1868 1866 struct nvmet_fc_fcp_iod *fod, u8 op) 1869 1867 { 1870 1868 struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq; 1871 1871 - struct scatterlist *sg, *datasg; 1872 1869 unsigned long flags; 1873 1873 - u32 tlen, sg_off; 1870 1870 + u32 tlen; 1874 1871 int ret; 1875 1872 1876 1873 fcpreq->op = op; 1877 1874 fcpreq->offset = fod->offset; 1878 1875 fcpreq->timeout = NVME_FC_TGTOP_TIMEOUT_SEC; 1879 1879 - tlen = min_t(u32, (NVMET_FC_MAX_KB_PER_XFR * 1024), 1876 1876 + 1877 1877 + tlen = min_t(u32, tgtport->max_sg_cnt * PAGE_SIZE, 1880 1878 (fod->total_length - fod->offset)); 1881 1881 - tlen = min_t(u32, tlen, NVME_FC_MAX_SEGMENTS * PAGE_SIZE); 1882 1882 - tlen = min_t(u32, tlen, fod->tgtport->ops->max_sgl_segments 1883 1883 - * PAGE_SIZE); 1884 1879 fcpreq->transfer_length = tlen; 1885 1880 fcpreq->transferred_length = 0; 1886 1881 fcpreq->fcp_error = 0; 1887 1882 fcpreq->rsplen = 0; 1888 1883 1889 1889 - fcpreq->sg_cnt = 0; 1890 1890 - 1891 1891 - datasg = fod->next_sg; 1892 1892 - sg_off = fod->next_sg_offset; 1893 1893 - 1894 1894 - for (sg = fcpreq->sg ; tlen; sg++) { 1895 1895 - *sg = *datasg; 1896 1896 - if (sg_off) { 1897 1897 - sg->offset += sg_off; 1898 1898 - sg->length -= sg_off; 1899 1899 - sg->dma_address += sg_off; 1900 1900 - sg_off = 0; 1901 1901 - } 1902 1902 - if (tlen < sg->length) { 1903 1903 - sg->length = tlen; 1904 1904 - fod->next_sg = datasg; 1905 1905 - fod->next_sg_offset += tlen; 1906 1906 - } else if (tlen == sg->length) { 1907 1907 - fod->next_sg_offset = 0; 1908 1908 - fod->next_sg = sg_next(datasg); 1909 1909 - } else { 1910 1910 - fod->next_sg_offset = 0; 1911 1911 - datasg = sg_next(datasg); 1912 1912 - } 1913 1913 - tlen -= sg->length; 1914 1914 - fcpreq->sg_cnt++; 1915 1915 - } 1884 1884 + fcpreq->sg = &fod->data_sg[fod->offset / PAGE_SIZE]; 1885 1885 + fcpreq->sg_cnt = DIV_ROUND_UP(tlen, PAGE_SIZE); 1916 1886 1917 1887 /* 1918 1888 * If the last READDATA request: check if LLDD supports ··· 2199 2225 fod->req.sg = fod->data_sg; 2200 2226 fod->req.sg_cnt = fod->data_sg_cnt; 2201 2227 fod->offset = 0; 2202 2202 - fod->next_sg = fod->data_sg; 2203 2203 - fod->next_sg_offset = 0; 2204 2228 2205 2229 if (fod->io_dir == NVMET_FCP_WRITE) { 2206 2230 /* pull the data over before invoking nvmet layer */

-3

drivers/nvme/target/fcloop.c

reviewed

··· 193 193 194 194 #define TGTPORT_OPTS (NVMF_OPT_WWNN | NVMF_OPT_WWPN) 195 195 196 196 - #define ALL_OPTS (NVMF_OPT_WWNN | NVMF_OPT_WWPN | NVMF_OPT_ROLES | \ 197 197 - NVMF_OPT_FCADDR | NVMF_OPT_LPWWNN | NVMF_OPT_LPWWPN) 198 198 - 199 196 200 197 static DEFINE_SPINLOCK(fcloop_lock); 201 198 static LIST_HEAD(fcloop_lports);

+1

drivers/nvme/target/loop.c

reviewed

··· 375 375 error = blk_mq_alloc_tag_set(&ctrl->admin_tag_set); 376 376 if (error) 377 377 goto out_free_sq; 378 378 + ctrl->ctrl.admin_tagset = &ctrl->admin_tag_set; 378 379 379 380 ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set); 380 381 if (IS_ERR(ctrl->ctrl.admin_q)) {

+1

drivers/nvme/target/nvmet.h

reviewed

··· 115 115 u32 cc; 116 116 u32 csts; 117 117 118 118 + uuid_t hostid; 118 119 u16 cntlid; 119 120 u32 kato; 120 121

+1 -1

include/linux/nvme-fc-driver.h

reviewed

··· 624 624 u32 timeout; 625 625 u32 transfer_length; 626 626 struct fc_ba_rjt ba_rjt; 627 627 - struct scatterlist sg[NVME_FC_MAX_SEGMENTS]; 627 627 + struct scatterlist *sg; 628 628 int sg_cnt; 629 629 void *rspaddr; 630 630 dma_addr_t rspdma;

+28 -9

include/linux/nvme.h

reviewed

··· 32 32 33 33 #define NVME_RDMA_IP_PORT 4420 34 34 35 35 + #define NVME_NSID_ALL 0xffffffff 36 36 + 35 37 enum nvme_subsys_type { 36 38 NVME_NQN_DISC = 1, /* Discovery type target subsystem */ 37 39 NVME_NQN_NVME = 2, /* NVME type target subsystem */ ··· 135 133 enum { 136 134 NVME_CC_ENABLE = 1 << 0, 137 135 NVME_CC_CSS_NVM = 0 << 4, 136 136 + NVME_CC_EN_SHIFT = 0, 137 137 + NVME_CC_CSS_SHIFT = 4, 138 138 NVME_CC_MPS_SHIFT = 7, 139 139 - NVME_CC_ARB_RR = 0 << 11, 140 140 - NVME_CC_ARB_WRRU = 1 << 11, 141 141 - NVME_CC_ARB_VS = 7 << 11, 142 142 - NVME_CC_SHN_NONE = 0 << 14, 143 143 - NVME_CC_SHN_NORMAL = 1 << 14, 144 144 - NVME_CC_SHN_ABRUPT = 2 << 14, 145 145 - NVME_CC_SHN_MASK = 3 << 14, 146 146 - NVME_CC_IOSQES = NVME_NVM_IOSQES << 16, 147 147 - NVME_CC_IOCQES = NVME_NVM_IOCQES << 20, 139 139 + NVME_CC_AMS_SHIFT = 11, 140 140 + NVME_CC_SHN_SHIFT = 14, 141 141 + NVME_CC_IOSQES_SHIFT = 16, 142 142 + NVME_CC_IOCQES_SHIFT = 20, 143 143 + NVME_CC_AMS_RR = 0 << NVME_CC_AMS_SHIFT, 144 144 + NVME_CC_AMS_WRRU = 1 << NVME_CC_AMS_SHIFT, 145 145 + NVME_CC_AMS_VS = 7 << NVME_CC_AMS_SHIFT, 146 146 + NVME_CC_SHN_NONE = 0 << NVME_CC_SHN_SHIFT, 147 147 + NVME_CC_SHN_NORMAL = 1 << NVME_CC_SHN_SHIFT, 148 148 + NVME_CC_SHN_ABRUPT = 2 << NVME_CC_SHN_SHIFT, 149 149 + NVME_CC_SHN_MASK = 3 << NVME_CC_SHN_SHIFT, 150 150 + NVME_CC_IOSQES = NVME_NVM_IOSQES << NVME_CC_IOSQES_SHIFT, 151 151 + NVME_CC_IOCQES = NVME_NVM_IOCQES << NVME_CC_IOCQES_SHIFT, 148 152 NVME_CSTS_RDY = 1 << 0, 149 153 NVME_CSTS_CFS = 1 << 1, 150 154 NVME_CSTS_NSSRO = 1 << 4, 155 155 + NVME_CSTS_PP = 1 << 5, 151 156 NVME_CSTS_SHST_NORMAL = 0 << 2, 152 157 NVME_CSTS_SHST_OCCUR = 1 << 2, 153 158 NVME_CSTS_SHST_CMPLT = 2 << 2, ··· 260 251 NVME_CTRL_ONCS_WRITE_UNCORRECTABLE = 1 << 1, 261 252 NVME_CTRL_ONCS_DSM = 1 << 2, 262 253 NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, 254 254 + NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, 263 255 NVME_CTRL_VWC_PRESENT = 1 << 0, 264 256 NVME_CTRL_OACS_SEC_SUPP = 1 << 0, 265 257 NVME_CTRL_OACS_DIRECTIVES = 1 << 5, ··· 386 376 __u8 rsvd216[296]; 387 377 }; 388 378 379 379 + struct nvme_fw_slot_info_log { 380 380 + __u8 afi; 381 381 + __u8 rsvd1[7]; 382 382 + __le64 frs[7]; 383 383 + __u8 rsvd64[448]; 384 384 + }; 385 385 + 389 386 enum { 390 387 NVME_SMART_CRIT_SPARE = 1 << 0, 391 388 NVME_SMART_CRIT_TEMPERATURE = 1 << 1, ··· 403 386 404 387 enum { 405 388 NVME_AER_NOTICE_NS_CHANGED = 0x0002, 389 389 + NVME_AER_NOTICE_FW_ACT_STARTING = 0x0102, 406 390 }; 407 391 408 392 struct nvme_lba_range_type { ··· 695 677 NVME_FEAT_ASYNC_EVENT = 0x0b, 696 678 NVME_FEAT_AUTO_PST = 0x0c, 697 679 NVME_FEAT_HOST_MEM_BUF = 0x0d, 680 680 + NVME_FEAT_TIMESTAMP = 0x0e, 698 681 NVME_FEAT_KATO = 0x0f, 699 682 NVME_FEAT_SW_PROGRESS = 0x80, 700 683 NVME_FEAT_HOST_ID = 0x81,

+30

include/linux/string.h

reviewed

··· 230 230 void fortify_panic(const char *name) __noreturn __cold; 231 231 void __read_overflow(void) __compiletime_error("detected read beyond size of object passed as 1st parameter"); 232 232 void __read_overflow2(void) __compiletime_error("detected read beyond size of object passed as 2nd parameter"); 233 233 + void __read_overflow3(void) __compiletime_error("detected read beyond size of object passed as 3rd parameter"); 233 234 void __write_overflow(void) __compiletime_error("detected write beyond size of object passed as 1st parameter"); 234 235 235 236 #if !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE) ··· 425 424 } 426 425 427 426 #endif 427 427 + 428 428 + /** 429 429 + * memcpy_and_pad - Copy one buffer to another with padding 430 430 + * @dest: Where to copy to 431 431 + * @dest_len: The destination buffer size 432 432 + * @src: Where to copy from 433 433 + * @count: The number of bytes to copy 434 434 + * @pad: Character to use for padding if space is left in destination. 435 435 + */ 436 436 + __FORTIFY_INLINE void memcpy_and_pad(void *dest, size_t dest_len, 437 437 + const void *src, size_t count, int pad) 438 438 + { 439 439 + size_t dest_size = __builtin_object_size(dest, 0); 440 440 + size_t src_size = __builtin_object_size(src, 0); 441 441 + 442 442 + if (__builtin_constant_p(dest_len) && __builtin_constant_p(count)) { 443 443 + if (dest_size < dest_len && dest_size < count) 444 444 + __write_overflow(); 445 445 + else if (src_size < dest_len && src_size < count) 446 446 + __read_overflow3(); 447 447 + } 448 448 + if (dest_size < dest_len) 449 449 + fortify_panic(__func__); 450 450 + if (dest_len > count) { 451 451 + memcpy(dest, src, count); 452 452 + memset(dest + count, pad, dest_len - count); 453 453 + } else 454 454 + memcpy(dest, src, dest_len); 455 455 + } 428 456 429 457 #endif /* _LINUX_STRING_H_ */

+1

include/uapi/linux/loop.h

reviewed

··· 88 88 #define LOOP_CHANGE_FD 0x4C06 89 89 #define LOOP_SET_CAPACITY 0x4C07 90 90 #define LOOP_SET_DIRECT_IO 0x4C08 91 91 + #define LOOP_SET_BLOCK_SIZE 0x4C09 91 92 92 93 /* /dev/loop-control interface */ 93 94 #define LOOP_CTL_ADD 0x4C80