docs: md: convert to ReST · tjh.dev/kernel@7ed44d5

+12

Documentation/md/index.rst

··· 1 + :orphan: 2 + 3 + ==== 4 + RAID 5 + ==== 6 + 7 + .. toctree:: 8 + :maxdepth: 1 9 + 10 + md-cluster 11 + raid5-cache 12 + raid5-ppl

+125 -65

Documentation/md/md-cluster.txt Documentation/md/md-cluster.rst

··· 1 + ========== 2 + MD Cluster 3 + ========== 4 + 1 5 The cluster MD is a shared-device RAID for a cluster, it supports 2 6 two levels: raid1 and raid10 (limited support). 3 7 4 8 5 9 1. On-disk format 10 + ================= 6 11 7 12 Separate write-intent-bitmaps are used for each cluster node. 8 13 The bitmaps record all writes that may have been started on that node, 9 - and may not yet have finished. The on-disk layout is: 14 + and may not yet have finished. The on-disk layout is:: 10 15 11 - 0 4k 8k 12k 12 - ------------------------------------------------------------------- 13 - | idle | md super | bm super [0] + bits | 14 - | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] | 15 - | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits | 16 - | bm bits [3, contd] | | | 16 + 0 4k 8k 12k 17 + ------------------------------------------------------------------- 18 + | idle | md super | bm super [0] + bits | 19 + | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] | 20 + | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits | 21 + | bm bits [3, contd] | | | 17 22 18 23 During "normal" functioning we assume the filesystem ensures that only 19 24 one node writes to any given block at a time, so a write request will ··· 33 28 34 29 35 30 2. DLM Locks for management 31 + =========================== 36 32 37 33 There are three groups of locks for managing the device: 38 34 39 35 2.1 Bitmap lock resource (bm_lockres) 36 + ------------------------------------- 40 37 41 38 The bm_lockres protects individual node bitmaps. They are named in 42 39 the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a ··· 55 48 joins the cluster. 56 49 57 50 2.2 Message passing locks 51 + ------------------------- 58 52 59 53 Each node has to communicate with other nodes when starting or ending 60 54 resync, and for metadata superblock updates. This communication is ··· 63 55 with the Lock Value Block (LVB) of one of the "message" lock. 64 56 65 57 2.3 new-device management 58 + ------------------------- 66 59 67 60 A single lock: "no-new-dev" is used to co-ordinate the addition of 68 61 new devices - this must be synchronized across the array. 69 62 Normally all nodes hold a concurrent-read lock on this device. 70 63 71 64 3. Communication 65 + ================ 72 66 73 67 Messages can be broadcast to all nodes, and the sender waits for all 74 68 other nodes to acknowledge the message before proceeding. Only one 75 69 message can be processed at a time. 76 70 77 71 3.1 Message Types 72 + ----------------- 78 73 79 74 There are six types of messages which are passed: 80 75 81 - 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has 76 + 3.1.1 METADATA_UPDATED 77 + ^^^^^^^^^^^^^^^^^^^^^^ 78 + 79 + informs other nodes that the metadata has 82 80 been updated, and the node must re-read the md superblock. This is 83 81 performed synchronously. It is primarily used to signal device 84 82 failure. 85 83 86 - 3.1.2 RESYNCING: informs other nodes that a resync is initiated or 84 + 3.1.2 RESYNCING 85 + ^^^^^^^^^^^^^^^ 86 + informs other nodes that a resync is initiated or 87 87 ended so that each node may suspend or resume the region. Each 88 88 RESYNCING message identifies a range of the devices that the 89 89 sending node is about to resync. This overrides any previous 90 90 notification from that node: only one ranged can be resynced at a 91 91 time per-node. 92 92 93 - 3.1.3 NEWDISK: informs other nodes that a device is being added to 93 + 3.1.3 NEWDISK 94 + ^^^^^^^^^^^^^ 95 + 96 + informs other nodes that a device is being added to 94 97 the array. Message contains an identifier for that device. See 95 98 below for further details. 96 99 97 - 3.1.4 REMOVE: A failed or spare device is being removed from the 100 + 3.1.4 REMOVE 101 + ^^^^^^^^^^^^ 102 + 103 + A failed or spare device is being removed from the 98 104 array. The slot-number of the device is included in the message. 99 105 100 - 3.1.5 RE_ADD: A failed device is being re-activated - the assumption 106 + 3.1.5 RE_ADD: 107 + 108 + A failed device is being re-activated - the assumption 101 109 is that it has been determined to be working again. 102 110 103 - 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap 111 + 3.1.6 BITMAP_NEEDS_SYNC: 112 + 113 + If a node is stopped locally but the bitmap 104 114 isn't clean, then another node is informed to take the ownership of 105 115 resync. 106 116 107 117 3.2 Communication mechanism 118 + --------------------------- 108 119 109 120 The DLM LVB is used to communicate within nodes of the cluster. There 110 121 are three resources used for the purpose: 111 122 112 - 3.2.1 token: The resource which protects the entire communication 123 + 3.2.1 token 124 + ^^^^^^^^^^^ 125 + The resource which protects the entire communication 113 126 system. The node having the token resource is allowed to 114 127 communicate. 115 128 116 - 3.2.2 message: The lock resource which carries the data to 117 - communicate. 129 + 3.2.2 message 130 + ^^^^^^^^^^^^^ 131 + The lock resource which carries the data to communicate. 118 132 119 - 3.2.3 ack: The resource, acquiring which means the message has been 133 + 3.2.3 ack 134 + ^^^^^^^^^ 135 + 136 + The resource, acquiring which means the message has been 120 137 acknowledged by all nodes in the cluster. The BAST of the resource 121 138 is used to inform the receiving node that a node wants to 122 139 communicate. 123 140 124 141 The algorithm is: 125 142 126 - 1. receive status - all nodes have concurrent-reader lock on "ack". 143 + 1. receive status - all nodes have concurrent-reader lock on "ack":: 127 144 128 - sender receiver receiver 129 - "ack":CR "ack":CR "ack":CR 145 + sender receiver receiver 146 + "ack":CR "ack":CR "ack":CR 130 147 131 - 2. sender get EX on "token" 132 - sender get EX on "message" 133 - sender receiver receiver 134 - "token":EX "ack":CR "ack":CR 135 - "message":EX 136 - "ack":CR 148 + 2. sender get EX on "token", 149 + sender get EX on "message":: 150 + 151 + sender receiver receiver 152 + "token":EX "ack":CR "ack":CR 153 + "message":EX 154 + "ack":CR 137 155 138 156 Sender checks that it still needs to send a message. Messages 139 157 received or other events that happened while waiting for the 140 158 "token" may have made this message inappropriate or redundant. 141 159 142 - 3. sender writes LVB. 160 + 3. sender writes LVB 161 + 143 162 sender down-convert "message" from EX to CW 163 + 144 164 sender try to get EX of "ack" 145 - [ wait until all receivers have *processed* the "message" ] 146 165 147 - [ triggered by bast of "ack" ] 148 - receiver get CR on "message" 149 - receiver read LVB 150 - receiver processes the message 151 - [ wait finish ] 152 - receiver releases "ack" 153 - receiver tries to get PR on "message" 166 + :: 154 167 155 - sender receiver receiver 156 - "token":EX "message":CR "message":CR 157 - "message":CW 158 - "ack":EX 168 + [ wait until all receivers have *processed* the "message" ] 169 + 170 + [ triggered by bast of "ack" ] 171 + receiver get CR on "message" 172 + receiver read LVB 173 + receiver processes the message 174 + [ wait finish ] 175 + receiver releases "ack" 176 + receiver tries to get PR on "message" 177 + 178 + sender receiver receiver 179 + "token":EX "message":CR "message":CR 180 + "message":CW 181 + "ack":EX 159 182 160 183 4. triggered by grant of EX on "ack" (indicating all receivers 161 184 have processed message) 162 - sender down-converts "ack" from EX to CR 163 - sender releases "message" 164 - sender releases "token" 165 - receiver upconvert to PR on "message" 166 - receiver get CR of "ack" 167 - receiver release "message" 168 185 169 - sender receiver receiver 170 - "ack":CR "ack":CR "ack":CR 186 + sender down-converts "ack" from EX to CR 187 + 188 + sender releases "message" 189 + 190 + sender releases "token" 191 + 192 + :: 193 + 194 + receiver upconvert to PR on "message" 195 + receiver get CR of "ack" 196 + receiver release "message" 197 + 198 + sender receiver receiver 199 + "ack":CR "ack":CR "ack":CR 171 200 172 201 173 202 4. Handling Failures 203 + ==================== 174 204 175 205 4.1 Node Failure 206 + ---------------- 176 207 177 208 When a node fails, the DLM informs the cluster with the slot 178 209 number. The node starts a cluster recovery thread. The cluster ··· 224 177 - cleans the bitmap of the failed node 225 178 - releases bitmap<number> lock of the failed node 226 179 - initiates resync of the bitmap on the current node 227 - md_check_recovery is invoked within recover_bitmaps, 228 - then md_check_recovery -> metadata_update_start/finish, 229 - it will lock the communication by lock_comm. 230 - Which means when one node is resyncing it blocks all 231 - other nodes from writing anywhere on the array. 180 + md_check_recovery is invoked within recover_bitmaps, 181 + then md_check_recovery -> metadata_update_start/finish, 182 + it will lock the communication by lock_comm. 183 + Which means when one node is resyncing it blocks all 184 + other nodes from writing anywhere on the array. 232 185 233 186 The resync process is the regular md resync. However, in a clustered 234 187 environment when a resync is performed, it needs to tell other nodes ··· 245 198 particular I/O range should be suspended or not. 246 199 247 200 4.2 Device Failure 201 + ================== 248 202 249 203 Device failures are handled and communicated with the metadata update 250 204 routine. When a node detects a device failure it does not allow ··· 253 205 acknowledged by all other nodes. 254 206 255 207 5. Adding a new Device 208 + ---------------------- 256 209 257 210 For adding a new device, it is necessary that all nodes "see" the new 258 211 device to be added. For this, the following algorithm is used: 259 212 260 - 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues 213 + 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues 261 214 ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD) 262 - 2. Node 1 sends a NEWDISK message with uuid and slot number 263 - 3. Other nodes issue kobject_uevent_env with uuid and slot number 215 + 2. Node 1 sends a NEWDISK message with uuid and slot number 216 + 3. Other nodes issue kobject_uevent_env with uuid and slot number 264 217 (Steps 4,5 could be a udev rule) 265 - 4. In userspace, the node searches for the disk, perhaps 218 + 4. In userspace, the node searches for the disk, perhaps 266 219 using blkid -t SUB_UUID="" 267 - 5. Other nodes issue either of the following depending on whether 220 + 5. Other nodes issue either of the following depending on whether 268 221 the disk was found: 269 222 ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and 270 - disc.number set to slot number) 223 + disc.number set to slot number) 271 224 ioctl(CLUSTERED_DISK_NACK) 272 - 6. Other nodes drop lock on "no-new-devs" (CR) if device is found 273 - 7. Node 1 attempts EX lock on "no-new-dev" 274 - 8. If node 1 gets the lock, it sends METADATA_UPDATED after 225 + 6. Other nodes drop lock on "no-new-devs" (CR) if device is found 226 + 7. Node 1 attempts EX lock on "no-new-dev" 227 + 8. If node 1 gets the lock, it sends METADATA_UPDATED after 275 228 unmarking the disk as SpareLocal 276 - 9. If not (get "no-new-dev" lock), it fails the operation and sends 229 + 9. If not (get "no-new-dev" lock), it fails the operation and sends 277 230 METADATA_UPDATED. 278 231 10. Other nodes get the information whether a disk is added or not 279 232 by the following METADATA_UPDATED. 280 233 281 - 6. Module interface. 234 + 6. Module interface 235 + =================== 282 236 283 237 There are 17 call-backs which the md core can make to the cluster 284 238 module. Understanding these can give a good overview of the whole 285 239 process. 286 240 287 241 6.1 join(nodes) and leave() 242 + --------------------------- 288 243 289 244 These are called when an array is started with a clustered bitmap, 290 245 and when the array is stopped. join() ensures the cluster is ··· 295 244 Only the first 'nodes' nodes in the cluster can use the array. 296 245 297 246 6.2 slot_number() 247 + ----------------- 298 248 299 249 Reports the slot number advised by the cluster infrastructure. 300 250 Range is from 0 to nodes-1. 301 251 302 252 6.3 resync_info_update() 253 + ------------------------ 303 254 304 255 This updates the resync range that is stored in the bitmap lock. 305 256 The starting point is updated as the resync progresses. The ··· 309 256 It does *not* send a RESYNCING message. 310 257 311 258 6.4 resync_start(), resync_finish() 259 + ----------------------------------- 312 260 313 261 These are called when resync/recovery/reshape starts or stops. 314 262 They update the resyncing range in the bitmap lock and also ··· 319 265 resync_finish() also sends a BITMAP_NEEDS_SYNC message which 320 266 allows some other node to take over. 321 267 322 - 6.5 metadata_update_start(), metadata_update_finish(), 323 - metadata_update_cancel(). 268 + 6.5 metadata_update_start(), metadata_update_finish(), metadata_update_cancel() 269 + ------------------------------------------------------------------------------- 324 270 325 271 metadata_update_start is used to get exclusive access to 326 272 the metadata. If a change is still needed once that access is ··· 329 275 can be used to release the lock. 330 276 331 277 6.6 area_resyncing() 278 + -------------------- 332 279 333 280 This combines two elements of functionality. 334 281 ··· 344 289 a node failure. 345 290 346 291 6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack() 292 + --------------------------------------------------------------- 347 293 348 294 These are used to manage the new-disk protocol described above. 349 295 When a new device is added, add_new_disk_start() is called before ··· 356 300 new_disk_ack() is called. 357 301 358 302 6.8 remove_disk() 303 + ----------------- 359 304 360 305 This is called when a spare or failed device is removed from 361 306 the array. It causes a REMOVE message to be send to other nodes. 362 307 363 308 6.9 gather_bitmaps() 309 + -------------------- 364 310 365 311 This sends a RE_ADD message to all other nodes and then 366 312 gathers bitmap information from all bitmaps. This combined 367 313 bitmap is then used to recovery the re-added device. 368 314 369 315 6.10 lock_all_bitmaps() and unlock_all_bitmaps() 316 + ------------------------------------------------ 370 317 371 318 These are called when change bitmap to none. If a node plans 372 319 to clear the cluster raid's bitmap, it need to make sure no other ··· 378 319 accordingly. 379 320 380 321 7. Unsupported features 322 + ======================= 381 323 382 324 There are somethings which are not supported by cluster MD yet. 383 325

+15 -13

Documentation/md/raid5-cache.txt Documentation/md/raid5-cache.rst

··· 1 - RAID5 cache 1 + ================ 2 + RAID 4/5/6 cache 3 + ================ 2 4 3 5 Raid 4/5/6 could include an extra disk for data cache besides normal RAID 4 6 disks. The role of RAID disks isn't changed with the cache disk. The cache disk ··· 8 6 since 4.4) or write-back mode (supported since 4.10). mdadm (supported since 9 7 3.4) has a new option '--write-journal' to create array with cache. Please 10 8 refer to mdadm manual for details. By default (RAID array starts), the cache is 11 - in write-through mode. A user can switch it to write-back mode by: 9 + in write-through mode. A user can switch it to write-back mode by:: 12 10 13 - echo "write-back" > /sys/block/md0/md/journal_mode 11 + echo "write-back" > /sys/block/md0/md/journal_mode 14 12 15 - And switch it back to write-through mode by: 13 + And switch it back to write-through mode by:: 16 14 17 - echo "write-through" > /sys/block/md0/md/journal_mode 15 + echo "write-through" > /sys/block/md0/md/journal_mode 18 16 19 17 In both modes, all writes to the array will hit cache disk first. This means 20 18 the cache disk must be fast and sustainable. 21 19 22 - ------------------------------------- 23 - write-through mode: 20 + write-through mode 21 + ================== 24 22 25 23 This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an unclean 26 24 shutdown can cause data in some stripes to not be in consistent state, eg, data ··· 44 42 In write-through mode, the cache disk isn't required to be big. Several 45 43 hundreds megabytes are enough. 46 44 47 - -------------------------------------- 48 - write-back mode: 45 + write-back mode 46 + =============== 49 47 50 48 write-back mode fixes the 'write hole' issue too, since all write data is 51 49 cached on cache disk. But the main goal of 'write-back' cache is to speed up ··· 66 64 In write-back mode, MD also caches data in memory. The memory cache includes 67 65 the same data stored on cache disk, so a power loss doesn't cause data loss. 68 66 The memory cache size has performance impact for the array. It's recommended 69 - the size is big. A user can configure the size by: 67 + the size is big. A user can configure the size by:: 70 68 71 - echo "2048" > /sys/block/md0/md/stripe_cache_size 69 + echo "2048" > /sys/block/md0/md/stripe_cache_size 72 70 73 71 Too small cache disk will make the write aggregation less efficient in this 74 72 mode depending on the workloads. It's recommended to use a cache disk with at 75 73 least several gigabytes size in write-back mode. 76 74 77 - -------------------------------------- 78 - The implementation: 75 + The implementation 76 + ================== 79 77 80 78 The write-through and write-back cache use the same disk format. The cache disk 81 79 is organized as a simple write log. The log consists of 'meta data' and 'data'

+2

Documentation/md/raid5-ppl.txt Documentation/md/raid5-ppl.rst

··· 1 + ================== 1 2 Partial Parity Log 3 + ================== 2 4 3 5 Partial Parity Log (PPL) is a feature available for RAID5 arrays. The issue 4 6 addressed by PPL is that after a dirty shutdown, parity of a particular stripe

Configure Feed

Configure Feed