+350

blog/fun-with-redirection-2021-09-22.markdown

reviewed

··· 1 1 + --- 2 2 + title: Fun with Redirection 3 3 + date: 2021-09-22 4 4 + author: Twi 5 5 + tags: 6 6 + - shell 7 7 + - redirection 8 8 + - osdev 9 9 + --- 10 10 + 11 11 + When you're hacking in the shell or in a script, sometimes you want to change 12 12 + how the output of a command is routed. Today I'm gonna cover common shell 13 13 + redirection tips and tricks that I use every day at work and how it all works 14 14 + under the hood. 15 15 + 16 16 + Let's say you're trying to capture the output of a command to a file, such as 17 17 + `uname -av`: 18 18 + 19 19 + ```console 20 20 + $ uname -av 21 21 + Linux shachi 5.13.15 #1-NixOS SMP Wed Sep 8 06:50:21 UTC 2021 x86_64 GNU/Linux 22 22 + ``` 23 23 + 24 24 + You could copy that to the clipboard and paste it into a file, but there is a 25 25 + better way thanks to the `>` operator: 26 26 + 27 27 + ```console 28 28 + $ uname -av > uname.txt 29 29 + $ cat uname.txt 30 30 + Linux shachi 5.13.15 #1-NixOS SMP Wed Sep 8 06:50:21 UTC 2021 x86_64 GNU/Linux 31 31 + ``` 32 32 + 33 33 + Let's say you want to run this on a few machines and put all of the output into 34 34 + `uname.txt`. You could write a shell script loop like this: 35 35 + 36 36 + ```sh 37 37 + # make sure the file doesn't already exist 38 38 + rm -f uname.txt 39 39 + 40 40 + for host in shachi chrysalis kos-mos ontos pneuma 41 41 + do 42 42 + ssh $host -- uname -av >> uname.txt 43 43 + done 44 44 + ``` 45 45 + 46 46 + Then `uname.txt` should look like this: 47 47 + 48 48 + ``` 49 49 + Linux shachi 5.13.15 #1-NixOS SMP Wed Sep 8 06:50:21 UTC 2021 x86_64 GNU/Linux 50 50 + Linux chrysalis 5.10.63 #1-NixOS SMP Wed Sep 8 06:49:02 UTC 2021 x86_64 GNU/Linux 51 51 + Linux kos-mos 5.10.45 #1-NixOS SMP Fri Jun 18 08:00:06 UTC 2021 x86_64 GNU/Linux 52 52 + Linux ontos 5.10.52 #1-NixOS SMP Tue Jul 20 14:05:59 UTC 2021 x86_64 GNU/Linux 53 53 + Linux pneuma 5.10.57 #1-NixOS SMP Sun Aug 8 07:05:24 UTC 2021 x86_64 GNU/Linux 54 54 + ``` 55 55 + 56 56 + Now let's say you want to extract all of the hostnames from that `uname.txt`. 57 57 + The pattern of the file seems to specify that fields are separated by spaces and 58 58 + the hostname seems to be the second space-separated field in each line. You can 59 59 + use the `cut` command to select that small subset from each line, and you can 60 60 + feed the `cut` command's standard input using the `<` operator: 61 61 + 62 62 + ```console 63 63 + $ cut -d' ' -f2 < uname.txt 64 64 + shachi 65 65 + chrysalis 66 66 + kos-mos 67 67 + ontos 68 68 + pneuma 69 69 + ``` 70 70 + 71 71 + [It's worth noting that a lot of these core CLI utilities are built on the idea 72 72 + that they are _filters_, or things that take one infinite stream of text in on 73 73 + one end and then return another stream of text out the other 74 74 + end. This is done through a channel called "standard input/output", where 75 75 + standard input refers to input to the command and standard output refers to the 76 76 + output of the command.](conversation://Mara/hacker) 77 77 + 78 78 + [That's a great metaphor, let's build onto it using the `|` (pipe) 79 79 + operator. The pipe operator lets you pipe the standard output of one command to 80 80 + the standard input of another.](conversation://Cadey/enby) 81 81 + 82 82 + [You mentioned that you can pass files as input and output for commands, does 83 83 + this mean that standard input and standard output are 84 84 + files?](conversation://Mara/happy) 85 85 + 86 86 + [Precisely! They are just files that are automatically open for every process. 87 87 + Usually commands will output to standard out and some will also accept input via 88 88 + standard in.](conversation://Cadey/enby) 89 89 + 90 90 + [Doesn't that have some level of overhead though? Isn't it expensive to spin up 91 91 + a whole heckin' `cat` process for that?](conversation://Mara/hmm) 92 92 + 93 93 + [Not on any decent system made in the last 20 years. This may have some impact 94 94 + on Windows (because they have core architectural mistakes that make processes 95 95 + take up to 100 milliseconds to spin up), but this is about Unix/Linux. I think 96 96 + these should work on Windows too if you use Cygwin, but if you're using WSL you 97 97 + shouldn't have any real issues there](conversation://Cadey/coffee) 98 98 + 99 99 + Let's say we want to rewrite that `cut` command above to use pipes. You could 100 100 + write it like this: 101 101 + 102 102 + ```sh 103 103 + cat uname.txt | cut -d' ' -f2 104 104 + ``` 105 105 + 106 106 + [The mnemonic we use for remembering the `cut` command is that fields are 107 107 + separated by the `d`elimiter and you cut out the nth 108 108 + `f`ield/s. You can use ](conversation://Mara/hacker) 109 109 + 110 110 + This will get you the exact same output: 111 111 + 112 112 + ```console 113 113 + $ cat uname.txt | cut -d' ' -f2 114 114 + shachi 115 115 + chrysalis 116 116 + kos-mos 117 117 + ontos 118 118 + pneuma 119 119 + ``` 120 120 + 121 121 + Personally I prefer writing shell pipelines like that as it makes it a bit 122 122 + easier to tack on more specific selectors or operations as you go along. For 123 123 + example, if you wanted to sort them you could pipe the result to `sort`: 124 124 + 125 125 + ```console 126 126 + $ cat uname.txt | cut -d' ' -f2 | sort 127 127 + chrysalis 128 128 + kos-mos 129 129 + ontos 130 130 + pneuma 131 131 + shachi 132 132 + ``` 133 133 + 134 134 + This lets you gradually build up a shell pipeline as you drill down to the data 135 135 + you want in the format you want. 136 136 + 137 137 + [I wanted to save this compiler error to a file but it didn't work. I tried 138 138 + doing this:](conversation://Mara/hmm) 139 139 + 140 140 + ```console 141 141 + $ rustc foo.rs > foo.log 142 142 + ``` 143 143 + 144 144 + But the output printed to the screen instead of the file: 145 145 + 146 146 + ```console 147 147 + $ rustc foo.rs > foo.log 148 148 + error: expected one of `!` or `::`, found `main` 149 149 + --> foo.rs:1:5 150 150 + | 151 151 + 1 | fun main() {} 152 152 + | ^^^^ expected one of `!` or `::` 153 153 + 154 154 + error: aborting due to previous error 155 155 + 156 156 + $ cat foo.log 157 157 + $ 158 158 + ``` 159 159 + 160 160 + This happens because there are actually _two_ output streams per program. There 161 161 + is the standard out stream and there is also a standard error stream. The reason 162 162 + that standard error exists is so that you can see if any errors have happened if 163 163 + you redirect standard out. 164 164 + 165 165 + Sometimes standard out may not be a stream of text, say you have a compressed 166 166 + file you want to analyze and there's an issue with the decompression. If the 167 167 + decompressor wrote its errors to the standard output stream, it could confuse or 168 168 + corrupt your analysis. 169 169 + 170 170 + However, we can redirect standard error in particular by modifying how we 171 171 + redirect to the file: 172 172 + 173 173 + ```console 174 174 + $ rustc foo.rs 2> foo.log 175 175 + $ cat foo.log 176 176 + error: expected one of `!` or `::`, found `main` 177 177 + --> foo.rs:1:5 178 178 + | 179 179 + 1 | fun main() {} 180 180 + | ^^^^ expected one of `!` or `::` 181 181 + 182 182 + error: aborting due to previous error 183 183 + ``` 184 184 + 185 185 + [Where did the `2` come from?](conversation://Mara/wat) 186 186 + 187 187 + So I mentioned earlier that redirection modifies the standard input and output 188 188 + of programs. This is not entirely true, but it was a convenient half-truth to 189 189 + help build this part of the explanation. 190 190 + 191 191 + For every process on a Unix-like system (such as Linux and macOS), the kernel 192 192 + stores a list of active file-like objects. This includes real files on the 193 193 + filesystem, pipes between processes, network sockets, and more. When a program 194 194 + reads or writes a file, they tell the kernel which file they want to use by 195 195 + giving it a number index into that list, starting at zero. Standard in/out/error 196 196 + are just the conventional names for the first three open files in the list, like 197 197 + this: 198 198 + 199 199 + | File Descriptor | Purpose | 200 200 + | :------ | :------- | 201 201 + | 0 | Standard input | 202 202 + | 1 | Standard output | 203 203 + | 2 | Standard error | 204 204 + 205 205 + Shell redirection simply changes which files are in that list of open files when 206 206 + the program starts running. 207 207 + 208 208 + That is why you use a `2` there, because you are telling the shell to change 209 209 + file descriptor number `2` of the `rustc` process to point to the filesystem 210 210 + file `foo.log`, which in turn makes the standard error of `rustc` get written to 211 211 + that file for you. 212 212 + 213 213 + In turn, this also means that `cat foo.txt > foo2.txt` is actually a shortcut 214 214 + for saying `cat foo.txt 1> foo2.txt`, but the `1` can be omitted there because 215 215 + standard out is usually the "default" output that most of these kind of 216 216 + pipelines cares about. 217 217 + 218 218 + [How would I get both standard output and standard error in the same 219 219 + file?](conversation://Mara/hmm) 220 220 + 221 221 + The cool part about the `>` operator is that it doesn't just stop with output to 222 222 + files on the desk, you can actually have one file descriptor get pointed to 223 223 + another. Let's say you have a need for both standard out and standard error to 224 224 + go to the same file. You can do this with a command like this: 225 225 + 226 226 + ``` 227 227 + $ rustc foo.rs 2>&1 > foo.log 228 228 + ``` 229 229 + 230 230 + This tells the shell to point standard error to standard out and then the 231 231 + combined output to `foo.log`. There's a short form of this too: 232 232 + 233 233 + ``` 234 234 + $ rustc foo.rs &> foo.log 235 235 + ``` 236 236 + 237 237 + [Where can I expect to use that?](conversation://Mara/hmm) 238 238 + 239 239 + [It's a bourne shell extension, but I've tested it in `zsh` and `fish`. You can 240 240 + also do `&|` to pipe both standard out and standard error at the same time in 241 241 + the same way you'd do `2>&1 | whatever`.](conversation://Cadey/enby) 242 242 + 243 243 + That will put standard out and standard error to `foo.log` the same way that 244 244 + `2>&1 > foo.log` will. You can also use this with `>>`: 245 245 + 246 246 + ``` 247 247 + $ rustc foo.rs &>> foo.log 248 248 + $ cat foo.log 249 249 + error: expected one of `!` or `::`, found `main` 250 250 + --> foo.rs:1:5 251 251 + | 252 252 + 1 | fun main() {} 253 253 + | ^^^^ expected one of `!` or `:: 254 254 + 255 255 + error: aborting due to previous error 256 256 + 257 257 + error: expected one of `!` or `::`, found `main` 258 258 + --> foo.rs:1:5 259 259 + | 260 260 + 1 | fun main() {} 261 261 + | ^^^^ expected one of `!` or `::` 262 262 + 263 263 + error: aborting due to previous error 264 264 + ``` 265 265 + 266 266 + [How do I redirect standard in to a file?](conversation://Mara/hmm) 267 267 + 268 268 + The answer there is not directly! There is a workaround in the form of a tool 269 269 + called `tee` which outputs its standard in to both standard out and a file. For 270 270 + example: 271 271 + 272 272 + ```console 273 273 + $ dmesg | tee dmesg.txt | grep 'msedge' 274 274 + [ 70.585463] traps: msedge[4715] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 275 275 + [ 70.702544] traps: msedge[4745] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 276 276 + [ 70.806296] traps: msedge[4781] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 277 277 + [ 70.918095] traps: msedge[4889] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 278 278 + [ 71.031938] traps: msedge[4926] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 279 279 + [ 71.138974] traps: msedge[4935] trap invalid opcode ip:5630ddcedc4c sp:7ffd41f67700 error:0 in msedge[5630d8fc2000+952d000] 280 280 + [ 1169.163603] traps: msedge[35719] trap invalid opcode ip:556a93951c4c sp:7ffc533f35c0 error:0 in msedge[556a8ec26000+952d000] 281 281 + [ 1213.301722] traps: msedge[36054] trap invalid opcode ip:55a245960c4c sp:7ffe6d169b40 error:0 in msedge[55a240c35000+952d000] 282 282 + [10963.234459] traps: msedge[104732] trap invalid opcode ip:55fdb864fc4c sp:7ffc996dfee0 error:0 in msedge[55fdb3924000+952d000] 283 283 + ``` 284 284 + 285 285 + This would put the output of the `dmesg` command (read from kernel logs) into 286 286 + `dmesg.txt`, as well as sending it into the grep command. You might want to do 287 287 + this when debugging long command pipelines to see exactly what is going into a 288 288 + program that isn't doing what you expect. 289 289 + 290 290 + Redirections also work in scripts too. You can also set "default" redirects for 291 291 + every command in a script using the `exec` command: 292 292 + 293 293 + ```sh 294 294 + exec > out.log 2> error.log 295 295 + 296 296 + ls 297 297 + rustc foo.rs 298 298 + ``` 299 299 + 300 300 + This will have the file listing from `ls` written to `out.log` and any errors 301 301 + from `rustc` written to `error.log`. 302 302 + 303 303 + A lot of other shell tricks and fun is built on top of these fundamentals. For 304 304 + example you can take a folder, zip it up and then unzip it over on another 305 305 + machine using a command like this: 306 306 + 307 307 + ``` 308 308 + $ tar cz ./blog | ssh pneuma tar xz -C ~/code/christine.website/blog 309 309 + ``` 310 310 + 311 311 + This will run `tar` to create a compressed copy of the `./blog` folder and then 312 312 + pipe that to tar on another computer to extract that into 313 313 + `~/code/christine.website/blog`. It's just pipes and redirection all the way 314 314 + down! Deep inside `ssh` it's really just piping output of commands back and 315 315 + forth over an encrypted network socket. Connecting to an IRC server is just 316 316 + piping in and out data to the chat server, even more so if you use TLS to 317 317 + connect there. In a way you can model just about everything in Unix with pipes 318 318 + and file descriptors because that is the cornerstone of its design: Everything 319 319 + is a file. 320 320 + 321 321 + [This doesn't mean it's literally a file on the disk, it means you can _interact 322 322 + with_ just about everything using the same system interface as you do with 323 323 + files. Even things like hard disks and video cards.](conversation://Mara/hacker) 324 324 + 325 325 + Here's a fun thing to do. Using [`curl`](https://curl.se/) to read the contents 326 326 + of a URL and [`jq`](https://stedolan.github.io/jq/) to select out bits from a 327 327 + JSON stream, you can make a script that lets you read the most recent title from 328 328 + my blog's [JSONFeed](/blog.json): 329 329 + 330 330 + ```sh 331 331 + #!/usr/bin/env bash 332 332 + # xeblog-post.sh 333 333 + 334 334 + curl -s https://christine.website/blog.json | jq -r '.items[0] | "\(.title) \(.url)"' 335 335 + ``` 336 336 + 337 337 + At the time of writing this post, here is the output I get from this command: 338 338 + 339 339 + ``` 340 340 + $ ./xeblog-post.sh 341 341 + Anbernic RG280M Review https://christine.website/blog/rg280m-review 342 342 + ``` 343 343 + 344 344 + What else could you do with pipes and redirection? The cloud's the limit! 345 345 + 346 346 + --- 347 347 + 348 348 + Thanks to violet spark for looking over this post and fact-checking as well as 349 349 + helping mend some of the brain dump and awkward wording into more polished 350 350 + sentences.

+8

config.dhall

reviewed

··· 70 70 , twitter = Some "BeJustFine" 71 71 , inSystem = True 72 72 } 73 73 + , Author::{ 74 74 + , name = "Nicole" 75 75 + , handle = "Twi" 76 76 + , picUrl = None Text 77 77 + , link = None Text 78 78 + , twitter = None Text 79 79 + , inSystem = True 80 80 + } 73 81 ] 74 82 , port = defaultPort 75 83 , clackSet = [ "Ashlynn" ]