The code and data behind xeiaso.net
talks: add static analysis talk
Signed-off-by: Xe Iaso <me@christine.website>
+442
-15
+2
-1
scripts/resize
+2
-1
scripts/resize
+11
-2
src/handlers/talks.rs
+11
-2
src/handlers/talks.rs
···
4
4
extract::{Extension, Path},
5
5
response::Html,
6
6
};
7
+
use http::header::HeaderMap;
7
8
use lazy_static::lazy_static;
8
9
use prometheus::{opts, register_int_counter_vec, IntCounterVec};
9
10
use std::sync::Arc;
···
25
26
Ok(Html(result))
26
27
}
27
28
28
-
#[instrument(skip(state))]
29
+
#[instrument(skip(state, headers))]
29
30
pub async fn post_view(
30
31
Path(name): Path<String>,
31
32
Extension(state): Extension<Arc<State>>,
33
+
headers: HeaderMap,
32
34
) -> Result {
33
35
let mut want: Option<Post> = None;
34
36
···
38
40
}
39
41
}
40
42
43
+
let referer = if let Some(referer) = headers.get(http::header::REFERER) {
44
+
let referer = referer.to_str()?.to_string();
45
+
Some(referer)
46
+
} else {
47
+
None
48
+
};
49
+
41
50
match want {
42
51
None => Err(PostNotFound(name).into()),
43
52
Some(post) => {
···
46
55
.inc();
47
56
let body = templates::Html(post.body_html.clone());
48
57
let mut result: Vec<u8> = vec![];
49
-
templates::talkpost_html(&mut result, post, body)?;
58
+
templates::talkpost_html(&mut result, post, body, referer)?;
50
59
Ok(Html(result))
51
60
}
52
61
}
+423
talks/conf42-static-analysis.markdown
+423
talks/conf42-static-analysis.markdown
···
1
+
---
2
+
title: How Static Code Analysis Prevents You From Waking Up at 3AM With Production on Fire
3
+
date: 2022-06-09
4
+
slides_link: https://cdn.xeiaso.net/file/christine-static/talks/Conf42+SRE+2022.pdf
5
+
---
6
+
7
+
# How Static Code Analysis Prevents You From Waking Up at 3AM With Production on Fire
8
+
9
+
<style>
10
+
img {
11
+
display: block;
12
+
margin-left: auto;
13
+
margin-right: auto;
14
+
}
15
+
</style>
16
+
17
+
<center><iframe width="560" height="315" src="https://www.youtube.com/embed/cVUrScvthqs"
18
+
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
19
+
allowfullscreen></iframe></center>
20
+
21
+
<xeblog-conv name="Cadey" mood="coffee">The talk video will be live at 2022 M06
22
+
10 at 13:00 EDT. It will not work if you are reading this at the exact
23
+
time of release or before it is released via Patreon.</xeblog-conv>
24
+
25
+
26
+
27
+
Hi, I’m Xe Iaso and today I’m going to talk about static analysis and how it
28
+
helps you engineer more reliable systems. This will help you make it harder for
29
+
incorrect code to blow up production at 3AM. There are a lot of tools out there
30
+
that can do this for a variety of languages, however I’m going to focus on Go
31
+
because that is what I am an expert in. In this talk I’ll cover the problem
32
+
space, some solutions you can apply today and how you can work with people to
33
+
engineer more reliable systems.
34
+
35
+
36
+
37
+
As I said, I’m Xe. I’m the Archmage of Infrastructure at Tailscale. I’ve been an
38
+
SRE for long enough that I have moved over into developer relations. As a
39
+
disclaimer, this talk may contain opinions. None of these opinions are of my
40
+
employer.
41
+
42
+
I’ll have a recording of this talk, slides, speaker notes, and a transcript of
43
+
up in a day or two after the conference. The QR code in the corner of the screen
44
+
will take you to my blog.
45
+
46
+
47
+
48
+
When starting to think about the problem, I find it helps to start thinking
49
+
about the problem space. This usually means thinking about the total problem at
50
+
an incredibly high level.
51
+
52
+
So let’s think about the problem space of compilers. At the highest possible
53
+
level, a compiler can take literally anything as input and maybe produce an
54
+
output.
55
+
56
+
57
+
58
+
A compiler’s job is to take this anything, see if it matches a set of rules and
59
+
then produce an output of some kind. In the case of the Go compiler, this means
60
+
that the input needs to match the rules that the Go language has defined in its
61
+
specification.
62
+
63
+
64
+
65
+
This human-readable specification outlines core rules of the Go language. These
66
+
include things like every `.go` file needs to be in a package, the need to
67
+
declare variables before using them, what core types are in the language, how to
68
+
deal with slices, etc.
69
+
70
+
However this specification doesn’t define what _correct_ Go code is. It only
71
+
defines what _valid_ Go code is. This is normal for specifications of this kind,
72
+
ensuring correctness is an active field of research in computer science that
73
+
small scrappy startups like Google, Microsoft and Apple struggle with.
74
+
75
+
76
+
77
+
As a result though, you can’t rely on the compiler itself from stopping
78
+
incorrect code to be deployed into production. A lot of trivial errors will be
79
+
stopped in the process, but it won’t stop more subtle errors. This is an
80
+
example of the kind of error that the Go compiler can catch by itself, if you
81
+
declare a value as an integer you can’t then put a string in it. They are
82
+
different types and the compiler will reject it.
83
+
84
+
85
+
86
+
I know one of you out there is probably thinking something like “What about
87
+
other languages like Rust and Haskell? Aren’t those compilers known for
88
+
correctness?”
89
+
90
+
91
+
92
+
That’s a good point, there are other languages that have more strict rules like
93
+
linear types and explicitly marking poking the outside world. However the kinds
94
+
of errors that are brought up in this talk can still happen in those languages,
95
+
even if it’s more difficult to do that by accident.
96
+
97
+
98
+
99
+
Static analysis on top of your existing compiler lets you move closer to
100
+
correctness without going the maximalist route like when using Rust or Haskell.
101
+
102
+
103
+
104
+
It’s a balance between pragmatism and correctness. The pragmatic solution and
105
+
the correct solution are always in conflict, so you need to find a way down the
106
+
middle.
107
+
108
+
109
+
110
+
In general, proving everything is correct with static analysis is impossible. It
111
+
takes a theoretically infinite amount of time to tell if absolutely every facet
112
+
of the code is correct in every single way. This is a case where the perfect is
113
+
the enemy of the good, so here are some patterns for things that can be proven
114
+
with static analysis in Go:
115
+
116
+
117
+
118
+
* Forgetting to close an HTTP response body
119
+
* Making typos in struct tags
120
+
* Ensuring that cancellable contexts get cancelled in trivially provable ways
121
+
* Writing invalid time formats
122
+
* Writing an invalid regular expression that would otherwise blow up at runtime
123
+
124
+
125
+
126
+
These kinds of things are easy to prove and are enabled by default in `go vet`
127
+
and staticcheck.
128
+
129
+
Also for the record, incorrect code won’t explode instantly upon it being run.
130
+
The devil is in the details of how it is incorrect and how those things can pile
131
+
up to create issues downstream. Incorrect code can also confuse you while trying
132
+
to debug it, which can make you waste time you could spend doing anything else.
133
+
134
+
135
+
136
+
This is an example of Go code that will compile, will likely work, but is incorrect.
137
+
138
+
139
+
140
+
This is incorrect because the HTTP response is read from, but never closed.
141
+
Failing to do this in Go will cause you to leak the resources associated with
142
+
the HTTP connection. When you close the response, it releases the connection so
143
+
that it can be used for other HTTP actions.
144
+
145
+
If you don’t do this, you can easily run into a state where your server
146
+
application will run out of available sockets at 3AM. So you may be tempted to
147
+
fix it like this:
148
+
149
+
150
+
151
+
However this is incorrect too. Look at where the `defer` is called.
152
+
153
+
Let’s think about how the program flow will work. I’m going to translate this
154
+
into a diagram of how this program will be executed.
155
+
156
+
157
+
158
+
This flowchart is another way to think about how this program is being executed.
159
+
It starts on the left side and flows to the end on the right.
160
+
161
+
162
+
163
+
In this case we start with the http dot Get call and then defer closing the
164
+
response body. Then we check to see if there was an error or not.
165
+
166
+
167
+
168
+
If there wasn’t an error, we can use the response and do something useful, then
169
+
the response body closes automatically due to the deferred close. Everything
170
+
works as expected.
171
+
172
+
173
+
174
+
However if there was an error, something different happens. The error is
175
+
returned and then the scheduled Close call runs. The Close call assumes that the
176
+
response is valid, but it’s not. This results in the program panicking which is
177
+
a crash at 3AM. This is the kind of place that static analysis comes in to save
178
+
you. Let’s take a look at what `go vet` says about this code:
179
+
180
+
181
+
182
+
It caught that error! To fix this we need to move the `defer` call to after the
183
+
error check like this:
184
+
185
+
186
+
187
+
The response body is closed after we know it’s usable. This will work as we
188
+
expect in production. This is an example of how trivial errors can be fixed with
189
+
a little extra tooling without having to use an entirely maximalist approach.
190
+
191
+
192
+
193
+
If you use `go test` then a large amount of `go vet` checks are run by default.
194
+
This covers a wide variety of common issues with trivial fixes that help move
195
+
your code towards the corresponding Go idioms. It’s limited to the subset of
196
+
tests that aren’t known to have false positives, so if you want to have more
197
+
assurance you will need to run `go vet` in your continuous integration step.
198
+
199
+
200
+
201
+
<xeblog-conv name="Mara" mood="hmm">If these are so trivially detectable, why
202
+
isn’t this part of the normal `go build` flow?</xeblog-conv>
203
+
204
+
205
+
206
+
The reason this isn’t done by default is kind of a matter of philosophy. Go
207
+
isn’t a language that wants to make it impossible to write buggy code. Go just
208
+
wants to give you tools to make your life easier.
209
+
210
+
In the Go team’s view, they would rather buggy code get compiled than have the
211
+
compiler reject your code on accident.
212
+
213
+
It’s the result a philosophy of trusting that there are gaps between the
214
+
programmer and production servers. During those gaps there are tools like
215
+
Staticcheck and `go vet` in addition to human review.
216
+
217
+
218
+
219
+
Here’s an example of a more complicated problem that Staticcheck can catch.
220
+
221
+
222
+
223
+
Go lets you make variables that are scoped to if statements. This lets you write
224
+
code like this:
225
+
226
+
227
+
228
+
Which is shorthand for writing out something like this:
229
+
230
+
231
+
232
+
This does the same thing, but it looks a bit more ugly. The `err` value isn’t in
233
+
scope at the end of the inline block, so it will be dropped by the garbage
234
+
collector.
235
+
236
+
237
+
238
+
However let’s also consider the other important part of this snippet: variable shadowing.
239
+
240
+
241
+
242
+
We have two different variables named `x` and they are different types and
243
+
declared at different places. To help tell them apart I’ve coloured the inner
244
+
one yellow and the outer one red.
245
+
246
+
In a type assertion like this the red variable is not an `int` but the yellow
247
+
variable is an `int` that might have failed to assert down. If it fails to
248
+
assert down, then the yellow `x` variable will always be an `int` have the value
249
+
`0`. This is probably not what you want, given that the log call with `%T`
250
+
format specifier would let you know what type the red `x` variable was.
251
+
252
+
When this code is run, you will get an error message like this:
253
+
254
+
255
+
256
+
This will confuse the living hell out of you. The correct fix here is to rename
257
+
the int version of `x`. You could do this in a few ways, but here’s a valid
258
+
approach:
259
+
260
+
261
+
262
+
This will get the correct result. You would need to change the `ok` branch of
263
+
the `if` statement to use `xInt` instead of `x`, but the Go language server can
264
+
usually automate this (in Emacs you’d press `M-x` and type in `lsp-rename` and
265
+
hit enter).
266
+
267
+
There are a bunch of other checks that Staticcheck runs by default and I could
268
+
easily talk about them for a few hours, but I’m gonna focus on one of the more
269
+
interestingly subtle checks.
270
+
271
+
272
+
273
+
In Go it’s a common pattern to write custom error types. With Go interfaces and
274
+
their “duck typing”, anything that matches the definition of the `error`
275
+
interface is able to be used as an `error` value.
276
+
277
+
278
+
279
+
The type Failure has an Error method, which means that we can treat it as an
280
+
error.
281
+
282
+
283
+
284
+
However the receiver of the function is a pointer value. Normally this means a
285
+
few things, but in this case it means that the receiver may be nil.
286
+
287
+
288
+
289
+
Because of this we can return a nil value of Failure, but then when you try to
290
+
use it from Go it will explode at runtime:
291
+
292
+
293
+
294
+
Boom! It crashed! Segfault!
295
+
296
+
297
+
298
+
This happens because under the hood each interface value is a box. This box
299
+
contains the type of the value in the box and a pointer to the actual value
300
+
itself. But, this box will always exist even if the underlying value is `nil`.
301
+
302
+
This is always frustrating when you run into it, but let’s see what Staticcheck
303
+
says when you run it against this code:
304
+
305
+
306
+
307
+
Staticcheck will reject it. If this code was checked into source control and
308
+
Staticcheck was run in CI, tests would fail.
309
+
310
+
311
+
312
+
The correct version of doWork should look like this.
313
+
314
+
315
+
316
+
Note how I changed the failure case to use an untyped `nil`. This prevents the
317
+
`nil` value from being boxed into an interface. This will do the right thing.
318
+
319
+
320
+
321
+
This will help you ensure that this kind of code never enters production so it
322
+
cannot fail at untold hours of the night while you are sleeping.
323
+
324
+
325
+
326
+
As SREs, we tend to sleep very little as is. Statistically we have higher rates
327
+
of burnout, mind fog, fatigue and likelihood of turning into angry, sad people
328
+
as we do this job longer and longer. Especially if the culture of a company is
329
+
broken enough that you end up being on call during sleeping hours.
330
+
331
+
This is not healthy. It is not sustainable for us to be woken up at obscene
332
+
hours of the night because of trivial and preventable errors. If we get woken up
333
+
in the night, it should be for things that are measurably novel and not caused
334
+
by errors that should have never been allowed to be deployed in the first place.
335
+
336
+
337
+
338
+
I don’t think I’ve heard my pager sound in years by this point, but the last
339
+
time I heard it I almost had a full blown panic attack. I have been in the kind
340
+
of place where burnout from the pager severely affected my health.
341
+
342
+
I’m still recovering from the after-effects of that tour of SRE duty, and it has
343
+
resulted in me making permanent career changes so that I am never put in that
344
+
kind of position again. I don’t wish this hell on anyone.
345
+
346
+
347
+
348
+
Normally things can feel like this when you are an SRE put in the line of pager
349
+
fire. It feels like both fixing production and being able to get more sleep are
350
+
unworkable and that you would have severe difficulty getting from one side to
351
+
the other.
352
+
353
+
354
+
355
+
Adding static analysis to your continuous integration setup can allow you to
356
+
walk down a middle path between these two extremes. It is not going to be
357
+
perfect, however gradually things will get better.
358
+
359
+
Trivial errors will be blocked from going into production and you will be able
360
+
to sleep easier.
361
+
362
+
363
+
364
+
The benefits of being able to rest easier like this are numerous and difficult
365
+
to summarize in the short amount of time I have left. It could save your
366
+
relationship with your loved ones. It could prevent people near you from
367
+
resenting you.
368
+
369
+
It could be the difference between a long and happy career or having to drop out
370
+
of tech at 25; burnt out to a crisp and unable to do much of anything.
371
+
372
+
373
+
374
+
It could be the difference between life and an early, untimely death from a
375
+
preventable heart attack.
376
+
377
+
378
+
379
+
In talks like these it’s easy to ignore the fact the people that are responsible
380
+
for making sure services are reliable are that. Human. Company culture may get
381
+
in the way, there may be a lack of people that are willing or able to take the
382
+
pager rotation.
383
+
384
+
385
+
386
+
However when the machines come to take our jobs, I hope this one is one of the
387
+
first that they take.
388
+
389
+
390
+
391
+
In the meantime, all we can do is get towards a more sustainable future. And the
392
+
best thing we can do is make sure people sleep well without having to worry
393
+
about being woken up because of errors that tools like Staticcheck can block
394
+
from getting into production.
395
+
396
+
397
+
398
+
If you use Go in production, I highly suggest using Staticcheck. If you find it
399
+
useful, sponsor Dominik on GitHub. Software like this is complicated to develop
400
+
and the best way to ensure Dominik can keep developing it is to pay him for his
401
+
efforts. The better he sleeps, the better you sleep as an SRE.
402
+
403
+
404
+
405
+
As for other languages, I don't know what the best practices are. You will have
406
+
to do research on this, you may have to work together with coworkers to figure
407
+
out what would be the best option for your team. It is worth the effort though.
408
+
This helps you make a better product for everyone, and it's worth the teething
409
+
pains at first.
410
+
411
+
412
+
413
+
I’m almost at the end of the presentation, but I wanted to give a special shout
414
+
out to all of these people who helped make this talk a reality. I want to also
415
+
give out a special shout out to my coworkers at Tailscale that let me load shed
416
+
so I could focus on making this talk shine.
417
+
418
+
419
+
420
+
Thanks for watching! I’ll stick around in the chat for questions, but if I miss
421
+
your question and you really want an answer to it, please email it to
422
+
code42sre2022@xeserv.us. I’m happy to answer questions and I enjoy writing up
423
+
responses.
+6
-12
templates/talkpost.rs.html
+6
-12
templates/talkpost.rs.html
···
1
-
@use super::{header_html, footer_html, mara};
2
-
@use crate::post::Post;
1
+
@use super::{header_html, footer_html};
2
+
@use crate::{post::Post, tmpl::nag};
3
3
@use chrono::prelude::*;
4
4
5
-
@(post: Post, body: impl ToHtml)
5
+
@(post: Post, body: impl ToHtml, referer: Option<String>)
6
6
7
7
@:header_html(Some(&post.front_matter.title.clone()), None)
8
8
···
46
46
@}
47
47
</script>
48
48
49
-
@if Utc::today().num_days_from_ce() < post.date.num_days_from_ce() {
50
-
<div class="warning">
51
-
@:mara("hacker", "Mara", Html(format!(r#"Hey, this post is set to go live to the public on {} UTC. Right now you are reading a pre-publication version of this post. Please do not share this on social media. This post will automatically go live for everyone on the intended publication date. If you want access to these posts, please join the <a href="https://patreon.com/cadey">Patreon</a>. It helps me afford the copyeditor that I contract for the technical content I write."#, post.detri())))
52
-
</div>
53
-
} else {
54
-
<script async src="https://media.ethicalads.io/media/client/ethicalads.min.js"></script>
55
-
}
49
+
@Html(nag::referer(referer).0)
50
+
51
+
@Html(nag::prerelease(&post).0)
56
52
57
53
@body
58
54
59
55
<a href="@post.front_matter.slides_link.as_ref().unwrap()">Link to the slides</a>
60
-
61
-
<div data-ea-publisher="christinewebsite" data-ea-type="text"></div>
62
56
63
57
<hr />
64
58