The code and data behind xeiaso.net
talks: reorganize into folders without breaking links
Signed-off-by: Xe Iaso <me@xeiaso.net>
+239
-190
+1
lume/src/talks/2018/_data.yml
+1
lume/src/talks/2018/_data.yml
···
1
+
year: 2018
+1
lume/src/talks/2019/_data.yml
+1
lume/src/talks/2019/_data.yml
···
1
+
year: 2019
+136
lume/src/talks/2019/webassembly-on-the-server-system-calls.md
+136
lume/src/talks/2019/webassembly-on-the-server-system-calls.md
···
1
+
---
2
+
title: "WebAssembly on the Server: How System Calls Work"
3
+
date: 2019-05-31
4
+
basename: ../webassembly-on-the-server-system-calls-2019-05-31
5
+
slides_link: https://cdn.xeiaso.net/file/christine-static/static/talks/wasm-on-the-server-system-calls.pdf
6
+
---
7
+
8
+
[Video](https://www.youtube.com/watch?v=G4l8RX0tA3E)
9
+
10
+
## My Speaker Notes
11
+
12
+
- Hi, my name is Christine. I work as a senior SRE for Lightspeed. Today I'm gonna talk about something I've been researching and learning a lot about: WebAssembly on the server.
13
+
- Something a lot of you might be asking: what is WebAssembly?
14
+
- WebAssembly is very new and there's a lot of confusing and overly vague coverage on it.
15
+
- In this talk, I will explain WebAssembly at a high level and show how to start solving one of the hardest problems in it: how to communicate with the outside world.
16
+
- When I say the "outside world" I mean anything that is not literally one of these 5 basic things:
17
+
- Externally imported functions, defined by the user
18
+
- The dynamic dispatch function table
19
+
- Global variables
20
+
- Linear memory, or basically ram
21
+
- Compiled functions, or your code that runs in the virtual machine
22
+
- WebAssembly is a Virtual Machine format for the Web
23
+
- The closest analogue to WASM in its current form is a CPU and supporting hardware
24
+
- However, because it's a virtual machine, the hardware is irrelevant
25
+
- Though it was intended for browsers, the implementation of it is really generic.
26
+
- WebAssembly provides:
27
+
- External functions
28
+
- A function table for dynamic dispatch
29
+
- Immutable globals (as of the MVP)
30
+
- Linear memory
31
+
- Compiled functions (these exist outside of linear memory like an AVR chip)
32
+
- Why WebAssembly on the Server?
33
+
- It makes hardware less relevant.
34
+
- Most of our industry targets a single vendor in basic configurations: Intel amd64 processors running Linux
35
+
- Intel has had many security bugs and it may not be a good idea to fundamentally design our architecture to rely on them.
36
+
- This also removes the OS from the equation for most compute tasks.
37
+
- What are system calls and why do they matter?
38
+
- System calls enforce abstractions to the outside world.
39
+
- Your code goes through system calls to reach things from the outside world, eg:
40
+
- Randomness
41
+
- Network sockets
42
+
- The filesystem
43
+
- Etc
44
+
- How are they implemented?
45
+
- The platform your program runs on exposes those system calls
46
+
- Programs pass pointers into linear memory (this will be shown later in the slides)
47
+
- Why is this relevant to WebAssembly?
48
+
- The WebAssembly Minimum Viable Product doesn't define any system calls
49
+
- WebAssembly System Calls Out of The Box
50
+
- Yeah, nothing. You're on your own. This is both very good and very very bad.
51
+
- So what's a pointer in WebAssembly?
52
+
- Simplified, a WebAssembly virtual machine is some structure that has a reference to a byte slice. That byte slice is treated as the linear memory of that VM.
53
+
- A pointer is just an offset into this slice
54
+
- Showing the WebAssembly world diagram from earlier: pointers apply to only this part of it. Function pointers _do_ exist in WebAssembly, just by the dynamic dispatch table from earlier.
55
+
- So what can we do about it?
56
+
- Let's introduce a pet project of mine for a few years. It's called Dagger, and it has been a fantastic stepping stone while other solutions are being invented.
57
+
- Dagger is a proof of concept system call API that I'll be walking through the high level implementation of
58
+
- It's got a very simple implementation (500-ish lines)
59
+
- It's intended for teaching and learning about the low levels of WebAssembly.
60
+
- It's based on a very very very simplistic view of the unix philosophy. In unix, everything is a file. With Dagger, everything is a stream, even HTTP.
61
+
- As such, there's no magic in Dagger.
62
+
- And even though it's so simple, it's still usable for more than just basic/trivial things.
63
+
- A dagger process has a bunch of streams in a slice.
64
+
- The API gives out and uses stream descriptors, or offsets into this slice.
65
+
- Dagger's API is really really simple, it's only got 5 calls:
66
+
- Opening a stream
67
+
- Closing a stream
68
+
- Reading from a stream
69
+
- Writing to a stream
70
+
- Flushing intermediately buffered data from a stream to its remote (or local) target
71
+
- Open
72
+
- Open opens a stream by URL, then returns its descriptor. It can also return an error instead.
73
+
- It's got 5 basic stream types:
74
+
- Logging
75
+
- Jailed filesystem access
76
+
- HTTP/S
77
+
- 5 system calls is all you need for HTTP!
78
+
- Randomness
79
+
- Standard input/output
80
+
- Let's walk through the code that implements it
81
+
- Here's a simplified view of the open function in a Dagger process.
82
+
- The system call arguments are here
83
+
- And the stream URL gets read from the VM memory here
84
+
- Remember that pointers are just integer offsets into memory
85
+
- Then this gets passed to the rest of the open file logic that isn't shown here
86
+
- Close
87
+
- Closes a stream by its descriptor.
88
+
- It returns a negative error if anything goes wrong, which is unlikely.
89
+
- Let's walk through its code:
90
+
- It grabs the arguments from the VM
91
+
- Then it passes that to the rest of the logic that isn't shown here
92
+
- Read
93
+
- Reads a limited amount of bytes from a stream
94
+
- Returns a negative error if things go wrong
95
+
- Let's walk through its code:
96
+
- This is a bigger function, so I've broken it up into a few slides.
97
+
- First it gets the arguments from the VM
98
+
- Then it creates the intermediate buffer to copy things into from the stream
99
+
- Then it does the reading into that buffer
100
+
- Then it copies the buffer into the VM ram
101
+
- Write
102
+
- Write is very similar to read, except it just copies the ram out of the VM and into the stream
103
+
- It returns the number of bytes written, which SHOULD equal the data length argument
104
+
- Let's walk through the code:
105
+
- Again, this function is bigger so I
106
+
- Flush
107
+
- Flush does just about what you'd think, it flushes intermediate buffers to the actual stream targets.
108
+
- This blocks until the flushing is complete
109
+
- Mostly used for the HTTP client
110
+
- Let's walk through its code:
111
+
- It gets the descriptor from the VM
112
+
- It runs the flush operation and returns the result
113
+
- So, with all this covered, let's talk about usage. Here's the famous "Hello, world" example:
114
+
- This is in Zig, mainly because Zig allows me to be really concise. Things work just about as you'd expect so it's less of a logical jump than you'd think.
115
+
- First we try to open the stream. Dagger doesn't have any streams open in its environment by default, so we open standard output.
116
+
- Then we try to write the message to the stream. The interface in Zig is a bit rough right now, but it takes the pointer to the message and how long the message is. Zig doesn't let us implicitly ignore the return value of this function, so we just explicitly ignore it instead.
117
+
- Finally we try to close the output stream.
118
+
- The beauty of zig is that if any of these things we try to do fails, the entire function will fail.
119
+
- However none of this fails so we can just run it with the dagger tool and get this output:
120
+
- What this can build to
121
+
- This basic idea can be used to build up to any of the following things:
122
+
- A functions as a service backend (See Olin)
123
+
- Generic event handlers
124
+
- Distributed computing
125
+
- Transactional computing
126
+
- What you can do
127
+
- Play with the code (link at the end)
128
+
- Implement this API from scratch
129
+
- It's really not that hard
130
+
- A possible project idea I was going to do but ran out of time (moving internationally sucks) is to make a Gopher server with every route powered by WebAssembly
131
+
- Got questions?
132
+
- Tweet or email me if you really want to make sure your questions get answered. That is one of the best ways to ensure I actually see it.
133
+
- I'm happy to go into detail, I can pull out code examples too.
134
+
- Thanks to all of these people who have given help, ideas and inspiration. Without them I would never have been able to get this far.
135
+
- Follow my progress on GitHub!
136
+
- I hope that QR code is big enough. If it's not let me know and I can make things like that bigger in the future somehow, hopefully.
+1
lume/src/talks/2021/_data.yml
+1
lume/src/talks/2021/_data.yml
···
1
+
year: 2021
+1
lume/src/talks/2022/_data.yml
+1
lume/src/talks/2022/_data.yml
···
1
+
year: 2022
+1
lume/src/talks/2023/_data.yml
+1
lume/src/talks/2023/_data.yml
···
1
+
year: 2023
+1
-1
lume/src/talks/2024/xeact.mdx
+1
-1
lume/src/talks/2024/xeact.mdx
+29
-9
lume/src/talks/asg-2023-nixos.mdx
lume/src/talks/2023/asg-nixos.mdx
+29
-9
lume/src/talks/asg-2023-nixos.mdx
lume/src/talks/2023/asg-nixos.mdx
···
2
2
title: "Making NixOS modules for fun and (hopefully) profit"
3
3
date: 2023-09-15
4
4
slides_link: https://drive.google.com/file/d/1h0Y8PUREuSrrSStmJMJ4MFHm0h53tf5a/view?usp=sharing
5
+
basename: ../asg-2023-nixos
5
6
tags:
6
-
- nix
7
-
- nixos
8
-
- tailscale
7
+
- nix
8
+
- nixos
9
+
- tailscale
9
10
---
10
11
11
-
<XeblogConv name="Cadey" mood="coffee" standalone>There was an A/V glitch with the recording, my slides were intended to be black and white, but they somehow came out as purple and green. This couldn't be fixed even when trying several HDMI cables. If this becomes an issue I may re-record this talk in my home studio.</XeblogConv>
12
+
<XeblogConv name="Cadey" mood="coffee" standalone>
13
+
There was an A/V glitch with the recording, my slides were intended to be
14
+
black and white, but they somehow came out as purple and green. This couldn't
15
+
be fixed even when trying several HDMI cables. If this becomes an issue I may
16
+
re-record this talk in my home studio.
17
+
</XeblogConv>
12
18
13
19
<XeblogVideo path="talks/2023/asg-nixos/video/proper"></XeblogVideo>
14
20
···
297
303
};
298
304
```
299
305
300
-
We defined the devShell to build the program development. We defined the package to build the software, and now we'll define the module to tell NixOS how to run the software. This is a basic NixOS module. It's defined inline to the flake for now, moving it to its own file is an exercise for the reader.
306
+
We defined the devShell to build the program development. We defined the package to build the software, and now we'll define the module to tell NixOS how to run the software. This is a basic NixOS module. It's defined inline to the flake for now, moving it to its own file is an exercise for the reader.
301
307
302
308
Like I said before, a NixOS module is a function that takes in the state of the world and returns new additions to the state of the world. This NixOS module provides some options under `xe.services.douglas-adams-quotes` and then if the module is enabled, it creates a new systemd service to run it in. We're in the future, so we can use fancy things like DynamicUser to avoid having to run this service as root.
303
309
···
360
366
}
361
367
```
362
368
363
-
To make it import this, first we add a new input that points to the Douglas Adams quotes flake. This then gets threaded into the outputs function, we import the module, and finally enable it on the system.
369
+
To make it import this, first we add a new input that points to the Douglas Adams quotes flake. This then gets threaded into the outputs function, we import the module, and finally enable it on the system.
364
370
365
371
(Pause)
366
372
···
389
395
390
396
Get your phones out, I'm gonna be showing a QR Code:
391
397
392
-
<XeblogConv name="Mara" mood="hacker" standalone>For those playing the Xe Iaso blog extended universe home game, visit [m85-kongir.shark-harmonic.ts.net](https://m85-kongir.shark-harmonic.ts.net) instead.</XeblogConv>
398
+
<XeblogConv name="Mara" mood="hacker" standalone>
399
+
For those playing the Xe Iaso blog extended universe home game, visit
400
+
[m85-kongir.shark-harmonic.ts.net](https://m85-kongir.shark-harmonic.ts.net)
401
+
instead.
402
+
</XeblogConv>
393
403
394
404
Scan this QR code. You can trust me, right? It's not gonna be a Rick Roll. I'm not that mean. When you do, you'll connect to my VM on my laptop on the conference wifi, yet still exposed to the public internet.
395
405
···
421
431
422
432
There was a question about encrypted secrets in NixOS. I suggest using [agenix](https://github.com/ryantm/agenix) to have age-encrypted secrets in your NixOS configs. It has you encrypt things against SSH host public keys for your machines. It's a bit of a hack, but it works well enough that it's what I use in prod for my own stuff. This really needs to be solved upstream with proper handling of secret values at the Nix level.
423
433
424
-
<XeblogConv name="Aoi" mood="wut">Why do you need to use something like agenix at all?</XeblogConv>
425
-
<XeblogConv name="Mara" mood="hacker">Every file in `/nix/store` is world-readable. Depending on your threat model and if your NixOS configs are open source, this can be fine. If your threat model includes public NixOS configs, this becomes less fine; especially when CI is brought into the mix. You wouldn't want someone to figure out what your secrets are in your CI flow and then exfiltrate [Tailscale authkeys](https://tailscale.com/kb/1085/auth-keys/) or something, that could be bad!</XeblogConv>
434
+
<XeblogConv name="Aoi" mood="wut">
435
+
Why do you need to use something like agenix at all?
436
+
</XeblogConv>
437
+
<XeblogConv name="Mara" mood="hacker">
438
+
Every file in `/nix/store` is world-readable. Depending on your threat model
439
+
and if your NixOS configs are open source, this can be fine. If your threat
440
+
model includes public NixOS configs, this becomes less fine; especially when
441
+
CI is brought into the mix. You wouldn't want someone to figure out what your
442
+
secrets are in your CI flow and then exfiltrate [Tailscale
443
+
authkeys](https://tailscale.com/kb/1085/auth-keys/) or something, that could
444
+
be bad!
445
+
</XeblogConv>
426
446
427
447
---
428
448
+1
lume/src/talks/conf42-static-analysis.mdx
lume/src/talks/2022/conf42-static-analysis.mdx
+1
lume/src/talks/conf42-static-analysis.mdx
lume/src/talks/2022/conf42-static-analysis.mdx
+1
lume/src/talks/how-my-website-works.mdx
lume/src/talks/2022/how-my-website-works.mdx
+1
lume/src/talks/how-my-website-works.mdx
lume/src/talks/2022/how-my-website-works.mdx
+1
lume/src/talks/irc-why-it-failed-2018-05-17.md
lume/src/talks/2018/irc-why-it-failed.md
+1
lume/src/talks/irc-why-it-failed-2018-05-17.md
lume/src/talks/2018/irc-why-it-failed.md
+5
-5
lume/src/talks/nixos-pain-2021-11-10.md
lume/src/talks/2021/nixos-pain.md
+5
-5
lume/src/talks/nixos-pain-2021-11-10.md
lume/src/talks/2021/nixos-pain.md
···
2
2
title: How Nix and NixOS Get So Close to Perfect
3
3
date: 2021-11-10
4
4
slides_link: https://cdn.xeiaso.net/file/christine-static/static/talks/nixos-pain.pdf
5
+
basename: ../nixos-pain-2021-11-10
5
6
tags:
6
-
- nix
7
-
- nixos
8
-
- docker
9
-
- packagingcon
7
+
- nix
8
+
- nixos
9
+
- docker
10
+
- packagingcon
10
11
---
11
12
12
13
## Author's Note
···
319
320
documentation on every single thing that ships with NixOS by default. There
320
321
should be no module in the library of modules without documentation on how to
321
322
use it and an example or two of where you'd use some of the weirder options.
322
-
323
323
324
324
<center>
325
325
<picture>
+1
lume/src/talks/progressive-webapp-conversion-in-5-minutes-2019-01-28.md
lume/src/talks/2019/progressive-webapp-conversion-in-5-minutes.md
+1
lume/src/talks/progressive-webapp-conversion-in-5-minutes-2019-01-28.md
lume/src/talks/2019/progressive-webapp-conversion-in-5-minutes.md
+1
lume/src/talks/rustconf-2022-sheer-terror-pam.mdx
lume/src/talks/2022/sheer-terror-pam.mdx
+1
lume/src/talks/rustconf-2022-sheer-terror-pam.mdx
lume/src/talks/2022/sheer-terror-pam.mdx
+3
-1
lume/src/talks/subtle-magic-tsnet.mdx
lume/src/talks/2023/subtle-magic-tsnet.mdx
+3
-1
lume/src/talks/subtle-magic-tsnet.mdx
lume/src/talks/2023/subtle-magic-tsnet.mdx
···
1
1
---
2
2
title: The Subtle Magic of tsnet
3
3
date: 2023-07-07
4
+
basename: ../subtle-magic-tsnet
4
5
tags:
5
6
- tsnet
6
7
- go
···
8
9
---
9
10
10
11
<div className="text-xl">
11
-
This post was written while I worked for Tailscale. It is archived here for posterity.
12
+
This post was written while I worked for Tailscale. It is archived here for
13
+
posterity.
12
14
</div>
13
15
14
16
<XeblogConv name="Cadey" mood="enby">
+8
-7
lume/src/talks/surreal-horror-pam-2021-11-09.md
lume/src/talks/2021/surreal-horror-pam.md
+8
-7
lume/src/talks/surreal-horror-pam-2021-11-09.md
lume/src/talks/2021/surreal-horror-pam.md
···
2
2
title: The Surreal Horror of PAM
3
3
date: 2021-11-09
4
4
slides_link: https://cdn.xeiaso.net/file/christine-static/static/talks/surreal-horror-pam.pdf
5
+
basename: ../surreal-horror-pam-2021-11-09
5
6
tags:
6
-
- alpinelinux
7
-
- pam
8
-
- satire
7
+
- alpinelinux
8
+
- pam
9
+
- satire
9
10
---
10
11
11
12
<iframe width="1043" height="587" src="https://www.youtube.com/embed/INjCiHUIjgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
···
137
138
really do not want to spend all day debugging PAM with gdb and strace on Ubuntu
138
139
to demonstrate it with that. PAM has a few kinds of modules:
139
140
140
-
* authentication, this is not just checking your password, but also making sure
141
+
- authentication, this is not just checking your password, but also making sure
141
142
that your account is allowed to be logged into and setting up things like your
142
143
preferred login shell
143
-
* account, the things that assign a user an account based on the circumstances
144
+
- account, the things that assign a user an account based on the circumstances
144
145
of their authentication or validate that somehow (this is also where an LDAP
145
146
server would get thrown into the mix if you really hate yourself)
146
-
* password, the things that check passwords or do other kinds of validation like
147
+
- password, the things that check passwords or do other kinds of validation like
147
148
that (if you want to use Google Authenticator TOTP codes, you’d do that here)
148
-
* session, these things handle other system errata like making sure the
149
+
- session, these things handle other system errata like making sure the
149
150
message-of-the-day (MOTD) is shown when you log in or letting logind know
150
151
about the session so it can make a cgroup for you
151
152
+9
-5
lume/src/talks/systemd-the-good-parts-2021-05-16.md
lume/src/talks/2021/systemd-the-good-parts.md
+9
-5
lume/src/talks/systemd-the-good-parts-2021-05-16.md
lume/src/talks/2021/systemd-the-good-parts.md
···
1
1
---
2
2
title: "systemd: The Good Parts"
3
3
date: 2021-05-16
4
+
basename: ../systemd-the-good-parts-2021-05-16
4
5
slides_link: https://docs.google.com/presentation/d/1a0XaGu87xUcpQQVLkrnXKoKrdpN1ObiPrG9aGYVMw7k/edit?usp=sharing
5
6
---
6
7
···
16
17
Alpine user for almost a decade and it's one of my favorite linux distributions.
17
18
18
19
The best things in life come with disclaimers and here are the disclaimers for this talk:
20
+
19
21
- This talk may contain opinions. These opinions are my own and not necessarily
20
22
the opinions of my employer.
21
23
- This talk is not evangelism. This talk is intended to show how green the grass
···
72
74
- What are the last few log lines?
73
75
- If you need to reboot the server right now for some reason, will that service
74
76
come back up on reboot?
75
-
77
+
76
78
77
79
78
80
systemd includes a tool called systemctl that allows you to query the status of
···
93
95
those numbers aren't reflected here but it's actually much higher.
94
96
- At the bottom we can see the last few log lines. These are just random
95
97
requests that people make to my blog.
96
-
98
+
97
99
If you haven't seen all of this in action before you might be wondering
98
100
something like "Wait, where did it get those logs from?"
99
101
···
126
128
127
129
Now there's at least four classes of benefits for systemd and I'm going to break
128
130
them down into the following groups:
131
+
129
132
- developers
130
133
- packagers
131
134
- system administrators
132
135
- users
133
-
136
+
134
137
In general people that are developing services that run on systemd get the following benefits:
135
138
136
139
- Predictability. systemd configuration files are declarative rather than
137
140
imperative. You declare units instead of imperatively building up init
138
141
scripts. Options are declared and enforced by the service manager. This makes
139
-
it a lot easier to review changes for correctness.
142
+
it a lot easier to review changes for correctness.
140
143
- Portability. when setting up a service with systemd there's only one syntax to
141
144
learn across 15 plus different distributions. This means that you don't have
142
145
to maintain a giant pile of hacks to make the program just start consistently
···
188
191
operator. systemd is set up so that it's hard to do the wrong thing. It is
189
192
hard to have logs go anywhere but the system journal. It is hard to write a
190
193
unit that doesn't tell you if the service is actually running or not. And it
191
-
makes it so that the path of least resistance will do most of what you want.
194
+
makes it so that the path of least resistance will do most of what you want.
192
195
- Sometimes system administrators have opinions that are different than the
193
196
opinions of the packager. Sometimes you need to change environment variables
194
197
for http proxies or something and sometimes you believe the packager has
···
219
222
incident response, so it is more difficult to have hidden dependencies.
220
223
221
224
As far as users go:
225
+
222
226
- systemd is not limited to just managing system level services systemd can also
223
227
manage user services with systemd user mode. I use this on my Linux system in
224
228
order to have a couple services running in the background querying for weather
+1
lume/src/talks/thinking-different-2018-11-03.md
lume/src/talks/2018/thinking-different.md
+1
lume/src/talks/thinking-different-2018-11-03.md
lume/src/talks/2018/thinking-different.md
+1
lume/src/talks/unix-philosophy-logical-extreme-wasm.mdx
lume/src/talks/2023/unix-philosophy-logical-extreme-wasm.mdx
+1
lume/src/talks/unix-philosophy-logical-extreme-wasm.mdx
lume/src/talks/2023/unix-philosophy-logical-extreme-wasm.mdx
+15
-12
lume/src/talks/virtual-networks-pulumi-tailscale.mdx
lume/src/talks/2023/virtual-networks-pulumi-tailscale.mdx
+15
-12
lume/src/talks/virtual-networks-pulumi-tailscale.mdx
lume/src/talks/2023/virtual-networks-pulumi-tailscale.mdx
···
1
1
---
2
2
title: Building Virtual Networks with Pulumi and Tailscale
3
3
date: 2023-01-11
4
+
basename: ../virtual-networks-pulumi-tailscale
4
5
tags:
5
-
- pulumi
6
-
- tailscale
6
+
- pulumi
7
+
- tailscale
7
8
skip_ads: true
8
9
---
9
10
10
-
<XeblogConv name="Cadey" mood="enby">This was a
11
-
[workshop](https://www.pulumi.com/resources/building-virtual-networks-with-pulumi-and-tailscale/)
12
-
that I helped with so that people could learn how to glue Tailscale and
13
-
[Pulumi](https://www.pulumi.com/) (think Terraform but you can declare resources
14
-
in programming languages such as TypeScript instead of HCL) together by creating
15
-
a Tailscale subnet router to connect you to a VPC in AWS. I'm including the
16
-
speaking bits that I did for the talk, but most of what I was there for was to
17
-
help field questions about Tailscale. Internet streamer brain is a useful tool
18
-
when properly harnessed.</XeblogConv>
11
+
<XeblogConv name="Cadey" mood="enby">
12
+
This was a
13
+
[workshop](https://www.pulumi.com/resources/building-virtual-networks-with-pulumi-and-tailscale/)
14
+
that I helped with so that people could learn how to glue Tailscale and
15
+
[Pulumi](https://www.pulumi.com/) (think Terraform but you can declare
16
+
resources in programming languages such as TypeScript instead of HCL) together
17
+
by creating a Tailscale subnet router to connect you to a VPC in AWS. I'm
18
+
including the speaking bits that I did for the talk, but most of what I was
19
+
there for was to help field questions about Tailscale. Internet streamer brain
20
+
is a useful tool when properly harnessed.
21
+
</XeblogConv>
19
22
20
23
<XeblogVideo path="talks/pulumi-workshop-2023"></XeblogVideo>
21
24
···
65
68
66
69
Tailscale doesn't stop there, there's SSH management, file sharing, an
67
70
ngrok-like tunnelling solution, and so much more.
68
-
71
+
69
72
I'll hand things back over to Josh so we can learn more about Pulumi.
+1
lume/src/talks/wasm-abi.mdx
lume/src/talks/2022/wasm-abi.mdx
+1
lume/src/talks/wasm-abi.mdx
lume/src/talks/2022/wasm-abi.mdx
+20
-15
lume/src/talks/wazero-lightning-2023.mdx
lume/src/talks/2023/carcinization-rust.mdx
+20
-15
lume/src/talks/wazero-lightning-2023.mdx
lume/src/talks/2023/carcinization-rust.mdx
···
1
1
---
2
2
title: "[talk] The carcinization of Go programs"
3
3
date: 2023-03-24
4
+
basename: ../wazero-lightning-2023
4
5
slides_link: https://drive.google.com/file/d/1ANxRJPzNeKbLogZz0wCH_o9E9jNLypja/view?usp=sharing
5
6
tags:
6
-
- wasm
7
-
- wazero
8
-
- rust
9
-
- golang
7
+
- wasm
8
+
- wazero
9
+
- rust
10
+
- golang
10
11
skip_ads: true
11
12
---
12
13
···
16
17
17
18
## Transcript
18
19
19
-
<XeblogConv standalone name="Cadey" mood="enby">This is a lightning
20
-
talk version of [this
21
-
post](https://xeiaso.net/blog/carcinization-golang).</XeblogConv>
20
+
<XeblogConv standalone name="Cadey" mood="enby">
21
+
This is a lightning talk version of [this
22
+
post](https://xeiaso.net/blog/carcinization-golang).
23
+
</XeblogConv>
22
24
23
25
<XeblogSlide name="2023/wazero-lightning/01" essential></XeblogSlide>
24
26
···
60
62
carcinization of Go programs via WebAssembly. This is how I snuck Rust
61
63
into a Go shop.
62
64
63
-
<XeblogConv name="Mara" mood="hacker">The "carcinization" refers to
64
-
the evolutionary tendency of programs becoming either crabs or trees
65
-
when time stretches to infinity. If you can imagine library use as
66
-
evolution, then this joke makes more sense.</XeblogConv>
65
+
<XeblogConv name="Mara" mood="hacker">
66
+
The "carcinization" refers to the evolutionary tendency of programs becoming
67
+
either crabs or trees when time stretches to infinity. If you can imagine
68
+
library use as evolution, then this joke makes more sense.
69
+
</XeblogConv>
67
70
68
71
<XeblogSlide name="2023/wazero-lightning/06"></XeblogSlide>
69
72
···
83
86
this problem is fairly novel. Anything that was close to this just
84
87
made atrocities out of the text in ways that I couldn't customize.
85
88
86
-
<XeblogConv name="Aoi" mood="wut">I guess some solutions could be out
87
-
there, but just locked in closed-source repos.</XeblogConv>
89
+
<XeblogConv name="Aoi" mood="wut">
90
+
I guess some solutions could be out there, but just locked in closed-source
91
+
repos.
92
+
</XeblogConv>
88
93
89
94
<XeblogSlide name="2023/wazero-lightning/09"></XeblogSlide>
90
95
···
100
105
101
106
When I wrote this program, I made it in a fairly naiive way. I took
102
107
HTML over standard input and had it spit out slackdown on standard
103
-
output.
108
+
output.
104
109
105
110
This idea just so happens to be one of the core parts of the Unix
106
111
philosophy: programs should be filters that take input in one form and
···
143
148
And now this atrocity is shipped to production and holds together the
144
149
Mastodon post announcement service. Most people aren't aware that it's
145
150
a thing, and it runs fast enough that nobody really cares that it's a
146
-
programming turducken of Go, Rust and WebAssembly. It works perfectly
151
+
programming turducken of Go, Rust and WebAssembly. It works perfectly
147
152
and it wouldn't be possible without the efforts of the Wazero team.
148
153
149
154
<XeblogSlide name="2023/wazero-lightning/16" essential></XeblogSlide>
-135
lume/src/talks/webassembly-on-the-server-system-calls-2019-05-31.md
-135
lume/src/talks/webassembly-on-the-server-system-calls-2019-05-31.md
···
1
-
---
2
-
title: "WebAssembly on the Server: How System Calls Work"
3
-
date: 2019-05-31
4
-
slides_link: https://cdn.xeiaso.net/file/christine-static/static/talks/wasm-on-the-server-system-calls.pdf
5
-
---
6
-
7
-
[Video](https://www.youtube.com/watch?v=G4l8RX0tA3E)
8
-
9
-
## My Speaker Notes
10
-
11
-
* Hi, my name is Christine. I work as a senior SRE for Lightspeed. Today I'm gonna talk about something I've been researching and learning a lot about: WebAssembly on the server.
12
-
* Something a lot of you might be asking: what is WebAssembly?
13
-
* WebAssembly is very new and there's a lot of confusing and overly vague coverage on it.
14
-
* In this talk, I will explain WebAssembly at a high level and show how to start solving one of the hardest problems in it: how to communicate with the outside world.
15
-
* When I say the "outside world" I mean anything that is not literally one of these 5 basic things:
16
-
* Externally imported functions, defined by the user
17
-
* The dynamic dispatch function table
18
-
* Global variables
19
-
* Linear memory, or basically ram
20
-
* Compiled functions, or your code that runs in the virtual machine
21
-
* WebAssembly is a Virtual Machine format for the Web
22
-
* The closest analogue to WASM in its current form is a CPU and supporting hardware
23
-
* However, because it's a virtual machine, the hardware is irrelevant
24
-
* Though it was intended for browsers, the implementation of it is really generic.
25
-
* WebAssembly provides:
26
-
* External functions
27
-
* A function table for dynamic dispatch
28
-
* Immutable globals (as of the MVP)
29
-
* Linear memory
30
-
* Compiled functions (these exist outside of linear memory like an AVR chip)
31
-
* Why WebAssembly on the Server?
32
-
* It makes hardware less relevant.
33
-
* Most of our industry targets a single vendor in basic configurations: Intel amd64 processors running Linux
34
-
* Intel has had many security bugs and it may not be a good idea to fundamentally design our architecture to rely on them.
35
-
* This also removes the OS from the equation for most compute tasks.
36
-
* What are system calls and why do they matter?
37
-
* System calls enforce abstractions to the outside world.
38
-
* Your code goes through system calls to reach things from the outside world, eg:
39
-
* Randomness
40
-
* Network sockets
41
-
* The filesystem
42
-
* Etc
43
-
* How are they implemented?
44
-
* The platform your program runs on exposes those system calls
45
-
* Programs pass pointers into linear memory (this will be shown later in the slides)
46
-
* Why is this relevant to WebAssembly?
47
-
* The WebAssembly Minimum Viable Product doesn't define any system calls
48
-
* WebAssembly System Calls Out of The Box
49
-
* Yeah, nothing. You're on your own. This is both very good and very very bad.
50
-
* So what's a pointer in WebAssembly?
51
-
* Simplified, a WebAssembly virtual machine is some structure that has a reference to a byte slice. That byte slice is treated as the linear memory of that VM.
52
-
* A pointer is just an offset into this slice
53
-
* Showing the WebAssembly world diagram from earlier: pointers apply to only this part of it. Function pointers _do_ exist in WebAssembly, just by the dynamic dispatch table from earlier.
54
-
* So what can we do about it?
55
-
* Let's introduce a pet project of mine for a few years. It's called Dagger, and it has been a fantastic stepping stone while other solutions are being invented.
56
-
* Dagger is a proof of concept system call API that I'll be walking through the high level implementation of
57
-
* It's got a very simple implementation (500-ish lines)
58
-
* It's intended for teaching and learning about the low levels of WebAssembly.
59
-
* It's based on a very very very simplistic view of the unix philosophy. In unix, everything is a file. With Dagger, everything is a stream, even HTTP.
60
-
* As such, there's no magic in Dagger.
61
-
* And even though it's so simple, it's still usable for more than just basic/trivial things.
62
-
* A dagger process has a bunch of streams in a slice.
63
-
* The API gives out and uses stream descriptors, or offsets into this slice.
64
-
* Dagger's API is really really simple, it's only got 5 calls:
65
-
* Opening a stream
66
-
* Closing a stream
67
-
* Reading from a stream
68
-
* Writing to a stream
69
-
* Flushing intermediately buffered data from a stream to its remote (or local) target
70
-
* Open
71
-
* Open opens a stream by URL, then returns its descriptor. It can also return an error instead.
72
-
* It's got 5 basic stream types:
73
-
* Logging
74
-
* Jailed filesystem access
75
-
* HTTP/S
76
-
* 5 system calls is all you need for HTTP!
77
-
* Randomness
78
-
* Standard input/output
79
-
* Let's walk through the code that implements it
80
-
* Here's a simplified view of the open function in a Dagger process.
81
-
* The system call arguments are here
82
-
* And the stream URL gets read from the VM memory here
83
-
* Remember that pointers are just integer offsets into memory
84
-
* Then this gets passed to the rest of the open file logic that isn't shown here
85
-
* Close
86
-
* Closes a stream by its descriptor.
87
-
* It returns a negative error if anything goes wrong, which is unlikely.
88
-
* Let's walk through its code:
89
-
* It grabs the arguments from the VM
90
-
* Then it passes that to the rest of the logic that isn't shown here
91
-
* Read
92
-
* Reads a limited amount of bytes from a stream
93
-
* Returns a negative error if things go wrong
94
-
* Let's walk through its code:
95
-
* This is a bigger function, so I've broken it up into a few slides.
96
-
* First it gets the arguments from the VM
97
-
* Then it creates the intermediate buffer to copy things into from the stream
98
-
* Then it does the reading into that buffer
99
-
* Then it copies the buffer into the VM ram
100
-
* Write
101
-
* Write is very similar to read, except it just copies the ram out of the VM and into the stream
102
-
* It returns the number of bytes written, which SHOULD equal the data length argument
103
-
* Let's walk through the code:
104
-
* Again, this function is bigger so I
105
-
* Flush
106
-
* Flush does just about what you'd think, it flushes intermediate buffers to the actual stream targets.
107
-
* This blocks until the flushing is complete
108
-
* Mostly used for the HTTP client
109
-
* Let's walk through its code:
110
-
* It gets the descriptor from the VM
111
-
* It runs the flush operation and returns the result
112
-
* So, with all this covered, let's talk about usage. Here's the famous "Hello, world" example:
113
-
* This is in Zig, mainly because Zig allows me to be really concise. Things work just about as you'd expect so it's less of a logical jump than you'd think.
114
-
* First we try to open the stream. Dagger doesn't have any streams open in its environment by default, so we open standard output.
115
-
* Then we try to write the message to the stream. The interface in Zig is a bit rough right now, but it takes the pointer to the message and how long the message is. Zig doesn't let us implicitly ignore the return value of this function, so we just explicitly ignore it instead.
116
-
* Finally we try to close the output stream.
117
-
* The beauty of zig is that if any of these things we try to do fails, the entire function will fail.
118
-
* However none of this fails so we can just run it with the dagger tool and get this output:
119
-
* What this can build to
120
-
* This basic idea can be used to build up to any of the following things:
121
-
* A functions as a service backend (See Olin)
122
-
* Generic event handlers
123
-
* Distributed computing
124
-
* Transactional computing
125
-
* What you can do
126
-
* Play with the code (link at the end)
127
-
* Implement this API from scratch
128
-
* It's really not that hard
129
-
* A possible project idea I was going to do but ran out of time (moving internationally sucks) is to make a Gopher server with every route powered by WebAssembly
130
-
* Got questions?
131
-
* Tweet or email me if you really want to make sure your questions get answered. That is one of the best ways to ensure I actually see it.
132
-
* I'm happy to go into detail, I can pull out code examples too.
133
-
* Thanks to all of these people who have given help, ideas and inspiration. Without them I would never have been able to get this far.
134
-
* Follow my progress on GitHub!
135
-
* I hope that QR code is big enough. If it's not let me know and I can make things like that bigger in the future somehow, hopefully.