this repo has no description
:memo:
+6
-2
+5
Data/Data Engineering.md
+5
Data/Data Engineering.md
···
2
2
3
3
- Gather as much domain knowledge as possible. Technical knowledge is not enough.
4
4
- If it can be solved with SQL, stick to SQL.
5
+
- To be successful, data engineers need to care about:
6
+
- **Fidelity:** how reliably can data be transferred and stored without corruption or loss?
7
+
- **Capacity:** how much data can be moved and how quickly?
8
+
- **Reliability:** how well can systems recover from outages and incidents?
9
+
- **Speed of execution:** how quickly can you get a new data source up and running?
5
10
6
11
## Data Pipelines
7
12
+1
-2
Teamwork.md
+1
-2
Teamwork.md
···
52
52
2) Remove more than you add.
53
53
3) Optimize what works.
54
54
4) Shorten iteration cycles. **[Boyd's Law of Iteration](https://blog.codinghorror.com/boyds-law-of-iteration/): speed of iteration beats quality of iteration**.
55
-
5) [[Automation | Automate]].
56
-
6) Standards, a straight pipe is easier to fix than a curved one
55
+
5) [[Automation | Automate]] and keep standards.
57
56
- Keep great global [[coordination]] and incentivize local experimentation.
58
57
- Being able to run small and compounding experiments (on the product or company [[processes]] and systems) is important.
59
58
- [Some experiments won't work](https://www.lesswrong.com/posts/97LgacucCxmyjYiNT/the-archipelago-model-of-community-standards). But oftentimes it _feels_ like it wont work when in fact you just haven't stuck with it long enough for it to bear fruit. This is hard enough for _solo_ experiments. For group experiments, where not just one but _many_ people must all try a thing at once and _get good at it_, all it takes is a little defection to spiral into a mass exodus.