+1

Data/Analytics Engineering.md

reviewed

··· 20 20 - Imagine your company today as a human society where only half the population can read (understand the data), one tenth can write (SQL queries), where half a dozen languages are spoken, and where most of the books ([[Dashboards]]/insight reports) in the library contain things that once were true but have since been outdated (but you don’t know which ones). Not a highly productive information ecosystem. 21 21 - [[Teamwork|Collaborate with your team]] and break down complex models into reusable pieces. 22 22 - Domain knowledge is more important than your coding skills. 23 23 + - Ground truth isn't a single place. Start by joining on common unique keys and counting things, then figure out what’s different and why. 23 24 - Working with data is like exploring the horizon. It changes as soon as you look it from a higher place (more data). 24 25 - Reduce the areas where business logic can be injected, create “time to live” policies on last mile transforms, build a culture of standardizing + celebrating access to cross-functional codebases. 25 26 - People default to writing business logic in the tool they are most comfortable with. The best way for data teams to prevent sprawling business logic is not just to limit last mile transforms in other tools, but also to invite others into their tools. [This logic will be written, and if the data team gate-keeps, it will be written outside of their visibility](https://ian-macomber.medium.com/data-systems-tend-towards-production-be5a86f65561). If a data team can educate and encourage contributions to their codebase, they invite code to be written where it most belongs.

+5 -5

Data/Data Culture.md

reviewed

··· 17 17 - The modern data team needs to have *real organizational power*—it needs to be able to say "no” and mean it. If your data team does not truly have the power to say no to stakeholders, it will get sent on all kinds of wild goose chases, be unproductive, experience employee churn, etc. 18 18 - Data should report to the CEO. Ideally at least with some weekly metrics split into (a) notable trends, (b) watching close, and (c) business as usual. 19 19 - If data is the most precious asset in a company, does it make sense to have only one team responsible for it? 20 20 - - [People talk about data as the new oil but for most companies it’s a lot closer uranium](https://news.ycombinator.com/item?id=27781286). Hard to find people who can to handle or process it correctly, nontrivial security/liabilities if PII is involved, expensive to store and a generally underwhelming return on effort relative to the anticipated utility. 20 20 + - [People talk about data as the new oil but for most companies it’s a lot closer uranium](https://news.ycombinator.com/item?id=27781286). Hard to find people who can to handle or process it correctly, nontrivial security/liabilities if PII is involved, expensive to store and a generally underwhelming return on effort relative to the anticipated utility. 21 21 - [The pain in data teams come from needing to influence PMs/peers with having little control of them. Data teams need to become really great internal marketers/persuaders](https://anchor.fm/census/episodes/The-evolution-of-the-data-industry--data-jobs-w-Avo-CEO-and-Co-founder-Stefania-Olafsdottir-e16hu1l). That said, it shouldn't be the data team job to convince the organization to be data driven. That's not an effective way of spending resources. 22 22 - People problems are orders of magnitude more difficult to solve than data problems. 23 23 - **Integrate data where the decision is made**. E.g: Google showing restaurant scores when you're looking something for dinner. ··· 40 40 - You won't have the best allocation of resources in a reactive team. Data teams need extra [[slack]]. [Balance user requests with actual needs](https://scientistemily.substack.com/p/product-management-skills-for-data). 41 41 - How can we measure the data team impact? 42 42 - Making a [[Writing a Roadmap|roadmap]] can help you telling if you are hitting milestone deadlines or letting them slip. 43 43 - - Embedded data team members need to help other teams build their roadmap too. 43 43 + - Embedded data team members need to help other teams build their roadmap too. 44 44 - Also, having a changelog ([do releases!](https://betterprogramming.pub/great-data-platforms-use-conventional-commits-51fc22a7417c)) will help show the team impact on the data product across time. 45 45 - [Push for a *centralization of the reporting structure*, but keeping the *work management decentralized*](https://erikbern.com/2021/07/07/the-data-team-a-short-story.html). 46 46 - Unify resources (datasets, entities, definitions, metrics). Have one source of truth for each one and make that clear to everyone. That source of truth needs heavy curation. Poor curation leads to confusion, distrust and…. lots of wasted effort. ··· 82 82 - [Data ownership is a hard problem](https://www.linkedin.com/posts/chad-sanderson_heres-why-data-ownership-is-an-incredibly-activity-6904107936533114880-gw8n/). Data is fundamentally generated by services (or front-end instrumentation) which is managed by engineers. CDC and other pipelines are built by data engineers. The delineation of ownership responsibilities is very rarely established, with each group wanting to push 'ownership' onto someone else so they can do the jobs they were hired for. 83 83 - [Becoming a data-driven organization is a journey, which unfolds over time and requires critical thinking, human judgement, and experimentation](https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard). Fail fast, learn faster. 84 84 - [Path to create a data-driven organization](https://twitter.com/_abhisivasailam/status/1520274838450888704): 85 85 - - 1. Get a well-placed leader with influence to message, model, and demand data-driven execution . 86 86 - - 2. Hire/fire based on data aptitude and usage. 87 87 - - 3. Create mechanisms that force analytical conversations. Sometimes there is no way around spending an afternoon breaking down metrics by different segments until you find The Thing. 85 85 + - 1. Get a well-placed leader with influence to message, model, and demand data-driven execution . 86 86 + - 2. Hire/fire based on data aptitude and usage. 87 87 + - 3. Create mechanisms that force analytical conversations. Sometimes there is no way around spending an afternoon breaking down metrics by different segments until you find The Thing. 88 88 - [Start small. Don't try to wrangle data for the entire company until you have the tools and process down for one team](https://data-columns.hightouch.io/your-first-60-days-as-a-first-data-hire-weeks-3-4/). 89 89 - Difficulty to work with data scales exponentially with size. 90 90 - [Data is used largely to answer these questions](https://roundup.getdbt.com/p/bring-back-scenario-analysis):

+1

Gardening.md

reviewed

··· 3 3 - Know your soil. Understanding the type of soil in your garden is crucial to determining which plants will thrive, and which won't. 4 4 - Use a watering can or a soaker hose to deliver water directly to the roots, and avoid getting the leaves wet. 5 5 - Soak weeds in a container to extract nutrient. Water your plants with that every now and then. If it looks sad, it needs water (a good rule for houseplants and humans). 6 6 + - Adopt [permaculture principles](https://youtu.be/acjpwIxZzlA). 6 7 7 8 ## Resources 8 9

+16

Recipes.md

reviewed

··· 78 78 1. Brew with a 1 part coffee to 8 parts water ratio. 79 79 2. Use a coarse grind. 80 80 3. Put the mix on the fridge from 24 to 96 hours. 81 81 + 82 82 + ## Nachos 83 83 + 84 84 + <img src="https://i2.wp.com/www.downshiftology.com/wp-content/uploads/2022/03/Nachos-8.jpg" width="350" /> 85 85 + 86 86 + ### Ingredients 87 87 + 88 88 + - Chips (not too much flavor) 89 89 + - Cheese (not pre-shredded) 90 90 + - Some kind of protein (soy, chorizo, beans) 91 91 + - Guacamole (avocado, red onion, garlic, lime, cumin, salt, pepper) 92 92 + - Salsa (tomato, onion, garlic, lime, salt, pepper) 93 93 + 94 94 + ## Steps 95 95 + 96 96 + - [Follow Babish](https://youtu.be/lMaF0iPeDFo).

+11 -10

Teamwork.md

reviewed

··· 3 3 - Explicitly define the [[values]] and desired [[culture]] of your team. 4 4 - Share a vision to make [loosely coupled, tightly aligned teams](https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/). Then, define the strategy with [[Writing Team Key Results|great key results]]. 5 5 - When coming up with a long-term vision is important to stay abstract. 6 6 - - Stick to defining components and keep concepts generic (cache, [[Databases]], algorithm, ...). Show how the components interact. 7 7 - - Define boundaries and limitations of each component. 6 6 + - Stick to defining components and keep concepts generic (cache, [[Databases]], algorithm, ...). Show how the components interact. 7 7 + - Define boundaries and limitations of each component. 8 8 - Work in the open and [[Documentation|document]] everything. Transparency increases understanding and reduces synchronization challenges. **[Emulate Open Source projects](https://tomayko.com/blog/2012/adopt-an-open-source-process-constraints) and [[Remote Jobs|remote companies]]**. 9 9 - To make everyone more productive and happy: **Make feedback loops fast**. [Some best practices](https://simonwillison.net/2022/Oct/1/software-engineering-practices/): 10 10 - Tested, automated process for new development environments. ··· 16 16 - Create a [[Company Handbooks|handbook]] to store your [[Company Knowledge Management|company knowledge]]. Document: 17 17 - [[Processes]]. Status updates, [[Design Docs]], [on-boarding docs/scripts](https://about.gitlab.com/blog/2020/02/10/lessons-learned-as-data-team-manager/), [[Checklist]], ... 18 18 - Decisions. Context and rationale can be documented in a durable location. 19 19 - - Each team should [keep a changelog](https://keepachangelog.com/en/1.0.0/). [The company too](https://medium.com/linear-app/startups-write-changelogs-c6a1d2ff4820). ^473cb4 20 20 - - Aim to [confirm and log decisions](https://understandlegacycode.com/blog/earn-maintainers-esteem-with-adrs/) to move them forward. [Everything must have an URL](https://ben.balter.com/2015/11/12/why-urls/). 21 21 - - Show your work. Capture who made what decision and when, along with a detailed, but _concise_ description of why and how that decision was made. 22 22 - - Consistent changelogs also communicate new features, the value they get from your product, and your commitment to improving it. 19 19 + - Each team should [keep a changelog](https://keepachangelog.com/en/1.0.0/). [The company too](https://medium.com/linear-app/startups-write-changelogs-c6a1d2ff4820). ^473cb4 20 20 + - Aim to [confirm and log decisions](https://understandlegacycode.com/blog/earn-maintainers-esteem-with-adrs/) to move them forward. [Everything must have an URL](https://ben.balter.com/2015/11/12/why-urls/). 21 21 + - Show your work. Capture who made what decision and when, along with a detailed, but _concise_ description of why and how that decision was made. 22 22 + - Consistent changelogs also communicate new features, the value they get from your product, and your commitment to improving it. 23 23 - [[Meetings]] agendas and conclusions. 24 24 - Responsibilities. Things that aren't your fault can still be your responsibility. If something is everyone's job, it's no one's job. 25 25 - Defaults. Each thing should have a place by default, docs, issues, ... 26 26 - Aim to be a completely autonomous team. Everyone should feel empower to make decisions. Those who are responsible for something must have the means and context to effect it. You build it, you run it! **The company strategy guides the team, it doesn't tell it what to do.** 27 27 - Run [Automated Check-ins](https://basecamp.com/features/checkins) to share things explicitly. What are people working on, what are they planning to work on next, ... 28 28 - - The right way to promote people is to give them meaningful goals for the organization and promote them if they hit the goals. 28 28 + - The right way to promote people is to give them [meaningful and clear goals](https://youtu.be/oIMvMb5wVO4) for the organization and promote them if they hit the goals. 29 29 - Lack of ownership is the root of all evil. 30 30 - Having skin in the game improves the decision making process. 31 31 - [Have direct responsible individuals for everything. Everyone's problem is no one's problem](https://nintil.com/programming). Having a name accountable instead of a vague "the team" or "the process" makes it easy to make changes. ··· 48 48 - **Focus on business outcomes, not on technologies.** 49 49 - When you start from a shared understanding – that you’re all doing your best you can – you can foster a compassionate working environment. 50 50 - Everyone on your team should assume that everyone else on the team is doing their best work, given their circumstances. 51 51 - - Trust people. Add [[Processes]] where you need to replace some level of trust. 51 51 + - Trust people with freedom. Add [[Processes]] where you need to replace some level of trust. 52 52 + - Share as much context as you can. 52 53 - Times change, trends change, cultures change. Make it explicit. 53 54 - Spend time at work thinking strategically. E.g: Think about the approach you will take to address the company's needs over the medium to long term. 54 55 - **How to drive change in a team**: find people who agree on the problem, start small, experiment, scale, repeat. Making big change is hard. Keeping things simple is hard. ··· 65 66 5. [[Automation|Automate]] and keep standards. 66 67 - Keep great global [[coordination]] and incentive local experimentation. 67 68 - Being able to run small and compounding experiments (on the product or company [[processes]] and systems) is important. **Work smaller**. 68 68 - - [Some experiments won't work](https://www.lesswrong.com/posts/97LgacucCxmyjYiNT/the-archipelago-model-of-community-standards). But oftentimes it _feels_ like it wont work when in fact you just haven't stuck with it long enough for it to bear fruit. This is hard enough for _solo_ experiments. For group experiments, where not just one but _many_ people must all try a thing at once and _get good at it_, all it takes is a little defection to spiral into a mass exodus. 69 69 + - [Some experiments won't work](https://www.lesswrong.com/posts/97LgacucCxmyjYiNT/the-archipelago-model-of-community-standards). But oftentimes it _feels_ like it wont work when in fact you just haven't stuck with it long enough for it to bear fruit. This is hard enough for _solo_ experiments. For group experiments, where not just one but _many_ people must all try a thing at once and _get good at it_, all it takes is a little defection to spiral into a mass exodus. 69 70 - The group with the most power determine the system that reflect and reinforce their own way of thinking. Aim for inclusion. _Diversity is being invited to the party. Inclusion is being asked to dance and help organizing the party_. 70 71 - [Brainstorm for questions first (explore). Then find the answers (exploit).](https://getpocket.com/explore/item/better-brainstorming) 71 72 - Strive for constructive conflict. Get people to[[Asking Questions|ask questions]]. Engage in passionate, unfiltered debate about what you need to do to succeed. ··· 95 96 - [Learned helplessness](https://en.wikipedia.org/wiki/Learned_helplessness) can happen in a team. Two of the main reasons of this [normalization of deviance](https://danluu.com/wat/): 96 97 - The team needs to follow processes that have either been externally imposed, or internally imposed but no-one remembers exactly why. 97 98 - The sheer scale and/or complexity of how things work. There is truly no-one who understands the emergent behavior of the [[Systems|system]]. 98 98 - - E.g: Slow _boiling frog_ situations where existing tools have become ineffective but no one noticed. 99 99 + - E.g: Slow _boiling frog_ situations where existing tools have become ineffective but no one noticed. 99 100 - [Act as if you might leave on short notice](https://jmmv.dev/2021/04/always-be-quitting.html). Document your knowledge, long-term plans, meetings, train people around you, empower other people, delegate and keep learning! 100 101 - You have to put in more effort to make something appear effortless. Effortless, elegant performances are often the result of a large volume of effortful. Praise this instead of complex solutions. 101 102 - Invisible work will happen. If you're doing it, make an effort to share and get credit for it. Build a narrative (story) for your work. Arm your manager and fight recency bias keeping track of all the things you've done.