+1

Data/Dashboards.md

reviewed

··· 22 22 - Make them so its easy to go one layer down (X went down in Y location, or for Z new users, etc). 23 23 - Recreate dashboard from first principles periodically. 24 24 - When plotting a rate, add the top of funnel and bottom of funnel numbers to make sure things are as expected. 25 25 + - A large change is not necessarily worth investigating, and a small change is not necessarily benign. What you want to know is if the change is exceptional. 25 26 - Be clear with your stakeholder about whether this is a one-off vs. something that should be referenced more than once. 26 27 - Add an [explicit expiration date so it doesn't mold](https://mikkeldengsoe.substack.com/p/moldy-data). 27 28 - [Have meetings where you check and discuss the metrics on the dashboard](https://counting.substack.com/p/the-utility-of-an-unwatched-dashboard). This creates a powerful forcing function to look at the thing.

+13

Data/Data Culture.md

reviewed

··· 22 22 - If data is the most precious asset in a company, does it make sense to have only one team responsible for it? 23 23 - [People talk about data as the new oil but for most companies it’s a lot closer uranium](https://news.ycombinator.com/item?id=27781286). Hard to find people who can to handle or process it correctly, nontrivial security/liabilities if PII is involved, expensive to store and a generally underwhelming return on effort relative to the anticipated utility. 24 24 - [The pain in data teams come from needing to influence PMs/peers with having little control of them. Data teams need to become really great internal marketers/persuaders](https://anchor.fm/census/episodes/The-evolution-of-the-data-industry--data-jobs-w-Avo-CEO-and-Co-founder-Stefania-Olafsdottir-e16hu1l). That said, it shouldn't be the data team job to convince the organization to be data driven. That's not an effective way of spending resources. 25 25 + - Executives are expected to be data driven, even if they don't know what it means. 25 26 - People problems are orders of magnitude more difficult to solve than data problems. 26 27 - **Integrate data where the decision is made**. E.g: Google showing restaurant scores when you're looking something for dinner. 27 28 - Reduce the time to insights. If the data is already in the tool you're using, then there's zero time to insights. Provide a set of tools with the same data and let people choose depending on the goal. ··· 42 43 - Good use of data is, ultimately, a question of good epistemology. (“Is this true? What can we conclude? How do we know that?”) Good epistemology is hard. It must be taught. 43 44 - **When things are going well, no one cares about data**. The right time to present data is when things are starting to go bad. Use your early warning detection systems to understand when it looks like it's gonna be time for data to step in and save the day and then position data as a solution in the context of whatever meaning makes sense. The stakeholders are decision makers and they don't have a ton of time. They're looking to make decisions, they're looking to solve problems. 44 45 - [So much of data work is about accumulating little bits of knowledge and building a shared context in your org so that it's possible to have the big, earth shattering revelations we all wish we could drive on a predictable schedule](https://twitter.com/imightbemary/status/1536368160961572864). 46 46 + - A big purpose of data is knowledge. Knowledge is "theories or models that allow you to predict the outcomes of your business actions". 45 47 - You won't have the best allocation of resources in a reactive team. Data teams need extra [[slack]]. [Balance user requests with actual needs](https://scientistemily.substack.com/p/product-management-skills-for-data). 46 48 - Do weekly recaps in Slack in to highlight key items, company-wide progress toward north-stars, improvements in certain areas, new customer highlights. All positive and fun stuff. 47 49 - How can we measure the data team impact? ··· 79 81 - [Culture eats strategy (and tools) for breakfast](https://news.ycombinator.com/item?id=29062266). Until there’s a cultural mindset shift towards how companies value data and metadata, nothing will change. 80 82 - [Tools eat process for breakfast](https://benn.substack.com/p/the-product-is-the-process). No matter how much you blog about best practices, or how many talks you give about better ways to work, people will eventually find their way back into the behavioral grooves cut by the products they use (e.g: dbt, GitHub, ...). 81 83 - Most of the work done in data is in an effort to **reduce entropy** — Model data to remove inaccuracies, turn commonly asked questions into self-serve reports, and funnel ad-hoc questions into a formalized request process. This kind of attitude the nature of data practitioners. In the case of driving decisions with data, **embrace the chaos**. 84 84 + - Data doesn't so much drift towards entropy, **but sprints at it**. 82 85 - [Navigating the chaos to arrive at a trustworthy recommendation is one of the most important jobs to be done.](https://roundup.getdbt.com/p/iterating-on-your-data-team). Decisions usually need to be taken faster and data analyst are [not invited to the table early enough](https://petrjanda.substack.com/p/bring-data-analyst-to-the-table). Again, be lean and iterate. 83 86 - Data is *not* a “set it and forget it” kind of activity. Your dashboard *will* get stale in less than six months. Your key metrics *will* eventually have bad data in them. That machine learning model you spent all of last quarter developing *will* **[drift](https://towardsdatascience.com/model-drift-in-machine-learning-models-8f7e7413b563)** from its original fit. The environment in which your business operates is constantly changing, and so will the product or service that your business delivers. As a result, what is knowable about your business, about your product or service, is constantly changing too. And fast. 84 87 - [Have regular cleanups and audits to keep data in check](https://www.avo.app/blog/data-literacy-why-people-dont-trust-data-tips-from-patreons-dir-of-data-science). They are crucial to keeping your data trust up to par. [Schedule time to delete stuff](https://twitter.com/EdDaWord/status/1532148425487097857). ··· 88 91 - [Doing the fundamentals really well almost always exposes how little is actually understood about why things are happening. It’s uncomfortable for high performing people to acknowledge that your grip on the levers is slippery](https://twitter.com/gwenwindflower/status/1498822586255519744). 89 92 - [Data ownership is a hard problem](https://www.linkedin.com/posts/chad-sanderson_heres-why-data-ownership-is-an-incredibly-activity-6904107936533114880-gw8n/). Data is fundamentally generated by services (or front-end instrumentation) which is managed by engineers. CDC and other pipelines are built by data engineers. The delineation of ownership responsibilities is very rarely established, with each group wanting to push 'ownership' onto someone else so they can do the jobs they were hired for. 90 93 - [Becoming a data-driven organization is a journey, which unfolds over time and requires critical thinking, human judgement, and experimentation](https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard). Fail fast, learn faster. 94 94 + - Data-drivenness is about building tools, abilities, and, most crucially, a culture that acts on data. 91 95 - [Path to create a data-driven organization](https://twitter.com/_abhisivasailam/status/1520274838450888704): 92 96 - 1. Get a well-placed leader with influence to message, model, and demand data-driven execution. 93 97 - 2. Hire/fire based on data aptitude and usage. ··· 112 116 - Have a documentation [entry-point for Data](https://github.com/mozilla/data-docs). 113 117 - [For self-serve, aim to own as little as possible but keep in mind you can't make people do what you want but can stop them for doing what you don't want](https://youtu.be/wyW6hQGZxgY) 114 118 - [You need to make a grocery store. You can’t give folks directions to the farm to pick their own produce](https://twitter.com/teej_m/status/1603205457992044545). 119 119 + - It's easy to lie with statistics, but it's hard to tell the truth without them. 120 120 + 121 121 + ## Tools 122 122 + 123 123 + Sometimes, https://commoncog.com/becoming-data-driven-first-principles/#the-trick. These might help your organization think better with data. 124 124 + 125 125 + - [Process Behaviour Charts](https://demingalliance.org/resources/articles/process-behaviour-charts-an-introduction) 126 126 + - [Time Lagged Conversions](https://better.engineering/modeling-conversion-rates-and-saving-millions-of-dollars-using-kaplan-meier-and-gamma-distributions/) 127 127 + - Change point detection

+2

Data/Experimentation.md

reviewed

··· 4 4 5 5 When you're not certain of the right answer, the best approach is to [have a portfolio](https://seths.blog/2022/01/portfolio-thinking/), a range of bets that reward us with resilience and significant upside. 6 6 7 7 + _[Insight is derived from action, not analysis](https://commoncog.com/becoming-data-driven-first-principles/)_: you only learn to improve your business when you test control factors, not when you discover them. 8 8 + 7 9 **A/B Test** (Including multivariate tests) are randomized controlled experiment where a single success metric is measured to determine which variant performs the best. 8 10 9 11 Normally this will consist of two groups: A **control** and **test** group, but it could also be implemented with multiple test groups (a multivariate test). The goal of an A/B test is to reach a statistically significant result, i.e. you can say that one variant is better than the other with a high level of confidence that observed difference did not occur by chance. While A/B testing is a very powerful conversion optimization instrument, it requires lots of hard work. [Most of your experiments will not produce a significant result](https://www.jitbit.com/news/185-most-of-your-ab-tests-will-fail).

+4

Data/Metrics.md

reviewed

··· 16 16 - Product metrics allow measuring product progress and creating alignment in an outcome-oriented way. There are many product frameworks available to help us think about the right key things to track. Think about **[product metrics that matter](https://uxdesign.cc/product-metrics-that-matter-951b9e4d4eca)** for you. 17 17 - Push a culture of metrics and goals as a source of learning, not promotions or success delegation. 18 18 - Vanity metrics are surface-level metrics. They're often large measures, like number of downloads, that impress others. **Clarity metrics** are operational metrics, like the number of minutes a day your product actually gets used or how long it took for a user to get service. These are the hidden gears that drive growth. 19 19 + - When people are pressured to meet a target value there are three ways they can proceed: 20 20 + 1. They can work to improve the system. 21 21 + 2. They can distort the system. 22 22 + 3. Or they can distort the data. 19 23 20 24 ## Good Metric Checklist 21 25

+3

Organizations.md

reviewed

··· 8 8 - Make processes [[Idempotence|idempotent]] and [[Automation|automated]]. 9 9 - [Put mechanisms that enable the organization to learn and adapt](https://www.remyevard.com/posts/2021/11/30/healthy-organizations.html). 10 10 - _Culture, Coordination, and Capital_ are the foundation of your ability to have an impact on your mission. 11 11 + - Every company is a process, and processes may be decomposed into multiple smaller processes. Each process you look at will have outputs that you care about and inputs that you must discover. 11 12 - A company may be looked as a combination of 3 things: 12 13 - The people who work at the company. 13 14 - The process the company uses to get work done. ··· 34 35 - Decision making should be pushed down the hierarchy to the practitioner. 35 36 - An organization exhibits risk aversion comparable to the most risk averse decision maker in the decision chain. 36 37 - Every business has an equation that describes how it generates revenue. Write it down and decompose it to better understand the relationships. [Don't try to optimize that number or it'll be gamed](https://www.fast.ai/2019/09/24/metrics/). Solve Problems, not [[metrics]]. When a measure becomes a target, it ceases to be a good measure. Not everything that counts can be measured, and not everything that can be measured counts. [The more important a metric is in social decision making, the more likely it is to be manipulated](https://en.wikipedia.org/wiki/Campbell%27s_law). 38 38 + - Your stated strategy must match the strategy revealed in the context of decision making (budgets and teams). 37 39 - [[Data Culture|Data]] can be valuable in helping you understand the world, test hypotheses, and move beyond gut instincts or hunches. [[Metrics]] can be useful when they are in their proper context and place. 38 40 - Use data to identify friction points, and constantly experiment with changes to make things easier for you and your customers. 39 41 - How does the [data-informed product loop look](https://cutlefish.substack.com/p/tbm-852-the-data-informed-product)? ··· 48 50 - Train employees well enough they could get another job, but treat them well enough so they never want to. 49 51 - [Apolitical](https://twitter.com/naval/status/1263322014372130817) work environment is hard to maintain. Choose your company battles and causes. 50 52 - Managers will hire more managers as the team scales and that creates new teams. Teams will fight to justify their existence. Beware of this build-up inertia. 53 53 + - [The causal structure of your business does not care about departments](https://commoncog.com/becoming-data-driven-first-principles/). 51 54 - [As organizations become less efficient / less effective, they need more and more managers to "manage" that inefficiency. This kicks off a wicked cycle, because they'll self-identify with managing a problem ... which reinforces it.](https://mobile.twitter.com/johncutlefish/status/1472669773410410504) 52 55 - It might be interesting to cap the core team size at N people (e.g: 150). Focus on solving one problem, and do it well. 53 56 - [The art of org design is essentially effective iteration towards form-context fit. You need four sub-skills to do effective iteration](https://commoncog.com/blog/org-design-skill/). To get good at org design, you need to build more accurate models of the people in your org, learn how they respond to [[incentives]], and in build enough power and credibility to get your org changes to take place.