feat: 🔄 Update various documents to enhance data practices and metric understanding

+1 -1

Data/Analytics Engineering.md

··· 11 11 - [Analytics code should be version controlled, tested, modular and maintainable](https://www.getdbt.com/analytics-engineering/why/). 12 12 - Define all resources ([[Dashboards]] in YAML, Cohorts in SQL, ...) as code. 13 13 - Treat data the same way engineers treat code. That means CI/CD, tests, frequent PRs, ... 14 - - Use [[Data Request Template]] when getting questions. 14 + - Use [[Data Practices#Data Request Template]] when getting questions. 15 15 - Analytics work can be roughly split in two buckets: 16 16 1. Building automated [[Systems]], from metrics to [[Dashboards]], to enable self-service use cases for business users. This is what we now typically call analytics engineering. 17 17 2. Doing ad-hoc analyses, to answer some questions directly.

+1

Data/Dashboards.md

··· 70 70 - That the data is complete. 71 71 - The dashboard is always just a snapshot of "what" is happening, but knowing the underlying base level data is always needed to understand "why" it’s happening. 72 72 - Usually, answers don't lead to Eureka moments, they lead to follow up questions and follow up questions. 73 + - Having 20 dashboards means one is always likely to be up or down by a statistically significant amount.

+25 -16

Data/Data Culture.md

··· 1 1 # Data Culture 2 2 3 3 - The data team needs to be focus on delivering insights and supporting decisions. The outcome of the data team are *decisions* and a *shared context across the organization* that makes coordination easier. 4 - - Your goal as a data professional is to facilitate [[Making Decisions|decision making]]. 4 + - Your goal as a data professional is to facilitate [[Making Decisions|decision making]] and [help surface/investigate the performance of a business](https://sqlpatterns.com/p/delivering-value-as-a-data-team) (e.g. [operational](https://twitter.com/ergestx/status/1731324299590479989)). 5 5 - Learning to drive decisions quickly, a bias to action, is a critical competency for an analyst. Every skill you learn – [[communication]], [[writing]], [[experimentation]], [[Metrics|metric design]] – supports this. 6 - - If analysis is not actionable, it does not really matter. Analysis must drive to action. [Clear results won't spur action themselves](https://www.linkedin.com/posts/eric-weber-060397b7_data-analytics-machinelearning-activity-6675746028144205824-CQxW/). The organization needs to be ready to pivot when something isn't working. 7 - - [Data's impact is tough to measure — it doesn't always translate to value](https://dfrieds.com/articles/data-science-reality-vs-expectations.html). 8 - - The Data Team should be building and iterating the [Data Product](https://locallyoptimistic.com/post/run-your-data-team-like-a-product-team/). 6 + - [If analysis is not actionable, it does not really matter](https://twitter.com/decisionleader/status/1661041373783441408). Analysis must drive to action. [Clear results won't spur action themselves](https://www.linkedin.com/posts/eric-weber-060397b7_data-analytics-machinelearning-activity-6675746028144205824-CQxW/). The organization needs to be ready to pivot when something isn't working. 7 + - [Data's impact is tough to measure — it doesn't always translate to value](https://dfrieds.com/articles/data-science-reality-vs-expectations.html) 8 + - The value of "insights" is often unknown. 9 + - The Data Team should be building and iterating the [Data Product](https://locallyoptimistic.com/post/run-your-data-team-like-a-product-team/). 9 10 - Notebooks are a workshop. Production systems are the factory. Not everything needs to be put into production. Not everything should be a notebook. You need both. Lean in to the strength of each. 10 11 - Data is fundamentally a collaborative design process rather than a tool, an analysis, or even a product. [Data works best when the entire feedback loop from idea to production is an iterative process](https://pedram.substack.com/p/data-can-learn-from-design). 11 12 - [To get buy in, explain how the business could benefit from better data](https://youtu.be/Mlz1VwxZuDs) (e.g: more and better insights). Start small and show value. 12 13 - Run *[Purpose Meetings](https://www.avo.app/blog/tracking-the-right-product-metrics)* or [Business Metrics Review](https://youtu.be/nlMn572Dabc). 13 14 - Purpose Meetings are 30 min meetings in which stakeholders, engineers and data align on the goal of a release and what is the best way to evaluate the impact and understand its success. Align on the goal, commit on metrics and design the data. 14 15 - Business Metrics Review is a 30 to 60 minutes meeting to chat and explore key metrics and teach how to think with data. 16 + - You don't hit a quantitative goal by focusing on the goal. You hit a quantitative goal by focusing on the process. 15 17 - Value of clear goals and expectations. Validate what you think your job is with your manager and stakeholders, repeatedly. 16 18 - [While the output of your team is what you want to maximize, you'll need some indicators that will help guide you day-to-day](https://data-columns.hightouch.io/your-first-60-days-as-a-first-data-hire-weeks-3-4/). Decide what's important to you (test coverage, documentation missing, queries run, models created, ...), and generate some internal reports for yourself. 17 - - [Data teams should be a part of the business conversations from the beginning](https://cultivating-algos.stitchfix.com/). Get the data team involved early, have open discussions with them about the existing work, and how to prioritize new work against the existing backlog. Don’t accept new work without addressing the existing bottlenecks, and don’t accept new work without requirements. **Organizational [[politics]] matter way more than any data methods or technical knowledge**. The hard bit about becoming data driven in business isn't the technical bits. It's the political bits. 19 + - [Data teams should be a part of the business conversations from the beginning](https://cultivating-algos.stitchfix.com/). Get the data team involved early, have open discussions with them about the existing work, and how to prioritize new work against the existing backlog. Don't accept new work without addressing the existing bottlenecks, and don't accept new work without requirements. **Organizational [[politics]] matter way more than any data methods or technical knowledge**. The hard bit about becoming data driven in business isn't the technical bits. It's the political bits. 18 20 - Including data people in meetings causes happy accidents! 19 21 - The layout of the organization impacts time of the information to propagate and adds losses. 20 - - The modern data team needs to have *real organizational power* — it needs to be able to say "no” and mean it. If your data team does not truly have the power to say no to stakeholders, it will get sent on all kinds of wild goose chases, be unproductive, experience employee churn, etc. 22 + - The modern data team needs to have *real organizational power* — it needs to be able to say "no" and mean it. If your data team does not truly have the power to say no to stakeholders, it will get sent on all kinds of wild goose chases, be unproductive, experience employee churn, etc. 21 23 - Data should report to the CEO. Ideally at least with some weekly metrics split into (a) notable trends, (b) watching close, and (c) business as usual. 22 24 - If data is the most precious asset in a company, does it make sense to have only one team responsible for it? 23 - - [People talk about data as the new oil but for most companies it’s a lot closer uranium](https://news.ycombinator.com/item?id=27781286). Hard to find people who can to handle or process it correctly, nontrivial security/liabilities if PII is involved, expensive to store and a generally underwhelming return on effort relative to the anticipated utility. 25 + - [People talk about data as the new oil but for most companies it's a lot closer uranium](https://news.ycombinator.com/item?id=27781286). Hard to find people who can to handle or process it correctly, nontrivial security/liabilities if PII is involved, expensive to store and a generally underwhelming return on effort relative to the anticipated utility. 24 26 - [The pain in data teams come from needing to influence PMs/peers with having little control of them. Data teams need to become really great internal marketers/persuaders](https://anchor.fm/census/episodes/The-evolution-of-the-data-industry--data-jobs-w-Avo-CEO-and-Co-founder-Stefania-Olafsdottir-e16hu1l). That said, it shouldn't be the data team job to convince the organization to be data driven. That's not an effective way of spending resources. 25 27 - Executives are expected to be data driven, even if they don't know what it means. 28 + - Epistemology of the leadership team really really matters. 26 29 - People problems are orders of magnitude more difficult to solve than data problems. 27 30 - **Integrate data where the decision is made**. E.g: Google showing restaurant scores when you're looking something for dinner. 28 31 - Reduce the time to insights. If the data is already in the tool you're using, then there's zero time to insights. Provide a set of tools with the same data and let people choose depending on the goal. ··· 31 34 - Data professionals can build consensus as the company becomes more diverse. 32 35 - Data systems can establish methods for understanding the world even as it becomes more complex. 33 36 - Data literacy can create pathways for anyone to contribute equally to the organization's reality. 34 - - Understanding variation is the beginning of data literacy. 35 - - Create a single space as the central place to post [[Data Request Template|data requests]]. 37 + - [Understanding variation is the beginning of data literacy](https://twitter.com/ejames_c/status/1732597443127382369). 38 + - Create a single space as the central place to post [[Data Practices#Data Request Template]]. 36 39 - On the other hand, data analysis and data science are domain level problems and cannot be centralized. 37 40 - Create a single space to [[Data Practices|share the results of analysis and decisions made based on them]]. 38 - - Log changes so everyone can jump in and be aware of what’s going on. 41 + - Log changes so everyone can jump in and be aware of what's going on. 39 42 - Log assumptions and lessons learned somewhere. This information should loop back into the data product. 40 43 - Make the warehouse the source of truth for all the teams. 41 44 - What data is Finance/HR/Marketing using to set their OKRs? Put that on the warehouse and model it. 42 45 - [[Metrics]] should be derived from the most realistic data sources. E.g: using internal databases instead of product tracking for "Users Created". 43 46 - Do you want better data? Hire people interested in data! 44 - - Good use of data is, ultimately, a question of good epistemology. (“Is this true? What can we conclude? How do we know that?”) Good epistemology is hard. It must be taught. 47 + - Having managers tell the data team to "Find Insights" is a telltale mark of bad data management and organizational structure. 48 + - Good use of data is, ultimately, a question of good epistemology. ("Is this true? What can we conclude? How do we know that?") Good epistemology is hard. It must be taught. 45 49 - **When things are going well, no one cares about data**. The right time to present data is when things are starting to go bad. Use your early warning detection systems to understand when it looks like it's gonna be time for data to step in and save the day and then position data as a solution in the context of whatever meaning makes sense. The stakeholders are decision makers and they don't have a ton of time. They're looking to make decisions, they're looking to solve problems. 46 50 - [So much of data work is about accumulating little bits of knowledge and building a shared context in your org so that it's possible to have the big, earth shattering revelations we all wish we could drive on a predictable schedule](https://twitter.com/imightbemary/status/1536368160961572864). 47 51 - A big purpose of data is knowledge. Knowledge is "theories or models that allow you to predict the outcomes of your business actions". Insights may originate from data but are confirmed through actions. ··· 60 64 - Use the questions people are asking to find data *hotspots* and focus our energy on those. That means some corners of your data will be messy, and some concepts will go undocumented. Data is perennially broken and messy. **Embrace the mess**. 61 65 - Get excited when people ask questions. Embrace confusion and curiosity. Offer help. Be friendly. 62 66 - [Reality is complex and multidimensional and often difficult to comprehend](https://mobile.twitter.com/rahulj51/status/1485429967131639808). 63 - - [Document data when it’s generated](https://davidsj.substack.com/p/the-data-chasm). Make it part of the process of adding a new event, table, or a replication job, when the change is already top of mind. If possible, embed it in the development process, and pester people when they don’t include the necessary updates. This shifts the burden of documentation upstream, making it part of the development cycle. 67 + - [Document data when it's generated](https://davidsj.substack.com/p/the-data-chasm). Make it part of the process of adding a new event, table, or a replication job, when the change is already top of mind. If possible, embed it in the development process, and pester people when they don't include the necessary updates. This shifts the burden of documentation upstream, making it part of the development cycle. 64 68 - To align stakeholders incentives with the data team, stakeholders should show their impact through data. This forces stakeholders to [[Product Analytics|plan tracking]] and think about metrics. 65 69 - [To achieve distribution, build for who your stakeholder truly is, not for the stakeholder you want them to be](https://ian-macomber.medium.com/launching-and-scaling-data-science-teams-three-years-later-f1fa6f25b4ae). 66 70 - You should have something that answers the following questions: ··· 79 83 - [Send surveys](https://docs.google.com/forms/d/e/1FAIpQLSfufs_0zOGlFiE6oqrdZU7xCi399CBYbIlZkAMe15GTRRcPZA/viewform) from time to time trying to get pain points and know where issues are. 80 84 - E.g: Do you have access to the data I need to make decisions in your role? 81 85 - [Bring the collaboration process inline with the assets to allow for better handoffs and feedback](https://pedram.substack.com/p/data-can-learn-from-design). 82 - - [Culture eats strategy (and tools) for breakfast](https://news.ycombinator.com/item?id=29062266). Until there’s a cultural mindset shift towards how companies value data and metadata, nothing will change. 86 + - [Culture eats strategy (and tools) for breakfast](https://news.ycombinator.com/item?id=29062266). Until there's a cultural mindset shift towards how companies value data and metadata, nothing will change. 83 87 - [Tools eat process for breakfast](https://benn.substack.com/p/the-product-is-the-process). No matter how much you blog about best practices, or how many talks you give about better ways to work, people will eventually find their way back into the behavioral grooves cut by the products they use (e.g: dbt, GitHub, ...). 84 88 - Most of the work done in data is in an effort to **reduce entropy** — Model data to remove inaccuracies, turn commonly asked questions into self-serve reports, and funnel ad-hoc questions into a formalized request process. This kind of attitude the nature of data practitioners. In the case of driving decisions with data, **embrace the chaos**. 85 89 - Data doesn't so much drift towards entropy, **but sprints at it**. 86 90 - [Navigating the chaos to arrive at a trustworthy recommendation is one of the most important jobs to be done.](https://roundup.getdbt.com/p/iterating-on-your-data-team). Decisions usually need to be taken faster and data analyst are [not invited to the table early enough](https://petrjanda.substack.com/p/bring-data-analyst-to-the-table). Again, be lean and iterate. 87 - - Data is *not* a “set it and forget it” kind of activity. Your dashboard *will* get stale in less than six months. Your key metrics *will* eventually have bad data in them. That machine learning model you spent all of last quarter developing *will* **[drift](https://towardsdatascience.com/model-drift-in-machine-learning-models-8f7e7413b563)** from its original fit. The environment in which your business operates is constantly changing, and so will the product or service that your business delivers. As a result, what is knowable about your business, about your product or service, is constantly changing too. And fast. 91 + - Data is *not* a "set it and forget it" kind of activity. Your dashboard *will* get stale in less than six months. Your key metrics *will* eventually have bad data in them. That machine learning model you spent all of last quarter developing *will* **[drift](https://towardsdatascience.com/model-drift-in-machine-learning-models-8f7e7413b563)** from its original fit. The environment in which your business operates is constantly changing, and so will the product or service that your business delivers. As a result, what is knowable about your business, about your product or service, is constantly changing too. And fast. 88 92 - [Have regular cleanups and audits to keep data in check](https://www.avo.app/blog/data-literacy-why-people-dont-trust-data-tips-from-patreons-dir-of-data-science). They are crucial to keeping your data trust up to par. [Schedule time to delete stuff](https://twitter.com/EdDaWord/status/1532148425487097857). 89 93 - We're moving from software consumers to data consumers. Data and BI will become more and more federated (you get data insights on your JIRA card without having to leave JIRA) 90 94 - Over time, data literacy across organizations will become commonplace the same way typewriting has. [Most professionals, at all levels of the business, will be capable of generating their own insights without requiring a data team](https://roundup.getdbt.com/p/data-expertise-everywhere). 91 95 - Data practitioners acknowledge that solid reporting is at the bottom of the data hierarchy of needs but few companies do even basic KPI reporting well. 92 - - [Doing the fundamentals really well almost always exposes how little is actually understood about why things are happening. It’s uncomfortable for high performing people to acknowledge that your grip on the levers is slippery](https://twitter.com/gwenwindflower/status/1498822586255519744). 96 + - [Doing the fundamentals really well almost always exposes how little is actually understood about why things are happening. It's uncomfortable for high performing people to acknowledge that your grip on the levers is slippery](https://twitter.com/gwenwindflower/status/1498822586255519744). 93 97 - [Data ownership is a hard problem](https://www.linkedin.com/posts/chad-sanderson_heres-why-data-ownership-is-an-incredibly-activity-6904107936533114880-gw8n/). Data is fundamentally generated by services (or front-end instrumentation) which is managed by engineers. CDC and other pipelines are built by data engineers. The delineation of ownership responsibilities is very rarely established, with each group wanting to push 'ownership' onto someone else so they can do the jobs they were hired for. 94 98 - [Becoming a data-driven organization is a journey, which unfolds over time and requires critical thinking, human judgement, and experimentation](https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard). Fail fast, learn faster. 95 99 - [Data-drivenness is about building tools, abilities, and, most crucially, a culture that acts on data](https://twitter.com/ejames_c/status/1732592768890057115). ··· 117 121 - Make your [modeling technique](https://data-columns.hightouch.io/untitled-2/) explicit. 118 122 - Have a documentation [entry-point for Data](https://github.com/mozilla/data-docs). 119 123 - [For self-serve, aim to own as little as possible but keep in mind you can't make people do what you want but can stop them for doing what you don't want](https://youtu.be/wyW6hQGZxgY) 120 - - [You need to make a grocery store. You can’t give folks directions to the farm to pick their own produce](https://twitter.com/teej_m/status/1603205457992044545). 124 + - [You need to make a grocery store. You can't give folks directions to the farm to pick their own produce](https://twitter.com/teej_m/status/1603205457992044545). 121 125 - It's easy to lie with statistics, but it's hard to tell the truth without them. 126 + - Most people approach data with an "optimisation worldview", thinking in terms of "make number go up". There is an alternative. The Process Control worldview, which is similar to"Here is a process. Your job is to discover all the control factors that affect this process.". 127 + - Your job is to figure out what you can control that affects the process, and then systematically pursue that. 128 + - You can discover these control factors through one of two ways: 129 + - [[Experimentation]] 130 + - Observe sudden, unexplained special variation in your data, which you must then investigate to uncover new control factors that you don't already know about. 122 131 123 132 ## Tools 124 133

+71 -7

Data/Data Practices.md

··· 2 2 3 3 Some useful practices to keep in mind when working in a data team. They can be proposed as Data Improvement Proposals (DIPs) and discussed in the team. 4 4 5 - ## Request For Analysis 5 + ## Data Request Template 6 + 7 + Inspired by [Caitlin Hudon Intake Form for Data Requests](https://caitlinhudon.com/2020/09/16/data-intake-form/). 6 8 7 - - It's an [intake form for data requests](https://www.caitlinhudon.com/posts/2020/09/16/data-intake-form). 9 + ### Why? 10 + 11 + - Quick way of gathering context needed for good analysis. 12 + - People need to prepare and that means better kickoff conversation. 13 + - Easier to triage and connect dots across requests. 14 + - Creates easily referenced records of requests. 15 + - Friction cuts down on lazy asks. Too much friction will disuade legit ask. 16 + 17 + ### Data Request Form 18 + 19 + - What kind of deliverable would be most helpful for your request? 20 + - Will you need this data again in the future? 21 + - Which stats is this going to change or which action will be taken based on this data and by whom? 22 + - How can we measure our progress/success for each step? 23 + - What happens if we don't hit the target? 24 + - What is the real problem you're trying to solve? 25 + - Frame the (initially vague) data questions this way: [How does _lever_ impact _KPI_?](https://www.narrator.ai/blog/how-i-frame-data-questions-to-make-analyses-more-useful/) 26 + - What is the business question you're trying to answer? 27 + - What decision will you make or action will you take with this data? 28 + - [What are we trying to improve](https://twitter.com/ergestx/status/1758538405695086829)? 29 + - **Request Description** 30 + - Do you know where is the related data? 31 + - What time period do you care about? 32 + - Can this analysis be done in our current BI tools? 33 + - If yes, do you have a starting link? 34 + - If no, can [[Reverse ETL]] help? 35 + - Which is more important? 36 + - Getting the answer quickly 37 + - Getting an accurate answer 38 + - Any gotchas we should know about? 39 + - What is the priority for this request? _Optional_ 40 + - Who will see this deliverable? 41 + - How will this deliver business value within 90 days? 42 + 43 + ## Weekly Metrics Review 44 + 45 + - Beware of the question "Why did metric X go down last week?" It's frustrating to not know. But the reality is that there's often random spikes of usage, seasonality, and other external factors. If you chase every dip, you'll make up narratives that aren't true. 46 + - [You'll find 95% of metrics fall into "business as usual"](https://twitter.com/teej_m/status/1293637500703924225) and don't need to be discussed. When you report the weekly metrics, split into: 47 + 1. Notable trends. 48 + 2. Watching close. 49 + 3. Business as usual. 50 + - Make people talk and explore their metrics 8 51 9 52 ## Data Platform UX Survey 53 + 54 + We need to know what works, what doesn't and what people are using. 10 55 11 56 - [Asking about the data experience](https://docs.google.com/forms/d/e/1FAIpQLSfufs_0zOGlFiE6oqrdZU7xCi399CBYbIlZkAMe15GTRRcPZA/viewform) 12 57 - [https://locallyoptimistic.com/post/surveys/](https://locallyoptimistic.com/post/surveys/) 13 58 - [https://docs.google.com/forms/d/e/1FAIpQLSc-z4yCYX5cpbPTMUInLYurxYgY1UXd7iJOMGI_DAGc-wB17w/viewform](https://docs.google.com/forms/d/e/1FAIpQLSc-z4yCYX5cpbPTMUInLYurxYgY1UXd7iJOMGI_DAGc-wB17w/viewform) 14 - - We need to know what works, what doesn’t and what people are using 15 59 - Is all the data you want available? 60 + - What data is painful now? 16 61 - Is it difficult to find the data you want? 62 + - Are there any known issues or limitations with the current set of tools or datasets? 17 63 - Do you trust the metrics/calculations presented? 18 - - Do you ever get confused about what metric or dimension means? 64 + - Which ones do you trust? 65 + - Which ones do you not trust? 66 + - Do you ever get confused about what a metric or data point represent? 19 67 - Do you know where to go for help with data questions? 20 - - Do you feel like our data platform hinders your job performance? 21 - - Are you happy with our data platform? 68 + - Do you feel like our data approach hinders your job performance? 69 + - Are you happy with our data setup? 22 70 - Can these two resources request be merged with an existing artifact(Dashboard/Cohort)? 23 71 - Does X resource needs to be tagged as "curated"? Does Y needs to be deprecated? 24 72 ··· 44 92 📝 Some extra information that might be useful. 45 93 🔍 Dig deeper on link.com! 46 94 47 - _Questions, concerns? Thread on!_ 🧵 95 + _Questions, concerns, ideas? Thread on!_ 🧵 48 96 ``` 97 + 98 + ## Charting Principles 99 + 100 + [Some principles to keep in mind when creating charts](https://www.eugenewei.com/blog/2017/11/13/remove-the-legend). 101 + 102 + - Don't include a legend; instead, label data series directly in the plot area. Usually labels to the right of the most recent data point are best. Some people argue that a legend is okay if you have more than one data series. My belief is that they're never needed on any well-constructed line graph. 103 + - Use thousands comma separators to make large figures easier to read 104 + - Related to that, never include more precision than is needed in data labels. For example, Excel often chooses two decimal places for currency formats, but most line graphs don't need that, and often you can round to 000's or millions to reduce data label size. If you're measuring figures in the billions and trillions, we don't need to see all those zeroes, in fact it makes it harder to read. 105 + - Format axis labels to match the format of the figures being measured; if it's US dollars, for example, format the labels as currency. 106 + - Look at the spacing of axis labels and increase the interval if they are too crowded. As Tufte counsels, always reduce non-data-ink as much as possible without losing communicative power. 107 + - Start your y-axis at zero (assuming you don't have negative values) 108 + - Try not to have too many data series; five to eight seems the usual limit, depending on how closely the lines cluster. On rare occasion, it's fine to exceed this; sometimes the sheer volume of data series is the point, to show a bunch of lines clustered. These are edge cases for a reason, however. 109 + - If you have too many data series, consider using small multiples if the situation warrants, for example if the y-axes can match in scale across all the multiples. 110 + - Include explanations for anomalous events directly on the graph; you may not always be there in person to explain your chart if it travels to other audiences. 111 + - Always note, usually below the graph, the source for the data. 112 + - Include targets for figures as asymptotes to help audiences see if you're on track to reach them.

-35

Data/Data Request Template.md

··· 1 - # Data Request Template 2 - 3 - Inspired by [Caitlin Hudon Intake Form for Data Requests](https://caitlinhudon.com/2020/09/16/data-intake-form/). 4 - 5 - ## Why? 6 - 7 - - Quick way of gathering context needed for good analysis. 8 - - People need to prepare and that means better kickoff conversation. 9 - - Easier to triage and connect dots across requests. 10 - - Creates easily referenced records of requests. 11 - - Friction cuts down on lazy asks. Too much friction will disuade legit ask. 12 - 13 - ## Data Request Form 14 - 15 - - What kind of deliverable would be most helpful for your request? 16 - - Will you need this data again in the future? 17 - - Which stats is this going to change or which action will be taken based on this data and by whom? 18 - - How can we measure our progress/success for each step? 19 - - What happens if we don't hit the target? 20 - - What is the real problem you're trying to solve? 21 - - Frame the (initially vague) data questions this way: [How does _lever_ impact _KPI_?](https://www.narrator.ai/blog/how-i-frame-data-questions-to-make-analyses-more-useful/) 22 - - What is the business question you're trying to answer? 23 - - What decision will you make or action will you take with this data? 24 - - **Request Description** 25 - - Do you know where is the related data? 26 - - Can this analysis be done in our current BI tools? 27 - - If yes, do you have a starting link? 28 - - If no, can [[Reverse ETL]] help? 29 - - Which is more important? 30 - - Getting the answer quickly 31 - - Getting an accurate answer 32 - - Any gotchas we should know about? 33 - - What is the priority for this request? _Optional_ 34 - - Who will see this deliverable? 35 - - How will this deliver business value within 90 days?

+20 -1

Data/Metrics.md

··· 2 2 3 3 ![[Quotes#^a5049d]] 4 4 5 + - Metrics are how we express how a company turns its inputs into outputs — how the company creates, captures, transforms, spends, and distributes value. 5 6 - Focus on 3 to 5 metrics. You may also want to consider counter-metrics (or pairing metrics) that keep you from [over-rotating on a singular metric](https://www.dataliftoff.com/wp-content/uploads/2022/10/tennis_balls-1536x2048.jpeg). 6 7 - [Design **north star metrics that capture value to the customer** rather than value to your organization](https://roundup.getdbt.com/p/the-perfect-north-star-metric). Beware, optimizing against at a single north star metric is like looking exclusively at the score to get insight into how to win the game. 7 8 - Common understanding of a metric matters more than the metric precision. That understanding requires some standardization (names, time spans, ...) and that needs [[Coordination]]. ··· 9 10 - Rely on the SMART framework (Specific, Measurable, Achievable, Results-Oriented, Targeted). 10 11 - Pick the simplest metric that works for you. Metrics definitions should be as easy as a tool-tip away to find. 11 12 - [Metrics are a tool, but not the destination](https://breakingpoint.substack.com/p/you-have-too-many-metrics)! You want to use the fewest metrics possible to cover all the fundamentals of your business. 13 + - Pick a metric you want to improve or affect! 12 14 - Organizations need three things related to metrics: 13 15 1. A [[Metrics|process for defining metrics]]. 14 16 2. A single source of truth for the metrics. ··· 20 22 1. They can work to improve the system. 21 23 2. They can distort the system. 22 24 3. Or they can distort the data. 23 - - Every metric you use should have an Operational Definition. 25 + - [Every metric you use should have an Operational Definition](https://twitter.com/ejames_c/status/1732621626259484953). 24 26 1. A criterion — the thing you want to measure. 25 27 2. Test procedure — how will you measure the thing? 26 28 3. Decision rule — how will you decide if the thing you’re measuring should be included in the count? 29 + - A process is predictable if it just contains routine variation. It is unpredictable if it contains both routine and special variation. You can only improve a process if you first make it predictable. 30 + - Metrics should help forming a working causal model of the business you're in so you know what interventions you can make and can predict the consequences. 31 + 27 32 ## Good Metric Checklist 28 33 29 34 - **Specific and sensitive**: Metrics should be specific to the product or feature, and need to be explicitly and quantitatively defined. The metric should also be sensitive enough to measure the impact we expect to see. ··· 73 78 - Is movable. You can affect the metric through changes to the product experience. 74 79 - Is not an average. The danger of averages is you may move the metric by inspiring a small subset of customers to do a lot more of something. But this may not affect enough members to improve the overall product experience. 75 80 - Correlates to your high-level engagement metric. 81 + 82 + ## Metric Trees 83 + 84 + A metric tree is a logical representation of a business' growth model in a graph form. It's a simplified representation of how inputs flow into outputs. 85 + 86 + - Metric trees are composed of: 87 + - Output metrics 88 + - Input metrics 89 + - Relationships amongst them. Output Metrics track the performance of key business processes while Input Metrics are the knobs and levers that allow you to manipulate the Output Metrics. 90 + - [Start from the top metric you care about (the north star metric)](https://sqlpatterns.com/p/designing-metrics-trees). 91 + - Gather a list of all the metrics your business measures. 92 + - Standardize metrics before you build the metrics tree. 93 + - Recursively break down inputs. Once you have a new metric, ask yourself, what are the components of that? 94 + - The relationship between the metrics isn't always mathematical. Sometimes it's causal or even correlational.

+1

Organizations.md

··· 68 68 - By [swinging the pendulum](https://twitter.com/BrandonMChu/status/1502312472644100105) and changing focus periodically, you accept more extreme (and clear) outcomes in the short term, but in the long term arrived at the middle ground you aim for. 69 69 - [Big organizations develop strategic inefficiency to carry on doing what they're doing](https://youtu.be/v1eWIshUzr8?t=1147). 70 70 - You can't sell something like versioning or encryption as the key feature, you need to put out a comparable product and have these features be the thing that tips the scale in your favour. 71 + - Your business may be thought of as a process. It has inputs, and it has outputs. One type of knowledge is understanding which inputs affect which outputs. 71 72 72 73 ## Resources 73 74

Configure Feed

Configure Feed