this repo has no description
💡 Add link on data coverage vs nuance and note that opinions are revealed after seeing results
+2
+2
Impact Evaluators.md
+2
Impact Evaluators.md
···
67
67
- Humans excel at relative judgments, but struggle with absolute judgments.
68
68
- Many algorithms can be used to convert pairwise comparisons into absolute scores.
69
69
- Pairwise shines when all the context is in the UX.
70
+
- [Data is good at providing comprehensive coverage of things that are countable. Data is bad at dealing with nuances and qualitative concepts that experts intuitively understand.](https://gov.optimism.io/t/lessons-learned-from-two-years-of-retroactive-public-goods-funding/9239)
70
71
- **Design for composability**. Define clear data structures (graphs, weight vectors) as APIs between layers.
71
72
- Multiple communities could share measurement infrastructure.
72
73
- Different evaluation methods can operate on the same data.
···
123
124
- The meta-layer can help compose and evaluate mechanisms. How do we know mechanism B is better than A? Or even better than A + B, how do we evolve things?
124
125
- Is the evaluation/reward better than a centralized/simpler alternative?
125
126
- E.g: on tabular clinical prediction datasets, standard logistic regression was found to be on par with deep recurrent models.
127
+
- People only reveal their true opinions after seeing the result (you need to show people something and iterate based on their reactions in order to build something they actually want).
126
128
- **Exploration vs Exploitation**. IEs are optimization processes with tend to exploit (more impact, more reward). This ends up with a monopoly (100% exploit). You probably want to always have some exploration.
127
129
- [IEs need to show how the solution is produced by the interactions of people each of whom possesses only partial knowledge](https://news.ycombinator.com/item?id=44232461).
128
130