this repo has no description
📝 Address juror reliability concerns
+2
-1
+2
-1
Deep Funding.md
+2
-1
Deep Funding.md
···
46
46
- **Juror Reliability**
47
47
- So far, expert juror's pairwise comparisons have been inconsistent, noisy, and low in statistical power
48
48
- Getting comparisons has been quite expensive in time and resources
49
+
- The jury (secret) pool diversity is not guaranteed
49
50
- Asking jurors "how much better" introduces order‑dependence and scale mismatch
50
51
- Messy jurors have [disproportionate impact on the weights](https://davidgasquez.github.io/deepfunding-trial-data-analysis/#-robustness-checks)
51
52
- Weights are not consistent due to the limited amount of data collected and the variance on it
···
104
105
- Recommendation systems
105
106
- Sports (elo)
106
107
- RLHF
107
-
- Pairwise make thins a decision (yes / no, this or that). No one knows what 3.4x better means
108
+
- Pairwise make choices a simple decision (yes / no, this or that). No one knows what 3.4x better means
108
109
- Occam's razor works here too: simple things generalize better
109
110
- Intensity makes the distribution curve arbitrary
110
111
- We should test the assumption experts jurors give good results. Jurors are messy and not well calibrated. Collecting more information from "expert" jurors will probably add more noise. We should instead assume noisy jurors and use techniques to deal with that.