this repo has no description
📝 Clarify weight derivation, add Current Approach, and revamp Open Problems - Explain weights as latent utilities and expand sourcing options - Add step by step workflow for jurors, aggregation, and market evaluation - Reorganize open issues into juror reliability, mechanism settings, weight evaluation, and graph maintenance with references - Introduce ideas on robust pairwise aggregation and active ranking - Remove redundant Open Problems section and refine wording
+43
-34
+43
-34
Deep Funding.md
+43
-34
Deep Funding.md
···
5
5
In Deep Funding, multiple mechanisms work together:
6
6
7
7
1. A mechanism that generates an up-to-date and comprehensive DAG of relevant dependencies given a source node
8
-
2. A mechanism that fills the graph with relevant weights. For example:
9
-
1. Aggregating human preferences
10
-
2. Scaling Human Judgement to fill the rest of the graph. This can be something like a [prediction market](https://ethresear.ch/t/deep-funding-a-prediction-market-for-open-source-dependencies/23101), an AI model, ...
8
+
2. A mechanism that fills the graph with relevant weights. These weights represent the latent item utilities. There can be many ways of getting to them!
9
+
1. Aggregating human preferences (polls, pairwise comparisons, ...)
10
+
2. Using prediction markets
11
+
3. Getting weights from an AI model
12
+
4. Having experts fill weights manually
11
13
3. A mechanism that takes that weight vector as input and distributes money to the projects
12
14
13
-
This problem touches data, mechanism design, and open source! Each layer can be optimized and iterated independently.
15
+
This problem touches data, mechanism design, and open source! Also, each layer can be optimized and iterated independently.
14
16
15
17
In its current shape, the graph's vertices are projects and the edges are the relative impact of each project in its parent. The same approach could be used for anything that matches the graph shape (e.g: science research).
16
18
···
28
30
- Skin in the Game. Participants have something to lose from bad assessments
29
31
- Project Independence (no need to participate in the process to get funded)
30
32
33
+
## Current Approach
34
+
35
+
So far, Deep Funding has been implemented like this:
36
+
37
+
1. A list of projects is choosen. This is usually provided by an external entity or process (e.g: the [best model from the ML competition](https://cryptopond.xyz/modelfactory/detail/2564617) choose the next 100 projects). So far a DAG/graph structure has not been needed since all projects have been compared for their impact on the "Ethereum Ecosystem".
38
+
2. Jurors do pairwise comparisons between projects. An aggregation method is choosen (Huber loss, L2 norm in log space, ...) to derive the "ground truth" relative project weights.
39
+
3. An ML competition and [a Prediction Market](https://ethresear.ch/t/deep-funding-a-prediction-market-for-open-source-dependencies/23101) are kicked off. Modelers and traders are evaluated against a holdout set of pairwise comparisons.
40
+
4. Partitipants are rewarded based on how close they get to the "jurors' ground truth".
41
+
42
+
### Open Problems
43
+
44
+
After participating in the ML competition and Prediction Market, and doing a few deep dives into the data and methodology, I think these are the main open problems.
45
+
46
+
- **Juror Reliability**
47
+
- So far, expert juror's pairwise comparisons have been inconsistent, noisy, and low in statistical power
48
+
- Getting comparisons has been quite expensive in time and resources
49
+
- Asking jurors "how much better" introduces order‑dependence and scale mismatch
50
+
- Messy jurors have [disproportionate impact on the weights](https://davidgasquez.github.io/deepfunding-trial-data-analysis/#-robustness-checks)
51
+
- Weights are not consistent due to the limited amount of data collected and the variance on it
52
+
- Large graphs (hundreds of projects) make getting accurate weights from the pairwise evaluation infeasible
53
+
- **Mechanism Settings**
54
+
- Some parameters have a large effect and haven't been adjusted
55
+
- The aggregation formula (huber, log loss, bradley terry, ...) has a very large impact on both modelers/traders and project rewards
56
+
- Need more process around who chooses the aggregation formula and why is choosen
57
+
- In the pilot (huber loss), some projects got weights on a scale jurors didn't feel reasonable (e.g: EIPs repo got 30%)
58
+
- The prediction market might cause good modelers to not participate as time of entry is more important than having a good model
59
+
- **Weights Evaluation**
60
+
- [How do we measure success?](https://davidgasquez.com/weight-allocation-mechanism-evals/) If the goal of pattern recognition was to classify objects in a scene, it made sense to score an algorithm by how often it succeeded in doing so. What is the equivalent for Deep Funding?
61
+
- Once the weights are set, there isn't [a process to evaluate how "fit" those are](https://davidgasquez.com/weight-allocation-mechanism-evals/)
62
+
- E.g: the current idea is to gather a connected graph of pairwise comparisons, why not use that to reward projects directly and skip the Prediction Market?
63
+
- We need a falsifiable hypotheses to validate Deep Funding is "better"
64
+
- **Graph Maintenance**
65
+
- If the process takes a few weeks, the weights might change significally (e.g: a project releases a major version)
66
+
- Jurors are also affected by temporal drift and their preferences evolve over time
67
+
31
68
## Ideas
32
69
70
+
- There are beter methods to derive weights from [noisy pairwise comparisons](https://arxiv.org/abs/2510.09333) ([from multiple annotators](https://arxiv.org/abs/1612.04413))
71
+
- Use active ranking or dueling bandits to [speed up the data gathering process](https://projecteuclid.org/journals/annals-of-statistics/volume-47/issue-6/Active-ranking-from-pairwise-comparisons-and-when-parametric-assumptions-do/10.1214/18-AOS1772.pdf)
33
72
- Use juror data as the human feedback side of RLHF
34
73
- Mix this with a [social choice perspective](https://iclr.cc/virtual/2024/invited-talk/21799). Benchmarks are the aggregation of "votes" to choose the best models. Arrow's impossibility theorem works here though!
35
74
- Create models/mechanisms that fill the graph using whatever techniques they want
···
77
116
- Self declaration needs a "contest process" to resolve issues/abuse.
78
117
- Harberger Tax on self declarations? Bayesian Truth Serum for Weight Elicitation?
79
118
- Projects continuously auction off "maintenance contracts" where funders bid on keeping projects maintained. The auction mechanism reveals willingness-to-pay for continued operation. Dependencies naturally emerge as projects that lose maintenance see their dependents bid up their contracts
80
-
81
-
## Open Problems
82
-
83
-
- Unreliable juror evidence (low statistical power & consistency)
84
-
- Very few edges achieve statistical significance
85
-
- The noise makes any downstream model "garbage‑in, garbage‑out"
86
-
- How to incentivize good jurors?
87
-
- How to deal with messy jurors?
88
-
- Intensity scoring in pairwise comparisons
89
-
- Asking jurors "how much better" introduces order‑dependence and scale mismatch (e.g., 999× vs 100× for the same comparison)
90
-
- Are there beter methods to derive weights from [noisy pairwise comparisons](https://arxiv.org/abs/2510.09333) ([from multiple annotators](https://arxiv.org/abs/1612.04413))?
91
-
- Dependency graph accuracy, completeness, and adversarial behavior
92
-
- How to keep the graph up to date?
93
-
- Process transparency, openness, and reproducibility
94
-
- Who chooses the aggregation formula?
95
-
- Why not Bradley-Terry (RLHF style)?
96
-
- What is the best metric to compute distances?
97
-
- The process is sensitive to irrelevant changes (e.g: updating some parameter on the data pipeline changes the leaderboard considerably)
98
-
- Temporal drift (dependencies and preferences)
99
-
- Snapshots get out of date
100
-
- Jurors evaluate projects that have changed
101
-
- Scalability to large graphs
102
-
- With hundreds of nodes, assigning a reasonable score to a random project requires local link structure...
103
-
- How to [speed up the data gathering process](https://projecteuclid.org/journals/annals-of-statistics/volume-47/issue-6/Active-ranking-from-pairwise-comparisons-and-when-parametric-assumptions-do/10.1214/18-AOS1772.pdf)?
104
-
- Incentive design for participants
105
-
- Static, concentrated rewards don’t encourage edge‑case discovery or diversity of modeling approaches.
106
-
- Fitness of the Deep Funding approach
107
-
- How is the final distribution evaluated?
108
-
- Falsifiable hypothesis to verify the weights are correct
109
-
- If the goal of pattern recognition was to classify objects in a scene, it made sense to score an algorithm by how often it succeeded in doing so. The goal of Deep Funding is to allocate resources to public goods in a way that rivals how private goods are funded, how do we measure success?