Code and data for arewedecentralizedyet.online and related projects
Add more background information and improve formatting
+131
-112
+12
-2
www/css/index.css
+12
-2
www/css/index.css
···
21
21
text-align: center;
22
22
}
23
23
24
+
div.container > * {
25
+
vertical-align: top;
26
+
}
27
+
24
28
div.container {
25
29
background: #ffffff11;
26
30
padding: 15px;
27
-
margin: 10px;
31
+
margin-right: auto;
32
+
margin-left: auto;
33
+
width: fit-content;
28
34
margin-bottom: 25px;
29
35
border-radius: 10px;
30
36
}
···
52
58
display: inline-block;
53
59
text-align: center;
54
60
width: 300px;
61
+
}
62
+
63
+
div.spacer {
64
+
height: 60px;
55
65
}
56
66
57
67
div.gage {
···
98
108
display: block;
99
109
margin-left: auto;
100
110
margin-right: auto;
101
-
color: #909296;
111
+
/* color: #909296; */
102
112
}
103
113
104
114
th {
+1
-36
www/data.js
+1
-36
www/data.js
···
96
96
]
97
97
]
98
98
},
99
-
"hosting": {
100
-
"HHI": 1186,
101
-
"shannon": 4.1174,
102
-
"simpson": 0.8814,
103
-
"servers": 34019,
104
-
"biggest_abs": 13370289,
105
-
"biggest_pct": 30.32,
106
-
"rest_abs": 30727592,
107
-
"rest_pct": 69.68,
108
-
"b_vals": [
109
-
[
110
-
25,
111
-
1
112
-
],
113
-
[
114
-
50,
115
-
4
116
-
],
117
-
[
118
-
75,
119
-
37
120
-
],
121
-
[
122
-
90,
123
-
338
124
-
],
125
-
[
126
-
99,
127
-
5436
128
-
],
129
-
[
130
-
99.5,
131
-
8603
132
-
]
133
-
]
134
-
},
99
+
"hosting": {"HHI": 744, "shannon": 4.9738, "simpson": 0.9256, "servers": 20955, "biggest_abs": 282775, "biggest_pct": 24.98, "rest_abs": 849111, "rest_pct": 75.02, "b_vals": [[25, 2], [50, 15], [75, 109], [90, 535], [99, 10312], [99.5, 15296]]},
135
100
"dns": {
136
101
"HHI": 1545,
137
102
"shannon": 3.7577,
+118
-74
www/index.html
+118
-74
www/index.html
···
92
92
<div class="data">
93
93
Data: <a href="https://relay1.us-east.bsky.network/">Bluesky</a> (<span id="atUpdated"></span>)
94
94
</div>
95
-
</div>
95
+
</div>
96
96
97
+
<div class="insert-metrics-help"></div>
98
+
99
+
</div>
97
100
98
-
<details class="help tab metric-help" id="social-metrics-hhi">
99
-
<summary>What is HHI?</summary>
100
-
<div class="panel">
101
-
<p>
102
-
The <a href="https://en.wikipedia.org/wiki/Herfindahl%E2%80%93Hirschman_index">Herfindahl–Hirschman
103
-
Index</a> (<strong>HHI</strong>) is an indicator from economics used to measure competition between
104
-
firms in an industry.
105
-
Mathematically, HHI is the sum of the squares of market shares of all servers.
106
-
Values close to zero indicate perfectly competitive markets (eg. many servers, with users
107
-
spread evenly), while values close to 10000 indicate highly concentrated monopolies (eg.
108
-
most users on a single server). In economics, values below 100 are considered
109
-
"Highly Competitive", below 1500 is "Unconcentrated", and above 2500 is
110
-
considered "Highly Concentrated".
111
-
</p>
112
-
<p>
113
-
This method of measuring centralization for Internet systems is described in the paper
114
-
<a href="https://doi.org/10.1145/3718958.3750492">"Formalizing Dependence of Web Infrastructure"
115
-
by Habib et al.</a>, which appeared in the <a href="https://conferences.sigcomm.org/sigcomm/2025/">ACM
116
-
SIGCOMM conference in 2025</a>. The authors motivate this measure by describing it as a special case
117
-
of the "Earth Mover Distance": the amount of "work" it would take to change one statistical distribution
118
-
into another. Here, the empirical distribution measured for the system's current deployment is
119
-
compared to an idealized fully-decentralized distribution.
120
-
</p>
121
-
</div>
122
-
</details>
123
-
<details class="help tab metric-help" id="social-metrics-shannon">
124
-
<summary>What is the Shannon Index?</summary>
125
-
<div class="panel">
101
+
102
+
<div class="container">
103
+
<h2>Hosting Services</h2>
104
+
105
+
<details class="help" id="hosting-details">
106
+
<summary>What does this measure?</summary>
107
+
<div>
126
108
<p>
127
-
The <a href="https://en.wikipedia.org/wiki/Diversity_index#Shannon_index">Shannon Index</a>
128
-
is an entropy-based measure used in ecological studies to measure population diversity.
129
-
It is computed as Shannon entropy using the natural log: the negative sum over all servers of the "market
130
-
share" times the log of the market share. Lower values indicate lower entropy (a high concentration of one species),
131
-
while higher values indicate a more even population. In this context,
132
-
the maximum value is the natural log of the number of servers.
109
+
For web serving, this site uses data from a number of sources that have been aggregated by the <a href="https://iyp.iijlab.net/">Internet Yellow Pages</a>
110
+
project from <a href="https://www.iijlab.net/">the IIJ Research Laboratory</a>. It uses the worldwide <a href="https://tranco-list.eu/">Tranco Top 1m</a> list of the million most popular websites, compiled by the DistriNet research group. The URLs of these websites'
111
+
front pages are mapped to IP addresses, which are then mapped to <a href="https://www.ripe.net/manage-ips-and-asns/as-numbers/">AS numbers.</a>
112
+
We use the <a href="https://blog.apnic.net/2023/03/21/improving-the-inference-of-sibling-autonomous-systems/">sibling AS</a> dataset from the <a href="https://inetintel.notion.site/Internet-Intelligence-Lab-Georgia-Tech-1a739459e28a4e8084556a5ef518c009">Internet Intelligence Research Lab</a> to aggregate ASes run by the same organization.
113
+
The process is similar to that used by <a href="https://doi.org/10.1145/3718958.3750492">Habib et al. in SIGCOMM 2025</a>, though we use a different data
114
+
source and compute statistics worldwide.
115
+
Note that because this style of measurement looks at the main URL for the site's front page, it will, in many cases,
116
+
count the CDN serving a site rather than the site's origin server.
133
117
</p>
134
118
<p>
135
-
Because the Shannon Index is logarithmic, it is more responsive than HHI to changes in the "smaller players":
136
-
participants that are already large do not change the index much when their size changes. Changes in the smaller
137
-
participants, on the other hand, are more noticeable in the computed index. Thus, it is a good way of looking
138
-
at systems that are undergoing change at the smaller end of the hosting scale.
119
+
For the public git forges, this site uses the number of "origins" of type "git" archived
120
+
by <a href="https://softwareheritage.org">Software Heritage</a>; this is
121
+
roughly equivalent to the number of git repositories they crawl. Software
122
+
Heritage's archive coverage can be found on <a href="https://archive.softwareheritage.org/">their coverage page</a>.
123
+
The likely undercounts the true number of repositories, and each "fork" counts as a separate repository.
139
124
</p>
140
125
</div>
141
-
</details>
142
-
</div>
143
-
144
-
145
-
<div class="container">
146
-
<h2>Hosting Services</h2>
126
+
</details>
147
127
148
128
<div class="rating">
149
-
<h3>Web Hosting</h3>
129
+
<h3>Web Serving</h3>
150
130
<h5>Cloudflare, AWS, etc.</h5>
151
131
<div id="gageHosting" class="gage"></div>
152
132
<div class="shannon">
153
133
<div class="index">Shannon Index: <span id="hostingShannon"></span></div>
154
134
</div>
135
+
<div class="spacer"></div>
155
136
<table>
156
137
<tr><th>Hosts</th><td><span id="hostingServers"></span></td></tr>
157
138
<tr><th>Biggest</th><td><span id="hostingBiggestAbs"></span></td><td>(<span id="hostingBiggestPct"></span>%)</td></tr>
158
139
<tr><th>Rest</th><td><span id="hostingRestAbs"></span></td><td>(<span id="hostingRestPct"></span>%)</td></tr>
159
140
</table>
160
-
Data: <a href="https://zenodo.org/records/15733582">Habib et al.</a>
141
+
<div class="data">
142
+
Data: <a href="https://iyp.iijlab.net/">IYP</a> (<span id="hostingUpdated"></span>)
143
+
</div>
161
144
</div>
162
145
163
146
<div class="rating">
···
174
157
<tr><th>Biggest</th><td><span id="gitBiggestAbs"></span></td><td>(<span id="gitBiggestPct"></span>%)</td></tr>
175
158
<tr><th>Rest</th><td><span id="gitRestAbs"></span></td><td>(<span id="gitRestPct"></span>%)</td></tr>
176
159
</table>
177
-
Data: <a href="https://archive.softwareheritage.org/">Software Heritage</a>
160
+
<div class="data">
161
+
Data: <a href="https://archive.softwareheritage.org/">Software Heritage</a> (<span id="gitUpdated"></span>)
162
+
</div>
178
163
</div>
164
+
165
+
<div class="insert-metrics-help"></div>
179
166
</div>
180
167
181
168
<div class="container">
182
169
<h2>Network Infrastructure</h2>
170
+
<details class="help" id="network-details">
171
+
<summary>What does this measure?</summary>
172
+
<div>
173
+
<p>
174
+
The measurements in this section come from the paper <a href="https://doi.org/10.1145/3718958.3750492">"Formalizing Dependence of Web Infrastructure"
175
+
by Habib et al.</a>, which appeared in the <a href="https://conferences.sigcomm.org/sigcomm/2025/">ACM
176
+
SIGCOMM conference in 2025</a>. The authors of this work collected information for the most popular
177
+
10,000 websites in 150 countries using <a href="https://developer.chrome.com/docs/crux">Google's CrUX</a> dataset. These measurements were taken once in May 2023.
178
+
</p>
179
+
180
+
<p>
181
+
DNS server data is reported by the network (AS) hosting the authoritative nameservers for the domains in the set.
182
+
Certificate authority (CA) data was collected by connecting to the front page of the webserver and finding the root CA of the certificate provided by the server.
183
+
</p>
184
+
</div>
185
+
</details>
183
186
<div class="rating">
184
187
<h3>DNS Hosting</h3>
185
188
<h5>Cloudflare DNS, Amazon Route 53, etc.</h5>
···
192
195
<tr><th>Biggest</th><td><span id="dnsBiggestAbs"></span></td><td>(<span id="dnsBiggestPct"></span>%)</td></tr>
193
196
<tr><th>Rest</th><td><span id="dnsRestAbs"></span></td><td>(<span id="dnsRestPct"></span>%)</td></tr>
194
197
</table>
195
-
Data: <a href="https://zenodo.org/records/15733582">Habib et al.</a>
198
+
<div class="data">
199
+
Data: <a href="https://zenodo.org/records/15733582">Habib et al.</a> (05-2023)
200
+
</div>
196
201
</div>
197
202
<div class="rating">
198
203
<h3>TLS Certificates</h3>
···
206
211
<tr><th>Biggest</th><td><span id="certBiggestAbs"></span></td><td>(<span id="certBiggestPct"></span>%)</td></tr>
207
212
<tr><th>Rest</th><td><span id="certRestAbs"></span></td><td>(<span id="certRestPct"></span>%)</td></tr>
208
213
</table>
209
-
Data: <a href="https://zenodo.org/records/15733582">Habib et al.</a>
214
+
<div class="data">
215
+
Data: <a href="https://zenodo.org/records/15733582">Habib et al.</a> (05-2023)
216
+
</div>
210
217
</div>
218
+
219
+
<div class="insert-metrics-help"></div>
220
+
211
221
</div>
212
222
213
223
<div class="container">
214
224
<p>
215
-
This page measures the concentration of user data on the <a
216
-
href="https://fediverse.party/">Fediverse</a>, the <a
217
-
href="https://atproto.com/">Atmosphere</a>, and public git forges according to the
225
+
This page measures the concentration of user data on a variety of systems according to the
218
226
<a href="https://en.wikipedia.org/wiki/Herfindahl%E2%80%93Hirschman_index">Herfindahl–Hirschman
219
227
Index</a> (HHI) and the <a href="https://en.wikipedia.org/wiki/Diversity_index#Shannon_index">Shannon Index</a>.
220
228
User data is only one way to measure centralization: others include network structure, legal exposure, and
···
222
230
</p>
223
231
224
232
<p>
225
-
</p>
226
-
227
-
<p>
228
-
The <b>Shannon Index</b> is an entropy-based measure used in ecological studies. Because it is logarithmic,
229
-
it is more responsive than HHI to changes in the "smaller players".
230
-
It is computed as Shannon entropy using the natural log: the negative sum over all servers of the "market
231
-
share" times the log of the market share. Lower values indicate lower entropy (a high concentration of one species),
232
-
while higher values indicate a more even population. In this context,
233
-
the maximum value is the natural log of the number of servers.
234
-
</p>
235
-
236
-
<p>
237
-
For the public git forges, this site uses the number of "origins" of type "git" archived
238
-
by <a href="https://softwareheritage.org">Software Heritage</a>; this is
239
-
roughly equivalent to the number of git repositories they crawl. Software
240
-
Heritage's archive coverage can be found on <a href="https://archive.softwareheritage.org/">their coverage page</a>.
241
-
242
-
</p>
243
-
244
-
<p>
245
233
Code and data are available <a href="https://codeberg.org/ricci/are-we-decentralized-yet">on
246
234
Codeberg</a>.
247
235
Comments and pull requests, including other metrics for measuring
···
249
237
</p>
250
238
251
239
<p>
252
-
By Rob Ricci: <a href="https://discuss.systems/@ricci">@ricci@discuss.systems</a> /
240
+
By <a href="https://ricci.io/">Rob Ricci</a>: <a href="https://discuss.systems/@ricci">@ricci@discuss.systems</a> /
253
241
<a href="https://bsky.app/profile/ricci.io">@ricci.io</a>
254
242
</p>
255
243
···
257
245
</div>
258
246
259
247
248
+
<template id="about-metrics">
249
+
<details>
250
+
<summary>What is HHI?</summary>
251
+
<div class="panel">
252
+
<p>
253
+
The <a href="https://en.wikipedia.org/wiki/Herfindahl%E2%80%93Hirschman_index">Herfindahl–Hirschman
254
+
Index</a> (<strong>HHI</strong>) is an indicator from economics used to measure competition between
255
+
firms in an industry.
256
+
Mathematically, HHI is the sum of the squares of market shares of all servers.
257
+
Values close to zero indicate perfectly competitive markets (eg. many servers, with users
258
+
spread evenly), while values close to 10000 indicate highly concentrated monopolies (eg.
259
+
most users on a single server). In economics, values below 100 are considered
260
+
"Highly Competitive", below 1500 is "Unconcentrated", and above 2500 is
261
+
considered "Highly Concentrated".
262
+
</p>
263
+
<p>
264
+
This method of measuring centralization for Internet systems is described in the paper
265
+
<a href="https://doi.org/10.1145/3718958.3750492">"Formalizing Dependence of Web Infrastructure"
266
+
by Habib et al.</a>, which appeared in the <a href="https://conferences.sigcomm.org/sigcomm/2025/">ACM
267
+
SIGCOMM conference in 2025</a>. The authors motivate this measure by describing it as a special case
268
+
of the "Earth Mover Distance": the amount of "work" it would take to change one statistical distribution
269
+
into another. Here, the empirical distribution measured for the system's current deployment is
270
+
compared to an idealized fully-decentralized distribution.
271
+
</p>
272
+
</div>
273
+
</details>
274
+
275
+
<details>
276
+
<summary>What is the Shannon Index?</summary>
277
+
<div class="panel">
278
+
<p>
279
+
The <a href="https://en.wikipedia.org/wiki/Diversity_index#Shannon_index">Shannon Index</a>
280
+
is an entropy-based measure used in ecological studies to measure population diversity.
281
+
It is computed as Shannon entropy using the natural log: the negative sum over all servers of the "market
282
+
share" times the log of the market share. Lower values indicate lower entropy (a high concentration of one species),
283
+
while higher values indicate a more even population. In this context,
284
+
the maximum value is the natural log of the number of servers.
285
+
</p>
286
+
<p>
287
+
Because the Shannon Index is logarithmic, it is more responsive than HHI to changes in the "smaller players":
288
+
participants that are already large do not change the index much when their size changes. Changes in the smaller
289
+
participants, on the other hand, are more noticeable in the computed index. Thus, it is a good way of looking
290
+
at systems that are undergoing change at the smaller end of the hosting scale.
291
+
</p>
292
+
</div>
293
+
</details>
294
+
</template>
260
295
<script>
296
+
document.querySelectorAll('.insert-metrics-help').forEach(el => {
297
+
const tmpl = document.getElementById('about-metrics');
298
+
el.appendChild(tmpl.content.cloneNode(true));
299
+
});
300
+
261
301
var defaults = {
262
302
min: 0,
263
303
max: 10000,
···
347
387
348
388
document.getElementById("fediUpdated").textContent = data.lastUpdate;
349
389
document.getElementById("atUpdated").textContent = data.lastUpdate;
390
+
document.getElementById("gitUpdated").textContent = data.lastUpdate;
391
+
392
+
// Should come from file
393
+
document.getElementById("hostingUpdated").textContent = "10-31-2025";
350
394
351
395
document.getElementById("fediServers").textContent = addCommas(data.fedi.servers);
352
396
document.getElementById("atServers").textContent = addCommas(data.at.servers);