@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Rate limit requests by IP
Summary:
Fixes T3923. On `secure.phabricator.com`, we occasionally get slowed to a crawl when someone runs a security scanner against us, or 5 search bots decide to simultaneously index every line of every file in Diffusion.
Every time a user makes a request, give their IP address some points. If they get too many points in 5 minutes, start blocking their requests automatically for a while.
We give fewer points for logged in requests. We could futher refine this (more points for a 404, more points for a really slow page, etc.) but let's start simply.
Also, provide a mechanism for configuring this, and configuring the LB environment stuff at the same time (this comes up rarely, but we don't have a good answer right now).
Test Plan: Used `ab` and reloading over and over again to hit rate limits. Read documentation.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: chad, epriestley
Maniphest Tasks: T3923
Differential Revision: https://secure.phabricator.com/D8713
+398
-1
+3
.gitignore
+3
.gitignore
+3
src/docs/user/configuration/configuration_guide.diviner
+3
src/docs/user/configuration/configuration_guide.diviner
···
197
197
@{article:Configuring Accounts and Registration}; or
198
198
- understanding advanced configuration topics with
199
199
@{article:Configuration User Guide: Advanced Configuration}; or
200
+
- configuring a preamble script to set up the environment properly behind a
201
+
load balancer, or adjust rate limiting with
202
+
@{article:Configuring a Preamble Script}; or
200
203
- configuring where uploaded files and attachments will be stored with
201
204
@{article:Configuring File Storage}; or
202
205
- configuring Phabricator so it can send mail with
+114
src/docs/user/configuration/configuring_preamble.diviner
+114
src/docs/user/configuration/configuring_preamble.diviner
···
1
+
@title Configuring a Preamble Script
2
+
@group config
3
+
4
+
Adjust environmental settings (SSL, remote IP, rate limiting) using a preamble
5
+
script.
6
+
7
+
= Overview =
8
+
9
+
If Phabricator is deployed in an environment where HTTP headers behave oddly
10
+
(usually, because it is behind a load balancer), it may not be able to detect
11
+
some environmental features (like the client's IP, or the presence of SSL)
12
+
correctly.
13
+
14
+
You can use a special preamble script to make arbitrary adjustments to the
15
+
environment and some parts of Phabricator's configuration in order to fix these
16
+
problems and set up the environment which Phabricator expects.
17
+
18
+
NOTE: This is an advanced feature. Most installs should not need to configure
19
+
a preamble script.
20
+
21
+
= Creating a Preamble Script =
22
+
23
+
To create a preamble script, write a file to:
24
+
25
+
phabricator/support/preamble.php
26
+
27
+
(This file is in Phabricator's `.gitignore`, so you do not need to worry about
28
+
colliding with `git` or interacting with updates.)
29
+
30
+
This file should be a valid PHP script. If you aren't very familiar with PHP,
31
+
you can check for syntax errors with `php -l`:
32
+
33
+
phabricator/ $ php -l support/preamble.php
34
+
No syntax errors detected in support/preamble.php
35
+
36
+
If present, this script will be executed at the very beginning of each web
37
+
request, allowing you to adjust the environment. For common adjustments and
38
+
examples, see the next sections.
39
+
40
+
= Adjusting Client IPs =
41
+
42
+
If your install is behind a load balancer, Phabricator may incorrectly detect
43
+
all requests as originating from the load balancer, rather than from the correct
44
+
client IPs. If this is the case and some other header (like `X-Forwarded-For`)
45
+
is known to be trustworthy, you can overwrite the `REMOTE_ADDR` setting so
46
+
Phabricator can figure out the client IP correctly:
47
+
48
+
```
49
+
name=Overwrite REMOTE_ADDR with X-Forwarded-For
50
+
<?php
51
+
52
+
$_SERVER['REMOTE_ADDR'] = $_SERVER['HTTP_X_FORWARDED_FOR'];
53
+
```
54
+
55
+
You should do this //only// if the `X-Forwarded-For` header is always
56
+
trustworthy. In particular, if users can make requests to the web server
57
+
directly, they can provide an arbitrary `X-Forwarded-For` header, and thereby
58
+
spoof an arbitrary client IP.
59
+
60
+
= Adjusting SSL =
61
+
62
+
If your install is behind an SSL terminating load balancer, Phabricator may
63
+
detect requests as HTTP when the client sees them as HTTPS. This can cause
64
+
Phabricator to generate links with the wrong protocol, issue cookies without
65
+
the SSL-only flag, or reject requests outright.
66
+
67
+
To fix this, you can set `$_SERVER['HTTPS']` explicitly:
68
+
69
+
```
70
+
name=Explicitly Configure SSL Availability
71
+
<?php
72
+
73
+
$_SERVER['HTTPS'] = true;
74
+
```
75
+
76
+
You can also set this value to `false` to explicitly tell Phabricator that a
77
+
request is not an SSL request.
78
+
79
+
= Adjusting Rate Limiting =
80
+
81
+
Phabricator performs coarse, IP-based rate limiting by default. In most
82
+
situations the default settings should be reasonable: they are set fairly high,
83
+
and intended to prevent only significantly abusive behavior.
84
+
85
+
However, if legitimate traffic is being rate limited (or you want to make the
86
+
limits more strict) you can adjust the limits in the preamble script.
87
+
88
+
```
89
+
name=Adjust Rate Limiting Behavior
90
+
<?php
91
+
92
+
// The default is 1000, so a value of 2000 increases the limit by a factor
93
+
// of 2: users will be able to make twice as many requests before being
94
+
// rate limited.
95
+
96
+
// You can set the limit to 0 to disable rate limiting.
97
+
98
+
PhabricatorStartup::setMaximumRate(2000);
99
+
```
100
+
101
+
By examining `$_SERVER['REMOTE_ADDR']` or similar parameters, you could also
102
+
adjust the rate limit dynamically: for example, remove it for requests from an
103
+
internal network, but impose a strict limit for external requests.
104
+
105
+
Rate limiting needs to be configured in this way in order to make it as cheap as
106
+
possible to activate after a client is rate limited. The limiting checks execute
107
+
before any libraries or configuration are loaded, and can emit a response within
108
+
a few milliseconds.
109
+
110
+
= Next Steps =
111
+
112
+
Continue by:
113
+
114
+
- returning to the @{article:Configuration Guide}.
+252
support/PhabricatorStartup.php
+252
support/PhabricatorStartup.php
···
8
8
* NOTE: This class MUST NOT have any dependencies. It runs before libraries
9
9
* load.
10
10
*
11
+
* Rate Limiting
12
+
* =============
13
+
*
14
+
* Phabricator limits the rate at which clients can request pages, and issues
15
+
* HTTP 429 "Too Many Requests" responses if clients request too many pages too
16
+
* quickly. Although this is not a complete defense against high-volume attacks,
17
+
* it can protect an install against aggressive crawlers, security scanners,
18
+
* and some types of malicious activity.
19
+
*
20
+
* To perform rate limiting, each page increments a score counter for the
21
+
* requesting user's IP. The page can give the IP more points for an expensive
22
+
* request, or fewer for an authetnicated request.
23
+
*
24
+
* Score counters are kept in buckets, and writes move to a new bucket every
25
+
* minute. After a few minutes (defined by @{method:getRateLimitBucketCount}),
26
+
* the oldest bucket is discarded. This provides a simple mechanism for keeping
27
+
* track of scores without needing to store, access, or read very much data.
28
+
*
29
+
* Users are allowed to accumulate up to 1000 points per minute, averaged across
30
+
* all of the tracked buckets.
31
+
*
11
32
* @task info Accessing Request Information
12
33
* @task hook Startup Hooks
13
34
* @task apocalypse In Case Of Apocalypse
14
35
* @task validation Validation
36
+
* @task ratelimit Rate Limiting
15
37
*/
16
38
final class PhabricatorStartup {
17
39
···
19
41
private static $globals = array();
20
42
private static $capturingOutput;
21
43
private static $rawInput;
44
+
private static $maximumRate = 1000;
22
45
23
46
24
47
/* -( Accessing Request Information )-------------------------------------- */
···
92
115
93
116
self::setupPHP();
94
117
self::verifyPHP();
118
+
119
+
if (isset($_SERVER['REMOTE_ADDR'])) {
120
+
self::rateLimitRequest($_SERVER['REMOTE_ADDR']);
121
+
}
95
122
96
123
self::normalizeInput();
97
124
···
519
546
"setting or reduce the size of the request.\n\n".
520
547
"Request size according to 'Content-Length' was '{$length}', ".
521
548
"'post_max_size' is set to '{$config}'.");
549
+
}
550
+
551
+
552
+
/* -( Rate Limiting )------------------------------------------------------ */
553
+
554
+
555
+
/**
556
+
* Adjust the permissible rate limit score.
557
+
*
558
+
* By default, the limit is `1000`. You can use this method to set it to
559
+
* a larger or smaller value. If you set it to `2000`, users may make twice
560
+
* as many requests before rate limiting.
561
+
*
562
+
* @param int Maximum score before rate limiting.
563
+
* @return void
564
+
* @task ratelimit
565
+
*/
566
+
public static function setMaximumRate($rate) {
567
+
self::$maximumRate = $rate;
568
+
}
569
+
570
+
571
+
/**
572
+
* Check if the user (identified by `$user_identity`) has issued too many
573
+
* requests recently. If they have, end the request with a 429 error code.
574
+
*
575
+
* The key just needs to identify the user. Phabricator uses both user PHIDs
576
+
* and user IPs as keys, tracking logged-in and logged-out users separately
577
+
* and enforcing different limits.
578
+
*
579
+
* @param string Some key which identifies the user making the request.
580
+
* @return void If the user has exceeded the rate limit, this method
581
+
* does not return.
582
+
* @task ratelimit
583
+
*/
584
+
public static function rateLimitRequest($user_identity) {
585
+
if (!self::canRateLimit()) {
586
+
return;
587
+
}
588
+
589
+
$score = self::getRateLimitScore($user_identity);
590
+
if ($score > (self::$maximumRate * self::getRateLimitBucketCount())) {
591
+
// Give the user some bonus points for getting rate limited. This keeps
592
+
// bad actors who keep slamming the 429 page locked out completely,
593
+
// instead of letting them get a burst of requests through every minute
594
+
// after a bucket expires.
595
+
self::addRateLimitScore($user_identity, 50);
596
+
self::didRateLimit($user_identity);
597
+
}
598
+
}
599
+
600
+
601
+
/**
602
+
* Add points to the rate limit score for some user.
603
+
*
604
+
* If users have earned more than 1000 points per minute across all the
605
+
* buckets they'll be locked out of the application, so awarding 1 point per
606
+
* request roughly corresponds to allowing 1000 requests per second, while
607
+
* awarding 50 points roughly corresponds to allowing 20 requests per second.
608
+
*
609
+
* @param string Some key which identifies the user making the request.
610
+
* @param float The cost for this request; more points pushes them toward
611
+
* the limit faster.
612
+
* @return void
613
+
* @task ratelimit
614
+
*/
615
+
public static function addRateLimitScore($user_identity, $score) {
616
+
if (!self::canRateLimit()) {
617
+
return;
618
+
}
619
+
620
+
$current = self::getRateLimitBucket();
621
+
622
+
// There's a bit of a race here, if a second process reads the bucket before
623
+
// this one writes it, but it's fine if we occasionally fail to record a
624
+
// user's score. If they're making requests fast enough to hit rate
625
+
// limiting, we'll get them soon.
626
+
627
+
$bucket_key = self::getRateLimitBucketKey($current);
628
+
$bucket = apc_fetch($bucket_key);
629
+
if (!is_array($bucket)) {
630
+
$bucket = array();
631
+
}
632
+
633
+
if (empty($bucket[$user_identity])) {
634
+
$bucket[$user_identity] = 0;
635
+
}
636
+
637
+
$bucket[$user_identity] += $score;
638
+
apc_store($bucket_key, $bucket);
639
+
}
640
+
641
+
642
+
/**
643
+
* Determine if rate limiting is available.
644
+
*
645
+
* Rate limiting depends on APC, and isn't available unless the APC user
646
+
* cache is available.
647
+
*
648
+
* @return bool True if rate limiting is available.
649
+
* @task ratelimit
650
+
*/
651
+
private static function canRateLimit() {
652
+
if (!self::$maximumRate) {
653
+
return false;
654
+
}
655
+
656
+
if (!function_exists('apc_fetch')) {
657
+
return false;
658
+
}
659
+
660
+
return true;
661
+
}
662
+
663
+
664
+
/**
665
+
* Get the current bucket for storing rate limit scores.
666
+
*
667
+
* @return int The current bucket.
668
+
* @task ratelimit
669
+
*/
670
+
private static function getRateLimitBucket() {
671
+
return (int)(time() / 60);
672
+
}
673
+
674
+
675
+
/**
676
+
* Get the total number of rate limit buckets to retain.
677
+
*
678
+
* @return int Total number of rate limit buckets to retain.
679
+
* @task ratelimit
680
+
*/
681
+
private static function getRateLimitBucketCount() {
682
+
return 5;
683
+
}
684
+
685
+
686
+
/**
687
+
* Get the APC key for a given bucket.
688
+
*
689
+
* @param int Bucket to get the key for.
690
+
* @return string APC key for the bucket.
691
+
* @task ratelimit
692
+
*/
693
+
private static function getRateLimitBucketKey($bucket) {
694
+
return 'rate:bucket:'.$bucket;
695
+
}
696
+
697
+
698
+
/**
699
+
* Get the APC key for the smallest stored bucket.
700
+
*
701
+
* @return string APC key for the smallest stored bucket.
702
+
* @task ratelimit
703
+
*/
704
+
private static function getRateLimitMinKey() {
705
+
return 'rate:min';
706
+
}
707
+
708
+
709
+
/**
710
+
* Get the current rate limit score for a given user.
711
+
*
712
+
* @param string Unique key identifying the user.
713
+
* @return float The user's current score.
714
+
* @task ratelimit
715
+
*/
716
+
private static function getRateLimitScore($user_identity) {
717
+
$min_key = self::getRateLimitMinKey();
718
+
719
+
// Identify the oldest bucket stored in APC.
720
+
$cur = self::getRateLimitBucket();
721
+
$min = apc_fetch($min_key);
722
+
723
+
// If we don't have any buckets stored yet, store the current bucket as
724
+
// the oldest bucket.
725
+
if (!$min) {
726
+
apc_store($min_key, $cur);
727
+
$min = $cur;
728
+
}
729
+
730
+
// Destroy any buckets that are older than the minimum bucket we're keeping
731
+
// track of. Under load this normally shouldn't do anything, but will clean
732
+
// up an old bucket once per minute.
733
+
$count = self::getRateLimitBucketCount();
734
+
for ($cursor = $min; $cursor < ($cur - $count); $cursor++) {
735
+
apc_delete(self::getRateLimitBucketKey($cursor));
736
+
apc_store($min_key, $cursor + 1);
737
+
}
738
+
739
+
// Now, sum up the user's scores in all of the active buckets.
740
+
$score = 0;
741
+
for (; $cursor <= $cur; $cursor++) {
742
+
$bucket = apc_fetch(self::getRateLimitBucketKey($cursor));
743
+
if (isset($bucket[$user_identity])) {
744
+
$score += $bucket[$user_identity];
745
+
}
746
+
}
747
+
748
+
return $score;
749
+
}
750
+
751
+
752
+
/**
753
+
* Emit an HTTP 429 "Too Many Requests" response (indicating that the user
754
+
* has exceeded application rate limits) and exit.
755
+
*
756
+
* @return exit This method **does not return**.
757
+
* @task ratelimit
758
+
*/
759
+
private static function didRateLimit() {
760
+
$message =
761
+
"TOO MANY REQUESTS\n".
762
+
"You are issuing too many requests too quickly.\n".
763
+
"To adjust limits, see \"Configuring a Preamble Script\" in the ".
764
+
"documentation.";
765
+
766
+
header(
767
+
'Content-Type: text/plain; charset=utf-8',
768
+
$replace = true,
769
+
$http_error = 429);
770
+
771
+
echo $message;
772
+
773
+
exit(1);
522
774
}
523
775
524
776
}
+26
-1
webroot/index.php
+26
-1
webroot/index.php
···
1
1
<?php
2
2
3
-
require_once dirname(dirname(__FILE__)).'/support/PhabricatorStartup.php';
3
+
$phabricator_root = dirname(dirname(__FILE__));
4
+
require_once $phabricator_root.'/support/PhabricatorStartup.php';
5
+
6
+
// If the preamble script exists, load it.
7
+
$preamble_path = $phabricator_root.'/support/preamble.php';
8
+
if (file_exists($preamble_path)) {
9
+
require_once $preamble_path;
10
+
}
11
+
4
12
PhabricatorStartup::didStartup();
5
13
6
14
$show_unexpected_traces = false;
···
142
150
));
143
151
144
152
DarkConsoleXHProfPluginAPI::saveProfilerSample($access_log);
153
+
154
+
// Add points to the rate limits for this request.
155
+
if (isset($_SERVER['REMOTE_ADDR'])) {
156
+
$user_ip = $_SERVER['REMOTE_ADDR'];
157
+
158
+
// The base score for a request allows users to make 30 requests per
159
+
// minute.
160
+
$score = (1000 / 30);
161
+
162
+
// If the user was logged in, let them make more requests.
163
+
if ($request->getUser() && $request->getUser()->getPHID()) {
164
+
$score = $score / 5;
165
+
}
166
+
167
+
PhabricatorStartup::addRateLimitScore($user_ip, $score);
168
+
}
169
+
145
170
} catch (Exception $ex) {
146
171
PhabricatorStartup::didEncounterFatalException(
147
172
'Core Exception',