Support (but do not actually enable) a maximum file size limit for Git repositories

Summary:
Depends on D19816. Ref T13216. See PHI908. See PHI750. In a few cases, users have pushed multi-gigabyte files full of various things that probably shouldn't be version controlled. This tends to create various headaches.

Add support for limiting the maximum size of any object. Specifically, we:

- list all the objects each commit touches;
- check their size after the commit applies;
- if it's over the limit, reject the commit.

This change doesn't actually hook the limit up (the limit is always "0", i.e. unlimited), and doesn't have Mercurial or SVN support. The actual parser bit would probably be better in some other `Query/Parser` class eventually, too. But it at least roughly works.

Test Plan:
Changed the hard-coded limit to other values, tried to push stuff, got sensible results:

```
$ echo pew >> magic_missile.txt && git commit -am pew && git push
[master 98d07af] pew
1 file changed, 1 insertion(+)
# Push received by "local.phacility.net", forwarding to cluster host.
# Acquiring write lock for repository "spellbook"...
# Acquired write lock immediately.
# Acquiring read lock for repository "spellbook" on device "local.phacility.net"...
# Acquired read lock immediately.
# Device "local.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "local.phacility.net".
Counting objects: 49, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (48/48), done.
Writing objects: 100% (49/49), 3.44 KiB | 1.72 MiB/s, done.
Total 49 (delta 30), reused 0 (delta 0)
remote: +---------------------------------------------------------------+
remote: | * * * PUSH REJECTED BY EVIL DRAGON BUREAUCRATS * * * |
remote: +---------------------------------------------------------------+
remote: \
remote: \ ^ /^
remote: \ / \ // \
remote: \ |\___/| / \// .\
remote: \ /V V \__ / // | \ \ *----*
remote: / / \/_/ // | \ \ \ |
remote: @___@` \/_ // | \ \ \/\ \
remote: 0/0/| \/_ // | \ \ \ \
remote: 0/0/0/0/| \/// | \ \ | |
remote: 0/0/0/0/0/_|_ / ( // | \ _\ | /
remote: 0/0/0/0/0/0/`/,_ _ _/ ) ; -. | _ _\.-~ / /
remote: ,-} _ *-.|.-~-. .~ ~
remote: * \__/ `/\ / ~-. _ .-~ /
remote: \____(Oo) *. } { /
remote: ( (..) .----~-.\ \-` .~
remote: //___\\ \ DENIED! ///.----..< \ _ -~
remote: // \\ ///-._ _ _ _ _ _ _{^ - - - - ~
remote:
remote:
remote: OVERSIZED FILE
remote: This repository ("spellbook") is configured with a maximum individual file size limit, but you are pushing a change ("98d07af863e799509e7c3a639404d216f9fc79c7") which causes the size of a file ("magic_missile.txt") to exceed the limit. The commit makes the file 317 bytes long, but the limit for this repository is 1 bytes.
remote:
# Released cluster write lock.
To ssh://local.phacility.com/source/spellbook.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://epriestley@local.phacility.com/source/spellbook.git'
```

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: joshuaspence

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19817

epriestley 7 years ago a0d4b6da ab14f49e

+147

2 changed files

expand all

src

applications

diffusion

engine

DiffusionCommitHookEngine.php

repository

storage

PhabricatorRepositoryPushLog.php

+143

src/applications/diffusion/engine/DiffusionCommitHookEngine.php

··· 166 166 167 167 try { 168 168 if (!$is_initial_import) { 169 + $this->rejectOversizedFiles($content_updates); 170 + } 171 + } catch (DiffusionCommitHookRejectException $ex) { 172 + // If we're rejecting oversized files, flag everything. 173 + $this->rejectCode = PhabricatorRepositoryPushLog::REJECT_OVERSIZED; 174 + throw $ex; 175 + } 176 + 177 + try { 178 + if (!$is_initial_import) { 169 179 $this->rejectEnormousChanges($content_updates); 170 180 } 171 181 } catch (DiffusionCommitHookRejectException $ex) { ··· 1253 1263 $info = $this->loadChangesetsForCommit($identifier); 1254 1264 list($changesets, $size) = $info; 1255 1265 return $changesets; 1266 + } 1267 + 1268 + private function rejectOversizedFiles(array $content_updates) { 1269 + $repository = $this->getRepository(); 1270 + 1271 + // TODO: Allow repositories to be configured for a maximum filesize. 1272 + $limit = 0; 1273 + 1274 + if (!$limit) { 1275 + return; 1276 + } 1277 + 1278 + foreach ($content_updates as $update) { 1279 + $identifier = $update->getRefNew(); 1280 + 1281 + $sizes = $this->loadFileSizesForCommit($identifier); 1282 + foreach ($sizes as $path => $size) { 1283 + if ($size <= $limit) { 1284 + continue; 1285 + } 1286 + 1287 + $message = pht( 1288 + 'OVERSIZED FILE'. 1289 + "\n". 1290 + 'This repository ("%s") is configured with a maximum individual '. 1291 + 'file size limit, but you are pushing a change ("%s") which causes '. 1292 + 'the size of a file ("%s") to exceed the limit. The commit makes '. 1293 + 'the file %s bytes long, but the limit for this repository is '. 1294 + '%s bytes.', 1295 + $repository->getDisplayName(), 1296 + $identifier, 1297 + $path, 1298 + new PhutilNumber($size), 1299 + new PhutilNumber($limit)); 1300 + 1301 + throw new DiffusionCommitHookRejectException($message); 1302 + } 1303 + } 1304 + } 1305 + 1306 + public function loadFileSizesForCommit($identifier) { 1307 + $repository = $this->getRepository(); 1308 + $vcs = $repository->getVersionControlSystem(); 1309 + 1310 + $path_sizes = array(); 1311 + 1312 + switch ($vcs) { 1313 + case PhabricatorRepositoryType::REPOSITORY_TYPE_GIT: 1314 + list($paths_raw) = $repository->execxLocalCommand( 1315 + 'diff-tree -z -r --no-commit-id %s --', 1316 + $identifier); 1317 + 1318 + // With "-z" we get "<fields>\0<filename>\0" for each line. Group the 1319 + // delimited text into "<fields>, <filename>" pairs. 1320 + $paths_raw = trim($paths_raw, "\0"); 1321 + $paths_raw = explode("\0", $paths_raw); 1322 + if (count($paths_raw) % 2) { 1323 + throw new Exception( 1324 + pht( 1325 + 'Unexpected number of output lines from "git diff-tree" when '. 1326 + 'processing commit ("%s"): got %s lines, expected an even '. 1327 + 'number.', 1328 + $identifier, 1329 + phutil_count($paths_raw))); 1330 + } 1331 + $paths_raw = array_chunk($paths_raw, 2); 1332 + 1333 + $paths = array(); 1334 + foreach ($paths_raw as $path_raw) { 1335 + list($fields, $pathname) = $path_raw; 1336 + $fields = explode(' ', $fields); 1337 + 1338 + // Fields are: 1339 + // 1340 + // :100644 100644 aaaa bbbb M 1341 + // 1342 + // [0] Old file mode. 1343 + // [1] New file mode. 1344 + // [2] Old object hash. 1345 + // [3] New object hash. 1346 + // [4] Change mode. 1347 + 1348 + $paths[] = array( 1349 + 'path' => $pathname, 1350 + 'newHash' => $fields[3], 1351 + ); 1352 + } 1353 + 1354 + if ($paths) { 1355 + $check_paths = array(); 1356 + foreach ($paths as $path) { 1357 + if ($path['newHash'] === self::EMPTY_HASH) { 1358 + $path_sizes[$path['path']] = 0; 1359 + continue; 1360 + } 1361 + $check_paths[$path['newHash']][] = $path['path']; 1362 + } 1363 + 1364 + if ($check_paths) { 1365 + $future = $repository->getLocalCommandFuture( 1366 + 'cat-file --batch-check=%s', 1367 + '%(objectsize)') 1368 + ->write(implode("\n", array_keys($check_paths))); 1369 + 1370 + list($sizes) = $future->resolvex(); 1371 + $sizes = trim($sizes); 1372 + $sizes = phutil_split_lines($sizes, false); 1373 + if (count($sizes) !== count($check_paths)) { 1374 + throw new Exception( 1375 + pht( 1376 + 'Unexpected number of output lines from "git cat-file" when '. 1377 + 'processing commit ("%s"): got %s lines, expected %s.', 1378 + $identifier, 1379 + phutil_count($sizes), 1380 + phutil_count($check_paths))); 1381 + } 1382 + 1383 + foreach ($check_paths as $object_hash => $path_names) { 1384 + $object_size = (int)array_shift($sizes); 1385 + foreach ($path_names as $path_name) { 1386 + $path_sizes[$path_name] = $object_size; 1387 + } 1388 + } 1389 + } 1390 + } 1391 + break; 1392 + default: 1393 + throw new Exception( 1394 + pht( 1395 + 'File size limits are not supported for this VCS.')); 1396 + } 1397 + 1398 + return $path_sizes; 1256 1399 } 1257 1400 1258 1401 public function loadCommitRefForCommit($identifier) {

src/applications/repository/storage/PhabricatorRepositoryPushLog.php

··· 24 24 const CHANGEFLAG_REWRITE = 8; 25 25 const CHANGEFLAG_DANGEROUS = 16; 26 26 const CHANGEFLAG_ENORMOUS = 32; 27 + const CHANGEFLAG_OVERSIZED = 64; 27 28 28 29 const REJECT_ACCEPT = 0; 29 30 const REJECT_DANGEROUS = 1; ··· 31 32 const REJECT_EXTERNAL = 3; 32 33 const REJECT_BROKEN = 4; 33 34 const REJECT_ENORMOUS = 5; 35 + const REJECT_OVERSIZED = 6; 34 36 35 37 protected $repositoryPHID; 36 38 protected $epoch; ··· 63 65 self::CHANGEFLAG_REWRITE => pht('Rewrite'), 64 66 self::CHANGEFLAG_DANGEROUS => pht('Dangerous'), 65 67 self::CHANGEFLAG_ENORMOUS => pht('Enormous'), 68 + self::CHANGEFLAG_OVERSIZED => pht('Oversized'), 66 69 ); 67 70 } 68 71 ··· 74 77 self::REJECT_EXTERNAL => pht('Rejected: External Hook'), 75 78 self::REJECT_BROKEN => pht('Rejected: Broken'), 76 79 self::REJECT_ENORMOUS => pht('Rejected: Enormous'), 80 + self::REJECT_OVERSIZED => pht('Rejected: Oversized File'), 77 81 ); 78 82 } 79 83

Configure Feed

Configure Feed