Fix some issues with Diffusion file data limits

Summary:
See <https://discourse.phabricator-community.org/t/files-created-from-repository-contents-slightly-over-one-chunk-in-size-are-truncated-to-exactly-one-chunk-in-size/988/1>. Three issues here:

- When we finish reading `git cat-file ...` or whatever, we can end up with more than one chunk worth of bytes left in the internal buffer if the read is fast. Use `while` instead of `if` to make sure we write the whole buffer.
- Limiting output with `setStdoutSizeLimit()` isn't really a reliable way to limit the size if we're also reading from the buffer. It's also pretty indirect and confusing. Instead, just let the `FileUploadSource` explicitly implement a byte limit in a straightforward way.
- We weren't setting the time limit correctly on the main path.

Overall, this could cause >4MB files to "write" as 4MB files, with the rest of the file left in the UploadSource buffer. Since these files were technically under the limit, they could return as valid. This was intermittent.

Test Plan:
- Pushed a ~4.2MB file.
- Reloaded Diffusion a bunch, sometimes saw the `while/if` buffer race and produce a 4MB file with a prompt to download it. (Other times, the buffer worked right and the page just says "this file is too big, sorry").
- Applied patches.
- Reloaded Diffusion a bunch, no longer saw bad behavior or truncated files.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D18885

epriestley 8 years ago 6a627970 c6376804

+36 -20

4 changed files

expand all

src

__phutil_library_map__.php

applications

diffusion

query

DiffusionFileFutureQuery.php

files

uploadsource

PhabricatorFileUploadSource.php

PhabricatorFileUploadSourceByteLimitException.php

src/__phutil_library_map__.php

··· 2938 2938 'PhabricatorFileUploadDialogController' => 'applications/files/controller/PhabricatorFileUploadDialogController.php', 2939 2939 'PhabricatorFileUploadException' => 'applications/files/exception/PhabricatorFileUploadException.php', 2940 2940 'PhabricatorFileUploadSource' => 'applications/files/uploadsource/PhabricatorFileUploadSource.php', 2941 + 'PhabricatorFileUploadSourceByteLimitException' => 'applications/files/uploadsource/PhabricatorFileUploadSourceByteLimitException.php', 2941 2942 'PhabricatorFileinfoSetupCheck' => 'applications/config/check/PhabricatorFileinfoSetupCheck.php', 2942 2943 'PhabricatorFilesApplication' => 'applications/files/application/PhabricatorFilesApplication.php', 2943 2944 'PhabricatorFilesApplicationStorageEnginePanel' => 'applications/files/applicationpanel/PhabricatorFilesApplicationStorageEnginePanel.php', ··· 8362 8363 'PhabricatorFileUploadDialogController' => 'PhabricatorFileController', 8363 8364 'PhabricatorFileUploadException' => 'Exception', 8364 8365 'PhabricatorFileUploadSource' => 'Phobject', 8366 + 'PhabricatorFileUploadSourceByteLimitException' => 'Exception', 8365 8367 'PhabricatorFileinfoSetupCheck' => 'PhabricatorSetupCheck', 8366 8368 'PhabricatorFilesApplication' => 'PhabricatorApplication', 8367 8369 'PhabricatorFilesApplicationStorageEnginePanel' => 'PhabricatorApplicationConfigurationPanel',

+7 -17

src/applications/diffusion/query/DiffusionFileFutureQuery.php

··· 88 88 } 89 89 90 90 final protected function executeQuery() { 91 - $future = $this->newQueryFuture(); 91 + $future = $this->newConfiguredQueryFuture(); 92 92 93 93 $drequest = $this->getRequest(); 94 94 ··· 105 105 ->setViewPolicy(PhabricatorPolicies::POLICY_NOONE) 106 106 ->setExecFuture($future); 107 107 108 + $byte_limit = $this->getByteLimit(); 109 + if ($byte_limit) { 110 + $source->setByteLimit($byte_limit); 111 + } 112 + 108 113 $unguarded = AphrontWriteGuard::beginScopedUnguardedWrites(); 109 114 $file = $source->uploadFile(); 110 115 unset($unguarded); ··· 116 121 117 122 $this->didHitTimeLimit = true; 118 123 $file = null; 119 - } 120 - 121 - $byte_limit = $this->getByteLimit(); 122 - 123 - if ($byte_limit && ($file->getByteSize() > $byte_limit)) { 124 + } catch (PhabricatorFileUploadSourceByteLimitException $ex) { 124 125 $this->didHitByteLimit = true; 125 - 126 - $unguarded = AphrontWriteGuard::beginScopedUnguardedWrites(); 127 - id(new PhabricatorDestructionEngine()) 128 - ->destroyObject($file); 129 - unset($unguarded); 130 - 131 126 $file = null; 132 127 } 133 128 ··· 139 134 140 135 if ($this->getTimeout()) { 141 136 $future->setTimeout($this->getTimeout()); 142 - } 143 - 144 - $byte_limit = $this->getByteLimit(); 145 - if ($byte_limit) { 146 - $future->setStdoutSizeLimit($byte_limit + 1); 147 137 } 148 138 149 139 return $future;

+23 -3

src/applications/files/uploadsource/PhabricatorFileUploadSource.php

··· 12 12 private $shouldChunk; 13 13 private $didRewind; 14 14 private $totalBytesWritten = 0; 15 + private $totalBytesRead = 0; 16 + private $byteLimit = 0; 15 17 16 18 public function setName($name) { 17 19 $this->name = $name; ··· 38 40 39 41 public function getViewPolicy() { 40 42 return $this->viewPolicy; 43 + } 44 + 45 + public function setByteLimit($byte_limit) { 46 + $this->byteLimit = $byte_limit; 47 + return $this; 48 + } 49 + 50 + public function getByteLimit() { 51 + return $this->byteLimit; 41 52 } 42 53 43 54 public function uploadFile() { ··· 81 92 return false; 82 93 } 83 94 95 + $read_bytes = $data->current(); 96 + $this->totalBytesRead += strlen($read_bytes); 97 + 98 + if ($this->byteLimit && ($this->totalBytesRead > $this->byteLimit)) { 99 + throw new PhabricatorFileUploadSourceByteLimitException(); 100 + } 101 + 84 102 $rope = $this->getRope(); 85 - $rope->append($data->current()); 103 + $rope->append($read_bytes); 86 104 87 105 return true; 88 106 } ··· 160 178 } 161 179 } 162 180 163 - // If we have extra bytes at the end, write them. 164 - if ($rope->getByteLength()) { 181 + // If we have extra bytes at the end, write them. Note that it's possible 182 + // that we have more than one chunk of bytes left if the read was very 183 + // fast. 184 + while ($rope->getByteLength()) { 165 185 $this->writeChunk($file, $engine); 166 186 } 167 187

src/applications/files/uploadsource/PhabricatorFileUploadSourceByteLimitException.php

··· 1 + <?php 2 + 3 + final class PhabricatorFileUploadSourceByteLimitException 4 + extends Exception {}

Configure Feed

Configure Feed