recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Support export of data in files larger than 8MB

Summary:
Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. `PhabricatorFile::newFromFileData()` always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use `IteratorUploadSource`, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan:
- Changed the JSON exporter to just export 8MB of the letter "q": `return str_repeat('q', 1024 * 1024 * 9);`.
- Before change: fatal, "no storage engine can store this file".
- After change: export works cleanly.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13049

Differential Revision: https://secure.phabricator.com/D18953

epriestley 8b8a3142 0de62108

+50 -11
+6 -2
src/applications/files/storage/PhabricatorFile.php
··· 272 272 $file->setByteSize($length); 273 273 274 274 // NOTE: Once we receive the first chunk, we'll detect its MIME type and 275 - // update the parent file. This matters for large media files like video. 276 - $file->setMimeType('application/octet-stream'); 275 + // update the parent file if a MIME type hasn't been provided. This matters 276 + // for large media files like video. 277 + $mime_type = idx($params, 'mime-type'); 278 + if (!strlen($mime_type)) { 279 + $file->setMimeType('application/octet-stream'); 280 + } 277 281 278 282 $chunked_hash = idx($params, 'chunkedHash'); 279 283
+30
src/applications/files/uploadsource/PhabricatorFileUploadSource.php
··· 6 6 private $name; 7 7 private $relativeTTL; 8 8 private $viewPolicy; 9 + private $mimeType; 10 + private $authorPHID; 9 11 10 12 private $rope; 11 13 private $data; ··· 49 51 50 52 public function getByteLimit() { 51 53 return $this->byteLimit; 54 + } 55 + 56 + public function setMIMEType($mime_type) { 57 + $this->mimeType = $mime_type; 58 + return $this; 59 + } 60 + 61 + public function getMIMEType() { 62 + return $this->mimeType; 63 + } 64 + 65 + public function setAuthorPHID($author_phid) { 66 + $this->authorPHID = $author_phid; 67 + return $this; 68 + } 69 + 70 + public function getAuthorPHID() { 71 + return $this->authorPHID; 52 72 } 53 73 54 74 public function uploadFile() { ··· 243 263 $ttl = $this->getRelativeTTL(); 244 264 if ($ttl !== null) { 245 265 $parameters['ttl.relative'] = $ttl; 266 + } 267 + 268 + $mime_type = $this->getMimeType(); 269 + if ($mime_type !== null) { 270 + $parameters['mime-type'] = $mime_type; 271 + } 272 + 273 + $author_phid = $this->getAuthorPHID(); 274 + if ($author_phid !== null) { 275 + $parameters['authorPHID'] = $author_phid; 246 276 } 247 277 248 278 return $parameters;
+14 -9
src/applications/search/controller/PhabricatorApplicationSearchController.php
··· 461 461 462 462 $export_result = $format->newFileData(); 463 463 464 - $file = PhabricatorFile::newFromFileData( 465 - $export_result, 466 - array( 467 - 'name' => $filename, 468 - 'authorPHID' => $viewer->getPHID(), 469 - 'ttl.relative' => phutil_units('15 minutes in seconds'), 470 - 'viewPolicy' => PhabricatorPolicies::POLICY_NOONE, 471 - 'mime-type' => $mime_type, 472 - )); 464 + // We have all the data in one big string and aren't actually 465 + // streaming it, but pretending that we are allows us to actviate 466 + // the chunk engine and store large files. 467 + $iterator = new ArrayIterator(array($export_result)); 468 + 469 + $source = id(new PhabricatorIteratorFileUploadSource()) 470 + ->setName($filename) 471 + ->setViewPolicy(PhabricatorPolicies::POLICY_NOONE) 472 + ->setMIMEType($mime_type) 473 + ->setRelativeTTL(phutil_units('60 minutes in seconds')) 474 + ->setAuthorPHID($viewer->getPHID()) 475 + ->setIterator($iterator); 476 + 477 + $file = $source->uploadFile(); 473 478 474 479 return $this->newDialog() 475 480 ->setTitle(pht('Download Results'))