recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Reduce the impact of `bin/storage dump`

Summary:
Ref T12646.

- Use "wb1" instead of "wb" to use level 1 gzip compression (faster, less compressy). Locally, this went about 2x faster and the output only grew 4% larger.
- LinesOfALargeExecFuture does a lot of unnecessary string operations, and can boil down to a busy wait. The process is pretty saturated by I/O so this isn't the end of the world, but just use raw ExecFuture with FutureIterator so that we wait in `select()`.
- Also, nice the process to +19 so we try to give other things CPU.

Test Plan:
- Ran `bin/storage dump --compress --output ...`.
- Saw CPU time for my local database drop from ~240s to ~90s, with a 4% larger output. Most of this was adding the `1`, but the ExecFuture thing helped a little, too.
- I'm not sure what a great way to test `nice` in a local environment is and it's system dependent anyway, but nothing got worse / blew up.
- Used `gzcat | head` and `gzcat | tail` on the result to sanity-check that everything was preserved.

Reviewers: chad, amckinley

Reviewed By: chad

Maniphest Tasks: T12646

Differential Revision: https://secure.phabricator.com/D17795

epriestley 85ff1d5c 6da73fb3

+34 -15
+34 -15
src/infrastructure/storage/management/workflow/PhabricatorStorageManagementDumpWorkflow.php
··· 138 138 $command = csprintf('mysqldump %Ls', $argv); 139 139 } 140 140 141 + // Decrease the CPU priority of this process so it doesn't contend with 142 + // other more important things. 143 + if (function_exists('proc_nice')) { 144 + proc_nice(19); 145 + } 146 + 147 + 141 148 // If we aren't writing to a file, just passthru the command. 142 149 if ($output_file === null) { 143 150 return phutil_passthru('%C', $command); ··· 148 155 // a full disk). See T6996 for discussion. 149 156 150 157 if ($is_compress) { 151 - $file = gzopen($output_file, 'wb'); 158 + $file = gzopen($output_file, 'wb1'); 152 159 } else { 153 160 $file = fopen($output_file, 'wb'); 154 161 } ··· 162 169 163 170 $future = new ExecFuture('%C', $command); 164 171 165 - $lines = new LinesOfALargeExecFuture($future); 172 + try { 173 + $iterator = id(new FutureIterator(array($future))) 174 + ->setUpdateInterval(0.100); 175 + foreach ($iterator as $ready) { 176 + list($stdout, $stderr) = $future->read(); 177 + $future->discardBuffers(); 178 + 179 + if (strlen($stderr)) { 180 + fwrite(STDERR, $stderr); 181 + } 182 + 183 + if (strlen($stdout)) { 184 + if ($is_compress) { 185 + $ok = gzwrite($file, $stdout); 186 + } else { 187 + $ok = fwrite($file, $stdout); 188 + } 166 189 167 - try { 168 - foreach ($lines as $line) { 169 - $line = $line."\n"; 170 - if ($is_compress) { 171 - $ok = gzwrite($file, $line); 172 - } else { 173 - $ok = fwrite($file, $line); 190 + if ($ok !== strlen($stdout)) { 191 + throw new Exception( 192 + pht( 193 + 'Failed to write %d byte(s) to file "%s".', 194 + new PhutilNumber(strlen($stdout)), 195 + $output_file)); 196 + } 174 197 } 175 198 176 - if ($ok !== strlen($line)) { 177 - throw new Exception( 178 - pht( 179 - 'Failed to write %d byte(s) to file "%s".', 180 - new PhutilNumber(strlen($line)), 181 - $output_file)); 199 + if ($ready !== null) { 200 + $ready->resolvex(); 182 201 } 183 202 } 184 203