recaptime.dev / phorge
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Limit memory usage of `ssh-exec` during large pull operations

Summary: Fixes T4241. Ref T4206. See T4241 for a description here. Generally, when we connect a fat pipe (`git-upload-pack`) to a narrow one (`git` over SSH) we currently read limitless data into memory. Instead, throttle reads until writes catch up. This is now possible because of the previous changes in this sequence.

Test Plan:
- Ran `git clone` and `git push` on the entire Wine repository.
- Observed CPU and memory usage.
- Memory usage was constant and low, CPU usage was high only during I/O (which is expected, since we have to actually do work, although thre might be room to further reduce this).

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4241, T4206

Differential Revision: https://secure.phabricator.com/D7776

epriestley f7464400 537f2eab

+45 -3
+45 -3
src/infrastructure/ssh/PhabricatorSSHPassthruCommand.php
··· 97 97 98 98 $channels = array($command_channel, $io_channel, $error_channel); 99 99 100 + // We want to limit the amount of data we'll hold in memory for this 101 + // process. See T4241 for a discussion of this issue in general. 102 + 103 + $buffer_size = (1024 * 1024); // 1MB 104 + $io_channel->setReadBufferSize($buffer_size); 105 + $command_channel->setReadBufferSize($buffer_size); 106 + 107 + // TODO: This just makes us throw away stderr after the first 1MB, but we 108 + // don't currently have the support infrastructure to buffer it correctly. 109 + // It's difficult to imagine this causing problems in practice, though. 110 + $this->execFuture->getStderrSizeLimit($buffer_size); 111 + 100 112 while (true) { 101 113 PhutilChannel::waitForAny($channels); 102 114 ··· 104 116 $command_channel->update(); 105 117 $error_channel->update(); 106 118 107 - $done = !$command_channel->isOpen(); 119 + // If any channel is blocked on the other end, wait for it to flush before 120 + // we continue reading. For example, if a user is running `git clone` on 121 + // a 1GB repository, the underlying `git-upload-pack` may 122 + // be able to produce data much more quickly than we can send it over 123 + // the network. If we don't throttle the reads, we may only send a few 124 + // MB over the I/O channel in the time it takes to read the entire 1GB off 125 + // the command channel. That leaves us with 1GB of data in memory. 126 + 127 + while ($command_channel->isOpen() && 128 + $io_channel->isOpenForWriting() && 129 + ($command_channel->getWriteBufferSize() >= $buffer_size || 130 + $io_channel->getWriteBufferSize() >= $buffer_size || 131 + $error_channel->getWriteBufferSize() >= $buffer_size)) { 132 + PhutilChannel::waitForActivity(array(), $channels); 133 + $io_channel->update(); 134 + $command_channel->update(); 135 + $error_channel->update(); 136 + } 137 + 138 + // If the subprocess has exited and we've read everything from it, 139 + // we're all done. 140 + $done = !$command_channel->isOpenForReading() && 141 + $command_channel->isReadBufferEmpty(); 108 142 109 143 $in_message = $io_channel->read(); 110 144 if ($in_message !== null) { ··· 133 167 134 168 // If the client has disconnected, kill the subprocess and bail. 135 169 if (!$io_channel->isOpenForWriting()) { 136 - $this->execFuture->resolveKill(); 170 + $this->execFuture 171 + ->setStdoutSizeLimit(0) 172 + ->setStderrSizeLimit(0) 173 + ->setReadBufferSize(null) 174 + ->resolveKill(); 137 175 break; 138 176 } 139 177 } 140 178 141 - list($err) = $this->execFuture->resolve(); 179 + list($err) = $this->execFuture 180 + ->setStdoutSizeLimit(0) 181 + ->setStderrSizeLimit(0) 182 + ->setReadBufferSize(null) 183 + ->resolve(); 142 184 143 185 return $err; 144 186 }