@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Support AES256 at-rest encryption in Files
Summary:
Ref T11140. This makes encryption actually work:
- Provide a new configuation option, `keyring`, for specifying encryption keys.
- One key may be marked as `default`. This activates AES256 encryption for Files.
- Add `bin/files generate-key`. This is helps when generating valid encryption keys.
- Add `bin/files encode`. This changes the storage encoding of a file, and helps test encodings and migrate existing data.
- Add `bin/files cycle`. This re-encodes the block key with a new master key, if your master key leaks or you're just paraonid.
- Document all these options and behaviors.
Test Plan:
- Configured a bad `keyring`, hit a bunch of different errors.
- Used `bin/files generate-key` to try to generate bad keys, got appropriate errors ("raw doesn't support keys", etc).
- Used `bin/files generate-key` to generate an AES256 key.
- Put the new AES256 key into the `keyring`, without `default`.
- Uploaded a new file, verified it still uploaded as raw data (no `default` key yet).
- Used `bin/files encode` to change a file to ROT13 and back to raw. Verified old data got deleted and new data got stored properly.
- Used `bin/files encode --key ...` to explicitly convert a file to AES256 with my non-default key.
- Forced a re-encode of an AES256 file, verified the old data was deleted and a new key and IV were generated.
- Used `bin/files cycle` to try to cycle raw/rot13 files, got errors.
- Used `bin/files cycle` to cycle AES256 files. Verified metadata changed but file data did not. Verified file data was still decryptable with metadata.
- Ran `bin/files cycle --all`.
- Ran `encode` and `cycle` on chunked files, saw commands fail properly. These commands operate on the underlying data blocks, not the chunk metadata.
- Set key to `default`, uploaded a file, saw it stored as AES256.
- Read documentation.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T11140
Differential Revision: https://secure.phabricator.com/D16127
+858
-25
+10
src/__phutil_library_map__.php
+10
src/__phutil_library_map__.php
···
2524
2524
'PhabricatorFilesConfigOptions' => 'applications/files/config/PhabricatorFilesConfigOptions.php',
2525
2525
'PhabricatorFilesManagementCatWorkflow' => 'applications/files/management/PhabricatorFilesManagementCatWorkflow.php',
2526
2526
'PhabricatorFilesManagementCompactWorkflow' => 'applications/files/management/PhabricatorFilesManagementCompactWorkflow.php',
2527
+
'PhabricatorFilesManagementCycleWorkflow' => 'applications/files/management/PhabricatorFilesManagementCycleWorkflow.php',
2528
+
'PhabricatorFilesManagementEncodeWorkflow' => 'applications/files/management/PhabricatorFilesManagementEncodeWorkflow.php',
2527
2529
'PhabricatorFilesManagementEnginesWorkflow' => 'applications/files/management/PhabricatorFilesManagementEnginesWorkflow.php',
2530
+
'PhabricatorFilesManagementGenerateKeyWorkflow' => 'applications/files/management/PhabricatorFilesManagementGenerateKeyWorkflow.php',
2528
2531
'PhabricatorFilesManagementMigrateWorkflow' => 'applications/files/management/PhabricatorFilesManagementMigrateWorkflow.php',
2529
2532
'PhabricatorFilesManagementPurgeWorkflow' => 'applications/files/management/PhabricatorFilesManagementPurgeWorkflow.php',
2530
2533
'PhabricatorFilesManagementRebuildWorkflow' => 'applications/files/management/PhabricatorFilesManagementRebuildWorkflow.php',
···
2627
2630
'PhabricatorJiraIssueHasObjectEdgeType' => 'applications/doorkeeper/edge/PhabricatorJiraIssueHasObjectEdgeType.php',
2628
2631
'PhabricatorJumpNavHandler' => 'applications/search/engine/PhabricatorJumpNavHandler.php',
2629
2632
'PhabricatorKeyValueDatabaseCache' => 'applications/cache/PhabricatorKeyValueDatabaseCache.php',
2633
+
'PhabricatorKeyring' => 'applications/files/keyring/PhabricatorKeyring.php',
2634
+
'PhabricatorKeyringConfigOptionType' => 'applications/files/keyring/PhabricatorKeyringConfigOptionType.php',
2630
2635
'PhabricatorLDAPAuthProvider' => 'applications/auth/provider/PhabricatorLDAPAuthProvider.php',
2631
2636
'PhabricatorLegalpadApplication' => 'applications/legalpad/application/PhabricatorLegalpadApplication.php',
2632
2637
'PhabricatorLegalpadConfigOptions' => 'applications/legalpad/config/PhabricatorLegalpadConfigOptions.php',
···
7173
7178
'PhabricatorFilesConfigOptions' => 'PhabricatorApplicationConfigOptions',
7174
7179
'PhabricatorFilesManagementCatWorkflow' => 'PhabricatorFilesManagementWorkflow',
7175
7180
'PhabricatorFilesManagementCompactWorkflow' => 'PhabricatorFilesManagementWorkflow',
7181
+
'PhabricatorFilesManagementCycleWorkflow' => 'PhabricatorFilesManagementWorkflow',
7182
+
'PhabricatorFilesManagementEncodeWorkflow' => 'PhabricatorFilesManagementWorkflow',
7176
7183
'PhabricatorFilesManagementEnginesWorkflow' => 'PhabricatorFilesManagementWorkflow',
7184
+
'PhabricatorFilesManagementGenerateKeyWorkflow' => 'PhabricatorFilesManagementWorkflow',
7177
7185
'PhabricatorFilesManagementMigrateWorkflow' => 'PhabricatorFilesManagementWorkflow',
7178
7186
'PhabricatorFilesManagementPurgeWorkflow' => 'PhabricatorFilesManagementWorkflow',
7179
7187
'PhabricatorFilesManagementRebuildWorkflow' => 'PhabricatorFilesManagementWorkflow',
···
7283
7291
'PhabricatorJiraIssueHasObjectEdgeType' => 'PhabricatorEdgeType',
7284
7292
'PhabricatorJumpNavHandler' => 'Phobject',
7285
7293
'PhabricatorKeyValueDatabaseCache' => 'PhutilKeyValueCache',
7294
+
'PhabricatorKeyring' => 'Phobject',
7295
+
'PhabricatorKeyringConfigOptionType' => 'PhabricatorConfigJSONOptionType',
7286
7296
'PhabricatorLDAPAuthProvider' => 'PhabricatorAuthProvider',
7287
7297
'PhabricatorLegalpadApplication' => 'PhabricatorApplication',
7288
7298
'PhabricatorLegalpadConfigOptions' => 'PhabricatorApplicationConfigOptions',
+12
src/applications/config/option/PhabricatorSecurityConfigOptions.php
+12
src/applications/config/option/PhabricatorSecurityConfigOptions.php
···
43
43
'255.255.255.255/32',
44
44
);
45
45
46
+
$keyring_type = 'custom:PhabricatorKeyringConfigOptionType';
47
+
$keyring_description = $this->deformat(pht(<<<EOTEXT
48
+
The keyring stores master encryption keys. For help with configuring a keyring
49
+
and encryption, see **[[ %s | Configuring Encryption ]]**.
50
+
EOTEXT
51
+
,
52
+
PhabricatorEnv::getDoclink('Configuring Encryption')));
53
+
46
54
return array(
47
55
$this->newOption('security.alternate-file-domain', 'string', null)
48
56
->setLocked(true)
···
276
284
'unsecured content over plain HTTP. It is very difficult to '.
277
285
'undo this change once users\' browsers have accepted the '.
278
286
'setting.')),
287
+
$this->newOption('keyring', $keyring_type, array())
288
+
->setHidden(true)
289
+
->setSummary(pht('Configure master encryption keys.'))
290
+
->setDescription($keyring_description),
279
291
);
280
292
}
281
293
+44
-21
src/applications/files/format/PhabricatorFileAES256StorageFormat.php
+44
-21
src/applications/files/format/PhabricatorFileAES256StorageFormat.php
···
9
9
const FORMATKEY = 'aes-256-cbc';
10
10
11
11
private $keyName;
12
-
private static $keyRing = array();
13
12
14
13
public function getStorageFormatName() {
15
14
return pht('Encrypted (AES-256-CBC)');
16
15
}
17
16
17
+
public function canGenerateNewKeyMaterial() {
18
+
return true;
19
+
}
20
+
21
+
public function generateNewKeyMaterial() {
22
+
$envelope = self::newAES256Key();
23
+
$material = $envelope->openEnvelope();
24
+
return base64_encode($material);
25
+
}
26
+
27
+
public function canCycleMasterKey() {
28
+
return true;
29
+
}
30
+
31
+
public function cycleStorageProperties() {
32
+
$file = $this->getFile();
33
+
list($key, $iv) = $this->extractKeyAndIV($file);
34
+
return $this->formatStorageProperties($key, $iv);
35
+
}
36
+
18
37
public function newReadIterator($raw_iterator) {
19
38
$file = $this->getFile();
20
39
$data = $file->loadDataFromIterator($raw_iterator);
···
42
61
$key_envelope = self::newAES256Key();
43
62
$iv_envelope = self::newAES256IV();
44
63
64
+
return $this->formatStorageProperties($key_envelope, $iv_envelope);
65
+
}
66
+
67
+
private function formatStorageProperties(
68
+
PhutilOpaqueEnvelope $key_envelope,
69
+
PhutilOpaqueEnvelope $iv_envelope) {
70
+
45
71
// Encode the raw binary data with base64 so we can wrap it in JSON.
46
72
$data = array(
47
73
'iv.base64' => base64_encode($iv_envelope->openEnvelope()),
···
54
80
// Encrypt the block key with the master key, using a unique IV.
55
81
$data_iv = self::newAES256IV();
56
82
$key_name = $this->getMasterKeyName();
57
-
$master_key = self::getMasterKeyFromKeyRing($key_name);
83
+
$master_key = $this->getMasterKeyMaterial($key_name);
58
84
$data_cipher = $this->encryptData($data_clear, $master_key, $data_iv);
59
85
60
86
return array(
···
73
99
$outer_payload = base64_decode($outer_payload);
74
100
75
101
$outer_key_name = $file->getStorageProperty('key.name');
76
-
$outer_key = self::getMasterKeyFromKeyRing($outer_key_name);
102
+
$outer_key = $this->getMasterKeyMaterial($outer_key_name);
77
103
78
104
$payload = $this->decryptData($outer_payload, $outer_key, $outer_iv);
79
105
$payload = phutil_json_decode($payload);
···
142
168
return new PhutilOpaqueEnvelope($iv);
143
169
}
144
170
145
-
public function selectKey($key_name) {
171
+
public function selectMasterKey($key_name) {
146
172
// Require that the key exist on the key ring.
147
-
self::getMasterKeyFromKeyRing($key_name);
173
+
$this->getMasterKeyMaterial($key_name);
148
174
149
175
$this->keyName = $key_name;
150
176
return $this;
151
177
}
152
178
153
-
public static function addKeyToKeyRing($name, PhutilOpaqueEnvelope $key) {
154
-
self::$keyRing[$name] = $key;
155
-
}
156
-
157
179
private function getMasterKeyName() {
158
-
if ($this->keyName === null) {
159
-
throw new Exception(pht('No master key selected for AES256 storage.'));
180
+
if ($this->keyName !== null) {
181
+
return $this->keyName;
160
182
}
161
183
162
-
return $this->keyName;
163
-
}
164
-
165
-
private static function getMasterKeyFromKeyRing($key_name) {
166
-
if (!isset(self::$keyRing[$key_name])) {
167
-
throw new Exception(
168
-
pht(
169
-
'No master key "%s" exists in key ring for AES256 storage.',
170
-
$key_name));
184
+
$default = PhabricatorKeyring::getDefaultKeyName(self::FORMATKEY);
185
+
if ($default !== null) {
186
+
return $default;
171
187
}
172
188
173
-
return self::$keyRing[$key_name];
189
+
throw new Exception(
190
+
pht(
191
+
'No AES256 key is specified in the keyring as a default encryption '.
192
+
'key, and no encryption key has been explicitly selected.'));
193
+
}
194
+
195
+
private function getMasterKeyMaterial($key_name) {
196
+
return PhabricatorKeyring::getKey($key_name, self::FORMATKEY);
174
197
}
175
198
176
199
}
+23
src/applications/files/format/PhabricatorFileStorageFormat.php
+23
src/applications/files/format/PhabricatorFileStorageFormat.php
···
26
26
return array();
27
27
}
28
28
29
+
public function canGenerateNewKeyMaterial() {
30
+
return false;
31
+
}
32
+
33
+
public function generateNewKeyMaterial() {
34
+
throw new PhutilMethodNotImplementedException();
35
+
}
36
+
37
+
public function canCycleMasterKey() {
38
+
return false;
39
+
}
40
+
41
+
public function cycleStorageProperties() {
42
+
throw new PhutilMethodNotImplementedException();
43
+
}
44
+
45
+
public function selectMasterKey($key_name) {
46
+
throw new Exception(
47
+
pht(
48
+
'This storage format ("%s") does not support key selection.',
49
+
$this->getStorageFormatName()));
50
+
}
51
+
29
52
final public function getStorageFormatKey() {
30
53
return $this->getPhobjectClassConstant('FORMATKEY');
31
54
}
+8
-3
src/applications/files/format/__tests__/PhabricatorFileStorageFormatTestCase.php
+8
-3
src/applications/files/format/__tests__/PhabricatorFileStorageFormatTestCase.php
···
39
39
$engine = new PhabricatorTestStorageEngine();
40
40
41
41
$key_name = 'test.abcd';
42
-
$key_text = new PhutilOpaqueEnvelope('abcdefghijklmnopABCDEFGHIJKLMNOP');
42
+
$key_text = 'abcdefghijklmnopABCDEFGHIJKLMNOP';
43
43
44
-
PhabricatorFileAES256StorageFormat::addKeyToKeyRing($key_name, $key_text);
44
+
PhabricatorKeyring::addKey(
45
+
array(
46
+
'name' => $key_name,
47
+
'type' => 'aes-256-cbc',
48
+
'material.base64' => base64_encode($key_text),
49
+
));
45
50
46
51
$format = id(new PhabricatorFileAES256StorageFormat())
47
-
->selectKey($key_name);
52
+
->selectMasterKey($key_name);
48
53
49
54
$data = 'The cow jumped over the full moon.';
50
55
+52
src/applications/files/keyring/PhabricatorKeyring.php
+52
src/applications/files/keyring/PhabricatorKeyring.php
···
1
+
<?php
2
+
3
+
final class PhabricatorKeyring extends Phobject {
4
+
5
+
private static $hasReadConfiguration;
6
+
private static $keyRing = array();
7
+
8
+
public static function addKey($spec) {
9
+
self::$keyRing[$spec['name']] = $spec;
10
+
}
11
+
12
+
public static function getKey($name, $type) {
13
+
self::readConfiguration();
14
+
15
+
if (empty(self::$keyRing[$name])) {
16
+
throw new Exception(
17
+
pht(
18
+
'No key "%s" exists in keyring.',
19
+
$name));
20
+
}
21
+
22
+
$spec = self::$keyRing[$name];
23
+
24
+
$material = base64_decode($spec['material.base64'], true);
25
+
return new PhutilOpaqueEnvelope($material);
26
+
}
27
+
28
+
public static function getDefaultKeyName($type) {
29
+
self::readConfiguration();
30
+
31
+
foreach (self::$keyRing as $name => $key) {
32
+
if (!empty($key['default'])) {
33
+
return $name;
34
+
}
35
+
}
36
+
37
+
return null;
38
+
}
39
+
40
+
private static function readConfiguration() {
41
+
if (self::$hasReadConfiguration) {
42
+
return true;
43
+
}
44
+
45
+
self::$hasReadConfiguration = true;
46
+
47
+
foreach (PhabricatorEnv::getEnvConfig('keyring') as $spec) {
48
+
self::addKey($spec);
49
+
}
50
+
}
51
+
52
+
}
+111
src/applications/files/keyring/PhabricatorKeyringConfigOptionType.php
+111
src/applications/files/keyring/PhabricatorKeyringConfigOptionType.php
···
1
+
<?php
2
+
3
+
final class PhabricatorKeyringConfigOptionType
4
+
extends PhabricatorConfigJSONOptionType {
5
+
6
+
public function validateOption(PhabricatorConfigOption $option, $value) {
7
+
if (!is_array($value)) {
8
+
throw new Exception(
9
+
pht(
10
+
'Keyring configuration is not valid: value must be a '.
11
+
'list of encryption keys.'));
12
+
}
13
+
14
+
foreach ($value as $index => $spec) {
15
+
if (!is_array($spec)) {
16
+
throw new Exception(
17
+
pht(
18
+
'Keyring configuration is not valid: each entry in the list must '.
19
+
'be a dictionary describing an encryption key, but the value '.
20
+
'with index "%s" is not a dictionary.',
21
+
$index));
22
+
}
23
+
}
24
+
25
+
26
+
$map = array();
27
+
$defaults = array();
28
+
foreach ($value as $index => $spec) {
29
+
try {
30
+
PhutilTypeSpec::checkMap(
31
+
$spec,
32
+
array(
33
+
'name' => 'string',
34
+
'type' => 'string',
35
+
'material.base64' => 'string',
36
+
'default' => 'optional bool',
37
+
));
38
+
} catch (Exception $ex) {
39
+
throw new Exception(
40
+
pht(
41
+
'Keyring configuration has an invalid key specification (at '.
42
+
'index "%s"): %s.',
43
+
$index,
44
+
$ex->getMessage()));
45
+
}
46
+
47
+
$name = $spec['name'];
48
+
if (isset($map[$name])) {
49
+
throw new Exception(
50
+
pht(
51
+
'Keyring configuration is invalid: it describes multiple keys '.
52
+
'with the same name ("%s"). Each key must have a unique name.',
53
+
$name));
54
+
}
55
+
$map[$name] = true;
56
+
57
+
if (idx($spec, 'default')) {
58
+
$defaults[] = $name;
59
+
}
60
+
61
+
$type = $spec['type'];
62
+
switch ($type) {
63
+
case 'aes-256-cbc':
64
+
if (!function_exists('openssl_encrypt')) {
65
+
throw new Exception(
66
+
pht(
67
+
'Keyring is configured with a "%s" key, but the PHP OpenSSL '.
68
+
'extension is not installed. Install the OpenSSL extension '.
69
+
'to enable encryption.',
70
+
$type));
71
+
}
72
+
73
+
$material = $spec['material.base64'];
74
+
$material = base64_decode($material, true);
75
+
if ($material === false) {
76
+
throw new Exception(
77
+
pht(
78
+
'Keyring specifies an invalid key ("%s"): key material '.
79
+
'should be base64 encoded.',
80
+
$name));
81
+
}
82
+
83
+
if (strlen($material) != 32) {
84
+
throw new Exception(
85
+
pht(
86
+
'Keyring specifies an invalid key ("%s"): key material '.
87
+
'should be 32 bytes (256 bits) but has length %s.',
88
+
$name,
89
+
new PhutilNumber(strlen($material))));
90
+
}
91
+
break;
92
+
default:
93
+
throw new Exception(
94
+
pht(
95
+
'Keyring configuration is invalid: it describes a key with '.
96
+
'type "%s", but this type is unknown.',
97
+
$type));
98
+
}
99
+
}
100
+
101
+
if (count($defaults) > 1) {
102
+
throw new Exception(
103
+
pht(
104
+
'Keyring configuration is invalid: it describes multiple default '.
105
+
'encryption keys. No more than one key may be the default key. '.
106
+
'Keys currently configured as defaults: %s.',
107
+
implode(', ', $defaults)));
108
+
}
109
+
}
110
+
111
+
}
+132
src/applications/files/management/PhabricatorFilesManagementCycleWorkflow.php
+132
src/applications/files/management/PhabricatorFilesManagementCycleWorkflow.php
···
1
+
<?php
2
+
3
+
final class PhabricatorFilesManagementCycleWorkflow
4
+
extends PhabricatorFilesManagementWorkflow {
5
+
6
+
protected function didConstruct() {
7
+
$this
8
+
->setName('cycle')
9
+
->setSynopsis(
10
+
pht('Cycle master key for encrypted files.'))
11
+
->setArguments(
12
+
array(
13
+
array(
14
+
'name' => 'key',
15
+
'param' => 'keyname',
16
+
'help' => pht('Select a specific storage key to cycle to.'),
17
+
),
18
+
array(
19
+
'name' => 'all',
20
+
'help' => pht('Change encoding for all files.'),
21
+
),
22
+
array(
23
+
'name' => 'names',
24
+
'wildcard' => true,
25
+
),
26
+
));
27
+
}
28
+
29
+
public function execute(PhutilArgumentParser $args) {
30
+
$iterator = $this->buildIterator($args);
31
+
if (!$iterator) {
32
+
throw new PhutilArgumentUsageException(
33
+
pht(
34
+
'Either specify a list of files to cycle, or use --all to cycle '.
35
+
'all files.'));
36
+
}
37
+
38
+
$format_map = PhabricatorFileStorageFormat::getAllFormats();
39
+
$engines = PhabricatorFileStorageEngine::loadAllEngines();
40
+
41
+
$key_name = $args->getArg('key');
42
+
43
+
$failed = array();
44
+
foreach ($iterator as $file) {
45
+
$monogram = $file->getMonogram();
46
+
47
+
$engine_key = $file->getStorageEngine();
48
+
$engine = idx($engines, $engine_key);
49
+
50
+
if (!$engine) {
51
+
echo tsprintf(
52
+
"%s\n",
53
+
pht(
54
+
'%s: Uses unknown storage engine "%s".',
55
+
$monogram,
56
+
$engine_key));
57
+
$failed[] = $file;
58
+
continue;
59
+
}
60
+
61
+
if ($engine->isChunkEngine()) {
62
+
echo tsprintf(
63
+
"%s\n",
64
+
pht(
65
+
'%s: Stored as chunks, declining to cycle directly.',
66
+
$monogram));
67
+
continue;
68
+
}
69
+
70
+
$format_key = $file->getStorageFormat();
71
+
if (empty($format_map[$format_key])) {
72
+
echo tsprintf(
73
+
"%s\n",
74
+
pht(
75
+
'%s: Uses unknown storage format "%s".',
76
+
$monogram,
77
+
$format_key));
78
+
$failed[] = $file;
79
+
continue;
80
+
}
81
+
82
+
$format = clone $format_map[$format_key];
83
+
$format->setFile($file);
84
+
85
+
if (!$format->canCycleMasterKey()) {
86
+
echo tsprintf(
87
+
"%s\n",
88
+
pht(
89
+
'%s: Storage format ("%s") does not support key cycling.',
90
+
$monogram,
91
+
$format->getStorageFormatName()));
92
+
continue;
93
+
}
94
+
95
+
echo tsprintf(
96
+
"%s\n",
97
+
pht(
98
+
'%s: Cycling master key.',
99
+
$monogram));
100
+
101
+
try {
102
+
if ($key_name) {
103
+
$format->selectMasterKey($key_name);
104
+
}
105
+
106
+
$file->cycleMasterStorageKey($format);
107
+
108
+
echo tsprintf(
109
+
"%s\n",
110
+
pht('Done.'));
111
+
} catch (Exception $ex) {
112
+
echo tsprintf(
113
+
"%B\n",
114
+
pht('Failed! %s', (string)$ex));
115
+
$failed[] = $file;
116
+
}
117
+
}
118
+
119
+
if ($failed) {
120
+
$monograms = mpull($failed, 'getMonogram');
121
+
122
+
echo tsprintf(
123
+
"%s\n",
124
+
pht('Failures: %s.', implode(', ', $monograms)));
125
+
126
+
return 1;
127
+
}
128
+
129
+
return 0;
130
+
}
131
+
132
+
}
+151
src/applications/files/management/PhabricatorFilesManagementEncodeWorkflow.php
+151
src/applications/files/management/PhabricatorFilesManagementEncodeWorkflow.php
···
1
+
<?php
2
+
3
+
final class PhabricatorFilesManagementEncodeWorkflow
4
+
extends PhabricatorFilesManagementWorkflow {
5
+
6
+
protected function didConstruct() {
7
+
$this
8
+
->setName('encode')
9
+
->setSynopsis(
10
+
pht('Change the storage encoding of files.'))
11
+
->setArguments(
12
+
array(
13
+
array(
14
+
'name' => 'as',
15
+
'param' => 'format',
16
+
'help' => pht('Select the storage format to use.'),
17
+
),
18
+
array(
19
+
'name' => 'key',
20
+
'param' => 'keyname',
21
+
'help' => pht('Select a specific storage key.'),
22
+
),
23
+
array(
24
+
'name' => 'all',
25
+
'help' => pht('Change encoding for all files.'),
26
+
),
27
+
array(
28
+
'name' => 'force',
29
+
'help' => pht(
30
+
'Re-encode files which are already stored in the target '.
31
+
'encoding.'),
32
+
),
33
+
array(
34
+
'name' => 'names',
35
+
'wildcard' => true,
36
+
),
37
+
));
38
+
}
39
+
40
+
public function execute(PhutilArgumentParser $args) {
41
+
$iterator = $this->buildIterator($args);
42
+
if (!$iterator) {
43
+
throw new PhutilArgumentUsageException(
44
+
pht(
45
+
'Either specify a list of files to encode, or use --all to '.
46
+
'encode all files.'));
47
+
}
48
+
49
+
$force = (bool)$args->getArg('force');
50
+
51
+
$format_list = PhabricatorFileStorageFormat::getAllFormats();
52
+
$format_list = array_keys($format_list);
53
+
$format_list = implode(', ', $format_list);
54
+
55
+
$format_key = $args->getArg('as');
56
+
if (!strlen($format_key)) {
57
+
throw new PhutilArgumentUsageException(
58
+
pht(
59
+
'Use --as <format> to select a target encoding format. Available '.
60
+
'formats are: %s.',
61
+
$format_list));
62
+
}
63
+
64
+
$format = PhabricatorFileStorageFormat::getFormat($format_key);
65
+
if (!$format) {
66
+
throw new PhutilArgumentUsageException(
67
+
pht(
68
+
'Storage format "%s" is not valid. Available formats are: %s.',
69
+
$format_key,
70
+
$format_list));
71
+
}
72
+
73
+
$key_name = $args->getArg('key');
74
+
if (strlen($key_name)) {
75
+
$format->selectMasterKey($key_name);
76
+
}
77
+
78
+
$engines = PhabricatorFileStorageEngine::loadAllEngines();
79
+
80
+
$failed = array();
81
+
foreach ($iterator as $file) {
82
+
$monogram = $file->getMonogram();
83
+
84
+
$engine_key = $file->getStorageEngine();
85
+
$engine = idx($engines, $engine_key);
86
+
87
+
if (!$engine) {
88
+
echo tsprintf(
89
+
"%s\n",
90
+
pht(
91
+
'%s: Uses unknown storage engine "%s".',
92
+
$monogram,
93
+
$engine_key));
94
+
$failed[] = $file;
95
+
continue;
96
+
}
97
+
98
+
if ($engine->isChunkEngine()) {
99
+
echo tsprintf(
100
+
"%s\n",
101
+
pht(
102
+
'%s: Stored as chunks, no data to encode directly.',
103
+
$monogram));
104
+
continue;
105
+
}
106
+
107
+
if (($file->getStorageFormat() == $format_key) && !$force) {
108
+
echo tsprintf(
109
+
"%s\n",
110
+
pht(
111
+
'%s: Already encoded in target format.',
112
+
$monogram));
113
+
continue;
114
+
}
115
+
116
+
echo tsprintf(
117
+
"%s\n",
118
+
pht(
119
+
'%s: Changing encoding from "%s" to "%s".',
120
+
$monogram,
121
+
$file->getStorageFormat(),
122
+
$format_key));
123
+
124
+
try {
125
+
$file->migrateToStorageFormat($format);
126
+
127
+
echo tsprintf(
128
+
"%s\n",
129
+
pht('Done.'));
130
+
} catch (Exception $ex) {
131
+
echo tsprintf(
132
+
"%B\n",
133
+
pht('Failed! %s', (string)$ex));
134
+
$failed[] = $file;
135
+
}
136
+
}
137
+
138
+
if ($failed) {
139
+
$monograms = mpull($failed, 'getMonogram');
140
+
141
+
echo tsprintf(
142
+
"%s\n",
143
+
pht('Failures: %s.', implode(', ', $monograms)));
144
+
145
+
return 1;
146
+
}
147
+
148
+
return 0;
149
+
}
150
+
151
+
}
+63
src/applications/files/management/PhabricatorFilesManagementGenerateKeyWorkflow.php
+63
src/applications/files/management/PhabricatorFilesManagementGenerateKeyWorkflow.php
···
1
+
<?php
2
+
3
+
final class PhabricatorFilesManagementGenerateKeyWorkflow
4
+
extends PhabricatorFilesManagementWorkflow {
5
+
6
+
protected function didConstruct() {
7
+
$this
8
+
->setName('generate-key')
9
+
->setSynopsis(
10
+
pht('Generate an encryption key.'))
11
+
->setArguments(
12
+
array(
13
+
array(
14
+
'name' => 'type',
15
+
'param' => 'keytype',
16
+
'help' => pht('Select the type of key to generate.'),
17
+
),
18
+
));
19
+
}
20
+
21
+
public function execute(PhutilArgumentParser $args) {
22
+
$type = $args->getArg('type');
23
+
if (!strlen($type)) {
24
+
throw new PhutilArgumentUsageException(
25
+
pht(
26
+
'Specify the type of key to generate with --type.'));
27
+
}
28
+
29
+
$format = PhabricatorFileStorageFormat::getFormat($type);
30
+
if (!$format) {
31
+
throw new PhutilArgumentUsageException(
32
+
pht(
33
+
'No key type "%s" exists.',
34
+
$type));
35
+
}
36
+
37
+
if (!$format->canGenerateNewKeyMaterial()) {
38
+
throw new PhutilArgumentUsageException(
39
+
pht(
40
+
'Storage format "%s" can not generate keys.',
41
+
$format->getStorageFormatName()));
42
+
}
43
+
44
+
$material = $format->generateNewKeyMaterial();
45
+
46
+
$structure = array(
47
+
'name' => 'generated-key-'.Filesystem::readRandomCharacters(12),
48
+
'type' => $type,
49
+
'material.base64' => $material,
50
+
);
51
+
52
+
$json = id(new PhutilJSON())->encodeFormatted($structure);
53
+
54
+
echo tsprintf(
55
+
"%s: %s\n\n%B\n",
56
+
pht('Key Material'),
57
+
$format->getStorageFormatName(),
58
+
$json);
59
+
60
+
return 0;
61
+
}
62
+
63
+
}
+54
-1
src/applications/files/storage/PhabricatorFile.php
+54
-1
src/applications/files/storage/PhabricatorFile.php
···
326
326
327
327
$file = self::initializeNewFile();
328
328
329
-
$default_key = PhabricatorFileRawStorageFormat::FORMATKEY;
329
+
$aes_type = PhabricatorFileAES256StorageFormat::FORMATKEY;
330
+
$has_aes = PhabricatorKeyring::getDefaultKeyName($aes_type);
331
+
if ($has_aes !== null) {
332
+
$default_key = PhabricatorFileAES256StorageFormat::FORMATKEY;
333
+
} else {
334
+
$default_key = PhabricatorFileRawStorageFormat::FORMATKEY;
335
+
}
330
336
$key = idx($params, 'format', $default_key);
331
337
332
338
// Callers can pass in an object explicitly instead of a key. This is
···
440
446
$old_engine,
441
447
$old_identifier,
442
448
$old_handle);
449
+
450
+
return $this;
451
+
}
452
+
453
+
public function migrateToStorageFormat(PhabricatorFileStorageFormat $format) {
454
+
if (!$this->getID() || !$this->getStorageHandle()) {
455
+
throw new Exception(
456
+
pht("You can not migrate a file which hasn't yet been saved."));
457
+
}
458
+
459
+
$data = $this->loadFileData();
460
+
$params = array(
461
+
'name' => $this->getName(),
462
+
);
463
+
464
+
$engine = $this->instantiateStorageEngine();
465
+
$old_handle = $this->getStorageHandle();
466
+
467
+
$properties = $format->newStorageProperties();
468
+
$this->setStorageFormat($format->getStorageFormatKey());
469
+
$this->setStorageProperties($properties);
470
+
471
+
list($identifier, $new_handle) = $this->writeToEngine(
472
+
$engine,
473
+
$data,
474
+
$params);
475
+
476
+
$this->setStorageHandle($new_handle);
477
+
$this->save();
478
+
479
+
$this->deleteFileDataIfUnused(
480
+
$engine,
481
+
$identifier,
482
+
$old_handle);
483
+
484
+
return $this;
485
+
}
486
+
487
+
public function cycleMasterStorageKey(PhabricatorFileStorageFormat $format) {
488
+
if (!$this->getID() || !$this->getStorageHandle()) {
489
+
throw new Exception(
490
+
pht("You can not cycle keys for a file which hasn't yet been saved."));
491
+
}
492
+
493
+
$properties = $format->cycleStorageProperties();
494
+
$this->setStorageProperties($properties);
495
+
$this->save();
443
496
444
497
return $this;
445
498
}
+196
src/docs/user/configuration/configuring_encryption.diviner
+196
src/docs/user/configuration/configuring_encryption.diviner
···
1
+
@title Configuring Encryption
2
+
@group config
3
+
4
+
Setup guide for configuring encryption.
5
+
6
+
Overview
7
+
========
8
+
9
+
Phabricator supports at-rest encryption of uploaded file data stored in the
10
+
"Files" application.
11
+
12
+
Configuring at-rest file data encryption does not encrypt any other data or
13
+
resources. In particular, it does not encrypt the database and does not encrypt
14
+
Passphrase credentials.
15
+
16
+
Attackers who compromise a Phabricator host can read the master key and decrypt
17
+
the data. In most configurations, this does not represent a significant
18
+
barrier above and beyond accessing the file data. Thus, configuring at-rest
19
+
encryption is primarily useful for two types of installs:
20
+
21
+
- If you maintain your own webserver and database hardware but want to use
22
+
Amazon S3 or a similar cloud provider as a blind storage server, file data
23
+
encryption can let you do so without needing to trust the cloud provider.
24
+
- If you face a regulatory or compliance need to encrypt data at rest but do
25
+
not need to actually secure this data, encrypting the data and placing the
26
+
master key in plaintext next to it may satisfy compliance requirements.
27
+
28
+
The remainder of this document discusses how to configure at-rest encryption.
29
+
30
+
31
+
Quick Start
32
+
===========
33
+
34
+
To configure encryption, you will generally follow these steps:
35
+
36
+
- Generate a master key with `bin/files generate-key`.
37
+
- Add the master key it to the `keyring`, but don't mark it as `default` yet.
38
+
- Use `bin/files encode ...` to test encrypting a few files.
39
+
- Mark the key as `default` to automatically encrypt new files.
40
+
- Use `bin/files encode --all ...` to encrypt any existing files.
41
+
42
+
See the following sections for detailed guidance on these steps.
43
+
44
+
45
+
Configuring a Keyring
46
+
=====================
47
+
48
+
To configure a keyring, set `keyring` with `bin/config` or by using another
49
+
configuration source. This option should be a list of keys in this format:
50
+
51
+
```lang=json
52
+
...
53
+
"keyring": [
54
+
{
55
+
"name": "master.key",
56
+
"type": "aes-256-cbc",
57
+
"material.base64": "UcHUJqq8MhZRwhvDV8sJwHj7bNJoM4tWfOIi..."
58
+
"default": true
59
+
},
60
+
...
61
+
]
62
+
...
63
+
```
64
+
65
+
Each key should have these properties:
66
+
67
+
- `name`: //Required string.// A unique key name.
68
+
- `type`: //Required string.// Type of the key. Only `aes-256-cbc` is
69
+
supported.
70
+
- `material.base64`: //Required string.// The key material. See below for
71
+
details.
72
+
- `default`: //Optional bool.// Optionally, mark exactly one key as the
73
+
default key to enable encryption of newly uploaded file data.
74
+
75
+
The key material is sensitive an an attacker who learns it can decrypt data
76
+
from the storage engine.
77
+
78
+
79
+
Format: Raw Data
80
+
================
81
+
82
+
The `raw` storage format is automatically selected for all newly uploaded
83
+
file data if no key is makred as the `default` key in the keyring. This is
84
+
the behavior of Phabricator if you haven't configured anything.
85
+
86
+
This format stores raw data without modification.
87
+
88
+
89
+
Format: AES256
90
+
==============
91
+
92
+
The `aes-256-cbc` storage format is automatically selected for all newly
93
+
uploaded file data if an AES256 key is marked as the `default` key in the
94
+
keyring.
95
+
96
+
This format uses AES256 in CBC mode. Each block of file data is encrypted with
97
+
a unique, randomly generated private key. That key is then encrypted with the
98
+
master key. Among other motivations, this strategy allows the master key to be
99
+
cycled relatively cheaply later (see "Cycling Master Keys" below).
100
+
101
+
AES256 keys should be randomly generated and 256 bits (32 characters) in
102
+
length, then base64 encoded when represented in `keyring`.
103
+
104
+
You can generate a valid, properly encoded AES256 master key with this command:
105
+
106
+
```
107
+
phabricator/ $ ./bin/files generate-key --type aes-256-cbc
108
+
```
109
+
110
+
This mode is generally similar to the default server-side encryption mode
111
+
supported by Amazon S3.
112
+
113
+
114
+
Format: ROT13
115
+
=============
116
+
117
+
The `rot13` format is a test format that is never selected by default. You can
118
+
select this format explicitly with `bin/files encode` to test storage and
119
+
encryption behavior.
120
+
121
+
This format applies ROT13 encoding to file data.
122
+
123
+
124
+
Changing File Storage Formats
125
+
=============================
126
+
127
+
To test configuration, you can explicitly change the storage format of a file.
128
+
129
+
This will read the file data, decrypt it if necessary, write a new copy of the
130
+
data with the desired encryption, then update the file to point at the new
131
+
data. You can use this to make sure encryption works before turning it on by
132
+
default.
133
+
134
+
To change the format of an individual file, run this command:
135
+
136
+
```
137
+
phabricator/ $ ./bin/files encode --as <format> F123 [--key <key>]
138
+
```
139
+
140
+
This will change the storage format of the sepcified file.
141
+
142
+
143
+
Verifying Storage Formats
144
+
=========================
145
+
146
+
You can review the storage format of a file from the web UI, in the
147
+
{nav Storage} tab under "Format". You can also use the "Engine" and "Handle"
148
+
properties to identify where the underlying data is stored and verify that
149
+
it is encrypted or encoded in the way you expect.
150
+
151
+
See @{article:Configuring File Storage} for more information on storage
152
+
engines.
153
+
154
+
155
+
Cycling Master Keys
156
+
===================
157
+
158
+
If you need to cycle your master key, some storage formats support key cycling.
159
+
160
+
Cycling a file's encryption key decodes the local key for the data using the
161
+
old master key, then re-encodes it using the new master key. This is primarily
162
+
useful if you believe your master key may have been compromised.
163
+
164
+
First, add a new key to the keyring and mark it as the default key. You need
165
+
to leave the old key in place for now so existing data can be decrypted.
166
+
167
+
To cycle an individual file, run this command:
168
+
169
+
```
170
+
phabricator/ $ ./bin/files cycle F123
171
+
```
172
+
173
+
Verify that cycling worked properly by examining the command output and
174
+
accessing the file to check that the data is present and decryptable. You
175
+
can cycle additional files to gain additional confidence.
176
+
177
+
You can cycle all files with this command:
178
+
179
+
```
180
+
phabricator/ $ ./bin/files cycle --all
181
+
```
182
+
183
+
Once all files have been cycled, remove the old master key from the keyring.
184
+
185
+
Not all storage formats support key cycling: cycling a file only has an effect
186
+
if the storage format is an encrypted format. For example, cycling a file that
187
+
uses the `raw` storage format has no effect.
188
+
189
+
190
+
Next Steps
191
+
==========
192
+
193
+
Continue by:
194
+
195
+
- understanding storage engines with @{article:Configuring File Storage}; or
196
+
- returning to the @{article:Configuration Guide}.