@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.)
hq.recaptime.dev/wiki/Phorge
phorge
phabricator
Modernize file storage engine selection
Summary:
Fixes T5843. File storage engines use a very old "selector" mechanism which makes them difficult to extend.
This mechanism predates widespread use of `PhutilSymbolLoader` to discover available implementations at runtime. Runtime discovery has generally proven more flexible and easier to use than explicit selection (although it sometimes needs more UI to support it in cases where order or enabled/disabled flags can not be directly determined).
Use a modern runtime discovery mechanism instead of an explicit selector. This might break any installs which subclassed the `Selector`, but I believe almost no such installs exist, and they'll receive a meaningful exception upon upgrading (any custom engines will no longer implement all of the required methods).
Looking forward, this modernizes infrastructure to prepare for new "virtual" chunked-storage engines, with the eventual goal of supporting very large file uploads and data import into the Phacility cluster.
This uses D12051 to add UI to make it easier to understand the state of storage engines.
Test Plan:
Used new UI panel to assess storage engines:
{F336270}
- Uploaded a small file, saw it go to MySQL engine.
- Uploaded a larger file, saw it go to S3 engine.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5843
Differential Revision: https://secure.phabricator.com/D12053
+422
-247
+52
-48
resources/celerity/map.php
+52
-48
resources/celerity/map.php
···
7
7
*/
8
8
return array(
9
9
'names' => array(
10
-
'core.pkg.css' => '70320e8a',
10
+
'core.pkg.css' => 'efdeeb14',
11
11
'core.pkg.js' => '5f50c01b',
12
12
'darkconsole.pkg.js' => '8ab24e01',
13
13
'differential.pkg.css' => '1940be3f',
···
34
34
'rsrc/css/aphront/typeahead.css' => '0e403212',
35
35
'rsrc/css/application/almanac/almanac.css' => 'dbb9b3af',
36
36
'rsrc/css/application/auth/auth.css' => '1e655982',
37
-
'rsrc/css/application/base/main-menu-view.css' => 'f9f5cd1b',
37
+
'rsrc/css/application/base/main-menu-view.css' => '58db7ad2',
38
38
'rsrc/css/application/base/notification-menu.css' => '6aa0a74b',
39
39
'rsrc/css/application/base/phabricator-application-launch-view.css' => '16ca323f',
40
40
'rsrc/css/application/base/standard-page-view.css' => 'df338a4b',
···
44
44
'rsrc/css/application/config/config-welcome.css' => '6abd79be',
45
45
'rsrc/css/application/config/setup-issue.css' => '22270af2',
46
46
'rsrc/css/application/config/unhandled-exception.css' => '37d4f9a2',
47
-
'rsrc/css/application/conpherence/durable-column.css' => '7abcc3f2',
47
+
'rsrc/css/application/conpherence/durable-column.css' => 'acefcb30',
48
48
'rsrc/css/application/conpherence/menu.css' => 'c6ac5299',
49
49
'rsrc/css/application/conpherence/message-pane.css' => '5930260a',
50
50
'rsrc/css/application/conpherence/notification.css' => '04a6e10a',
···
105
105
'rsrc/css/application/tokens/tokens.css' => '3d0f239e',
106
106
'rsrc/css/application/uiexample/example.css' => '528b19de',
107
107
'rsrc/css/core/core.css' => '86bfbe8c',
108
-
'rsrc/css/core/remarkup.css' => '2dbff225',
108
+
'rsrc/css/core/remarkup.css' => 'bc65f3cc',
109
109
'rsrc/css/core/syntax.css' => '56c1ba38',
110
110
'rsrc/css/core/z-index.css' => '2db67397',
111
111
'rsrc/css/diviner/diviner-shared.css' => '38813222',
112
112
'rsrc/css/font/font-awesome.css' => 'ae9a7b4d',
113
-
'rsrc/css/font/font-source-sans-pro.css' => '0d859f60',
113
+
'rsrc/css/font/font-source-sans-pro.css' => '4a2430d7',
114
114
'rsrc/css/font/phui-font-icon-base.css' => '3dad2ae3',
115
115
'rsrc/css/layout/phabricator-filetree-view.css' => 'fccf9f82',
116
116
'rsrc/css/layout/phabricator-hovercard-view.css' => '893f4783',
···
128
128
'rsrc/css/phui/phui-crumbs-view.css' => '594d719e',
129
129
'rsrc/css/phui/phui-document.css' => '0f83a7df',
130
130
'rsrc/css/phui/phui-feed-story.css' => 'c9f3a0b5',
131
-
'rsrc/css/phui/phui-fontkit.css' => 'd30f4fa3',
131
+
'rsrc/css/phui/phui-fontkit.css' => '1fa79503',
132
132
'rsrc/css/phui/phui-form-view.css' => '78d729fe',
133
133
'rsrc/css/phui/phui-form.css' => 'f535f938',
134
134
'rsrc/css/phui/phui-header-view.css' => '083669db',
···
352
352
'rsrc/js/application/aphlict/behavior-aphlict-status.js' => 'ea681761',
353
353
'rsrc/js/application/auth/behavior-persona-login.js' => '9414ff18',
354
354
'rsrc/js/application/config/behavior-reorder-fields.js' => '14a827de',
355
-
'rsrc/js/application/conpherence/ConpherenceThreadManager.js' => 'efef202b',
356
-
'rsrc/js/application/conpherence/behavior-durable-column.js' => 'aa3b6c22',
357
-
'rsrc/js/application/conpherence/behavior-menu.js' => 'e476c952',
355
+
'rsrc/js/application/conpherence/ConpherenceThreadManager.js' => '0324970d',
356
+
'rsrc/js/application/conpherence/behavior-durable-column.js' => '9142e483',
357
+
'rsrc/js/application/conpherence/behavior-menu.js' => 'c4151295',
358
358
'rsrc/js/application/conpherence/behavior-pontificate.js' => '21ba5861',
359
359
'rsrc/js/application/conpherence/behavior-quicksand-blacklist.js' => '7927a7d3',
360
360
'rsrc/js/application/conpherence/behavior-widget-pane.js' => '2c1cd7f5',
···
514
514
'changeset-view-manager' => '88be0133',
515
515
'config-options-css' => '7fedf08b',
516
516
'config-welcome-css' => '6abd79be',
517
-
'conpherence-durable-column-view' => '7abcc3f2',
517
+
'conpherence-durable-column-view' => 'acefcb30',
518
518
'conpherence-menu-css' => 'c6ac5299',
519
519
'conpherence-message-pane-css' => '5930260a',
520
520
'conpherence-notification-css' => '04a6e10a',
521
-
'conpherence-thread-manager' => 'efef202b',
521
+
'conpherence-thread-manager' => '0324970d',
522
522
'conpherence-update-css' => '1099a660',
523
523
'conpherence-widget-pane-css' => '3d575438',
524
524
'differential-changeset-view-css' => '6a8b172a',
···
534
534
'diffusion-source-css' => '66fdf661',
535
535
'diviner-shared-css' => '38813222',
536
536
'font-fontawesome' => 'ae9a7b4d',
537
-
'font-source-sans-pro' => '0d859f60',
537
+
'font-source-sans-pro' => '4a2430d7',
538
538
'global-drag-and-drop-css' => '697324ad',
539
539
'harbormaster-css' => '49d64eb4',
540
540
'herald-css' => '826075fa',
···
558
558
'javelin-behavior-boards-dropdown' => '0ec56e1d',
559
559
'javelin-behavior-choose-control' => '6153c708',
560
560
'javelin-behavior-config-reorder-fields' => '14a827de',
561
-
'javelin-behavior-conpherence-menu' => 'e476c952',
561
+
'javelin-behavior-conpherence-menu' => 'c4151295',
562
562
'javelin-behavior-conpherence-pontificate' => '21ba5861',
563
563
'javelin-behavior-conpherence-widget-pane' => '2c1cd7f5',
564
564
'javelin-behavior-countdown-timer' => 'e4cc26b3',
···
585
585
'javelin-behavior-diffusion-locate-file' => '6d3e1947',
586
586
'javelin-behavior-diffusion-pull-lastmodified' => '2b228192',
587
587
'javelin-behavior-doorkeeper-tag' => 'e5822781',
588
-
'javelin-behavior-durable-column' => 'aa3b6c22',
588
+
'javelin-behavior-durable-column' => '9142e483',
589
589
'javelin-behavior-error-log' => '6882e80a',
590
590
'javelin-behavior-fancy-datepicker' => 'c51ae228',
591
591
'javelin-behavior-global-drag-and-drop' => '07f199d8',
···
730
730
'phabricator-hovercard-view-css' => '893f4783',
731
731
'phabricator-keyboard-shortcut' => '1ae869f2',
732
732
'phabricator-keyboard-shortcut-manager' => 'c1700f6f',
733
-
'phabricator-main-menu-view' => 'f9f5cd1b',
733
+
'phabricator-main-menu-view' => '58db7ad2',
734
734
'phabricator-nav-view-css' => '7aeaf435',
735
735
'phabricator-notification' => '0c6946e7',
736
736
'phabricator-notification-css' => '9c279160',
···
739
739
'phabricator-phtize' => 'd254d646',
740
740
'phabricator-prefab' => '72da38cc',
741
741
'phabricator-profile-css' => '1a20dcbf',
742
-
'phabricator-remarkup-css' => '2dbff225',
742
+
'phabricator-remarkup-css' => 'bc65f3cc',
743
743
'phabricator-search-results-css' => '559cc554',
744
744
'phabricator-shaped-request' => '7cbe244b',
745
745
'phabricator-side-menu-view-css' => '7e8c6341',
···
783
783
'phui-document-view-css' => '0f83a7df',
784
784
'phui-feed-story-css' => 'c9f3a0b5',
785
785
'phui-font-icon-base-css' => '3dad2ae3',
786
-
'phui-fontkit-css' => 'd30f4fa3',
786
+
'phui-fontkit-css' => '1fa79503',
787
787
'phui-form-css' => 'f535f938',
788
788
'phui-form-view-css' => '78d729fe',
789
789
'phui-header-view-css' => '083669db',
···
845
845
'029a133d' => array(
846
846
'aphront-dialog-view-css',
847
847
),
848
+
'0324970d' => array(
849
+
'javelin-dom',
850
+
'javelin-util',
851
+
'javelin-stratcom',
852
+
'javelin-install',
853
+
'javelin-workflow',
854
+
'javelin-router',
855
+
'javelin-behavior-device',
856
+
'javelin-vector',
857
+
),
848
858
'03d6ed07' => array(
849
859
'javelin-behavior',
850
860
'javelin-stratcom',
···
1140
1150
'javelin-dom',
1141
1151
'javelin-request',
1142
1152
'javelin-util',
1153
+
),
1154
+
'4a2430d7' => array(
1155
+
'phui-fontkit-css',
1143
1156
),
1144
1157
'4d94d9c3' => array(
1145
1158
'javelin-behavior',
···
1533
1546
'javelin-uri',
1534
1547
'phabricator-notification',
1535
1548
),
1549
+
'9142e483' => array(
1550
+
'javelin-behavior',
1551
+
'javelin-dom',
1552
+
'javelin-stratcom',
1553
+
'javelin-behavior-device',
1554
+
'javelin-scrollbar',
1555
+
'javelin-quicksand',
1556
+
'phabricator-keyboard-shortcut',
1557
+
'conpherence-thread-manager',
1558
+
),
1536
1559
'92eb531d' => array(
1537
1560
'javelin-behavior',
1538
1561
'javelin-dom',
···
1651
1674
'javelin-util',
1652
1675
'phabricator-prefab',
1653
1676
),
1654
-
'aa3b6c22' => array(
1655
-
'javelin-behavior',
1656
-
'javelin-dom',
1657
-
'javelin-stratcom',
1658
-
'javelin-scrollbar',
1659
-
'javelin-quicksand',
1660
-
'phabricator-keyboard-shortcut',
1661
-
'conpherence-thread-manager',
1662
-
),
1663
1677
'b1f0ccee' => array(
1664
1678
'javelin-install',
1665
1679
'javelin-dom',
···
1744
1758
'javelin-dom',
1745
1759
'javelin-vector',
1746
1760
),
1761
+
'c4151295' => array(
1762
+
'javelin-behavior',
1763
+
'javelin-dom',
1764
+
'javelin-util',
1765
+
'javelin-stratcom',
1766
+
'javelin-workflow',
1767
+
'javelin-behavior-device',
1768
+
'javelin-history',
1769
+
'javelin-vector',
1770
+
'phabricator-shaped-request',
1771
+
'conpherence-thread-manager',
1772
+
),
1747
1773
'c51ae228' => array(
1748
1774
'javelin-behavior',
1749
1775
'javelin-util',
···
1847
1873
'javelin-dom',
1848
1874
'javelin-uri',
1849
1875
),
1850
-
'e476c952' => array(
1851
-
'javelin-behavior',
1852
-
'javelin-dom',
1853
-
'javelin-util',
1854
-
'javelin-stratcom',
1855
-
'javelin-workflow',
1856
-
'javelin-behavior-device',
1857
-
'javelin-history',
1858
-
'javelin-vector',
1859
-
'phabricator-shaped-request',
1860
-
'conpherence-thread-manager',
1861
-
),
1862
1876
'e4cc26b3' => array(
1863
1877
'javelin-behavior',
1864
1878
'javelin-dom',
···
1905
1919
'efe49472' => array(
1906
1920
'javelin-install',
1907
1921
'javelin-util',
1908
-
),
1909
-
'efef202b' => array(
1910
-
'javelin-dom',
1911
-
'javelin-util',
1912
-
'javelin-stratcom',
1913
-
'javelin-install',
1914
-
'javelin-workflow',
1915
-
'javelin-router',
1916
-
'javelin-behavior-device',
1917
-
'javelin-vector',
1918
1922
),
1919
1923
'f24f3253' => array(
1920
1924
'javelin-behavior',
+2
-3
src/__phutil_library_map__.php
+2
-3
src/__phutil_library_map__.php
···
1689
1689
'PhabricatorDataNotAttachedException' => 'infrastructure/storage/lisk/PhabricatorDataNotAttachedException.php',
1690
1690
'PhabricatorDatabaseSetupCheck' => 'applications/config/check/PhabricatorDatabaseSetupCheck.php',
1691
1691
'PhabricatorDebugController' => 'applications/system/controller/PhabricatorDebugController.php',
1692
-
'PhabricatorDefaultFileStorageEngineSelector' => 'applications/files/engineselector/PhabricatorDefaultFileStorageEngineSelector.php',
1693
1692
'PhabricatorDefaultSearchEngineSelector' => 'applications/search/selector/PhabricatorDefaultSearchEngineSelector.php',
1694
1693
'PhabricatorDestructibleInterface' => 'applications/system/interface/PhabricatorDestructibleInterface.php',
1695
1694
'PhabricatorDestructionEngine' => 'applications/system/engine/PhabricatorDestructionEngine.php',
···
1813
1812
'PhabricatorFileStorageBlob' => 'applications/files/storage/PhabricatorFileStorageBlob.php',
1814
1813
'PhabricatorFileStorageConfigurationException' => 'applications/files/exception/PhabricatorFileStorageConfigurationException.php',
1815
1814
'PhabricatorFileStorageEngine' => 'applications/files/engine/PhabricatorFileStorageEngine.php',
1816
-
'PhabricatorFileStorageEngineSelector' => 'applications/files/engineselector/PhabricatorFileStorageEngineSelector.php',
1817
1815
'PhabricatorFileTemporaryGarbageCollector' => 'applications/files/garbagecollector/PhabricatorFileTemporaryGarbageCollector.php',
1818
1816
'PhabricatorFileTestCase' => 'applications/files/storage/__tests__/PhabricatorFileTestCase.php',
1819
1817
'PhabricatorFileTestDataGenerator' => 'applications/files/lipsum/PhabricatorFileTestDataGenerator.php',
···
1826
1824
'PhabricatorFileUploadException' => 'applications/files/exception/PhabricatorFileUploadException.php',
1827
1825
'PhabricatorFileinfoSetupCheck' => 'applications/config/check/PhabricatorFileinfoSetupCheck.php',
1828
1826
'PhabricatorFilesApplication' => 'applications/files/application/PhabricatorFilesApplication.php',
1827
+
'PhabricatorFilesApplicationStorageEnginePanel' => 'applications/files/applicationpanel/PhabricatorFilesApplicationStorageEnginePanel.php',
1829
1828
'PhabricatorFilesConfigOptions' => 'applications/files/config/PhabricatorFilesConfigOptions.php',
1830
1829
'PhabricatorFilesManagementCompactWorkflow' => 'applications/files/management/PhabricatorFilesManagementCompactWorkflow.php',
1831
1830
'PhabricatorFilesManagementEnginesWorkflow' => 'applications/files/management/PhabricatorFilesManagementEnginesWorkflow.php',
···
4977
4976
'PhabricatorDataNotAttachedException' => 'Exception',
4978
4977
'PhabricatorDatabaseSetupCheck' => 'PhabricatorSetupCheck',
4979
4978
'PhabricatorDebugController' => 'PhabricatorController',
4980
-
'PhabricatorDefaultFileStorageEngineSelector' => 'PhabricatorFileStorageEngineSelector',
4981
4979
'PhabricatorDefaultSearchEngineSelector' => 'PhabricatorSearchEngineSelector',
4982
4980
'PhabricatorDestructionEngine' => 'Phobject',
4983
4981
'PhabricatorDeveloperConfigOptions' => 'PhabricatorApplicationConfigOptions',
···
5122
5120
'PhabricatorFileUploadException' => 'Exception',
5123
5121
'PhabricatorFileinfoSetupCheck' => 'PhabricatorSetupCheck',
5124
5122
'PhabricatorFilesApplication' => 'PhabricatorApplication',
5123
+
'PhabricatorFilesApplicationStorageEnginePanel' => 'PhabricatorApplicationConfigurationPanel',
5125
5124
'PhabricatorFilesConfigOptions' => 'PhabricatorApplicationConfigOptions',
5126
5125
'PhabricatorFilesManagementCompactWorkflow' => 'PhabricatorFilesManagementWorkflow',
5127
5126
'PhabricatorFilesManagementEnginesWorkflow' => 'PhabricatorFilesManagementWorkflow',
+3
src/applications/config/check/PhabricatorExtraConfigSetupCheck.php
+3
src/applications/config/check/PhabricatorExtraConfigSetupCheck.php
···
211
211
'phd.start-taskmasters' => pht(
212
212
'Taskmasters now use an autoscaling pool. You can configure the '.
213
213
'pool size with `phd.taskmasters`.'),
214
+
'storage.engine-selector' => pht(
215
+
'Phabricator now automatically discovers available storage engines '.
216
+
'at runtime.'),
214
217
);
215
218
216
219
return $ancient_config;
+100
src/applications/files/applicationpanel/PhabricatorFilesApplicationStorageEnginePanel.php
+100
src/applications/files/applicationpanel/PhabricatorFilesApplicationStorageEnginePanel.php
···
1
+
<?php
2
+
3
+
final class PhabricatorFilesApplicationStorageEnginePanel
4
+
extends PhabricatorApplicationConfigurationPanel {
5
+
6
+
public function getPanelKey() {
7
+
return 'storage';
8
+
}
9
+
10
+
public function shouldShowForApplication(
11
+
PhabricatorApplication $application) {
12
+
return ($application instanceof PhabricatorFilesApplication);
13
+
}
14
+
15
+
public function buildConfigurationPagePanel() {
16
+
$viewer = $this->getViewer();
17
+
$application = $this->getApplication();
18
+
19
+
$engines = PhabricatorFileStorageEngine::loadAllEngines();
20
+
$writable_engines = PhabricatorFileStorageEngine::loadWritableEngines();
21
+
22
+
$yes = pht('Yes');
23
+
$no = pht('No');
24
+
25
+
$rows = array();
26
+
$rowc = array();
27
+
foreach ($engines as $key => $engine) {
28
+
$limited = $no;
29
+
$limit = null;
30
+
if ($engine->hasFilesizeLimit()) {
31
+
$limited = $yes;
32
+
$limit = phutil_format_bytes($engine->getFilesizeLimit());
33
+
}
34
+
35
+
if ($engine->canWriteFiles()) {
36
+
$writable = $yes;
37
+
} else {
38
+
$writable = $no;
39
+
}
40
+
41
+
if ($engine->isTestEngine()) {
42
+
$test = $yes;
43
+
} else {
44
+
$test = $no;
45
+
}
46
+
47
+
if (isset($writable_engines[$key])) {
48
+
$rowc[] = 'highlighted';
49
+
} else {
50
+
$rowc[] = null;
51
+
}
52
+
53
+
$rows[] = array(
54
+
$key,
55
+
get_class($engine),
56
+
$test,
57
+
$writable,
58
+
$limited,
59
+
$limit,
60
+
);
61
+
}
62
+
63
+
$table =
64
+
65
+
$table = id(new AphrontTableView($rows))
66
+
->setNoDataString(pht('No storage engines available.'))
67
+
->setHeaders(
68
+
array(
69
+
pht('Key'),
70
+
pht('Class'),
71
+
pht('Unit Test'),
72
+
pht('Writable'),
73
+
pht('Has Limit'),
74
+
pht('Limit'),
75
+
))
76
+
->setRowClasses($rowc)
77
+
->setColumnClasses(
78
+
array(
79
+
'',
80
+
'wide',
81
+
'',
82
+
'',
83
+
'',
84
+
'n',
85
+
));
86
+
87
+
$box = id(new PHUIObjectBoxView())
88
+
->setHeaderText(pht('Storage Engines'))
89
+
->appendChild($table);
90
+
91
+
return $box;
92
+
}
93
+
94
+
public function handlePanelRequest(
95
+
AphrontRequest $request,
96
+
PhabricatorController $controller) {
97
+
return new Aphront404Response();
98
+
}
99
+
100
+
}
-16
src/applications/files/config/PhabricatorFilesConfigOptions.php
-16
src/applications/files/config/PhabricatorFilesConfigOptions.php
···
147
147
"Set this to a valid Amazon S3 bucket to store files there. You ".
148
148
"must also configure S3 access keys in the 'Amazon Web Services' ".
149
149
"group.")),
150
-
$this->newOption(
151
-
'storage.engine-selector',
152
-
'class',
153
-
'PhabricatorDefaultFileStorageEngineSelector')
154
-
->setBaseClass('PhabricatorFileStorageEngineSelector')
155
-
->setSummary(pht('Storage engine selector.'))
156
-
->setDescription(
157
-
pht(
158
-
'Phabricator uses a storage engine selector to choose which '.
159
-
'storage engine to use when writing file data. If you add new '.
160
-
'storage engines or want to provide very custom rules (e.g., '.
161
-
'write images to one storage engine and other files to a '.
162
-
'different one), you can provide an alternate implementation '.
163
-
'here. The default engine will use choose MySQL, Local Disk, and '.
164
-
'S3, in that order, if they have valid configurations above and '.
165
-
'a file fits within configured limits.')),
166
150
$this->newOption('storage.upload-size-limit', 'string', null)
167
151
->setSummary(
168
152
pht('Limit to users in interfaces which allow uploading.'))
+150
src/applications/files/engine/PhabricatorFileStorageEngine.php
+150
src/applications/files/engine/PhabricatorFileStorageEngine.php
···
41
41
abstract public function getEngineIdentifier();
42
42
43
43
44
+
/**
45
+
* Prioritize this engine relative to other engines.
46
+
*
47
+
* Engines with a smaller priority number get an opportunity to write files
48
+
* first. Generally, lower-latency filestores should have lower priority
49
+
* numbers, and higher-latency filestores should have higher priority
50
+
* numbers. Setting priority to approximately the number of milliseconds of
51
+
* read latency will generally produce reasonable results.
52
+
*
53
+
* In conjunction with filesize limits, the goal is to store small files like
54
+
* profile images, thumbnails, and text snippets in lower-latency engines,
55
+
* and store large files in higher-capacity engines.
56
+
*
57
+
* @return float Engine priority.
58
+
* @task meta
59
+
*/
60
+
abstract public function getEnginePriority();
61
+
62
+
63
+
/**
64
+
* Return `true` if the engine is currently writable.
65
+
*
66
+
* Engines that are disabled or missing configuration should return `false`
67
+
* to prevent new writes. If writes were made with this engine in the past,
68
+
* the application may still try to perform reads.
69
+
*
70
+
* @return bool True if this engine can support new writes.
71
+
* @task meta
72
+
*/
73
+
abstract public function canWriteFiles();
74
+
75
+
76
+
/**
77
+
* Return `true` if the engine has a filesize limit on storable files.
78
+
*
79
+
* The @{method:getFilesizeLimit} method can retrieve the actual limit. This
80
+
* method just removes the ambiguity around the meaning of a `0` limit.
81
+
*
82
+
* @return bool `true` if the engine has a filesize limit.
83
+
* @task meta
84
+
*/
85
+
abstract public function hasFilesizeLimit();
86
+
87
+
88
+
/**
89
+
* Return maximum storable file size, in bytes.
90
+
*
91
+
* Not all engines have a limit; use @{method:getFilesizeLimit} to check if
92
+
* an engine has a limit. Engines without a limit can store files of any
93
+
* size.
94
+
*
95
+
* @return int Maximum storable file size, in bytes.
96
+
* @task meta
97
+
*/
98
+
public function getFilesizeLimit() {
99
+
throw new PhutilMethodNotImplementedException();
100
+
}
101
+
102
+
103
+
/**
104
+
* Identifies storage engines that support unit tests.
105
+
*
106
+
* These engines are not used for production writes.
107
+
*
108
+
* @return bool True if this is a test engine.
109
+
* @task meta
110
+
*/
111
+
public function isTestEngine() {
112
+
return false;
113
+
}
114
+
115
+
44
116
/* -( Managing File Data )------------------------------------------------- */
45
117
46
118
···
89
161
* @task file
90
162
*/
91
163
abstract public function deleteFile($handle);
164
+
165
+
166
+
/**
167
+
* Select viable default storage engines according to configuration. We'll
168
+
* select the MySQL and Local Disk storage engines if they are configured
169
+
* to allow a given file.
170
+
*
171
+
* @param int File size in bytes.
172
+
*/
173
+
public static function loadStorageEngines($length) {
174
+
$engines = self::loadWritableEngines();
175
+
176
+
$writable = array();
177
+
foreach ($engines as $key => $engine) {
178
+
if ($engine->hasFilesizeLimit()) {
179
+
$limit = $engine->getFilesizeLimit();
180
+
if ($limit < $length) {
181
+
continue;
182
+
}
183
+
}
184
+
185
+
$writable[$key] = $engine;
186
+
}
187
+
188
+
return $writable;
189
+
}
190
+
191
+
public static function loadAllEngines() {
192
+
static $engines;
193
+
194
+
if ($engines === null) {
195
+
$objects = id(new PhutilSymbolLoader())
196
+
->setAncestorClass(__CLASS__)
197
+
->loadObjects();
198
+
199
+
$map = array();
200
+
foreach ($objects as $engine) {
201
+
$key = $engine->getEngineIdentifier();
202
+
if (empty($map[$key])) {
203
+
$map[$key] = $engine;
204
+
} else {
205
+
throw new Exception(
206
+
pht(
207
+
'Storage engines "%s" and "%s" have the same engine '.
208
+
'identifier "%s". Each storage engine must have a unique '.
209
+
'identifier.',
210
+
get_class($engine),
211
+
get_class($map[$key]),
212
+
$key));
213
+
}
214
+
}
215
+
216
+
$map = msort($map, 'getEnginePriority');
217
+
218
+
$engines = $map;
219
+
}
220
+
221
+
return $engines;
222
+
}
223
+
224
+
public static function loadWritableEngines() {
225
+
$engines = self::loadAllEngines();
226
+
227
+
$writable = array();
228
+
foreach ($engines as $key => $engine) {
229
+
if ($engine->isTestEngine()) {
230
+
continue;
231
+
}
232
+
233
+
if (!$engine->canWriteFiles()) {
234
+
continue;
235
+
}
236
+
237
+
$writable[$key] = $engine;
238
+
}
239
+
240
+
return $writable;
241
+
}
92
242
93
243
}
+18
-3
src/applications/files/engine/PhabricatorLocalDiskFileStorageEngine.php
+18
-3
src/applications/files/engine/PhabricatorLocalDiskFileStorageEngine.php
···
4
4
* Local disk storage engine. Keeps files on local disk. This engine is easy
5
5
* to set up, but it doesn't work if you have multiple web frontends!
6
6
*
7
-
* @task impl Implementation
8
7
* @task internal Internals
9
8
*/
10
9
final class PhabricatorLocalDiskFileStorageEngine
11
10
extends PhabricatorFileStorageEngine {
12
11
13
12
14
-
/* -( Implementation )----------------------------------------------------- */
13
+
/* -( Engine Metadata )---------------------------------------------------- */
15
14
16
15
17
16
/**
18
17
* This engine identifies as "local-disk".
19
-
* @task impl
20
18
*/
21
19
public function getEngineIdentifier() {
22
20
return 'local-disk';
23
21
}
22
+
23
+
public function getEnginePriority() {
24
+
return 5;
25
+
}
26
+
27
+
public function canWriteFiles() {
28
+
$path = PhabricatorEnv::getEnvConfig('storage.local-disk.path');
29
+
return (bool)strlen($path);
30
+
}
31
+
32
+
33
+
public function hasFilesizeLimit() {
34
+
return false;
35
+
}
36
+
37
+
38
+
/* -( Managing File Data )------------------------------------------------- */
24
39
25
40
26
41
/**
+23
-8
src/applications/files/engine/PhabricatorMySQLFileStorageEngine.php
+23
-8
src/applications/files/engine/PhabricatorMySQLFileStorageEngine.php
···
7
7
* It uses the @{class:PhabricatorFileStorageBlob} to actually access the
8
8
* underlying database table.
9
9
*
10
-
* @task impl Implementation
11
10
* @task internal Internals
12
11
*/
13
12
final class PhabricatorMySQLFileStorageEngine
14
13
extends PhabricatorFileStorageEngine {
15
14
16
-
/* -( Implementation )----------------------------------------------------- */
15
+
16
+
/* -( Engine Metadata )---------------------------------------------------- */
17
17
18
18
19
19
/**
20
20
* For historical reasons, this engine identifies as "blob".
21
-
*
22
-
* @task impl
23
21
*/
24
22
public function getEngineIdentifier() {
25
23
return 'blob';
26
24
}
27
25
26
+
public function getEnginePriority() {
27
+
return 1;
28
+
}
29
+
30
+
public function canWriteFiles() {
31
+
return ($this->getFilesizeLimit() > 0);
32
+
}
33
+
34
+
35
+
public function hasFilesizeLimit() {
36
+
return true;
37
+
}
38
+
39
+
40
+
public function getFilesizeLimit() {
41
+
return PhabricatorEnv::getEnvConfig('storage.mysql-engine.max-size');
42
+
}
43
+
44
+
45
+
/* -( Managing File Data )------------------------------------------------- */
46
+
28
47
29
48
/**
30
49
* Write file data into the big blob store table in MySQL. Returns the row
31
50
* ID as the file data handle.
32
-
*
33
-
* @task impl
34
51
*/
35
52
public function writeFile($data, array $params) {
36
53
$blob = new PhabricatorFileStorageBlob();
···
43
60
44
61
/**
45
62
* Load a stored blob from MySQL.
46
-
* @task impl
47
63
*/
48
64
public function readFile($handle) {
49
65
return $this->loadFromMySQLFileStorage($handle)->getData();
···
52
68
53
69
/**
54
70
* Delete a blob from MySQL.
55
-
* @task impl
56
71
*/
57
72
public function deleteFile($handle) {
58
73
$this->loadFromMySQLFileStorage($handle)->delete();
+21
-1
src/applications/files/engine/PhabricatorS3FileStorageEngine.php
+21
-1
src/applications/files/engine/PhabricatorS3FileStorageEngine.php
···
10
10
extends PhabricatorFileStorageEngine {
11
11
12
12
13
-
/* -( Implementation )----------------------------------------------------- */
13
+
/* -( Engine Metadata )---------------------------------------------------- */
14
14
15
15
16
16
/**
···
19
19
public function getEngineIdentifier() {
20
20
return 'amazon-s3';
21
21
}
22
+
23
+
public function getEnginePriority() {
24
+
return 100;
25
+
}
26
+
27
+
public function canWriteFiles() {
28
+
$bucket = PhabricatorEnv::getEnvConfig('storage.s3.bucket');
29
+
$access_key = PhabricatorEnv::getEnvConfig('amazon-s3.access-key');
30
+
$secret_key = PhabricatorEnv::getEnvConfig('amazon-s3.secret-key');
31
+
32
+
return (strlen($bucket) && strlen($access_key) && strlen($secret_key));
33
+
}
34
+
35
+
36
+
public function hasFilesizeLimit() {
37
+
return false;
38
+
}
39
+
40
+
41
+
/* -( Managing File Data )------------------------------------------------- */
22
42
23
43
24
44
/**
+16
src/applications/files/engine/PhabricatorTestStorageEngine.php
+16
src/applications/files/engine/PhabricatorTestStorageEngine.php
···
13
13
return 'unit-test';
14
14
}
15
15
16
+
public function getEnginePriority() {
17
+
return 1000;
18
+
}
19
+
20
+
public function isTestEngine() {
21
+
return true;
22
+
}
23
+
24
+
public function canWriteFiles() {
25
+
return true;
26
+
}
27
+
28
+
public function hasFilesizeLimit() {
29
+
return false;
30
+
}
31
+
16
32
public function writeFile($data, array $params) {
17
33
AphrontWriteGuard::willWrite();
18
34
self::$storage[self::$nextHandle] = $data;
-52
src/applications/files/engineselector/PhabricatorDefaultFileStorageEngineSelector.php
-52
src/applications/files/engineselector/PhabricatorDefaultFileStorageEngineSelector.php
···
1
-
<?php
2
-
3
-
/**
4
-
* Default storage engine selector. See
5
-
* @{class:PhabricatorFileStorageEngineSelector} and @{article:File Storage
6
-
* Technical Documentation} for more information.
7
-
*/
8
-
final class PhabricatorDefaultFileStorageEngineSelector
9
-
extends PhabricatorFileStorageEngineSelector {
10
-
11
-
/**
12
-
* Select viable default storage engines according to configuration. We'll
13
-
* select the MySQL and Local Disk storage engines if they are configured
14
-
* to allow a given file.
15
-
*/
16
-
public function selectStorageEngines($data, array $params) {
17
-
$length = strlen($data);
18
-
19
-
$mysql_key = 'storage.mysql-engine.max-size';
20
-
$mysql_limit = PhabricatorEnv::getEnvConfig($mysql_key);
21
-
22
-
$engines = array();
23
-
if ($mysql_limit && $length <= $mysql_limit) {
24
-
$engines[] = new PhabricatorMySQLFileStorageEngine();
25
-
}
26
-
27
-
$local_key = 'storage.local-disk.path';
28
-
$local_path = PhabricatorEnv::getEnvConfig($local_key);
29
-
if ($local_path) {
30
-
$engines[] = new PhabricatorLocalDiskFileStorageEngine();
31
-
}
32
-
33
-
$s3_key = 'storage.s3.bucket';
34
-
if (PhabricatorEnv::getEnvConfig($s3_key)) {
35
-
$engines[] = new PhabricatorS3FileStorageEngine();
36
-
}
37
-
38
-
if ($mysql_limit && empty($engines)) {
39
-
// If we return no engines, an exception will be thrown but it will be
40
-
// a little vague ("No valid storage engines"). Since this is a default
41
-
// case, throw a more specific exception.
42
-
throw new Exception(
43
-
'This file exceeds the configured MySQL storage engine filesize '.
44
-
'limit, but no other storage engines are configured. Increase the '.
45
-
'MySQL storage engine limit or configure a storage engine suitable '.
46
-
'for larger files.');
47
-
}
48
-
49
-
return $engines;
50
-
}
51
-
52
-
}
-46
src/applications/files/engineselector/PhabricatorFileStorageEngineSelector.php
-46
src/applications/files/engineselector/PhabricatorFileStorageEngineSelector.php
···
1
-
<?php
2
-
3
-
/**
4
-
* Chooses appropriate storage engine(s) for files. When Phabricator needs
5
-
* to write a blob of file data, it uses the configured selector to get a list
6
-
* of suitable @{class:PhabricatorFileStorageEngine}s. For more information,
7
-
* see @{article:File Storage Technical Documentation}.
8
-
*
9
-
* @task select Selecting Storage Engines
10
-
*/
11
-
abstract class PhabricatorFileStorageEngineSelector {
12
-
13
-
final public function __construct() {
14
-
// <empty>
15
-
}
16
-
17
-
18
-
/* -( Selecting Storage Engines )------------------------------------------ */
19
-
20
-
21
-
/**
22
-
* Select valid storage engines for a file. This method will be called by
23
-
* Phabricator when it needs to store a file permanently. It must return a
24
-
* list of valid @{class:PhabricatorFileStorageEngine}s.
25
-
*
26
-
* If you are extending this class to provide a custom selector, you
27
-
* probably just want it to look like this:
28
-
*
29
-
* return array(new MyCustomFileStorageEngine());
30
-
*
31
-
* ...that is, store every file in whatever storage engine you're using.
32
-
* However, you can also provide multiple storage engines, or store some files
33
-
* in one engine and some files in a different engine by implementing a more
34
-
* complex selector.
35
-
*
36
-
* @param string File data.
37
-
* @param dict Dictionary of optional file metadata. This may be empty, or
38
-
* have some additional keys like 'file' and 'author' which
39
-
* provide metadata.
40
-
* @return list List of @{class:PhabricatorFileStorageEngine}s, ordered by
41
-
* preference.
42
-
* @task select
43
-
*/
44
-
abstract public function selectStorageEngines($data, array $params);
45
-
46
-
}
+11
-4
src/applications/files/storage/PhabricatorFile.php
+11
-4
src/applications/files/storage/PhabricatorFile.php
···
269
269
if (isset($params['storageEngines'])) {
270
270
$engines = $params['storageEngines'];
271
271
} else {
272
-
$selector = PhabricatorEnv::newObjectFromConfig(
273
-
'storage.engine-selector');
274
-
$engines = $selector->selectStorageEngines($data, $params);
272
+
$size = strlen($data);
273
+
$engines = PhabricatorFileStorageEngine::loadStorageEngines($size);
274
+
275
+
if (!$engines) {
276
+
throw new Exception(
277
+
pht(
278
+
'No configured storage engine can store this file. See '.
279
+
'"Configuring File Storage" in the documentation for '.
280
+
'information on configuring storage engines.'));
281
+
}
275
282
}
276
283
277
284
assert_instances_of($engines, 'PhabricatorFileStorageEngine');
278
285
if (!$engines) {
279
-
throw new Exception('No valid storage engines are available!');
286
+
throw new Exception(pht('No valid storage engines are available!'));
280
287
}
281
288
282
289
$file = PhabricatorFile::initializeNewFile();
-39
src/docs/tech/files.diviner
-39
src/docs/tech/files.diviner
···
1
-
@title File Storage Technical Documentation
2
-
@group filestorage
3
-
4
-
Phabricator file storage details.
5
-
6
-
= Overview =
7
-
8
-
Phabricator has a simple, general-purpose file storage system with configurable
9
-
storage backends that allows you to choose where files are stored. For a user
10
-
guide, see @{article:Configuring File Storage}.
11
-
12
-
= Class Relationships =
13
-
14
-
@{class:PhabricatorFile} holds file metadata (name, author, phid), including an
15
-
identifier for a @{class:PhabricatorFileStorageEngine} where the actual file
16
-
data is stored, and a data handle which identifies the data within that storage
17
-
engine.
18
-
19
-
When writing data, a @{class:PhabricatorFileStorageEngineSelector} is
20
-
instantiated (by default, @{class:PhabricatorDefaultFileStorageEngineSelector},
21
-
but you can change this by setting the ##storage.engine-selector## key in your
22
-
configuration). The selector returns a list of satisfactory
23
-
@{class:PhabricatorFileStorageEngine}s, in order of preference.
24
-
25
-
For instance, suppose the user is uploading a picture. The upload pipeline would
26
-
instantiate the configured selector, which might return a
27
-
@{class:PhabricatorMySQLFileStorageEngine} and a
28
-
@{class:PhabricatorLocalDiskFileStorageEngine}, indicating that the picture may
29
-
be stored in either storage engine but MySQL is preferred. If a given storage
30
-
engine fails to perform the write, it will fall back to the next engine.
31
-
32
-
= Adding New Storage Engines =
33
-
34
-
To add a new storage engine, extend @{class:PhabricatorFileStorageEngine}. In
35
-
order to make files actually get written to it, you also need to extend
36
-
@{class:PhabricatorFileStorageEngineSelector}, provide an implementation which
37
-
selects your storage engine for whatever files you want to store there, and then
38
-
configure Phabricator to use your selector by setting
39
-
`storage.engine-selector`.
+26
-27
src/docs/user/configuration/configuring_file_storage.diviner
+26
-27
src/docs/user/configuration/configuring_file_storage.diviner
···
3
3
4
4
Setup how Phabricator will store files.
5
5
6
-
= Overview =
6
+
Overview
7
+
========
7
8
8
9
Phabricator allows users to upload files, and several applications use file
9
10
storage (for instance, Maniphest allows you to attach files to tasks). You can
10
-
configure several different storage systems:
11
+
configure several different storage systems.
11
12
12
-
- you can store data in MySQL: this is the easiest to set up, but doesn't
13
-
scale well;
14
-
- you can store data on local disk: this is also easy to set up but won't
15
-
scale to multiple web frontends without NFS;
16
-
- or you can build a custom storage engine.
13
+
| System | Setup | Cost | Notes |
14
+
|========|=======|======|=======|
15
+
| MySQL | Automatic | Free | May not scale well. |
16
+
| Local Disk | Easy | Free | Does not scale well. |
17
+
| Amazon S3 | Easy | Cheap | Scales well. |
18
+
| Custom | Hard | Varies | Implement a custom storage engine. |
17
19
18
20
By default, Phabricator is configured to store files up to 1MB in MySQL, and
19
-
reject files larger than 1MB. It is recommended you set up local disk storage
20
-
for files larger than 1MB. This should be sufficient for most installs. If you
21
-
have a larger install or more unique requirements, you may want to customize
22
-
this further.
21
+
reject files larger than 1MB. To store larger files, you can either:
22
+
23
+
- configure local disk storage; or
24
+
- configure Amazon S3 storage; or
25
+
- raise the limits on MySQL.
23
26
24
-
For technical documentation (including instructions on building custom storage
25
-
engines) see @{article:File Storage Technical Documentation}.
27
+
See the rest of this document for some additional discussion of engines.
26
28
27
29
You don't have to fully configure this immediately, the defaults are okay until
28
30
you need to upload larger files and it's relatively easy to port files between
29
31
storage engines later.
30
32
31
-
= Storage Engines =
33
+
Storage Engines
34
+
===============
32
35
33
36
Builtin storage engines and information on how to configure them.
34
37
···
41
44
MySQL storage is configured by default, for files up to (just under) 1MB. You
42
45
can configure it with these keys:
43
46
44
-
- ##storage.mysql-engine.max-size##: Change the filesize limit. Set to 0
47
+
- `storage.mysql-engine.max-size`: Change the filesize limit. Set to 0
45
48
to disable.
46
49
47
-
For most installs, it is recommended you configure local disk storage below,
48
-
and then either leave this as is or disable it, depending on how upset you feel
49
-
about putting files in a database.
50
+
For most installs, it is reasonable to leave this engine as-is and let small
51
+
files (like thumbnails and profile images) be stored in MySQL, which is usually
52
+
the lowest-latency filestore.
53
+
54
+
To support larger files, configure another engine or increase this limit.
50
55
51
56
== Local Disk ==
52
57
53
58
- **Pros**: Very simple. Almost no setup required.
54
59
- **Cons**: Doesn't scale to multiple web frontends without NFS.
55
60
56
-
For most installs, it is **strongly recommended** that you configure local disk
57
-
storage. To do this, set the configuration key:
61
+
To upload larger files:
58
62
59
-
- ##storage.local-disk.path##: Set to some writable directory on local disk.
60
-
Make that directory. You're done.
63
+
- `storage.local-disk.path`: Set to some writable directory on local disk.
64
+
Make that directory.
61
65
62
66
== Amazon S3 ==
63
67
···
69
73
- ##amazon-s3.access-key## Your AWS access key.
70
74
- ##amazon-s3.secret-key## Your AWS secret key.
71
75
- ##storage.s3.bucket## S3 bucket name where files should be stored.
72
-
73
-
== Custom Engine ==
74
-
75
-
For details about writing a custom storage engine, see @{article:File Storage
76
-
Technical Documentation}.
77
76
78
77
= Testing Storage Engines =
79
78