@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Add "Autoplans" to Harbormaster

Summary:
Ref T8095. Two general problems:

- I want Harbormaster to own all lint and unit test results.
- I don't want users to have to configure anything for `arc` to keep working automatically.

These are in conflict because generic lint/unit test ownership in Harbormaster requires that build targets exist which we can attach build results to. However, we can't currently create build targets on demand: Harbormaster assumes it is responsible for creating targets, then running code or making third-party service calls to actually run the builds.

I considered two broad approaches to let `arc` push results into Harbormaster without requiring administrators to configure some kind of "arc results" build plan:

# Add magic target PHIDs like `PHID-MAGIC-this-is-really-arc-unit`.
# Add new code to build real targets with real PHIDs.

(1) is probably a bit less work to get off the ground, but I think it's worse overall and very likely to create more problems in the long run. I particularly worry that it will lead to a small amount of special casing in a very large number of places, which seems more fragile.

(2) is more work upfront but I think does a better job of putting all the special casing in one place that we can, e.g., more reasonably unit test, and letting the rest of the code rarely/never care about this case since it's just dealing with normal plans/steps/targets as far as it can tell.

This diff introduces "autoplans", which are source templates for plans/steps. This let us "push" these targets into Harbormaster. Hypthetically, any process "like" arc can use autoplans to upload test/lint/etc results. In practice, probably only `arc` will ever use this, but I think it's still quite a bit cleaner than the alternative despite all the generality.

Workflow is basically:

- `arc` creates a diff.
- `arc` calls `harbormaster.queryautotargets`, passing the diff PHID and saying "I have some lint and unit results I want to stick on this thing".
- Harbormaster builds the plan, steps, and targets (if any of them don't already exist), and hands back the target PHIDs so `arc` has a completely standard-looking place to put results.
- `arc` uploads the test results to the right targets, as though Harbormaster had asked it to run unit/lint in the first place.

(This doesn't actually do any of that yet, just sets things up.)

I'll maybe doc turn that ^^^^^^ into a doc for posterity since I think it's hard to guess what an "autotarget" is, but I'm going to grab some lunch first.

Test Plan:
- Added unit tests to make sure we can build these things properly.
- Used `harbormaster.queryautotargets` to build autotargets for a bunch of diffs.
- Verified targets come up in "waiting for message" state.
- Verified plans and steps are not editable.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: hach-que, epriestley

Maniphest Tasks: T8095

Differential Revision: https://secure.phabricator.com/D13345

+733 -15
+2
resources/sql/autopatches/20150618.harbor.1.planauto.sql
··· 1 + ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_buildplan 2 + ADD planAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};
+2
resources/sql/autopatches/20150618.harbor.2.stepauto.sql
··· 1 + ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_buildstep 2 + ADD stepAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};
+2
resources/sql/autopatches/20150618.harbor.3.buildauto.sql
··· 1 + ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_build 2 + ADD planAutoKey VARCHAR(32) COLLATE {$COLLATE_TEXT};
+16
src/__phutil_library_map__.php
··· 357 357 'DifferentialDiffTableOfContentsView' => 'applications/differential/view/DifferentialDiffTableOfContentsView.php', 358 358 'DifferentialDiffTestCase' => 'applications/differential/storage/__tests__/DifferentialDiffTestCase.php', 359 359 'DifferentialDiffTransaction' => 'applications/differential/storage/DifferentialDiffTransaction.php', 360 + 'DifferentialDiffTransactionQuery' => 'applications/differential/query/DifferentialDiffTransactionQuery.php', 360 361 'DifferentialDiffViewController' => 'applications/differential/controller/DifferentialDiffViewController.php', 361 362 'DifferentialDoorkeeperRevisionFeedStoryPublisher' => 'applications/differential/doorkeeper/DifferentialDoorkeeperRevisionFeedStoryPublisher.php', 362 363 'DifferentialDraft' => 'applications/differential/storage/DifferentialDraft.php', ··· 823 824 'FundInitiativeTransactionQuery' => 'applications/fund/query/FundInitiativeTransactionQuery.php', 824 825 'FundInitiativeViewController' => 'applications/fund/controller/FundInitiativeViewController.php', 825 826 'FundSchemaSpec' => 'applications/fund/storage/FundSchemaSpec.php', 827 + 'HarbormasterArcLintBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterArcLintBuildStepImplementation.php', 828 + 'HarbormasterArcUnitBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterArcUnitBuildStepImplementation.php', 829 + 'HarbormasterAutotargetsTestCase' => 'applications/harbormaster/__tests__/HarbormasterAutotargetsTestCase.php', 826 830 'HarbormasterBuild' => 'applications/harbormaster/storage/build/HarbormasterBuild.php', 827 831 'HarbormasterBuildAbortedException' => 'applications/harbormaster/exception/HarbormasterBuildAbortedException.php', 828 832 'HarbormasterBuildActionController' => 'applications/harbormaster/controller/HarbormasterBuildActionController.php', 833 + 'HarbormasterBuildArcanistAutoplan' => 'applications/harbormaster/autoplan/HarbormasterBuildArcanistAutoplan.php', 829 834 'HarbormasterBuildArtifact' => 'applications/harbormaster/storage/build/HarbormasterBuildArtifact.php', 830 835 'HarbormasterBuildArtifactQuery' => 'applications/harbormaster/query/HarbormasterBuildArtifactQuery.php', 836 + 'HarbormasterBuildAutoplan' => 'applications/harbormaster/autoplan/HarbormasterBuildAutoplan.php', 831 837 'HarbormasterBuildCommand' => 'applications/harbormaster/storage/HarbormasterBuildCommand.php', 832 838 'HarbormasterBuildDependencyDatasource' => 'applications/harbormaster/typeahead/HarbormasterBuildDependencyDatasource.php', 833 839 'HarbormasterBuildEngine' => 'applications/harbormaster/engine/HarbormasterBuildEngine.php', ··· 897 903 'HarbormasterPlanRunController' => 'applications/harbormaster/controller/HarbormasterPlanRunController.php', 898 904 'HarbormasterPlanViewController' => 'applications/harbormaster/controller/HarbormasterPlanViewController.php', 899 905 'HarbormasterPublishFragmentBuildStepImplementation' => 'applications/harbormaster/step/HarbormasterPublishFragmentBuildStepImplementation.php', 906 + 'HarbormasterQueryAutotargetsConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryAutotargetsConduitAPIMethod.php', 900 907 'HarbormasterQueryBuildablesConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryBuildablesConduitAPIMethod.php', 901 908 'HarbormasterQueryBuildsConduitAPIMethod' => 'applications/harbormaster/conduit/HarbormasterQueryBuildsConduitAPIMethod.php', 902 909 'HarbormasterRemarkupRule' => 'applications/harbormaster/remarkup/HarbormasterRemarkupRule.php', ··· 907 914 'HarbormasterStepAddController' => 'applications/harbormaster/controller/HarbormasterStepAddController.php', 908 915 'HarbormasterStepDeleteController' => 'applications/harbormaster/controller/HarbormasterStepDeleteController.php', 909 916 'HarbormasterStepEditController' => 'applications/harbormaster/controller/HarbormasterStepEditController.php', 917 + 'HarbormasterTargetEngine' => 'applications/harbormaster/engine/HarbormasterTargetEngine.php', 910 918 'HarbormasterTargetWorker' => 'applications/harbormaster/worker/HarbormasterTargetWorker.php', 911 919 'HarbormasterThrowExceptionBuildStep' => 'applications/harbormaster/step/HarbormasterThrowExceptionBuildStep.php', 912 920 'HarbormasterUIEventListener' => 'applications/harbormaster/event/HarbormasterUIEventListener.php', ··· 3704 3712 'DifferentialDiffTableOfContentsView' => 'AphrontView', 3705 3713 'DifferentialDiffTestCase' => 'PhutilTestCase', 3706 3714 'DifferentialDiffTransaction' => 'PhabricatorApplicationTransaction', 3715 + 'DifferentialDiffTransactionQuery' => 'PhabricatorApplicationTransactionQuery', 3707 3716 'DifferentialDiffViewController' => 'DifferentialController', 3708 3717 'DifferentialDoorkeeperRevisionFeedStoryPublisher' => 'DoorkeeperFeedStoryPublisher', 3709 3718 'DifferentialDraft' => 'DifferentialDAO', ··· 4235 4244 'FundInitiativeTransactionQuery' => 'PhabricatorApplicationTransactionQuery', 4236 4245 'FundInitiativeViewController' => 'FundController', 4237 4246 'FundSchemaSpec' => 'PhabricatorConfigSchemaSpec', 4247 + 'HarbormasterArcLintBuildStepImplementation' => 'HarbormasterBuildStepImplementation', 4248 + 'HarbormasterArcUnitBuildStepImplementation' => 'HarbormasterBuildStepImplementation', 4249 + 'HarbormasterAutotargetsTestCase' => 'PhabricatorTestCase', 4238 4250 'HarbormasterBuild' => array( 4239 4251 'HarbormasterDAO', 4240 4252 'PhabricatorApplicationTransactionInterface', ··· 4242 4254 ), 4243 4255 'HarbormasterBuildAbortedException' => 'Exception', 4244 4256 'HarbormasterBuildActionController' => 'HarbormasterController', 4257 + 'HarbormasterBuildArcanistAutoplan' => 'HarbormasterBuildAutoplan', 4245 4258 'HarbormasterBuildArtifact' => array( 4246 4259 'HarbormasterDAO', 4247 4260 'PhabricatorPolicyInterface', 4248 4261 ), 4249 4262 'HarbormasterBuildArtifactQuery' => 'PhabricatorCursorPagedPolicyAwareQuery', 4263 + 'HarbormasterBuildAutoplan' => 'Phobject', 4250 4264 'HarbormasterBuildCommand' => 'HarbormasterDAO', 4251 4265 'HarbormasterBuildDependencyDatasource' => 'PhabricatorTypeaheadDatasource', 4252 4266 'HarbormasterBuildEngine' => 'Phobject', ··· 4342 4356 'HarbormasterPlanRunController' => 'HarbormasterController', 4343 4357 'HarbormasterPlanViewController' => 'HarbormasterPlanController', 4344 4358 'HarbormasterPublishFragmentBuildStepImplementation' => 'HarbormasterBuildStepImplementation', 4359 + 'HarbormasterQueryAutotargetsConduitAPIMethod' => 'HarbormasterConduitAPIMethod', 4345 4360 'HarbormasterQueryBuildablesConduitAPIMethod' => 'HarbormasterConduitAPIMethod', 4346 4361 'HarbormasterQueryBuildsConduitAPIMethod' => 'HarbormasterConduitAPIMethod', 4347 4362 'HarbormasterRemarkupRule' => 'PhabricatorObjectRemarkupRule', ··· 4352 4367 'HarbormasterStepAddController' => 'HarbormasterController', 4353 4368 'HarbormasterStepDeleteController' => 'HarbormasterController', 4354 4369 'HarbormasterStepEditController' => 'HarbormasterController', 4370 + 'HarbormasterTargetEngine' => 'Phobject', 4355 4371 'HarbormasterTargetWorker' => 'HarbormasterWorker', 4356 4372 'HarbormasterThrowExceptionBuildStep' => 'HarbormasterBuildStepImplementation', 4357 4373 'HarbormasterUIEventListener' => 'PhabricatorEventListener',
+10
src/applications/differential/query/DifferentialDiffTransactionQuery.php
··· 1 + <?php 2 + 3 + final class DifferentialDiffTransactionQuery 4 + extends PhabricatorApplicationTransactionQuery { 5 + 6 + public function getTemplateApplicationTransaction() { 7 + return new DifferentialDiffTransaction(); 8 + } 9 + 10 + }
+9 -7
src/applications/differential/storage/DifferentialDiff.php
··· 381 381 $results = array(); 382 382 383 383 $results['buildable.diff'] = $this->getID(); 384 - $revision = $this->getRevision(); 385 - $results['buildable.revision'] = $revision->getID(); 386 - $repo = $revision->getRepository(); 384 + if ($this->revisionID) { 385 + $revision = $this->getRevision(); 386 + $results['buildable.revision'] = $revision->getID(); 387 + $repo = $revision->getRepository(); 387 388 388 - if ($repo) { 389 - $results['repository.callsign'] = $repo->getCallsign(); 390 - $results['repository.vcs'] = $repo->getVersionControlSystem(); 391 - $results['repository.uri'] = $repo->getPublicCloneURI(); 389 + if ($repo) { 390 + $results['repository.callsign'] = $repo->getCallsign(); 391 + $results['repository.vcs'] = $repo->getVersionControlSystem(); 392 + $results['repository.uri'] = $repo->getPublicCloneURI(); 393 + } 392 394 } 393 395 394 396 return $results;
+61
src/applications/harbormaster/__tests__/HarbormasterAutotargetsTestCase.php
··· 1 + <?php 2 + 3 + final class HarbormasterAutotargetsTestCase extends PhabricatorTestCase { 4 + 5 + protected function getPhabricatorTestCaseConfiguration() { 6 + return array( 7 + self::PHABRICATOR_TESTCONFIG_BUILD_STORAGE_FIXTURES => true, 8 + ); 9 + } 10 + 11 + public function testGenerateHarbormasterAutotargets() { 12 + $viewer = $this->generateNewTestUser(); 13 + 14 + $raw_diff = <<<EODIFF 15 + diff --git a/fruit b/fruit 16 + new file mode 100644 17 + index 0000000..1c0f49d 18 + --- /dev/null 19 + +++ b/fruit 20 + @@ -0,0 +1,2 @@ 21 + +apal 22 + +banan 23 + EODIFF; 24 + 25 + $parser = new ArcanistDiffParser(); 26 + $changes = $parser->parseDiff($raw_diff); 27 + 28 + $diff = DifferentialDiff::newFromRawChanges($viewer, $changes) 29 + ->setLintStatus(DifferentialLintStatus::LINT_AUTO_SKIP) 30 + ->setUnitStatus(DifferentialUnitStatus::UNIT_AUTO_SKIP) 31 + ->attachRevision(null) 32 + ->save(); 33 + 34 + $params = array( 35 + 'objectPHID' => $diff->getPHID(), 36 + 'targetKeys' => array( 37 + HarbormasterArcLintBuildStepImplementation::STEPKEY, 38 + HarbormasterArcUnitBuildStepImplementation::STEPKEY, 39 + ), 40 + ); 41 + 42 + // Creation of autotargets should work from an empty state. 43 + $result = id(new ConduitCall('harbormaster.queryautotargets', $params)) 44 + ->setUser($viewer) 45 + ->execute(); 46 + 47 + $targets = idx($result, 'targetMap'); 48 + foreach ($params['targetKeys'] as $target_key) { 49 + $this->assertTrue((bool)$result['targetMap'][$target_key]); 50 + } 51 + 52 + // Querying the same autotargets again should produce the same results, 53 + // not make new ones. 54 + $retry = id(new ConduitCall('harbormaster.queryautotargets', $params)) 55 + ->setUser($viewer) 56 + ->execute(); 57 + 58 + $this->assertEqual($result, $retry); 59 + } 60 + 61 + }
+16
src/applications/harbormaster/autoplan/HarbormasterBuildArcanistAutoplan.php
··· 1 + <?php 2 + 3 + final class HarbormasterBuildArcanistAutoplan 4 + extends HarbormasterBuildAutoplan { 5 + 6 + const PLANKEY = 'arcanist'; 7 + 8 + public function getAutoplanPlanKey() { 9 + return self::PLANKEY; 10 + } 11 + 12 + public function getAutoplanName() { 13 + return pht('Arcanist Client Results'); 14 + } 15 + 16 + }
+44
src/applications/harbormaster/autoplan/HarbormasterBuildAutoplan.php
··· 1 + <?php 2 + 3 + abstract class HarbormasterBuildAutoplan extends Phobject { 4 + 5 + abstract public function getAutoplanPlanKey(); 6 + abstract public function getAutoplanName(); 7 + 8 + public static function getAutoplan($key) { 9 + return idx(self::getAllAutoplans(), $key); 10 + } 11 + 12 + public static function getAllAutoplans() { 13 + static $plans; 14 + 15 + if ($plans === null) { 16 + $objects = id(new PhutilSymbolLoader()) 17 + ->setAncestorClass(__CLASS__) 18 + ->loadObjects(); 19 + 20 + $map = array(); 21 + foreach ($objects as $object) { 22 + $key = $object->getAutoplanPlanKey(); 23 + if (!empty($map[$key])) { 24 + $other = $map[$key]; 25 + throw new Exception( 26 + pht( 27 + 'Two build autoplans (of classes "%s" and "%s") define the same '. 28 + 'key ("%s"). Each autoplan must have a unique key.', 29 + get_class($other), 30 + get_class($object), 31 + $key)); 32 + } 33 + $map[$key] = $object; 34 + } 35 + 36 + ksort($map); 37 + 38 + $plans = $map; 39 + } 40 + 41 + return $plans; 42 + } 43 + 44 + }
+76
src/applications/harbormaster/conduit/HarbormasterQueryAutotargetsConduitAPIMethod.php
··· 1 + <?php 2 + 3 + final class HarbormasterQueryAutotargetsConduitAPIMethod 4 + extends HarbormasterConduitAPIMethod { 5 + 6 + public function getAPIMethodName() { 7 + return 'harbormaster.queryautotargets'; 8 + } 9 + 10 + public function getMethodDescription() { 11 + return pht('Load or create build autotargets.'); 12 + } 13 + 14 + protected function defineParamTypes() { 15 + return array( 16 + 'objectPHID' => 'phid', 17 + 'targetKeys' => 'list<string>', 18 + ); 19 + } 20 + 21 + protected function defineReturnType() { 22 + return 'map<string, phid>'; 23 + } 24 + 25 + protected function execute(ConduitAPIRequest $request) { 26 + $viewer = $request->getUser(); 27 + 28 + $phid = $request->getValue('objectPHID'); 29 + 30 + // NOTE: We use withNames() to let monograms like "D123" work, which makes 31 + // this a little easier to test. Real PHIDs will still work as expected. 32 + 33 + $object = id(new PhabricatorObjectQuery()) 34 + ->setViewer($viewer) 35 + ->withNames(array($phid)) 36 + ->executeOne(); 37 + if (!$object) { 38 + throw new Exception( 39 + pht( 40 + 'No such object "%s" exists.', 41 + $phid)); 42 + } 43 + 44 + if (!($object instanceof HarbormasterBuildableInterface)) { 45 + throw new Exception( 46 + pht( 47 + 'Object "%s" does not implement interface "%s". Autotargets may '. 48 + 'only be queried for buildable objects.', 49 + $phid, 50 + 'HarbormasterBuildableInterface')); 51 + } 52 + 53 + $autotargets = $request->getValue('targetKeys', array()); 54 + 55 + if ($autotargets) { 56 + $targets = id(new HarbormasterTargetEngine()) 57 + ->setViewer($viewer) 58 + ->setObject($object) 59 + ->setAutoTargetKeys($autotargets) 60 + ->buildTargets(); 61 + } else { 62 + $targets = array(); 63 + } 64 + 65 + // Reorder the results according to the request order so we can make test 66 + // assertions that subsequent calls return the same results. 67 + 68 + $map = mpull($targets, 'getPHID'); 69 + $map = array_select_keys($map, $autotargets); 70 + 71 + return array( 72 + 'targetMap' => $map, 73 + ); 74 + } 75 + 76 + }
+8 -2
src/applications/harbormaster/controller/HarbormasterStepAddController.php
··· 24 24 $plan_id = $plan->getID(); 25 25 $cancel_uri = $this->getApplicationURI("plan/{$plan_id}/"); 26 26 27 + $all = HarbormasterBuildStepImplementation::getImplementations(); 28 + foreach ($all as $key => $impl) { 29 + if ($impl->shouldRequireAutotargeting()) { 30 + unset($all[$key]); 31 + } 32 + } 33 + 27 34 $errors = array(); 28 35 if ($request->isFormPost()) { 29 36 $class = $request->getStr('class'); 30 - if (!HarbormasterBuildStepImplementation::getImplementation($class)) { 37 + if (empty($all[$class])) { 31 38 $errors[] = pht('Choose the type of build step you want to add.'); 32 39 } 33 40 if (!$errors) { ··· 39 46 $control = id(new AphrontFormRadioButtonControl()) 40 47 ->setName('class'); 41 48 42 - $all = HarbormasterBuildStepImplementation::getImplementations(); 43 49 foreach ($all as $class => $implementation) { 44 50 $control->addButton( 45 51 $class,
+5
src/applications/harbormaster/controller/HarbormasterStepEditController.php
··· 47 47 return new Aphront404Response(); 48 48 } 49 49 50 + if ($impl->shouldRequireAutotargeting()) { 51 + // No manual creation of autotarget steps. 52 + return new Aphront404Response(); 53 + } 54 + 50 55 $step = HarbormasterBuildStep::initializeNewStep($viewer) 51 56 ->setBuildPlanPHID($plan->getPHID()) 52 57 ->setClassName($class);
+251
src/applications/harbormaster/engine/HarbormasterTargetEngine.php
··· 1 + <?php 2 + 3 + final class HarbormasterTargetEngine extends Phobject { 4 + 5 + private $viewer; 6 + private $object; 7 + private $autoTargetKeys; 8 + 9 + public function setViewer($viewer) { 10 + $this->viewer = $viewer; 11 + return $this; 12 + } 13 + 14 + public function getViewer() { 15 + return $this->viewer; 16 + } 17 + 18 + public function setObject(HarbormasterBuildableInterface $object) { 19 + $this->object = $object; 20 + return $this; 21 + } 22 + 23 + public function getObject() { 24 + return $this->object; 25 + } 26 + 27 + public function setAutoTargetKeys(array $auto_keys) { 28 + $this->autoTargetKeys = $auto_keys; 29 + return $this; 30 + } 31 + 32 + public function getAutoTargetKeys() { 33 + return $this->autoTargetKeys; 34 + } 35 + 36 + public function buildTargets() { 37 + $object = $this->getObject(); 38 + $viewer = $this->getViewer(); 39 + 40 + $step_map = $this->generateBuildStepMap($this->getAutoTargetKeys()); 41 + 42 + $buildable = HarbormasterBuildable::createOrLoadExisting( 43 + $viewer, 44 + $object->getHarbormasterBuildablePHID(), 45 + $object->getHarbormasterContainerPHID()); 46 + 47 + $target_map = $this->generateBuildTargetMap($buildable, $step_map); 48 + 49 + return $target_map; 50 + } 51 + 52 + 53 + /** 54 + * Get a map of the @{class:HarbormasterBuildStep} objects for a list of 55 + * autotarget keys. 56 + * 57 + * This method creates the steps if they do not yet exist. 58 + * 59 + * @param list<string> Autotarget keys, like `"core.arc.lint"`. 60 + * @return map<string, object> Map of keys to step objects. 61 + */ 62 + private function generateBuildStepMap(array $autotargets) { 63 + $viewer = $this->getViewer(); 64 + 65 + $autosteps = $this->getAutosteps($autotargets); 66 + $autosteps = mgroup($autosteps, 'getBuildStepAutotargetPlanKey'); 67 + 68 + $plans = id(new HarbormasterBuildPlanQuery()) 69 + ->setViewer($viewer) 70 + ->withPlanAutoKeys(array_keys($autosteps)) 71 + ->needBuildSteps(true) 72 + ->execute(); 73 + $plans = mpull($plans, null, 'getPlanAutoKey'); 74 + 75 + // NOTE: When creating the plan and steps, we save the autokeys as the 76 + // names. These won't actually be shown in the UI, but make the data more 77 + // consistent for secondary consumers like typeaheads. 78 + 79 + $step_map = array(); 80 + foreach ($autosteps as $plan_key => $steps) { 81 + $plan = idx($plans, $plan_key); 82 + if (!$plan) { 83 + $plan = HarbormasterBuildPlan::initializeNewBuildPlan($viewer) 84 + ->setName($plan_key) 85 + ->setPlanAutoKey($plan_key); 86 + } 87 + 88 + $current = $plan->getBuildSteps(); 89 + $current = mpull($current, null, 'getStepAutoKey'); 90 + $new_steps = array(); 91 + 92 + foreach ($steps as $step_key => $step) { 93 + if (isset($current[$step_key])) { 94 + $step_map[$step_key] = $current[$step_key]; 95 + continue; 96 + } 97 + 98 + $new_step = HarbormasterBuildStep::initializeNewStep($viewer) 99 + ->setName($step_key) 100 + ->setClassName(get_class($step)) 101 + ->setStepAutoKey($step_key); 102 + 103 + $new_steps[$step_key] = $new_step; 104 + } 105 + 106 + if ($new_steps) { 107 + $plan->openTransaction(); 108 + if (!$plan->getPHID()) { 109 + $plan->save(); 110 + } 111 + foreach ($new_steps as $step_key => $step) { 112 + $step->setBuildPlanPHID($plan->getPHID()); 113 + $step->save(); 114 + 115 + $step->attachBuildPlan($plan); 116 + $step_map[$step_key] = $step; 117 + } 118 + $plan->saveTransaction(); 119 + } 120 + } 121 + 122 + return $step_map; 123 + } 124 + 125 + 126 + /** 127 + * Get all of the @{class:HarbormasterBuildStepImplementation} objects for 128 + * a list of autotarget keys. 129 + * 130 + * @param list<string> Autotarget keys, like `"core.arc.lint"`. 131 + * @return map<string, object> Map of keys to implementations. 132 + */ 133 + private function getAutosteps(array $autotargets) { 134 + $all_steps = HarbormasterBuildStepImplementation::getImplementations(); 135 + $all_steps = mpull($all_steps, null, 'getBuildStepAutotargetStepKey'); 136 + 137 + // Make sure all the targets really exist. 138 + foreach ($autotargets as $autotarget) { 139 + if (empty($all_steps[$autotarget])) { 140 + throw new Exception( 141 + pht( 142 + 'No build step provides autotarget "%s"!', 143 + $autotarget)); 144 + } 145 + } 146 + 147 + return array_select_keys($all_steps, $autotargets); 148 + } 149 + 150 + 151 + /** 152 + * Get a list of @{class:HarbormasterBuildTarget} objects for a list of 153 + * autotarget keys. 154 + * 155 + * If some targets or builds do not exist, they are created. 156 + * 157 + * @param HarbormasterBuildable A buildable. 158 + * @param map<string, object> Map of keys to steps. 159 + * @return map<string, object> Map of keys to targets. 160 + */ 161 + private function generateBuildTargetMap( 162 + HarbormasterBuildable $buildable, 163 + array $step_map) { 164 + 165 + $viewer = $this->getViewer(); 166 + $plan_map = mgroup($step_map, 'getBuildPlanPHID'); 167 + 168 + $builds = id(new HarbormasterBuildQuery()) 169 + ->setViewer($viewer) 170 + ->withBuildablePHIDs(array($buildable->getPHID())) 171 + ->withBuildPlanPHIDs(array_keys($plan_map)) 172 + ->needBuildTargets(true) 173 + ->execute(); 174 + 175 + $autobuilds = array(); 176 + foreach ($builds as $build) { 177 + $plan_key = $build->getBuildPlan()->getPlanAutoKey(); 178 + $autobuilds[$plan_key] = $build; 179 + } 180 + 181 + $new_builds = array(); 182 + foreach ($plan_map as $plan_phid => $steps) { 183 + $plan = head($steps)->getBuildPlan(); 184 + $plan_key = $plan->getPlanAutoKey(); 185 + 186 + $build = idx($autobuilds, $plan_key); 187 + if ($build) { 188 + // We already have a build for this set of targets, so we don't need 189 + // to do any work. (It's possible the build is an older build that 190 + // doesn't have all of the right targets if new autotargets were 191 + // recently introduced, but we don't currently try to construct them.) 192 + continue; 193 + } 194 + 195 + // NOTE: Normally, `applyPlan()` does not actually generate targets. 196 + // We need to apply the plan in-process to perform target generation. 197 + // This is fine as long as autotargets are empty containers that don't 198 + // do any work, which they always should be. 199 + 200 + PhabricatorWorker::setRunAllTasksInProcess(true); 201 + try { 202 + 203 + // NOTE: We might race another process here to create the same build 204 + // with the same `planAutoKey`. The database will prevent this and 205 + // using autotargets only currently makes sense if you just created the 206 + // resource and "own" it, so we don't try to handle this, but may need 207 + // to be more careful here if use of autotargets expands. 208 + 209 + $build = $buildable->applyPlan($plan); 210 + PhabricatorWorker::setRunAllTasksInProcess(false); 211 + } catch (Exception $ex) { 212 + PhabricatorWorker::setRunAllTasksInProcess(false); 213 + throw $ex; 214 + } 215 + 216 + $new_builds[] = $build; 217 + } 218 + 219 + if ($new_builds) { 220 + $all_targets = id(new HarbormasterBuildTargetQuery()) 221 + ->setViewer($viewer) 222 + ->withBuildPHIDs(mpull($new_builds, 'getPHID')) 223 + ->execute(); 224 + } else { 225 + $all_targets = array(); 226 + } 227 + 228 + foreach ($builds as $build) { 229 + foreach ($build->getBuildTargets() as $target) { 230 + $all_targets[] = $target; 231 + } 232 + } 233 + 234 + $target_map = array(); 235 + foreach ($all_targets as $target) { 236 + $target_key = $target 237 + ->getImplementation() 238 + ->getBuildStepAutotargetStepKey(); 239 + if (!$target_key) { 240 + continue; 241 + } 242 + $target_map[$target_key] = $target; 243 + } 244 + 245 + $target_map = array_select_keys($target_map, array_keys($step_map)); 246 + 247 + return $target_map; 248 + } 249 + 250 + 251 + }
+39
src/applications/harbormaster/query/HarbormasterBuildPlanQuery.php
··· 7 7 private $phids; 8 8 private $statuses; 9 9 private $datasourceQuery; 10 + private $planAutoKeys; 11 + private $needBuildSteps; 10 12 11 13 public function withIDs(array $ids) { 12 14 $this->ids = $ids; ··· 28 30 return $this; 29 31 } 30 32 33 + public function withPlanAutoKeys(array $keys) { 34 + $this->planAutoKeys = $keys; 35 + return $this; 36 + } 37 + 38 + public function needBuildSteps($need) { 39 + $this->needBuildSteps = $need; 40 + return $this; 41 + } 42 + 31 43 public function newResultObject() { 32 44 return new HarbormasterBuildPlan(); 33 45 } ··· 36 48 return $this->loadStandardPage($this->newResultObject()); 37 49 } 38 50 51 + protected function didFilterPage(array $page) { 52 + if ($this->needBuildSteps) { 53 + $plan_phids = mpull($page, 'getPHID'); 54 + 55 + $steps = id(new HarbormasterBuildStepQuery()) 56 + ->setParentQuery($this) 57 + ->setViewer($this->getViewer()) 58 + ->withBuildPlanPHIDs($plan_phids) 59 + ->execute(); 60 + $steps = mgroup($steps, 'getBuildPlanPHID'); 61 + 62 + foreach ($page as $plan) { 63 + $plan_steps = idx($steps, $plan->getPHID(), array()); 64 + $plan->attachBuildSteps($plan_steps); 65 + } 66 + } 67 + 68 + return $page; 69 + } 70 + 39 71 protected function buildWhereClauseParts(AphrontDatabaseConnection $conn) { 40 72 $where = parent::buildWhereClauseParts($conn); 41 73 ··· 65 97 $conn, 66 98 'name LIKE %>', 67 99 $this->datasourceQuery); 100 + } 101 + 102 + if ($this->planAutoKeys !== null) { 103 + $where[] = qsprintf( 104 + $conn, 105 + 'planAutoKey IN (%Ls)', 106 + $this->planAutoKeys); 68 107 } 69 108 70 109 return $where;
+4
src/applications/harbormaster/query/HarbormasterBuildPlanSearchEngine.php
··· 88 88 $item->setDisabled(true); 89 89 } 90 90 91 + if ($plan->isAutoplan()) { 92 + $item->addIcon('fa-lock grey', pht('Autoplan')); 93 + } 94 + 91 95 $item->setHref($this->getApplicationURI("plan/{$id}/")); 92 96 93 97 $list->addItem($item);
+38
src/applications/harbormaster/step/HarbormasterArcLintBuildStepImplementation.php
··· 1 + <?php 2 + 3 + final class HarbormasterArcLintBuildStepImplementation 4 + extends HarbormasterBuildStepImplementation { 5 + 6 + const STEPKEY = 'arcanist.lint'; 7 + 8 + public function getBuildStepAutotargetPlanKey() { 9 + return HarbormasterBuildArcanistAutoplan::PLANKEY; 10 + } 11 + 12 + public function getBuildStepAutotargetStepKey() { 13 + return self::STEPKEY; 14 + } 15 + 16 + public function shouldRequireAutotargeting() { 17 + return true; 18 + } 19 + 20 + public function getName() { 21 + return pht('Arcanist Lint Results'); 22 + } 23 + 24 + public function getGenericDescription() { 25 + return pht('Automatic `arc lint` step.'); 26 + } 27 + 28 + public function execute( 29 + HarbormasterBuild $build, 30 + HarbormasterBuildTarget $build_target) { 31 + return; 32 + } 33 + 34 + public function shouldWaitForMessage(HarbormasterBuildTarget $target) { 35 + return true; 36 + } 37 + 38 + }
+38
src/applications/harbormaster/step/HarbormasterArcUnitBuildStepImplementation.php
··· 1 + <?php 2 + 3 + final class HarbormasterArcUnitBuildStepImplementation 4 + extends HarbormasterBuildStepImplementation { 5 + 6 + const STEPKEY = 'arcanist.unit'; 7 + 8 + public function getBuildStepAutotargetPlanKey() { 9 + return HarbormasterBuildArcanistAutoplan::PLANKEY; 10 + } 11 + 12 + public function getBuildStepAutotargetStepKey() { 13 + return self::STEPKEY; 14 + } 15 + 16 + public function shouldRequireAutotargeting() { 17 + return true; 18 + } 19 + 20 + public function getName() { 21 + return pht('Arcanist Unit Results'); 22 + } 23 + 24 + public function getGenericDescription() { 25 + return pht('Automatic `arc unit` step.'); 26 + } 27 + 28 + public function execute( 29 + HarbormasterBuild $build, 30 + HarbormasterBuildTarget $build_target) { 31 + return; 32 + } 33 + 34 + public function shouldWaitForMessage(HarbormasterBuildTarget $target) { 35 + return true; 36 + } 37 + 38 + }
+19
src/applications/harbormaster/step/HarbormasterBuildStepImplementation.php
··· 1 1 <?php 2 2 3 + /** 4 + * @task autotarget Automatic Targets 5 + */ 3 6 abstract class HarbormasterBuildStepImplementation extends Phobject { 4 7 5 8 private $settings; ··· 247 250 return $future->resolve(); 248 251 } 249 252 } 253 + } 254 + 255 + 256 + /* -( Automatic Targets )-------------------------------------------------- */ 257 + 258 + 259 + public function getBuildStepAutotargetStepKey() { 260 + return null; 261 + } 262 + 263 + public function getBuildStepAutotargetPlanKey() { 264 + throw new PhutilMethodNotImplementedException(); 265 + } 266 + 267 + public function shouldRequireAutotargeting() { 268 + return false; 250 269 } 251 270 252 271 }
+8 -2
src/applications/harbormaster/storage/HarbormasterBuildable.php
··· 141 141 $build = HarbormasterBuild::initializeNewBuild($viewer) 142 142 ->setBuildablePHID($this->getPHID()) 143 143 ->setBuildPlanPHID($plan->getPHID()) 144 - ->setBuildStatus(HarbormasterBuild::STATUS_PENDING) 145 - ->save(); 144 + ->setBuildStatus(HarbormasterBuild::STATUS_PENDING); 145 + 146 + $auto_key = $plan->getPlanAutoKey(); 147 + if ($auto_key) { 148 + $build->setPlanAutoKey($auto_key); 149 + } 150 + 151 + $build->save(); 146 152 147 153 PhabricatorWorker::scheduleTask( 148 154 'HarbormasterBuildWorker',
+6
src/applications/harbormaster/storage/build/HarbormasterBuild.php
··· 9 9 protected $buildPlanPHID; 10 10 protected $buildStatus; 11 11 protected $buildGeneration; 12 + protected $planAutoKey; 12 13 13 14 private $buildable = self::ATTACHABLE; 14 15 private $buildPlan = self::ATTACHABLE; ··· 148 149 self::CONFIG_COLUMN_SCHEMA => array( 149 150 'buildStatus' => 'text32', 150 151 'buildGeneration' => 'uint32', 152 + 'planAutoKey' => 'text32?', 151 153 ), 152 154 self::CONFIG_KEY_SCHEMA => array( 153 155 'key_buildable' => array( ··· 158 160 ), 159 161 'key_status' => array( 160 162 'columns' => array('buildStatus'), 163 + ), 164 + 'key_planautokey' => array( 165 + 'columns' => array('buildablePHID', 'planAutoKey'), 166 + 'unique' => true, 161 167 ), 162 168 ), 163 169 ) + parent::getConfiguration();
+9 -1
src/applications/harbormaster/storage/build/HarbormasterBuildTarget.php
··· 175 175 return $this->implementation; 176 176 } 177 177 178 + public function isAutotarget() { 179 + try { 180 + return (bool)$this->getImplementation()->getBuildStepAutotargetPlanKey(); 181 + } catch (Exception $e) { 182 + return false; 183 + } 184 + } 185 + 178 186 public function getName() { 179 - if (strlen($this->name)) { 187 + if (strlen($this->name) && !$this->isAutotarget()) { 180 188 return $this->name; 181 189 } 182 190
+57 -2
src/applications/harbormaster/storage/configuration/HarbormasterBuildPlan.php
··· 1 1 <?php 2 2 3 + /** 4 + * @task autoplan Autoplans 5 + */ 3 6 final class HarbormasterBuildPlan extends HarbormasterDAO 4 7 implements 5 8 PhabricatorApplicationTransactionInterface, ··· 8 11 9 12 protected $name; 10 13 protected $planStatus; 14 + protected $planAutoKey; 11 15 12 16 const STATUS_ACTIVE = 'active'; 13 17 const STATUS_DISABLED = 'disabled'; ··· 16 20 17 21 public static function initializeNewBuildPlan(PhabricatorUser $actor) { 18 22 return id(new HarbormasterBuildPlan()) 19 - ->setPlanStatus(self::STATUS_ACTIVE); 23 + ->setName('') 24 + ->setPlanStatus(self::STATUS_ACTIVE) 25 + ->attachBuildSteps(array()); 20 26 } 21 27 22 28 protected function getConfiguration() { ··· 25 31 self::CONFIG_COLUMN_SCHEMA => array( 26 32 'name' => 'sort128', 27 33 'planStatus' => 'text32', 34 + 'planAutoKey' => 'text32?', 28 35 ), 29 36 self::CONFIG_KEY_SCHEMA => array( 30 37 'key_status' => array( ··· 32 39 ), 33 40 'key_name' => array( 34 41 'columns' => array('name'), 42 + ), 43 + 'key_planautokey' => array( 44 + 'columns' => array('planAutoKey'), 45 + 'unique' => true, 35 46 ), 36 47 ), 37 48 ) + parent::getConfiguration(); ··· 57 68 } 58 69 59 70 71 + /* -( Autoplans )---------------------------------------------------------- */ 72 + 73 + 74 + public function isAutoplan() { 75 + return ($this->getPlanAutoKey() !== null); 76 + } 77 + 78 + 79 + public function getAutoplan() { 80 + if (!$this->isAutoplan()) { 81 + return null; 82 + } 83 + 84 + return HarbormasterBuildAutoplan::getAutoplan($this->getPlanAutoKey()); 85 + } 86 + 87 + 88 + public function getName() { 89 + $autoplan = $this->getAutoplan(); 90 + if ($autoplan) { 91 + return $autoplan->getAutoplanName(); 92 + } 93 + 94 + return parent::getName(); 95 + } 96 + 97 + 60 98 /* -( PhabricatorSubscribableInterface )----------------------------------- */ 61 99 62 100 ··· 113 151 case PhabricatorPolicyCapability::CAN_EDIT: 114 152 // NOTE: In practice, this policy is always limited by the "Mangage 115 153 // Build Plans" policy. 154 + 155 + if ($this->isAutoplan()) { 156 + return PhabricatorPolicies::POLICY_NOONE; 157 + } 158 + 116 159 return PhabricatorPolicies::getMostOpenPolicy(); 117 160 } 118 161 } ··· 122 165 } 123 166 124 167 public function describeAutomaticCapability($capability) { 125 - return null; 168 + $messages = array(); 169 + 170 + switch ($capability) { 171 + case PhabricatorPolicyCapability::CAN_EDIT: 172 + if ($this->isAutoplan()) { 173 + $messages[] = pht( 174 + 'This is an autoplan (a builtin plan provided by an application) '. 175 + 'so it can not be edited.'); 176 + } 177 + break; 178 + } 179 + 180 + return $messages; 126 181 } 127 182 128 183 }
+13 -1
src/applications/harbormaster/storage/configuration/HarbormasterBuildStep.php
··· 12 12 protected $className; 13 13 protected $details = array(); 14 14 protected $sequence = 0; 15 + protected $stepAutoKey; 15 16 16 17 private $buildPlan = self::ATTACHABLE; 17 18 private $customFields = self::ATTACHABLE; 18 19 private $implementation; 19 20 20 21 public static function initializeNewStep(PhabricatorUser $actor) { 21 - return id(new HarbormasterBuildStep()); 22 + return id(new HarbormasterBuildStep()) 23 + ->setName('') 24 + ->setDescription(''); 22 25 } 23 26 24 27 protected function getConfiguration() { ··· 37 40 // which predated editable names. These should be backfilled with 38 41 // default names, then the code for handling `null` shoudl be removed. 39 42 'name' => 'text255?', 43 + 'stepAutoKey' => 'text32?', 40 44 ), 41 45 self::CONFIG_KEY_SCHEMA => array( 42 46 'key_plan' => array( 43 47 'columns' => array('buildPlanPHID'), 48 + ), 49 + 'key_stepautokey' => array( 50 + 'columns' => array('buildPlanPHID', 'stepAutoKey'), 51 + 'unique' => true, 44 52 ), 45 53 ), 46 54 ) + parent::getConfiguration(); ··· 86 94 } 87 95 88 96 return $this->implementation; 97 + } 98 + 99 + public function isAutostep() { 100 + return ($this->getStepAutoKey() !== null); 89 101 } 90 102 91 103