Migrate old reviewer edges to new storage

Summary:
Ref T10967. We still have double writes, so all reviewers are being written to both old and new storage. This migrates all the data in the old storage to the new storage, so both storage tables should have a complete set of data and be getting identical updates as we move forward.

After this, I can move readers over one at a time and eventually get rid of the old writes and old storage.

This loads all of the edge data into memory in a big chunk. I reached out to one install to get some more information about their data size. Ours is quite manageable and I think even large installs will probably fit into memory, but we can do this in chunks if not.

However, because the Edge table doesn't have an `id` column, we can't use either the `RawMigrationIterator` or the `MigrationIterator`, and would need to write a new `EdgeMigrationIterator`. This isn't tons of work but might not be necessary.

Test Plan: Ran the migration locally, spot-checked the results in the database for sanity and correctness.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17515

epriestley 9 years ago 8ad5d286 794b4565

+125

1 changed file

expand all

resources

sql

autopatches

20170320.reviewers.03.migrate.php

+125

resources/sql/autopatches/20170320.reviewers.03.migrate.php

··· 1 + <?php 2 + 3 + $table = new DifferentialRevision(); 4 + $diff_table = new DifferentialDiff(); 5 + $reviewer_table = new DifferentialReviewer(); 6 + 7 + $table_name = PhabricatorEdgeConfig::TABLE_NAME_EDGE; 8 + $data_name = PhabricatorEdgeConfig::TABLE_NAME_EDGEDATA; 9 + 10 + $conn = $table->establishConnection('w'); 11 + 12 + // Previously "DifferentialRevisionHasReviewerEdgeType::EDGECONST". 13 + $edge_type = 35; 14 + 15 + // NOTE: We can't use normal migration iterators for edges because they don't 16 + // have an "id" column. For now, try just loading the whole result set: the 17 + // actual size of the rows is small. If we run into issues, we could write an 18 + // EdgeIterator. 19 + $every_edge = queryfx_all( 20 + $conn, 21 + 'SELECT * FROM %T edge LEFT JOIN %T data ON edge.dataID = data.id 22 + WHERE edge.type = %d', 23 + $table_name, 24 + $data_name, 25 + $edge_type); 26 + 27 + foreach ($every_edge as $edge) { 28 + if ($edge['type'] != $edge_type) { 29 + // Ignore edges which aren't "reviewers", like subscribers. 30 + continue; 31 + } 32 + 33 + try { 34 + $data = phutil_json_decode($edge['data']); 35 + $data = idx($data, 'data'); 36 + } catch (Exception $ex) { 37 + // Just ignore any kind of issue with the edge data, we'll use a default 38 + // below. 39 + $data = null; 40 + } 41 + 42 + if (!$data) { 43 + $data = array( 44 + 'status' => 'added', 45 + ); 46 + } 47 + 48 + $status = idx($data, 'status'); 49 + 50 + $diff_phid = null; 51 + 52 + // NOTE: At one point, the code to populate "diffID" worked correctly, but 53 + // it seems to have later been broken. Salvage it if we can, and look up 54 + // the corresponding diff PHID. 55 + $diff_id = idx($data, 'diffID'); 56 + if ($diff_id) { 57 + $row = queryfx_one( 58 + $conn, 59 + 'SELECT phid FROM %T WHERE id = %d', 60 + $diff_table->getTableName(), 61 + $diff_id); 62 + if ($row) { 63 + $diff_phid = $row['phid']; 64 + } 65 + } 66 + 67 + if (!$diff_phid) { 68 + // If the status is "accepted" or "rejected", look up the current diff 69 + // PHID so we can distinguish between "accepted" and "accepted older". 70 + switch ($status) { 71 + case 'accepted': 72 + case 'rejected': 73 + case 'commented': 74 + $row = queryfx_one( 75 + $conn, 76 + 'SELECT diff.phid FROM %T diff JOIN %T revision 77 + ON diff.revisionID = revision.id 78 + WHERE revision.phid = %s 79 + ORDER BY diff.id DESC LIMIT 1', 80 + $diff_table->getTableName(), 81 + $table->getTableName(), 82 + $edge['src']); 83 + if ($row) { 84 + $diff_phid = $row['phid']; 85 + } 86 + break; 87 + } 88 + } 89 + 90 + // We now represent some states (like "Commented" and "Accepted Older") as 91 + // a primary state plus an extra flag, instead of making "Commented" a 92 + // primary state. Map old states to new states and flags. 93 + 94 + if ($status == 'commented') { 95 + $status = 'added'; 96 + $comment_phid = $diff_phid; 97 + $action_phid = null; 98 + } else { 99 + $comment_phid = null; 100 + $action_phid = $diff_phid; 101 + } 102 + 103 + if ($status == 'accepted-older') { 104 + $status = 'accepted'; 105 + } 106 + 107 + if ($status == 'rejected-older') { 108 + $status = 'rejected'; 109 + } 110 + 111 + queryfx( 112 + $conn, 113 + 'INSERT INTO %T (revisionPHID, reviewerPHID, reviewerStatus, 114 + lastActionDiffPHID, lastCommentDiffPHID, dateCreated, dateModified) 115 + VALUES (%s, %s, %s, %ns, %ns, %d, %d) 116 + ON DUPLICATE KEY UPDATE dateCreated = VALUES(dateCreated)', 117 + $reviewer_table->getTableName(), 118 + $edge['src'], 119 + $edge['dst'], 120 + $status, 121 + $action_phid, 122 + $comment_phid, 123 + $edge['dateCreated'], 124 + $edge['dateCreated']); 125 + }

Configure Feed

Configure Feed