Something went wrong. Try again.
This string has no description
Something went wrong. Try again.
9.7 kB · 193 lines · Created 2w ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193Ticket 1Summary: Convert lkp_* lookup datasets from .sas7bdat to SAS program text sources
Type: TaskLabels: repo-hygiene, qa-package, technical-debt
== Background
Analysts commit package output (.sas7bdat) to the release repo. Output filesrange from 128 KB to 1.2 GB. With 14 DPs and 1 to 4 refreshes per DP per year,14 to 56 sets of output land in the repo annually. Git stores this poorly:
* History is permanent. Deleting a file in a later commit does not shrink the repo. Every version ever committed stays in history, and every clone downloads all of it.* Binary files do not compress across versions. .sas7bdat files are page-structured binaries. A small data change rewrites large parts of the file, so Git stores each version close to full size.* Regenerated files look new even when the data is unchanged. The .sas7bdat header records creation and modification timestamps, so a rebuild with identical data produces a new blob.
We cannot add *.sas7bdat to .gitignore today. The package commits 20 lookupdatasets in .sas7bdat format, and a blanket ignore rule would exclude them.This ticket moves those lookups to SAS program text sources. Once no packageinput is stored as .sas7bdat, a later ticket can ignore the extensionrepo-wide and block output commits.
Text sources also give readable diffs in review, efficient storage acrossversions, and no timestamp churn.
== Current state (verified in repo)
* 20 tracked lkp_*.sas7bdat files sit in inputfiles/. Total size is about 5.3 MB. File sizes run from 192 KB to 832 KB (lkp_dem_zip and lkp_all_flags are the largest).* Churn is high. Each file has 94 to 159 commits. Commits that touch inputfiles/*.sas7bdat total 218.* Every consumer reads the lookups through the INFOLDER libref as infolder.lkp_*. There are about 90 references across a dozen program files. Consumer code must not change.* Dataset names are referenced three ways, so they are a data contract: - Direct references, for example infolder.lkp_all_flags. - Macro-built names, for example lkp_lab_&ds and lkp_pre_&ds in scdm_formats.sas lines 171 and 220. - Data-driven names. scdm_standard_macros.sas lines 944 to 975 build infolder.<lookup_table> from column values inside lkp_all_flags. lkp_rate_threshold appears only inside lkp_all_flags data, never in program code.* Observation order is a contract. scdm_formats.sas line 27 derives row numbers with monotonic(), and scdm_data_qa_review-level3comp.sas lines 265 to 271 select rows by position.* The run-time plumbing already exists. scdm_data_qa_review_master_file.sas lines 397 and 398 assign INFOLDER as a concatenated libref (inputfiles/custom then inputfiles) and assign the CUSTOM libref. %soc_lib supports multi-path librefs, proven at scdm_standard_macros.sas line 1494. The macro _customize_input.sas already writes customized copies to custom.lkp_* and relies on concat order to make them win.
== Approach
Store each lookup as a SAS program and build the datasets at run time.
1. Text source per lookup. Create inputfiles/lookups/. Add one .sas file per dataset, named lkp_<name>.sas. Each file contains an explicit ATTRIB statement (variable name, type, length, format, informat, label), a DATALINES4 block with one observation per line, and a trailing ;;;;. Variable order and observation order match the current datasets exactly.
2. Build step. Add a %build_lookups macro. It runs each text source into a request-scoped library outside the tracked tree, for example next to dplocal and msoc under the request root. It then reassigns INFOLDER as custom, then the built library, then inputfiles. Customized copies keep precedence, which matches today's behavior. The master file calls %build_lookups once, right after the existing libname assignments at lines 394 to 398 and before the program body. Data partners do nothing new. The build cost is seconds for 5.3 MB of lookups.
3. One-time exporter. Write a small SAS program that reads each current .sas7bdat and emits its text source deterministically. The exporter proves the conversion is lossless and stays in the repo for future bulk refreshes. The exporter must scan character values and fail loudly if a value contains a line break or four consecutive semicolons, because either would corrupt the DATALINES4 block.
4. Verification and removal. For each dataset, compare the built dataset against the archived original with PROC COMPARE and a dictionary.columns check on types, lengths, and variable order. After all 20 compare clean, run the test suite and one full package run on test SCDM data. Then git rm the 20 .sas7bdat files.
Why SAS program sources and not JSON or CSV: the lookups carry WHERE-conditionstrings that feed call execute, plus exact lengths that drive formatconstruction in scdm_formats.sas. Explicit ATTRIB plus DATALINES4 preservestypes, lengths, and arbitrary string content with no parser and no enginedependency. The package supports SAS 9.4 (readme.md, Requirements). DATALINES4works on every SAS 9.4 release. JSON and CSV need sidecar metadata for typesand lengths and add quoting risk for embedded quotes and commas.
== Scope
In scope: the 20 lkp_*.sas7bdat files directly under inputfiles/.
Out of scope:* Removing existing output from repo history (tracked separately in <TICKET>).* Adding the *.sas7bdat ignore rule (dependent ticket).* The 21 tracked .sas7bdat files inside embedded packages (mir subtree 17, qa_common_components 3, scdm_snapshot 1). The ignore-rule ticket must account for them with negations or follow-up conversion.
== Acceptance criteria
AC.1: git ls-files 'inputfiles/lkp_*.sas7bdat' returns nothing on the release branch.AC.2: inputfiles/lookups/ contains exactly 20 lkp_*.sas files. Each declares every variable with an explicit ATTRIB statement.AC.3: For all 20 datasets, PROC COMPARE of built dataset against the archived original reports zero discrepancies. A dictionary.columns comparison reports identical variable order, types, lengths, and labels.AC.4: Every test in inputfiles/_tests passes without edits.AC.5: A full package run on the standard test SCDM data produces msoc output identical to a run of the pre-conversion package on the same data.AC.6: No consumer program file changes. The diff touches only the master file (one %build_lookups call), the new macro, the new lookups folder, the exporter, and the removed binaries.AC.7: The master file header (lines 93 to 112) lists exactly the 20 lookups that exist as text sources. The stale entries control_flow.sas7bdat, lkp_dia_l2, and lkp_pro_l2 are gone.
== Test plan
* AC.1, AC.2, AC.6, AC.7: git ls-files and git diff review.* AC.3: SAS verification program that loops all 20 datasets through PROC COMPARE and dictionary.columns, and writes a pass or fail per dataset to the log. The program ships with the exporter so any future refresh can re-run it.* AC.4: run each test in inputfiles/_tests on the built lookups.* AC.5: two full runs (pre- and post-conversion package) on the same test data, then compare msoc datasets. Requires the synthetic SCDM test data. This is the one criterion that needs environment access rather than repo tooling.
== Risks
* Character values with line breaks or four consecutive semicolons would corrupt a DATALINES4 source. The exporter must detect both and stop. Risk is low but unverified until the exporter runs, because the current content is binary.* A data partner may have edited a lookup .sas7bdat in place instead of using the custom folder. Such a site loses its edit at cutover. Ask partners to report local lookup deltas before the release.* lkp_dem_zip will produce the longest source file. Review still works, because a diff shows only changed lines.* Future content edits happen directly in the text sources. Announce this in the release notes so nobody edits a rebuilt .sas7bdat and expects it to persist.────────────────────────────────────────
Ticket 2 (optional stub — the dependent ignore-rule ticket you mentioned; adjust or skip if it already exists)Summary: Add repo-wide *.sas7bdat ignore rule and block output commits
Type: TaskLabels: repo-hygiene, qa-packageDepends on: <Ticket 1>
== Background
Ticket <Ticket 1> removes the last package input stored as .sas7bdat frominputfiles/. After that lands, git can ignore the extension repo-wide. Thisstops package output from entering the repo and stops clone size growth atthe source.
== Approach
1. Replace the current .gitignore entries (msoc/*.sas7bdat, dplocal/*.sas7bdat, and the *.log rules) with *.sas7bdat plus *.log.2. Add negation rules for the 21 intentionally shipped binaries inside embedded packages: inputfiles/mir/inputfiles/ (17), inputfiles/qa_common_components/ (4, including cc_test fixtures and the msoc placeholder), and inputfiles/scdm_snapshot/inputfiles/ (1). Decide whether to keep these as binaries with negations or convert them in follow-up tickets. Mir content syncs from an upstream subtree, so a negation is the short-term answer.3. Verify with git check-ignore that an output .sas7bdat written to a request folder is ignored and that all 21 shipped binaries still track.
== Acceptance criteria
AC.1: git check-ignore reports a .sas7bdat file placed in an arbitrary repo folder as ignored.AC.2: git ls-files '*.sas7bdat' lists exactly the 21 shipped embedded-package binaries and nothing else.