Knou/Climate-Mood-Analysis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document restructuring and the beginnings of Filter 2 changes

Browse Source 
Split the pipeline documentation by purpose so each fact has one home:
- docs/pipeline-plan.md keeps the plan, checklist, tracker, and guardrails
- docs/decisions.md holds open decisions and the dated decision log
- docs/reviews/ holds findings and tasks: one file per filter, plus
  00-cross-filter.md for findings that span filters
- scripts/common/README.md holds the shared-helper rules (formerly Phase 2)
- filter-calculations.md now describes calculations only

Filed findings 12-22 from a consistency audit of the app, docs, and scripts.

Filter 1 (Köppen-Geiger): use "Köppen" with the umlaut in all prose, labels,
docstrings, help text, and checker messages (finding 21), and correct the
base build's "majority" docstring (finding 22).

Filter 2 (annual avg temperature): record the adopted definition in
filter-calculations.md §2: equally weighted 1991-2020 monthly normals, per
WMO-No. 1203 and NOAA's 2020 methodology; area-weighted county means; blank
unless all 12 months exist. Code changes for this filter are still pending.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Knou
2026-09-15 16:13:11 -04:00
co-authored by Claude Opus 5
parent a9e722791d
commit e855d583e3
27 changed files with 807 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reviews/00-cross-filter.md
+121
View File
@@ -0,0 +1,121 @@
# Cross-filter review
Findings that affect more than one filter. Each finding lives in exactly one
review file; the filter reviews it affects link here and record only how it
applied to them. Findings keep their numbers across all review files, and new
findings take the next number wherever they are filed. A finding that turns out
to affect other filters moves here, leaving a link behind.
## Findings
**2. Base NOAA aggregation is not area-weighted.** Every touched raster cell
receives equal weight, including cells that intersect only a small portion
of a county. This can matter most for small or narrow counties and along
coastlines. *Resolved for Köppen on 2026-09-13: class shares are now
area-weighted ([filter-calculations.md](../filter-calculations.md) §1).*
Affects filters 1 (resolved), 2, 6, and 7.
**11. Spatial weighting is inconsistent across metric families.** Base NOAA
normals use equal touched-cell weights, Köppen uses area-weighted class
shares, gridMET humidity uses
\(\cos(\phi)\) weights on cell centers, and NSRDB polygon metrics use
estimated overlap areas. Cross-metric comparisons should account for these
different county aggregation methods.
Affects filters 1, 2, 6, 7, 10, 11, and 12. The planned fix is item 3 of the
target design in [pipeline-plan.md](../pipeline-plan.md): one shared
county-aggregation module used by every raster-based metric.
**12. Alaska and Hawaii are not covered by the NOAA and gridMET sources.**
NOAA nClimGrid and gridMET cover the contiguous U.S. only; the nClimGrid grid
spans latitude 24.56 to 49.35 and longitude −124.69 to −67.02. All 29 Alaska
and 5 Hawaii counties are therefore blank in filters 2–10. Solar GHI and
clear-sky reduction (filters 11 and 12) cover both states.
`check_climate_data.py` allows these blanks (`OUTSIDE_CONUS`). The notes on
the base build in `scripts/county_data_sources.md` say "fallback values are
applied" for counties outside NOAA coverage, but `build_county_climate_data.py`
leaves them blank (lines 499–520).
Affects filters 2–10. Open decision: "Alaska and Hawaii coverage" in
[decisions.md](../decisions.md).
**13. Lexington, VA (51678) is missing from the NOAA daily county files.**
Diurnal temperature range and extreme temperature days are blank for it, and
`check_climate_data.py` allows those blanks (`NOAA_DAILY_MISSING_FIPS`).
Heat-index days instead use the surrounding Rockbridge County (51163) as a
proxy, recorded in `humidHeatSourceFips` and `humidHeatFipsAdjustment`. The
three filters handle the same gap differently.
Affects filters 3, 4, and 5.
**14. Helper functions are duplicated across scripts, and some copies have
drifted.** A 2026-09-13 survey found 19 functions with identical copies in
several scripts and 19 with copies that have drifted apart. Most identical
copies are NSRDB helpers, which belong in an NSRDB module rather than
`common/`; `read_csv_rows` (4 identical copies in `apply_*` scripts) is
cross-source. Drifted copies need a decision on which version is correct
before merging. Notable drifts: `summarize_county_gridmet_humidity.py` has its
own county loader and FIPS normalizer, and the state FIPS table is also copied
in `build_county_representative_points.py`,
`summarize_county_gridmet_humidity.py`, and
`request_nsrdb_county_polygon_archives.py`.
Affects every script that holds a copy. The rules for moving shared code are
in [scripts/common/README.md](../../scripts/common/README.md).
**15. The app's source text for three NOAA metrics names the wrong product.**
In `app.js`, `avgTempF`, `annualPrecipIn`, and `seasonalityIndex` credit
"NOAA NCEI 1991-2020 U.S. Climate Normals" and link the station-based Normals
page. Their values are computed from the nClimGrid-Monthly series averaged
over 1991–2020, which is NCEI's gridded-normals method rather than the station
product. Source 2 in `scripts/county_data_sources.md` likewise says the build
reads monthly normals files, while the build command passes
`data/noaa/nclimgrid/nclimgrid_tavg.nc` and `nclimgrid_prcp.nc`. Found during
the filter 2 review (2026-09-15).
Affects filters 2, 6, and 7. Filter 2's task list covers the `avgTempF` text.
**16. Source links point to retired or missing pages.** Checked 2026-09-15.
In `app.js`, the "NOAA nClimGrid Monthly" link used by wettest and driest month
(`https://www.ncei.noaa.gov/products/land-based-station/nclimgrid`) returns
404, and the "NREL National Solar Radiation Database" link used by solar GHI
and clear-sky GHI reduction (`https://nsrdb.nrel.gov/`) no longer resolves.
NREL's sites have moved to `nlr.gov`: `https://nsrdb.nlr.gov/` loads, and the
NSRDB scripts already call `developer.nlr.gov`. In Source 4 of
`scripts/county_data_sources.md`, `https://developer.nrel.gov/docs/solar/nsrdb/`
also fails (its `developer.nlr.gov` counterpart loads), and
`https://www.nrel.gov/gis/solar-resource-maps` fails with no counterpart at the
same path on `nlr.gov`. Every other link in `app.js`, `README.md`, and
`scripts/county_data_sources.md` loads.
Affects filters 8, 9, 11, and 12.
**17. The data-sources metric list is out of date.** The "Metric definitions
in generated output" list in `scripts/county_data_sources.md` describes
`extremeDays` / `oldExtremeDays` as an audit column kept in the app CSV, but
neither column is in `data/climate-data.csv`. The list has no entry for
`avgDiurnalTempRangeF` or `absoluteExtremeDays`, 2 of the 12 app metrics.
Affects filters 3 and 4.
## Decisions
In [decisions.md](../decisions.md): Shared helpers (2026-09-13), and the open
decision on Alaska and Hawaii coverage.
## Tasks
Fixes for these findings are carried out in the affected filters' reviews.
## Original review
`filter-calculations.md` was derived from the checked-in calculation and merge
scripts, not solely from UI descriptions. No calculation code was changed. The
automated test suite was not executed during this review because `pytest` is
not installed in either the system Python environment or the project virtual
environment. The suite uses `unittest` and runs without pytest:
`python -m unittest discover -s tests`.
Section 1 and findings 2, 4, and 11 were updated on 2026-09-14, after the
Köppen classification was reworked and applied.