Document restructuring and the beginnings of Filter 2 changes
Split the pipeline documentation by purpose so each fact has one home: - docs/pipeline-plan.md keeps the plan, checklist, tracker, and guardrails - docs/decisions.md holds open decisions and the dated decision log - docs/reviews/ holds findings and tasks: one file per filter, plus 00-cross-filter.md for findings that span filters - scripts/common/README.md holds the shared-helper rules (formerly Phase 2) - filter-calculations.md now describes calculations only Filed findings 12-22 from a consistency audit of the app, docs, and scripts. Filter 1 (Köppen-Geiger): use "Köppen" with the umlaut in all prose, labels, docstrings, help text, and checker messages (finding 21), and correct the base build's "majority" docstring (finding 22). Filter 2 (annual avg temperature): record the adopted definition in filter-calculations.md §2: equally weighted 1991-2020 monthly normals, per WMO-No. 1203 and NOAA's 2020 methodology; area-weighted county means; blank unless all 12 months exist. Code changes for this filter are still pending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+86
-78
@@ -54,7 +54,7 @@ in `app.js`.
|
||||
|
||||
| Group | UI label | Data key | Type | Display unit |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Climate Classification | Köppen-Geiger Climate Class | `koppenZone` | Categorical | Class |
|
||||
| Köppen-Geiger Classification | Köppen-Geiger Climate Class | `koppenZone` | Categorical | Class |
|
||||
| Temperature & Extremes | Annual Avg Temperature (Normals) | `avgTempF` | Numeric | °F |
|
||||
| Temperature & Extremes | Diurnal Temperature Range | `avgDiurnalTempRangeF` | Numeric | °F difference |
|
||||
| Temperature & Extremes | Annual Extreme Temperature Days | `absoluteExtremeDays` | Numeric | Days/year |
|
||||
@@ -194,8 +194,20 @@ the Köppen apply step must run after it.
|
||||
|
||||
**Data key:** `avgTempF`
|
||||
|
||||
If the NOAA source is a historical monthly series, the script first forms a
|
||||
1991--2020 climatology for each calendar month and cell:
|
||||
The annual value is a 1991--2020 climatological standard normal: the mean of
|
||||
the 12 monthly normals of NOAA nClimGrid-Monthly average temperature, averaged
|
||||
over each county's area. The definition was adopted on 2026-09-15. It is not
|
||||
yet applied to `data/climate-data.csv`, whose values still use the current
|
||||
method below.
|
||||
|
||||
**Source.** nClimGrid-Monthly is a 1/24° (about 5 km) grid of monthly values
|
||||
interpolated from GHCN station data, covering the contiguous U.S. from 1895 to
|
||||
the present. Its average temperature, `tavg`, is the mean of maximum and
|
||||
minimum temperature, (Tmax + Tmin)/2, not a 24-hour mean. Alaska and Hawaii
|
||||
are outside the grid, so their counties are blank.
|
||||
|
||||
**Cell normals.** For each grid cell \(i\) and calendar month \(m\), the
|
||||
normal is the mean over the 1991--2020 years with a valid value:
|
||||
|
||||
$$
|
||||
T_{i,m}^{\mathrm{norm}}
|
||||
@@ -204,17 +216,78 @@ T_{i,m}^{\mathrm{norm}}
|
||||
\sum_{y\in V_{i,m}}T_{i,m,y}.
|
||||
$$
|
||||
|
||||
The monthly county value is an unweighted mean of valid touched raster cells:
|
||||
This is NCEI's own method for its gridded normals, which it describes as "a
|
||||
simple 30-year average of monthly grids" (Rennie and Palecki,
|
||||
[*U.S. Monthly Gridded Precipitation and Temperature Climate Normals*](https://www.ncei.noaa.gov/sites/default/files/2022-04/Readme_Monthly_Gridded_Normals.pdf)).
|
||||
Our copy of nClimGrid is a later version than the June 2021 data NCEI used, so
|
||||
values can differ slightly from NCEI's published grids. The result is not
|
||||
NCEI's station-based U.S. Climate Normals product.
|
||||
|
||||
**County monthly values.** Each cell is weighted by the area of the cell that
|
||||
lies inside the county, \(a_{c,i}\), estimated as for Köppen (§1). Cells
|
||||
without data, such as ocean, are excluded:
|
||||
|
||||
$$
|
||||
T_{c,m}
|
||||
=
|
||||
\frac{\sum_{i}a_{c,i}\,T_{i,m}^{\mathrm{norm}}}{\sum_{i}a_{c,i}}.
|
||||
$$
|
||||
|
||||
**Annual value.** Every month has equal weight. The value is defined only when
|
||||
all 12 monthly values \(T_{c,m}\) exist; otherwise the county is blank:
|
||||
|
||||
$$
|
||||
T_c(^\circ\mathrm{F})
|
||||
=
|
||||
\left(
|
||||
\frac{1}{12}\sum_{m=1}^{12}T_{c,m}
|
||||
\right)\frac{9}{5}+32.
|
||||
$$
|
||||
|
||||
Values are kept at full precision until the stored value is rounded to
|
||||
0.1 °F.
|
||||
|
||||
**Rationale.** The method follows the WMO rules for annual normals, which
|
||||
NOAA also applies to its 1991--2020 Normals:
|
||||
|
||||
- *Equal month weights.* For a mean, WMO-No. 1203 §4.3.3(a) defines the annual
|
||||
normal as "the mean of the monthly normals", and its footnote says weighting
|
||||
months by their number of days "is not recommended for internationally
|
||||
exchanged products"
|
||||
([WMO Guidelines on the Calculation of Climate Normals, 2017](https://library.wmo.int/records/item/55797-wmo-guidelines-on-the-calculation-of-climate-normals)).
|
||||
NOAA's methodology states that "each month is treated equally in
|
||||
calculating seasonal and annual averages; they are not weighted by the length
|
||||
of month"
|
||||
([Normals Calculation Methodology 2020](https://www.ncei.noaa.gov/data/normals-annualseasonal/1991-2020/doc/Normals_Calculation_Methodology_2020.pdf),
|
||||
p. 2). Day-length weighting would have raised values by only 0.03 to
|
||||
0.13 °F at five sample points.
|
||||
- *From monthly normals.* WMO §4.3.3: annual normals "should be calculated
|
||||
from the monthly normals, and not from the individual annual values."
|
||||
- *Completeness.* WMO §4.3.3: "If the monthly normal for any of the constituent
|
||||
months of the period of interest is missing, then the multimonth normal
|
||||
should also be considered as missing."
|
||||
- *Area weighting.* Equal weights for every touched cell give full weight to
|
||||
cells that lie mostly outside the county, which matters most for small,
|
||||
narrow, and coastal counties. Area weighting matches the Köppen shares (§1).
|
||||
|
||||
Implementation: pending; see
|
||||
[reviews/02-annual-avg-temperature.md](reviews/02-annual-avg-temperature.md).
|
||||
|
||||
### Current method
|
||||
|
||||
Until the new values are applied, the cell normals are computed as above, but
|
||||
the two later steps differ. Each county month is the unweighted mean of every
|
||||
valid cell the county polygon touches:
|
||||
|
||||
$$
|
||||
T_{c,m}
|
||||
=
|
||||
\frac{1}{|G_{c,m}|}
|
||||
\sum_{i\in G_{c,m}}T_{i,m}^{\mathrm{norm}}.
|
||||
\sum_{i\in G_{c,m}}T_{i,m}^{\mathrm{norm}},
|
||||
$$
|
||||
|
||||
The annual value is the equally weighted mean of the available monthly values,
|
||||
converted from Celsius to Fahrenheit:
|
||||
and the annual value averages whichever of the \(M_c\) monthly values are
|
||||
available, normally 12:
|
||||
|
||||
$$
|
||||
T_c(^\circ\mathrm{F})
|
||||
@@ -224,8 +297,6 @@ T_c(^\circ\mathrm{F})
|
||||
\right)\frac{9}{5}+32.
|
||||
$$
|
||||
|
||||
Normally \(M_c=12\). The stored value is rounded to 0.1 °F.
|
||||
|
||||
Implementation: `scripts/build_county_climate_data.py:124--166`,
|
||||
`scripts/build_county_climate_data.py:206--221`, and
|
||||
`scripts/build_county_climate_data.py:478--514`.
|
||||
@@ -548,73 +619,10 @@ Implementation:
|
||||
`scripts/summarize_nsrdb_county_polygon_cloud_archives.py:138--218` and
|
||||
`scripts/summarize_nsrdb_county_polygon_cloud_archives.py:222--304`.
|
||||
|
||||
## Calculation review findings
|
||||
## Review findings
|
||||
|
||||
1. **Annual temperature weights months equally.** February has the same weight
|
||||
as January or July. If the intended label means an average across all days,
|
||||
monthly normals should instead be weighted by the number of days in each
|
||||
month.
|
||||
|
||||
2. **Base NOAA aggregation is not area-weighted.** Every touched raster cell
|
||||
receives equal weight, including cells that intersect only a small portion
|
||||
of a county. This can matter most for small or narrow counties and along
|
||||
coastlines. *Resolved for Köppen on 2026-09-13: class shares are now
|
||||
area-weighted (§1).*
|
||||
|
||||
3. **Partial precipitation years are accepted.** One valid monthly precipitation
|
||||
value is sufficient to produce `annualPrecipIn`; absent months silently lower
|
||||
the annual sum. Requiring all 12 months, or recording completeness, would be
|
||||
safer.
|
||||
|
||||
4. **The Köppen fallback can create false data.** A county with no valid raster
|
||||
cells is labeled `Cfa` instead of missing. A null value plus an audit flag
|
||||
would distinguish missing coverage from a genuine humid-subtropical class.
|
||||
*Resolved on 2026-09-13: the Köppen builder leaves such counties blank. The
|
||||
fallback remains only in `build_county_climate_data.py`, whose Köppen
|
||||
value is replaced by the apply step.*
|
||||
|
||||
5. **Extreme-day counts are not completeness-normalized.** A partially observed
|
||||
year contributes a raw count and receives the same weight as a complete year.
|
||||
Consider requiring a minimum number of valid days or annualizing partial
|
||||
counts explicitly.
|
||||
|
||||
6. **The absolute-extreme metric depends on unrelated percentile thresholds.**
|
||||
`build_annual_counts` skips a county when its retired local p95/p05 thresholds
|
||||
are missing, even though the active 95 °F / 0 °F calculation does not require
|
||||
those percentiles. The absolute calculation should be separated from that
|
||||
prerequisite.
|
||||
|
||||
7. **Heat Index days are a daily-extrema proxy.** Daily Tmax and daily minimum
|
||||
relative humidity are paired even though their observation times may differ.
|
||||
The result should not be described as an observed hourly maximum Heat Index.
|
||||
|
||||
8. **Heat-year completeness is permissive.** Any year with at least one valid
|
||||
Tmax/RH pair is included in the equal-year average. A minimum valid-day rule
|
||||
would reduce low-biased partial-year counts.
|
||||
|
||||
9. **The GHI formula assumes hourly, 365-day input.** It is correct for the
|
||||
current 60-minute, `leap_day=false` requests. If the request interval changes,
|
||||
the energy sum needs an interval-hours multiplier; leap-day handling would
|
||||
also need to change the divisor.
|
||||
|
||||
10. **Clear-sky reduction averages ratios rather than energy totals.** This is a
|
||||
valid but specific definition. It gives each retained time row equal weight,
|
||||
rather than weighting rows by available clear-sky energy. The label and
|
||||
documentation should retain this distinction.
|
||||
|
||||
11. **Spatial weighting is inconsistent across metric families.** Base NOAA
|
||||
normals use equal touched-cell weights, Köppen uses area-weighted class
|
||||
shares, gridMET humidity uses
|
||||
\(\cos(\phi)\) weights on cell centers, and NSRDB polygon metrics use
|
||||
estimated overlap areas. Cross-metric comparisons should account for these
|
||||
different county aggregation methods.
|
||||
|
||||
## Verification status
|
||||
|
||||
This reference was derived from the checked-in calculation and merge scripts,
|
||||
not solely from UI descriptions. No calculation code was changed. The automated
|
||||
test suite was not executed during this review because `pytest` is not installed
|
||||
in either the system Python environment or the project virtual environment.
|
||||
|
||||
Section 1 and findings 2, 4, and 11 were updated on 2026-09-14, after the
|
||||
Köppen classification was reworked and applied.
|
||||
Review findings and their status are kept with the filter reviews in
|
||||
[reviews/](reviews/): findings specific to one filter in that filter's file,
|
||||
and findings that affect several filters in
|
||||
[reviews/00-cross-filter.md](reviews/00-cross-filter.md). Findings keep their
|
||||
original numbers.
|
||||
|
||||
Reference in New Issue
Block a user