Climate-Mood-Analysis/scripts/county_data_sources.md

# County Climate Data Sources and Mapping

This project now supports county polygons and county-level filter records keyed by 5-digit county FIPS.

## Launching the app

The browser blocks `fetch("data/climate-data.csv")` when `index.html` is opened directly as a `file://` URL, so run the site through a local HTTP server:

```powershell
.\serve.ps1
```

Then open [http://localhost:8000/](http://localhost:8000/). This keeps the app CSV-only while allowing the map and filters to load normally.

## Source 1: Koppen-Geiger classes (`koppenZone`)

- Dataset: Beck et al. updated 1-km Koppen-Geiger climate classes (historical + future windows)
- Landing page: [https://www.gloh2o.org/koppen/](https://www.gloh2o.org/koppen/)
- Primary paper for updated release: [https://www.nature.com/articles/s41597-023-02549-6](https://www.nature.com/articles/s41597-023-02549-6)
- Coverage: 1901-2099 (use historical 1991-2020 layer for this project to align with NOAA baselines)
- License shown on dataset page: CC BY 4.0

## Source 2: NOAA 1991-2020 gridded normals (`avgTempF`, `annualPrecipIn`, `seasonalityIndex`, previous `extremeDays`)

- Main product page: [https://www.ncei.noaa.gov/products/land-based-station/us-climate-normals](https://www.ncei.noaa.gov/products/land-based-station/us-climate-normals)
- Monthly gridded normals readme: [https://www.ncei.noaa.gov/sites/default/files/2022-04/Readme_Monthly_Gridded_Normals.pdf](https://www.ncei.noaa.gov/sites/default/files/2022-04/Readme_Monthly_Gridded_Normals.pdf)
- Daily gridded normals readme: [https://www.ncei.noaa.gov/sites/default/files/2022-09/Readme%20for%20Daily%20Gridded%20Normals%201991-2020.pdf](https://www.ncei.noaa.gov/sites/default/files/2022-09/Readme%20for%20Daily%20Gridded%20Normals%201991-2020.pdf)
- Daily gridded normals documentation (includes units in C/mm): [https://www.ncei.noaa.gov/sites/default/files/2022-09/Documentation_Daily_Gridded_Normals%20V1.0.pdf](https://www.ncei.noaa.gov/sites/default/files/2022-09/Documentation_Daily_Gridded_Normals%20V1.0.pdf)

The ETL script uses:

- Monthly `tavg` normals (C)
- Monthly `prcp` normals (mm)
- Daily `tmax` normals (C)
- Daily `tmin` normals (C)

If you have monthly nClimGrid history files (for example `nclimgrid_tavg.nc`) rather than 12-slice normals files:

- The script now computes a 1991-2020 monthly climatology from the time series automatically.
- `extremeDays` can run in `auto` mode, which uses a documented monthly proxy when daily grids are not provided.

## Source 3: County polygons / FIPS join geometry

- Current app geometry file: `data/geojson-counties-fips.json`
- Original Plotly county geometry source: [https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json](https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json)
- Official county geometry reference (Census TIGER/Line): [https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html](https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html)

## Source 4: Solar resource (`avgSolarGhiKwhM2Day`)

- Recommended dataset: NREL National Solar Radiation Database (NSRDB)
- Data/API page: [https://developer.nrel.gov/docs/solar/nsrdb/](https://developer.nrel.gov/docs/solar/nsrdb/)
- Maps/geospatial data page: [https://www.nrel.gov/gis/solar-resource-maps](https://www.nrel.gov/gis/solar-resource-maps)
- Fallback point API option: NASA POWER `ALLSKY_SFC_SW_DWN` (surface shortwave downwelling radiation), [https://power.larc.nasa.gov/docs/tutorials/service-data-request/api/](https://power.larc.nasa.gov/docs/tutorials/service-data-request/api/)

The app metric is designed for annual average daily global horizontal irradiance (GHI), in `kWh/m2/day`.
For county means, use a gridded annual GHI raster and pass it to the generator with `--solar-ghi-raster`.

### First test: NSRDB county representative points

For a lightweight first pass, sample each county at one interior representative point instead of requesting full county polygons.
This creates a point CSV:

```powershell
.venv\Scripts\python.exe scripts\build_county_representative_points.py
```

Then fetch a small NSRDB GOES TMY test batch.
If `--email` or `--api-key` are omitted, the script prompts you for them:

```powershell
.venv\Scripts\python.exe scripts\fetch_nsrdb_representative_point_ghi.py --limit 10
```

The fetch script uses GOES TMY first.
For high-latitude points, it can automatically retry the NSRDB Polar TMY endpoint when GOES reports no data.
If a county FIPS appears in `data/nrel/county_representative_point_ghi_error_log.csv` from an earlier run, that county tries the Polar endpoint first on the next run.
Disable this behavior with `--no-polar-fallback`.

Outputs:

- `data/nrel/county_representative_points.csv`: county FIPS, name, state, latitude, and longitude.
- `data/nrel/representative_point_csv/`: cached raw NSRDB CSV responses by county FIPS.
- `data/nrel/county_representative_point_ghi_summary.csv`: summarized `avgSolarGhiKwhM2Day` values.

The calculation is:

```text
avgSolarGhiKwhM2Day = sum(hourly GHI) / 1000 / 365
```

If the raw cache is complete but the summary CSV only contains the last fetched batch, rebuild the summary from cached files without calling the API:

```powershell
.venv\Scripts\python.exe scripts\rebuild_nsrdb_representative_point_ghi_summary.py
```

This point-based workflow is easier to validate and resume than full polygon downloads, but it is an approximation of county sunlight rather than an area-weighted county mean.

### First cloud-cover pass: NSRDB representative points

For a fast cloud-cover input layer, fetch representative-point NSRDB CSVs with observed GHI, Clearsky GHI, and Cloud Type:

```powershell
.venv\Scripts\python.exe scripts\fetch_nsrdb_representative_point_cloud_metrics.py --limit 10
```

Once the first batch looks right, fetch every county:

```powershell
.venv\Scripts\python.exe scripts\fetch_nsrdb_representative_point_cloud_metrics.py --all
```

Outputs:

- `data/nrel/representative_point_cloud_csv/`: cached raw NSRDB CSV responses with `ghi,clearsky_ghi,cloud_type`.
- `data/nrel/county_representative_point_cloud_summary.csv`: summarized cloud-cover proxy fields.
- `data/nrel/county_representative_point_cloud_error_log.csv`: failed county requests with redacted API context.

The primary calculation uses daylight rows where Clearsky GHI is at least 50 W/m2:

```text
cloudinessIndexPct = 1 - mean(clamped(GHI / Clearsky GHI, 0, 1))
```

The same summary also stores daylight row counts, observed-to-clear-sky ratio, and broad Cloud Type frequency buckets. This is faster than the polygon archive workflow, but it remains a representative-point county approximation.

Apply the representative-point cloudiness metric to the browser app CSV:

```powershell
.venv\Scripts\python.exe scripts\apply_nsrdb_cloud_metric_to_climate_data.py
```

The current cloud summary covers the 3,143 county representative points and leaves Puerto Rico rows blank in `data/climate-data.csv`.

### County-average target: NSRDB polygon cloud archive requests

For a less noisy county-level cloudiness layer, submit county polygons to the NSRDB archive workflow with `ghi,clearsky_ghi,cloud_type`. This uses the same polygon tiling and pacing logic as the GHI archive workflow, but writes separate cloud manifests, response JSON files, archives, and summaries.

Start with a dry run:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_cloud_archives.py `
  --dry-run `
  --skip-site-count `
  --limit 5
```

Then submit a small real batch:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_cloud_archives.py `
  --limit 5
```

Continue in resumable chunks:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_cloud_archives.py `
  --start 5 `
  --limit 100
```

Download completed archives:

```powershell
.venv\Scripts\python.exe scripts\download_nsrdb_county_polygon_cloud_archives.py
```

Summarize downloaded archives into area-weighted county cloudiness:

```powershell
.venv\Scripts\python.exe scripts\summarize_nsrdb_county_polygon_cloud_archives.py --reuse-existing-output
```

Outputs:

- `data/nrel/county_polygon_cloud_request_manifest.csv`: submitted cloud archive requests.
- `data/nrel/county_polygon_cloud_error_log.csv`: cloud archive request errors.
- `data/nrel/polygon_cloud_request_responses/`: raw NSRDB cloud acknowledgement JSON files.
- `data/nrel/polygon_cloud_archives/`: downloaded cloud ZIP archives.
- `data/nrel/county_polygon_cloud_summary.csv`: area-weighted county cloudiness summaries.

Then update the browser app CSV. Polygon area-weighted values are used first when `data/nrel/county_polygon_cloud_summary.csv` exists; representative-point values remain the fallback:

```powershell
.venv\Scripts\python.exe scripts\apply_nsrdb_cloud_metric_to_climate_data.py
```

### County-average target: NSRDB polygon archive requests

For the final county-average solar layer, submit county polygons to the NSRDB archive workflow instead of sampling one representative point.
The polygon request returns a generated archive URL per county. Download those archives, then summarize them into a county polygon GHI CSV that can replace the representative-point solar values where available.

Start with a small dry run that does not touch the API:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_ghi_archives.py `
  --dry-run `
  --skip-site-count `
  --limit 5
```

Then submit a small real batch:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_ghi_archives.py `
  --limit 5
```

Once the first batch looks right, continue in resumable chunks:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_ghi_archives.py `
  --start 5 `
  --limit 100
```

Or request every county from the current start point:

```powershell
.venv\Scripts\python.exe scripts\request_nsrdb_county_polygon_ghi_archives.py `
  --all
```

Outputs:

- `data/nrel/county_polygon_ghi_request_manifest.csv`: one row per submitted county, including NSRDB profile, site count, request weight, response JSON path, and `download_url`.
- `data/nrel/county_polygon_ghi_error_log.csv`: counties that need retrying or tiling.
- `data/nrel/polygon_request_responses/`: raw API acknowledgement JSON files.
- `data/nrel/polygon_archives/`: downloaded county or tiled county ZIP archives.
- `data/nrel/county_polygon_ghi_summary.csv`: summarized polygon archive GHI, including `avgSolarGhiKwhM2Day`.

Download completed GHI archives:

```powershell
.venv\Scripts\python.exe scripts\download_nsrdb_county_polygon_ghi_archives.py
```

The script uses POST requests because county polygon WKT can be long.
It checks NSRDB site count by default and skips counties whose estimated request weight exceeds the API maximum.
Default pacing is one request every 2.1 seconds, matching the NSRDB archive limit.
Use `--include-puerto-rico` if you want to test Puerto Rico polygons too.

After the archives are downloaded, rebuild or resume the polygon summary:

```powershell
.venv\Scripts\python.exe scripts\summarize_nsrdb_county_polygon_archives.py --reuse-existing-output
```

Then update the app CSV. Polygon archive GHI is used first; representative-point GHI remains as a fallback for counties without polygon archive data:

```powershell
.venv\Scripts\python.exe scripts\apply_locally_extreme_metric_to_climate_data.py
```

## Metric definitions in generated output

- `koppenZone`: majority class within county polygon from Koppen raster.
- `avgTempF`: mean of 12 monthly county mean temperatures, converted C -> F.
- `annualPrecipIn`: sum of 12 monthly county mean precipitation totals, converted mm -> inches.
- `seasonalityIndex`: coefficient of variation of monthly precipitation totals, scaled to 0-100.
- `wettestPrecipMonth`: month with the highest 1991-2020 county mean precipitation total.
- `driestPrecipMonth`: month with the lowest 1991-2020 county mean precipitation total.
- previous `extremeDays` / `oldExtremeDays`: count of daily-normal or monthly-proxy days where county mean `tmax >= 95F` or `tmin <= 32F` (thresholds configurable in script). This is preserved only as an audit column after the NOAA nClimGrid-Daily metrics are applied.
- `avgSolarGhiKwhM2Day`: county mean annual average daily GHI, in `kWh/m2/day`. The current app CSV uses NSRDB polygon archive area-weighted values where available, with representative-point values kept as fallback.
- `cloudinessIndexPct`: representative-point NSRDB daylight cloudiness proxy derived from observed GHI divided by Clearsky GHI. Higher values mean observed irradiance is lower relative to modeled clear-sky irradiance. Despite the legacy field name, values are stored on a 0-1 scale.

## Run the generator

Install dependencies:

```powershell
pip install -r scripts/requirements_county_etl.txt
```

Run:

```powershell
python scripts/build_county_climate_data.py `
  --counties-geojson data/geojson-counties-fips.json `
  --koppen-raster data/koppen_geiger_tif/1991_2020/koppen_geiger_0p00833333.tif `
  --koppen-legend data/koppen_geiger_tif/legend.txt `
  --monthly-tavg-nc data/noaa/nclimgrid/nclimgrid_tavg.nc `
  --monthly-prcp-nc data/noaa/nclimgrid/nclimgrid_prcp.nc `
  --daily-tmax-nc data/noaa/nclimgrid/nclimgrid_tmax.nc `
  --daily-tmin-nc data/noaa/nclimgrid/nclimgrid_tmin.nc `
  --climatology-start-year 1991 `
  --climatology-end-year 2020 `
  --extreme-days-mode auto `
  --solar-ghi-csv data/nrel/county_polygon_ghi_summary.csv `
  --out data/climate-data.csv
```

Notes:

- This computes all counties in your geometry file, not just the sample records.
- For counties outside CONUS coverage in NOAA gridded files, fallback values are applied by the script when no valid grid values intersect.
- For physically-based daily `extremeDays`, provide true daily grids and set `--extreme-days-mode require-daily`.
- If `--counties-geojson` does not exist locally, the script will try to download the county GeoJSON automatically from the Plotly URL above and cache it at that path.
- `--solar-ghi-csv` is optional and can load county-keyed solar summaries into `avgSolarGhiKwhM2Day`.
- `--solar-ghi-raster` is optional and takes precedence over `--solar-ghi-csv`. If you have a gridded annual GHI raster, add `--solar-ghi-raster path/to/annual_ghi_kwh_m2_day.tif` for a true county-area raster mean.

## Source 5: NOAA nClimGrid-Daily county area averages (`locallyExtremeDays`)

- Main product page: [https://www.ncei.noaa.gov/products/land-based-station/nclimgrid-daily](https://www.ncei.noaa.gov/products/land-based-station/nclimgrid-daily)
- County/monthly area averages root: [https://www.ncei.noaa.gov/data/nclimgrid-daily/access/averages/](https://www.ncei.noaa.gov/data/nclimgrid-daily/access/averages/)
- User guide: [https://www.ncei.noaa.gov/data/nclimgrid-daily/doc/nclimgrid-daily_v1-0-0_user-guide.pdf](https://www.ncei.noaa.gov/data/nclimgrid-daily/doc/nclimgrid-daily_v1-0-0_user-guide.pdf)
- NCEI state-code to FIPS crosswalk: [https://www.ncei.noaa.gov/data/nclimgrid-daily/doc/us-state-codes_ncei-to-fips.csv](https://www.ncei.noaa.gov/data/nclimgrid-daily/doc/us-state-codes_ncei-to-fips.csv)
- Census boundary change notes: [https://www.census.gov/programs-surveys/geography/technical-documentation/boundary-change-notes.html](https://www.census.gov/programs-surveys/geography/technical-documentation/boundary-change-notes.html)
- Census county changes: [https://www.census.gov/programs-surveys/geography/technical-documentation/county-changes.html](https://www.census.gov/programs-surveys/geography/technical-documentation/county-changes.html)

The locally extreme metric uses county-specific 1991-2020 thresholds:

```text
p95_tmax_c = 95th percentile of county-average daily Tmax
p05_tmin_c = 5th percentile of county-average daily Tmin
locallyExtremeDays = count(Tmax >= p95_tmax_c OR Tmin <= p05_tmin_c)
```

The same daily county Tmax/Tmin records now also produce an absolute climate-bucket metric:

```text
absoluteExtremeDays = count(Tmax >= 95F OR Tmin <= 0F)
```

The heat threshold follows the EPA extreme-heat example of days at or above 95F.
The cold threshold is intentionally stricter than the older 32F freeze bucket because NWS cold guidance treats freezing as a freeze-warning/crop threshold and describes human extreme cold as region-dependent and often wind-chill based. The current NOAA cache does not include wind, so `Tmin <= 0F` is used as a configurable air-temperature proxy.

Run from the local NOAA CSV cache:

```powershell
.venv\Scripts\python.exe scripts\build_county_locally_extreme_data.py --skip-download
```

Outputs:

- `data/noaa/county_locally_extreme_thresholds.csv`
- `data/noaa/county_locally_extreme_days.csv`
- `data/noaa/county_locally_extreme_days_comparison.csv`

Apply the locally extreme average to the browser app CSV:

```powershell
.venv\Scripts\python.exe scripts\apply_locally_extreme_metric_to_climate_data.py
```

This keeps `data/climate-data.csv` compatible with the existing app by writing the locally extreme average into the active `extremeDays` column, while preserving the earlier proxy metric in `oldExtremeDays`. It also adds detail/audit columns:

- `locallyExtremeDays`
- `locallyExtremeHotDays`
- `locallyExtremeColdDays`
- `absoluteExtremeDays`
- `oldExtremeDays`
- `locallyExtremeAnalysisYears`
- `locallyExtremeSourceFips`
- `locallyExtremeFipsAdjustment`

FIPS/geography-vintage policy:

- Use current NOAA/Census county-equivalent FIPS for the locally extreme outputs.
- Correct NOAA's District of Columbia county row from source region code `18511` to Census FIPS `11001`.
- Apply only one-to-one old app comparison concordances, currently `46113 -> 46102` for Shannon/Oglala Lakota and `51515 -> 51019` for Bedford city/Bedford County.
- Do not guess split or many-to-one geography changes without a Census relationship file; leave those rows flagged in the comparison output.