Solar clock: storage and refresh deep dive

This doc explains how location data is stored and refreshed so the solar clock map stays up to date, and how to verify that everything is working.

Data pipeline (how the map gets its data)

Config
All locations shown on the map come from backend/app/config.py: COUNTRIES → LOCATIONS (one entry per city). The solar clock API returns every location in LOCATIONS; there is no “loaded countries” filter for the map.
Database
Daily metrics (sunrise, sunset, shortwave_radiation_sum_mj, temperature, etc.) are stored in daily_metrics with (location_id, date) unique. The solar clock reads from this table only; it does not call Open-Meteo at request time.
Ways data gets into daily_metrics
- Scheduled refresh (primary for “today” and near future)
  refresh_all_locations() runs every 6 hours (see main.py lifespan). It iterates over every location_id in LOCATIONS, calls Open-Meteo Forecast API for that location (past_days=0, forecast_days=14), and upserts the returned daily (and hourly) data into the DB. So in normal operation, every configured location should get fresh data at least 4× per day.
- Backfill (historical and gaps)
  POST /backfill (optional) runs run_backfill(), which uses the Open-Meteo Historical API to fill daily_metrics from a start date (default 1990) to today. Use this to populate the past so the map and history charts have data for older dates.
- Dashboard “Load” / forecast API
  When a user loads a country on the dashboard or calls the forecast endpoint, the server fetches forecast for that location and can persist it in the background via persist_metrics_background. That only affects the locations that were explicitly requested, not all locations.

So for an up-to-date map for today you rely on refresh_all_locations running every 6 hours. For past dates you need to have run backfill (or have had refresh/forecast runs that wrote those dates).

What can go wrong?

Scheduler not running
If the app never starts or the scheduler is disabled, no refresh runs. In Docker, the backend container must stay up so the 6-hour job can run.
Refresh fails for some locations
refresh_location(location_id) is called sequentially for each location. If Open-Meteo fails (rate limit, network, etc.) or the DB write fails, that location is skipped for that run (rollback + warning log). Next run (6 h later) will try again.
No backfill for past dates
If you pick a date in the past and never ran backfill (or backfill didn’t include that date range), those location+date rows won’t exist → grey dots (no data) in daylight.
Open-Meteo daily response missing shortwave_radiation_sum
The store maps shortwave_radiation_sum → shortwave_radiation_sum_mj. If the API omits it or returns null, the row can exist but shortwave_radiation_sum_mj is null → dot stays grey in daylight (treated as “no data” for solar).

How to verify storage and refresh

1. Debug endpoint: `GET /debug/solar-clock-storage`

Use this to check that all locations have a row for a given date and that solar data is present.

URL: GET /debug/solar-clock-storage?date=YYYY-MM-DD
Omit date to use today (UTC).
Response shape:
- date: the date checked.
- summary:
  - total_locations_in_config: number of locations in LOCATIONS.
  - locations_with_any_row: how many have any row in daily_metrics for that date (row exists).
  - locations_with_positive_solar: how many have shortwave_radiation_sum_mj > 0 (so the map can show red/orange/green).
  - locations_missing: count of locations with no row for that date.
- missing_location_ids: list of location_id with no row (capped at 50 in the list).
- locations: per-location list with location_id, has_row, shortwave_radiation_sum_mj.

What to aim for for “today” (after at least one successful refresh):

locations_with_any_row and locations_with_positive_solar should equal total_locations_in_config (or very close; a few failures can happen occasionally).
locations_missing should be 0 (or small and transient). If it’s large, refresh is failing for many locations or hasn’t run yet.

Example (curl):

curl -s "http://localhost:8000/debug/solar-clock-storage" | jq .
# Or for a specific date:
curl -s "http://localhost:8000/debug/solar-clock-storage?date=2025-03-08" | jq .

2. Logs (Docker / process output)

After adding logging in jobs.py:

Refresh: On failure, you’ll see:
WARNING ... refresh_location failed for <location_id>: <exception>
Backfill: On failure:
WARNING ... run_backfill failed for <location_id> (<year_start>–<year_end>): <exception>

So you can confirm that refresh/backfill are running and see which locations fail when they do.

3. Manual refresh trigger (if you add one)

There is no “trigger refresh now” HTTP endpoint by default. To force a refresh you can:

Restart the backend (scheduler will run on startup; the interval job then runs every 6 h), or
Add a POST /debug/refresh-all-locations that calls refresh_all_locations() (and optionally run it from a cron or script).

Checking GET /debug/solar-clock-storage before and after a refresh (or after waiting 6 h) shows whether storage is being filled.

Summary checklist for an up-to-date map

Check	Action
Map shows many grey dots (in daylight) for today	Call `GET /debug/solar-clock-storage`. If `locations_missing` is high or `locations_with_any_row` < total, refresh isn’t populating all locations. Check logs for `refresh_location failed for ...`.
Map shows grey for past dates	Run backfill for that date range: `POST /backfill` (or with `start`/`end` if your API supports it). Then re-check the debug endpoint for that date.
Scheduler running?	Backend must be up; scheduler starts in `lifespan`. Check that the process (e.g. Docker container) is running and not restarting constantly.
Open-Meteo issues	Failures are logged per location. If many locations fail, check network, rate limits, and Open-Meteo status.

Using the debug endpoint and logs together gives you a clear picture of whether all locations are stored properly and refreshed regularly so the solar clock map stays up to date.

6.4 KiB Raw Permalink Blame History Unescape Escape