gdegelas/solar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
solar/PLAN.md

14 KiB
Raw Permalink Blame History

Solar Production Forecast – Plan & current state

Overview

  • Backend: Python 3.12 + FastAPI. Forecast API (temperature, solar, sun times), multi-country locations, PostgreSQL for metrics and prices, Open-Meteo for weather/solar, Utilitarian Spot for day-ahead electricity prices.
  • Frontend: Vite + React + TypeScript. Country selector → dashboard (location cards, history chart), plus dedicated pages: hourly, compare, electricity prices, trading (production value + backtest + expected), battery trading (arbitrage backtest), world solar clock.
  • Data: Daily and hourly metrics and electricity prices are stored; scheduler refreshes locations (6h) and prices (6h); optional backfill for historical daily data.

Versioning (bump globally)

  • Single source: version.json at repo root:
    • frontend, backend: app versions (e.g. 1.2.0); bump for releases.
    • javascript, css: target/framework versions (e.g. 2026.3 for JS baseline Mar 2026, 1.1 for CSS); bump when you change language or design baseline.
  • Backend: Serves GET /version (reads version.json from /app/version.json in Docker, or repo root when run locally). Docker Compose mounts ./version.json:/app/version.json.
  • Frontend: Footer fetches /version and displays e.g. Frontend v1.2.0 · Backend v1.2.0 · JS 2026.3 · CSS v1.1.
  • To bump: Edit version.json, then restart backend (or rebuild if not using the mount).

Changelog (summary):

  • v1.1.0: Trading page (historical + expected earnings), GET /trading/expected, materialized backtest_daily.
  • v1.2.0: World solar clock, Battery trading page + backtest, GET /debug/solar-clock-storage, POST /metrics/refresh-all-locations, air-quality 404 no longer fails refresh; CSS 1.1.

What is implemented (current state)

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Frontend      │────▶│   Backend       │────▶│   PostgreSQL   │
│   Vite/React    │     │   FastAPI       │     │   :5432        │
│   :5173         │     │   :8000         │     │   (daily_metrics,
└─────────────────┘     └─────────────────┘     │    hourly_metrics,
        │                        │              │    electricity_prices,
        │                        │              │    backtest_daily,
        │                        ▼              │    price_predictions)
        │               ┌─────────────────┐     └─────────────────┘
        │               │ Open-Meteo      │
        │               │ Utilitarian Spot│
        │               └─────────────────┘
        ▼
   version.json (footer versions)
  • Backend: REST API; reads/writes PostgreSQL; calls Open-Meteo (forecast + historical archive) and Open-Meteo Air Quality (optional, 404 handled); fetches day-ahead prices from Utilitarian Spot (ENTSO-E).
  • Frontend: SPA; calls backend only; Recharts for time series; footer shows versions from GET /version.

Backend (Python 3.12 + FastAPI)

  • Runtime: python:3.12-slim (Docker).
  • Libraries: FastAPI, uvicorn, SQLAlchemy (async), httpx, apscheduler, pydantic; Open-Meteo and Spot clients in app/.
  • Data sources: Open-Meteo (forecast, historical archive, air quality), Utilitarian Spot (day-ahead prices per zone).
  • Scheduler (APScheduler): refresh_all_locations every 6h (forecast + daily/hourly upsert for all LOCATIONS); refresh_electricity_prices every 6h (rolling window + baseline price predictor). Air-quality 404 does not roll back location data.

Implemented endpoints (by group):

Group Method Path Purpose
Health / version GET /health Liveness.
GET /version Frontend/backend/JS/CSS versions from version.json.
Config GET /config Solar config (panel count, kWp, PR).
Countries / locations GET /countries List countries (id, name) for selector.
GET /locations List locations, optional ?country=be.
Forecast / metrics GET /forecast Live forecast (Open-Meteo) for one location; optional persist.
GET /metrics Metrics for one location (DB or live fetch).
GET /metrics/all Metrics for all locations of a country (from DB).
GET /metrics/solar-clock All locations worldwide for one date (solar clock map).
POST /metrics/refresh-all-locations Trigger full refresh of all locations (background).
GET /hourly Hourly metrics for one location (from DB).
GET /history Daily metrics for one location (from DB).
Prices GET /prices/zones List price zones.
GET /prices Stored prices for a zone (optional from_date, to_date).
GET /prices/compare Aligned prices for 2+ zones.
POST /prices/refresh Trigger price refresh (quick or full).
Trading GET /trading/production-value Hourly production × price (location, date range).
GET /trading/backtest Backtest “sell at day-ahead”; daily revenue + total.
GET /trading/expected Expected earnings next N days (live forecast + prices).
GET /trading/battery-backtest Battery arbitrage backtest (zone, dates, capacity, power, efficiency).
GET /trading/forecast-eval Price forecast model vs actuals (MAE, MAPE, etc.).
POST /trading/baseline-predict Run baseline price predictor (rolling mean).
GET /trading/forecast-benchmark Revenue: predicted vs benchmark prices.
Backfill / reports POST /backfill Start historical backfill (daily_metrics from archive API).
GET /reports/daily-metrics CSV report of daily metrics.
Debug GET /debug/solar-clock-storage Coverage of daily_metrics per location for a date (solar clock health).

Backend modules (app/): main.py, config.py, database.py, models.py, store.py, jobs.py, openmeteo.py, openmeteo_archive.py, openmeteo_airquality.py, spot_prices.py, sun_times.py, trading.py, battery_trading.py, baseline_price_predictor.py, price_forecast_eval.py, reports.py, cache.py.

Frontend (Vite + React + TypeScript)

  • Stack: Vite 6, React 18, TypeScript, Recharts.
  • Entry: Country selector (first screen); after selecting a country, dashboard with location cards and history chart.
  • Navigation (from dashboard header): Hourly by location, Compare locations, Electricity prices, Trading, Battery trading, Solar clock; Change country, Load forecast data, Backfill 1990–today.

Implemented views:

View Route / trigger Content
Country selector First page Grid of countries; “Solar clock” link.
Dashboard After country selected Location cards (metrics, expand), History chart (location select), Load forecast, Backfill.
Hourly Header “Hourly by location” Hourly metrics + chart per location.
Compare Header “Compare locations” Compare locations: table of metrics, optional chart.
Prices Header “Electricity prices” Zone selector, stored prices chart/table, “Fetch latest from market”.
Trading Header “Trading” Location + date range; production value series, backtest (historical), expected earnings (next 7 days).
Battery trading Header “Battery trading” Zone, dates, capacity, power, efficiency; run battery arbitrage backtest; summary + daily table.
Solar clock “Solar clock” from selector or header World map (azimuthal), date + UTC hour; dots by solar intensity; night = dark blue, no data = grey; “Refresh all locations” if >10% grey.

Database (PostgreSQL 16)

  • Tables (implemented):

    • daily_metrics — Per location, per date: sunrise, sunset, daylight/sunshine, shortwave_radiation_sum_mj, temp min/max, wind, precipitation, weather_code, optional AQ (PM2.5, PM10, AQI).
    • hourly_metrics — Per location, per hour (UTC): shortwave_radiation, temp, wind, rain, etc.
    • electricity_prices — Per zone, per time_utc: value_eur_mwh (day-ahead).
    • backtest_daily — Materialized per location, per date: revenue_eur, kwh (sell at day-ahead).
    • price_predictions — Per model_id, zone, time_utc: predicted price (e.g. baseline-7d-mean).
    • backtest_daily (materialized) and any eval-specific tables as in models.py.

  • Location ↔ zone: config.py: COUNTRY_TO_SPOT_ZONE; get_zone_for_location(location_id) for trading/price alignment. Zones: BE, NL, FR, DE_LU, AT, PL, ES, PT, IT-NORTH.

Docker Compose

  • Services:
    • postgres: postgres:16-alpine, port 5445→5432, healthcheck, volume degelas_pgdata.
    • backend: Build from backend/, port 8000, mount ./backend, ./version.json; env DATABASE_URL, solar config (SOLAR_PANEL_COUNT, etc.); uvicorn with reload.
    • frontend: Build from frontend/, port 5173, mount ./frontend; env VITE_API_URL=http://localhost:8000; npm run dev.
  • Network: degelas (bridge). Frontend calls backend at http://localhost:8000 from the browser.

Config: countries and locations

  • Single place: backend/app/config.py. COUNTRIES dict: id, name, list of locations (id, name, lat, lon). LOCATIONS is built from COUNTRIES (flat id → (lat, lon)). Add a country or city there; API and frontend pick it up. No separate “locations” API beyond list.

File layout (actual)

degelas/
├── docker-compose.yml
├── version.json
├── PLAN.md
├── docs/
│   ├── TRADING_PLATFORM.md
│   ├── BATTERY_TRADING.md
│   ├── SOLAR_CLOCK_GREY_DOTS.md
│   └── SOLAR_CLOCK_STORAGE_DEBUG.md
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app/
│       ├── main.py
│       ├── config.py
│       ├── database.py
│       ├── models.py
│       ├── store.py
│       ├── jobs.py
│       ├── openmeteo.py
│       ├── openmeteo_archive.py
│       ├── openmeteo_airquality.py
│       ├── spot_prices.py
│       ├── sun_times.py
│       ├── trading.py
│       ├── battery_trading.py
│       ├── baseline_price_predictor.py
│       ├── price_forecast_eval.py
│       ├── reports.py
│       ├── cache.py
│       └── ...
└── frontend/
    ├── Dockerfile
    ├── package.json
    ├── vite.config.ts
    ├── index.html
    └── src/
        ├── main.tsx
        ├── App.tsx
        ├── Footer.tsx
        ├── CountrySelector.tsx
        ├── HistoryChart.tsx
        ├── HourlyView.tsx
        ├── CompareView.tsx
        ├── PricesView.tsx
        ├── TradingView.tsx
        ├── BatteryTradingView.tsx
        ├── SolarClockView.tsx
        ├── TypewriterTitle.tsx
        └── ...

Running

docker compose up --build

Future / brainstorm (evolutions, trading agents, endpoints)

Use this section for prioritisation and design of next steps. Nothing here is committed to implementation.

Trading & strategies

  • Strategy registry: Parameterised strategies (e.g. “sell at day-ahead”, “threshold above X €/MWh”, “battery + solar”) with backtest and optional live signals.
  • Unified backtest API: One endpoint that takes strategy_id + params + date range; returns PnL, Sharpe, drawdown, daily curve.
  • Multi-location / portfolio: Backtest and expected earnings across multiple locations (same or different zones); aggregate or weighted views.
  • Battery: Solar-coupled strategy (charge from surplus first); payback calculator (capex, O&M) in UI; optional degradation model.

Trading agents framework (possible endpoints)

  • Agent config: CRUD or config file for “agents” (e.g. “solar_seller”, “battery_arbitrage”) with params (location, zone, thresholds).
  • Signals API: GET /trading/signals?agent_id=...&date=... — what should the agent do today? (e.g. hourly schedule: charge/discharge/sell).
  • Paper trading: POST trades (time, zone, side, volume, price); store in DB; PnL per agent or portfolio.
  • Live execution (later): Integration with exchange/broker (EPEX, Nord Pool); risk limits; audit log and alerts.

Data & analytics

  • Cache / materialise: Precompute production value or backtest by (location, date) for faster dashboards.
  • Price forecasting: More models (e.g. ML); evaluate vs actuals; optional “revenue if we used this forecast” in UI.
  • Reports: More CSV/Excel exports (e.g. battery backtest summary, trading PnL by month).

Infrastructure & product

  • Nginx: Reverse proxy (single entry, CORS in one place, optional TLS).
  • Env / secrets: API keys for optional data providers; feature flags for new strategies or agents.
  • Monitoring: Health checks, metrics (e.g. refresh success rate), optional alerting.

Frontend

  • Dashboard widgets: Configurable widgets (e.g. “today’s expected revenue”, “battery profit this week”).
  • Trading dashboard: Single view with tabs or cards: backtest, expected, battery, prices, signals (when implemented).
  • Solar clock: Optional “solar-coupled” layer or link to battery backtest for selected zone.

References

  • Trading roadmap & APIs: docs/TRADING_PLATFORM.md
  • Battery design & formulae: docs/BATTERY_TRADING.md
  • Solar clock grey dots: docs/SOLAR_CLOCK_GREY_DOTS.md
  • Solar clock storage debug: docs/SOLAR_CLOCK_STORAGE_DEBUG.md
  • History limits & backfill: docs/HISTORY_AND_LIMITS.md
  • Data sources (Open-Meteo, PVGIS): docs/DATA_SOURCES.md
  • Stored vs optional data (comparisons): docs/AVAILABLE_DATA.md