diff --git a/dao/DOCS.md b/dao/DOCS.md index cb1be64..c96816d 100644 --- a/dao/DOCS.md +++ b/dao/DOCS.md @@ -234,7 +234,12 @@ Ook het ophalen van dynamische uurprijzen (day ahead prices) stel je in via Daar is alles al ingevuld, maar je kunt het aanpassen aan jouw situatie:
De belangrijkste onderdelen zijn:
* source day ahead: waar haal je de data vandaan: nordpool is een goede - eerste keuze
+ eerste keuze.
+ * forecast extension provider: optionele provider voor een aanvullende forecast-extensie voorbij de officiele day-ahead horizon. Beschikbaar zijn `energypriceforecast` en `dayaheadprediction`.
+ * forecast extension hours: aantal extra uren dat je voorbij de officiele horizon wilt toevoegen.
+ * energypriceforecast-extension-api-url: optionele override voor de Energy Price Forecast EU extensie-feed.
+ * energypriceforecast-extension-country: optionele marktcode zoals `nl`, `de`, `dk1` of `no3` voor die extensie-feed.
+ * day-ahead-prediction-extension-url: optionele override voor de `corneel27/day-ahead-prediction` feed. Deze provider is momenteel alleen geschikt voor `NL`.
* energy taxes consumption: energiebelasting (euro/kWh, ex BTW) bij afname
* energy taxes production: energiebelasting bij teruglevering (euro/kWh, ex BTW)
* cost supplier consumption: kosten leverancier voor levering (euro/kWh, ex BTW)
@@ -654,6 +659,11 @@ Het is allemaal optioneel. | **meteoserver-attempts** | | getal | 2 | aantal ophaal pogingen | | **prices** | source day ahead | string | nordpool | keuze uit: nordpool / entsoe / easyenergy / tibber | | | entsoe-api-key | string | | alleen bij entsoe als source | +| | forecast extension provider | string | none | keuze uit: none / energypriceforecast / dayaheadprediction | +| | forecast extension hours | integer | 0 | extra uren voorbij officiele horizon | +| | energypriceforecast-extension-api-url | string, url | https://api.energypriceforecast.eu/api/v1/dao/prices | optioneel, alleen bij energypriceforecast extensie | +| | energypriceforecast-extension-country | string | | optioneel, alleen bij energypriceforecast extensie | +| | day-ahead-prediction-extension-url | string, url | https://raw.githubusercontent.com/corneel27/day-ahead-prediction/main/dap/data/prediction.json | optioneel, alleen bij dayaheadprediction extensie (NL) | | | regular high | getal | | | | | regular low | getal | | | | | switch to low | integer | 23 | | @@ -923,12 +933,28 @@ De meteodata worden opgehaald bij meteoserver. Ook hiervoor heb je een key nodig ### **prices**
* source day ahead, default "nordpool" - Hier bepaal je waar je je day ahead prijzen vandaan wilt halen. Je hebt de keuze uit drie bronnen: + Hier bepaal je waar je je officiele day ahead prijzen vandaan wilt halen. Je hebt de keuze uit vier bronnen: * nordpool * entsoe * easyenergy * tibber
+ De officiele bron blijft altijd leidend. Een forecast-provider kan alleen als aparte extensie worden gebruikt bovenop de al geimporteerde day-ahead prijzen. + * forecast extension provider: + Optionele provider voor een horizon-extensie voorbij de officiele day-ahead prijzen. Momenteel ondersteunt DAO hiervoor: + * none + * energypriceforecast + * dayaheadprediction + * forecast extension hours: + Het aantal uren dat je voorbij de officiele horizon wilt aanvullen. DAO vertaalt deze instelling intern naar de `hours` parameter van de gekozen provider-URL. + * energypriceforecast-extension-api-url: + Optionele override voor de Energy Price Forecast EU extensie-feed. Standaard: + `https://api.energypriceforecast.eu/api/v1/dao/prices` + * energypriceforecast-extension-country: + Optionele expliciete marktcode voor de extensie-feed, bijvoorbeeld `nl`, `de`, `dk1` of `no3`. Laat je dit leeg, dan probeert DAO te mappen vanuit de ingestelde landcode. + * day-ahead-prediction-extension-url: + Optionele override voor de gepubliceerde `prediction.json` van `corneel27/day-ahead-prediction`. Deze provider verwacht een JSON-array met minimaal `time_ts` en `prediction` en is momenteel alleen bedoeld voor `NL`. + Als je kiest voor **entsoe** dan moet je hieronder een api key invullen. * entsoe-api-key: Deze key genereer je op de site van entsoe en heb je nodig om daar de energieprijzen van de volgende op te halen. diff --git a/dao/data/options_example.json b/dao/data/options_example.json index 8043d5f..dc4b498 100644 --- a/dao/data/options_example.json +++ b/dao/data/options_example.json @@ -13,6 +13,11 @@ "prices": { "source day ahead": "nordpool", "entsoe-api-key": "!secret entsoe-api-key", + "forecast extension provider": "energypriceforecast", + "forecast extension hours": 24, + "energypriceforecast-extension-api-url": "https://api.energypriceforecast.eu/api/v1/dao/prices", + "energypriceforecast-extension-country": "nl", + "day-ahead-prediction-extension-url": "https://raw.githubusercontent.com/corneel27/day-ahead-prediction/main/dap/data/prediction.json", "regular high": 0.5, "regular low": 0.4, "switch to low": 23, @@ -590,4 +595,4 @@ } ] } -} \ No newline at end of file +} diff --git a/dao/data/options_start.json b/dao/data/options_start.json index 359b22b..35a1d65 100644 --- a/dao/data/options_start.json +++ b/dao/data/options_start.json @@ -12,6 +12,11 @@ "meteoserver-key": "!secret meteoserver-key", "prices": { "source day ahead": "nordpool", + "forecast extension provider": "energypriceforecast", + "forecast extension hours": 24, + "energypriceforecast-extension-api-url": "https://api.energypriceforecast.eu/api/v1/dao/prices", + "energypriceforecast-extension-country": "nl", + "day-ahead-prediction-extension-url": "https://raw.githubusercontent.com/corneel27/day-ahead-prediction/main/dap/data/prediction.json", "regular high": 0.50, "regular low": 0.40, "switch to low": 23, diff --git a/dao/lib/da_prices.py b/dao/lib/da_prices.py index 55cedce..577e124 100644 --- a/dao/lib/da_prices.py +++ b/dao/lib/da_prices.py @@ -13,6 +13,11 @@ import logging class DaPrices: + EXTENSION_CODE = "da_ext" + EXTENSION_RECORD_ID = 25 + EXTENSION_RECORD_NAME = "Tarief forecast extension" + EXTENSION_RECORD_DIM = "euro/kWh" + def __init__(self, config, db_da: DBmanagerObj, country: str = None, secrets: dict = None): self.config = config self.db_da = db_da @@ -20,6 +25,265 @@ class DaPrices: self.interval = str(config.interval or "1hour").lower() self.country = country if country is not None else "NL" + def _resolve_market_country(self, configured): + if configured: + return str(configured).strip().lower() + mapping = { + "NL": "nl", + "BE": "be", + "DE": "de", + "FR": "fr", + "AT": "at", + "CZ": "cz", + "DK1": "dk1", + "DK2": "dk2", + "NO1": "no1", + "NO2": "no2", + "NO3": "no3", + "NO4": "no4", + "NO5": "no5", + } + return mapping.get(str(self.country or "").upper(), "nl") + + def _forecast_extension_provider(self) -> str: + provider = getattr(self.config.prices, "forecast_extension_provider", "none") + return str(provider or "none").strip().lower() + + def _forecast_extension_hours(self) -> int: + hours = getattr(self.config.prices, "forecast_extension_hours", 0) + try: + return max(0, min(168, int(hours))) + except (TypeError, ValueError): + return 0 + + def _energypriceforecast_extension_country(self): + configured = getattr( + self.config.prices, + "energypriceforecast_extension_country", + None, + ) + return self._resolve_market_country(configured) + + def _energypriceforecast_extension_api_url(self): + configured = getattr( + self.config.prices, + "energypriceforecast_extension_api_url", + None, + ) + configured = str(configured or "").strip() + if configured: + return configured + return "https://api.energypriceforecast.eu/api/v1/dao/prices" + + def _day_ahead_prediction_extension_url(self): + configured = getattr( + self.config.prices, + "day_ahead_prediction_extension_url", + None, + ) + configured = str(configured or "").strip() + if configured: + return configured + return ( + "https://raw.githubusercontent.com/" + "corneel27/day-ahead-prediction/main/dap/data/prediction.json" + ) + + def _ensure_extension_code_available(self): + self.db_da.ensure_variabel_record( + record_id=self.EXTENSION_RECORD_ID, + code=self.EXTENSION_CODE, + name=self.EXTENSION_RECORD_NAME, + dim=self.EXTENSION_RECORD_DIM, + ) + + def _build_energypriceforecast_df( + self, + *, + start, + end, + api_url: str, + country: str, + code: str, + forecast_only: bool = False, + min_timestamp: int | None = None, + max_timestamp: int | None = None, + hours_override: int | None = None, + ) -> pd.DataFrame: + now_ts = datetime.datetime.now(datetime.timezone.utc).timestamp() + end_ts = end.timestamp() if hasattr(end, "timestamp") else now_ts + 48 * 3600 + hours = hours_override if hours_override is not None else max( + 1, min(168, math.ceil((end_ts - now_ts) / 3600)) + ) + mode_suffix = "&mode=forecast_only" if forecast_only else "" + url = f"{api_url}?country={country}&hours={hours}{mode_suffix}" + resp = get(url, timeout=15) + resp.raise_for_status() + payload = json.loads(resp.text) + entries = payload.get("entries") or [] + df_db = pd.DataFrame(columns=["time", "code", "value"]) + for entry in entries: + start_raw = str(entry.get("start") or "") + end_raw = str(entry.get("end") or "") + value = entry.get("value") + if not start_raw: + continue + try: + start_dt = datetime.datetime.fromisoformat(start_raw.replace("Z", "+00:00")) + end_dt = ( + datetime.datetime.fromisoformat(end_raw.replace("Z", "+00:00")) + if end_raw + else start_dt + datetime.timedelta(hours=1) + ) + value = float(value) + except (TypeError, ValueError): + continue + timestamps: list[int] + if self.interval == "15min" and end_dt > start_dt: + quarter_count = max(1, int(round((end_dt - start_dt).total_seconds() / 900))) + timestamps = [ + int((start_dt + datetime.timedelta(minutes=15 * idx)).timestamp()) + for idx in range(quarter_count) + ] + else: + timestamps = [int(start_dt.timestamp())] + for time_stamp in timestamps: + if min_timestamp is not None and time_stamp <= min_timestamp: + continue + if max_timestamp is not None and time_stamp >= max_timestamp: + continue + df_db.loc[df_db.shape[0]] = [str(time_stamp), code, value] + return df_db + + def _build_day_ahead_prediction_df( + self, + *, + api_url: str, + code: str, + min_timestamp: int | None = None, + max_timestamp: int | None = None, + ) -> pd.DataFrame: + resp = get(api_url, timeout=15) + resp.raise_for_status() + payload = json.loads(resp.text) + entries = payload if isinstance(payload, list) else payload.get("entries") or [] + df_db = pd.DataFrame(columns=["time", "code", "value"]) + for entry in entries: + try: + time_stamp = int(entry.get("time_ts")) + value = float(entry.get("prediction")) + except (TypeError, ValueError, AttributeError): + continue + if min_timestamp is not None and time_stamp <= min_timestamp: + continue + if max_timestamp is not None and time_stamp >= max_timestamp: + continue + if self.interval == "15min": + for idx in range(4): + quarter_ts = time_stamp + idx * 900 + if max_timestamp is not None and quarter_ts >= max_timestamp: + continue + df_db.loc[df_db.shape[0]] = [str(quarter_ts), code, value] + else: + df_db.loc[df_db.shape[0]] = [str(time_stamp), code, value] + return df_db + + def get_price_forecast_extension(self): + self._ensure_extension_code_available() + provider = self._forecast_extension_provider() + extension_hours = self._forecast_extension_hours() + if provider == "none" or extension_hours <= 0: + self.db_da.delete_code_range( + self.EXTENSION_CODE, + start=int(datetime.datetime.now().timestamp()), + ) + logging.info("Geen day-ahead forecast-extensie geconfigureerd.") + return + official_source = str(self.config.prices.source_day_ahead or "").strip().lower() + if official_source not in {"nordpool", "entsoe", "tibber", "easyenergy"}: + logging.warning( + "Forecast-extensie overgeslagen: source day ahead moet een officiele provider zijn." + ) + return + present = self.db_da.get_time_border_record("da") + if present is None: + logging.info( + "Forecast-extensie overgeslagen: er is nog geen officiele day-ahead horizon aanwezig." + ) + return + resolution_seconds = 3600 if self.interval == "1hour" else 900 + official_last_ts = int(present.timestamp()) + extension_start_ts = official_last_ts + resolution_seconds + now_ts = int(datetime.datetime.now().timestamp()) + if extension_start_ts <= now_ts: + logging.info( + "Forecast-extensie overgeslagen: officiele day-ahead horizon loopt niet verder dan nu." + ) + return + target_end_ts = extension_start_ts + extension_hours * 3600 + request_hours = max( + 1, + min(168, math.ceil((target_end_ts - now_ts) / 3600)), + ) + self.db_da.delete_code_range(self.EXTENSION_CODE, start=extension_start_ts) + + if provider == "energypriceforecast": + api_url = self._energypriceforecast_extension_api_url() + country = self._energypriceforecast_extension_country() + df_db = self._build_energypriceforecast_df( + start=datetime.datetime.fromtimestamp(extension_start_ts), + end=datetime.datetime.fromtimestamp(target_end_ts), + api_url=api_url, + country=country, + code=self.EXTENSION_CODE, + forecast_only=True, + min_timestamp=official_last_ts, + max_timestamp=target_end_ts, + hours_override=request_hours, + ) + if df_db.empty: + logging.info( + "Energy Price Forecast extensie: geen aanvullende slots beschikbaar (%s).", + country, + ) + return + logging.info( + "Energy Price Forecast extensie (%s): %d slot(s) toegevoegd voorbij officiele horizon, +%d uur.", + country, + len(df_db), + extension_hours, + ) + self.db_da.savedata(df_db) + return + + if provider == "dayaheadprediction": + if str(self.country or "").strip().upper() != "NL": + logging.warning( + "day-ahead-prediction extensie overgeslagen: provider ondersteunt momenteel alleen NL." + ) + return + api_url = self._day_ahead_prediction_extension_url() + df_db = self._build_day_ahead_prediction_df( + api_url=api_url, + code=self.EXTENSION_CODE, + min_timestamp=official_last_ts, + max_timestamp=target_end_ts, + ) + if df_db.empty: + logging.info( + "day-ahead-prediction extensie: geen aanvullende slots beschikbaar." + ) + return + logging.info( + "day-ahead-prediction extensie: %d slot(s) toegevoegd voorbij officiele horizon, +%d uur.", + len(df_db), + extension_hours, + ) + self.db_da.savedata(df_db) + return + + logging.warning("Onbekende forecast-extensie provider: %s", provider) + def get_prices( self, source, _start: datetime.datetime = None, _end: datetime.datetime = None ): diff --git a/dao/lib/db_manager.py b/dao/lib/db_manager.py index 206235d..24726d0 100644 --- a/dao/lib/db_manager.py +++ b/dao/lib/db_manager.py @@ -12,6 +12,7 @@ from sqlalchemy import ( and_, text, TIMESTAMP, + delete, ) import sqlalchemy_utils import os @@ -313,6 +314,85 @@ class DBmanagerObj(object): connection.close() self.log_pool_status() + def ensure_variabel_record( + self, + *, + record_id: int, + code: str, + name: str, + dim: str, + ) -> int: + """ + Ensure that a variabel record exists for the given code. + If the code already exists, return its id. + Otherwise create or update the requested record id and return it. + """ + connection = self.engine.connect() + try: + variabel_table = Table("variabel", self.metadata, autoload_with=self.engine) + + select_by_code = select(variabel_table.c.id).where( + variabel_table.c.code == code + ) + existing_by_code = connection.execute(select_by_code).first() + if existing_by_code: + return int(existing_by_code[0]) + + select_by_id = select(variabel_table.c.id).where( + variabel_table.c.id == record_id + ) + existing_by_id = connection.execute(select_by_id).first() + if existing_by_id: + connection.execute( + update(variabel_table) + .where(variabel_table.c.id == record_id) + .values(code=code, name=name, dim=dim) + ) + else: + connection.execute( + insert(variabel_table).values( + id=record_id, + code=code, + name=name, + dim=dim, + ) + ) + connection.commit() + return int(record_id) + finally: + connection.close() + + def delete_code_range( + self, + code: str, + start: int | None = None, + end: int | None = None, + tablename: str = "values", + ): + """ + Delete rows for a variable code in an optional unix-timestamp range. + """ + connection = self.engine.connect() + try: + values_table = Table(tablename, self.metadata, autoload_with=self.engine) + variabel_table = Table("variabel", self.metadata, autoload_with=self.engine) + select_variabel = select(variabel_table.c.id).where( + variabel_table.c.code == code + ) + variabel_result = connection.execute(select_variabel).first() + if not variabel_result: + return + variabel_id = variabel_result[0] + query = delete(values_table).where(values_table.c.variabel == variabel_id) + if start is not None: + query = query.where(values_table.c.time >= int(start)) + if end is not None: + query = query.where(values_table.c.time < int(end)) + connection.execute(query) + connection.commit() + finally: + connection.close() + def get_time_border_record( self, code: str, latest: bool = True, table_name: str = "values" ) -> datetime.datetime: diff --git a/dao/prog/check_db.py b/dao/prog/check_db.py index 2736ff2..9c16546 100644 --- a/dao/prog/check_db.py +++ b/dao/prog/check_db.py @@ -271,6 +271,11 @@ class CheckDB: print(f'Meteo-data verhuizen van "values" naar "prognoses"') self.move_meteodata_to_prognoses() + if l_version < 20260620: + record_2026_6_20 = [25, "da_ext", "Tarief forecast extension", "euro/kWh"] + self.upsert_variabel(variabel_tabel, record_2026_6_20) + print('Table "variabel" geupdated met day-ahead forecast extension.') + """ if l_version < 20250700: # update variabel with records voor calculated pv_ac, pv_dc and corr. factors diff --git a/dao/prog/config/models/pricing.py b/dao/prog/config/models/pricing.py index e63d9ba..168583c 100644 --- a/dao/prog/config/models/pricing.py +++ b/dao/prog/config/models/pricing.py @@ -10,13 +10,13 @@ from datetime import date class PricingConfig(BaseModel): """Day-ahead pricing and tariff configuration.""" - - source_day_ahead: Literal['nordpool', 'entsoe', 'tibber'] = Field( + + source_day_ahead: Literal['nordpool', 'entsoe', 'easyenergy', 'tibber'] = Field( default='nordpool', alias="source day ahead", description="Source for day-ahead prices", json_schema_extra={ - "x-help": "Data source for day-ahead electricity market prices. 'nordpool' for Nordic/Baltic, 'entsoe' for European markets, 'tibber' if using Tibber integration.", + "x-help": "Data source for imported official day-ahead electricity market prices. 'nordpool' for Nordic/Baltic, 'entsoe' for European markets, 'easyenergy' for the EasyEnergy public tariff feed, 'tibber' if using Tibber integration.", "x-ui-section": "Prices" } ) @@ -40,6 +40,79 @@ class PricingConfig(BaseModel): } } ) + forecast_extension_provider: Literal['none', 'energypriceforecast', 'dayaheadprediction'] = Field( + default='none', + alias="forecast extension provider", + description="Optional provider for extending the day-ahead horizon with forecast data", + json_schema_extra={ + "x-help": "Optional provider that extends the imported official day-ahead horizon with forecast prices. The extension never replaces already imported official prices.", + "x-ui-section": "Prices", + } + ) + forecast_extension_hours: int = Field( + default=0, + alias="forecast extension hours", + description="How many additional hours should be appended beyond the official day-ahead horizon", + json_schema_extra={ + "x-help": "Number of hours to extend beyond the imported official day-ahead horizon. DAO translates this into the provider-specific URL parameter.", + "x-ui-section": "Prices", + "x-validation-hint": "Integer between 0 and 168" + } + ) + energypriceforecast_extension_api_url: Optional[str] = Field( + default="https://api.energypriceforecast.eu/api/v1/dao/prices", + alias="energypriceforecast-extension-api-url", + description="Energy Price Forecast EU extension API URL", + json_schema_extra={ + "x-help": "Provider-specific URL for the Energy Price Forecast EU horizon extension feed. Expected response: format=dao-prices with entries[].", + "x-ui-section": "Prices", + "x-ui-rules": { + "effect": "SHOW", + "condition": { + "scope": "#/properties/forecast_extension_provider", + "schema": { + "const": "energypriceforecast" + } + } + } + } + ) + energypriceforecast_extension_country: Optional[str] = Field( + default=None, + alias="energypriceforecast-extension-country", + description="Override country code for Energy Price Forecast EU extension", + json_schema_extra={ + "x-help": "Optional explicit country/market code for the Energy Price Forecast EU extension feed, for example 'nl', 'de', 'dk1' or 'no3'. Leave empty to map from DAO country automatically.", + "x-ui-section": "Prices", + "x-ui-rules": { + "effect": "SHOW", + "condition": { + "scope": "#/properties/forecast_extension_provider", + "schema": { + "const": "energypriceforecast" + } + } + } + } + ) + day_ahead_prediction_extension_url: Optional[str] = Field( + default="https://raw.githubusercontent.com/corneel27/day-ahead-prediction/main/dap/data/prediction.json", + alias="day-ahead-prediction-extension-url", + description="day-ahead-prediction extension URL", + json_schema_extra={ + "x-help": "Provider-specific URL for the corneel27/day-ahead-prediction extension feed. Expected response: JSON array with fields like time_ts and prediction. This provider currently only fits the NL market.", + "x-ui-section": "Prices", + "x-ui-rules": { + "effect": "SHOW", + "condition": { + "scope": "#/properties/forecast_extension_provider", + "schema": { + "const": "dayaheadprediction" + } + } + } + } + ) # Date-based tariff configurations (date string -> value) energy_taxes_consumption: dict[str, float] = Field( @@ -152,6 +225,13 @@ class PricingConfig(BaseModel): if not (0 <= percentage <= 100): raise ValueError(f"VAT percentage must be between 0 and 100, got {percentage} for date {date}") return v + + @field_validator('forecast_extension_hours') + @classmethod + def validate_forecast_extension_hours(cls, v: int) -> int: + if not (0 <= v <= 168): + raise ValueError("forecast extension hours must be between 0 and 168") + return v model_config = ConfigDict( extra='allow', @@ -167,7 +247,7 @@ Configure electricity market prices and tariff components for accurate cost opti ## Price Components Total electricity cost consists of: -1. **Market price**: Day-ahead spot price (nordpool/entsoe/tibber) +1. **Market price**: Imported official day-ahead spot price (nordpool/entsoe/easyenergy/tibber) 2. **Energy taxes**: Government energy taxes 3. **Supplier costs**: Your supplier's markup/fees 4. **VAT**: Value-added tax on sum of above @@ -191,8 +271,15 @@ System uses tariff active on optimization date. - **nordpool**: Nord Pool (Nordic/Baltic markets) - **entsoe**: ENTSO-E Transparency Platform (all European markets) +- **easyenergy**: EasyEnergy tariff endpoint - **tibber**: Tibber API (if using Tibber as supplier) +## Optional Horizon Extension + +- **forecast extension provider**: Optional forecast provider for extending the imported official horizon +- **forecast extension hours**: How many additional hours should be appended beyond the official horizon +- **energypriceforecast**: Provider-specific extension feed from Energy Price Forecast EU + ## Tips - All prices are EXCLUDING VAT (VAT applied separately) diff --git a/dao/prog/da_report.py b/dao/prog/da_report.py index b11d8c6..986abe2 100644 --- a/dao/prog/da_report.py +++ b/dao/prog/da_report.py @@ -3041,6 +3041,21 @@ class Report(DaBase): df_da = self.db_da.get_column_data( "values", "da", start=start, end=end, agg_func=agg_func ) + ext_provider = str( + getattr(self.prices_options, "forecast_extension_provider", "none") or "none" + ).lower() + ext_hours = int(getattr(self.prices_options, "forecast_extension_hours", 0) or 0) + if ext_provider != "none" and ext_hours > 0: + df_da_ext = self.db_da.get_column_data( + "values", "da_ext", start=start, end=end, agg_func=agg_func + ) + if len(df_da_ext) > 0: + df_da["source_rank"] = 0 + df_da_ext["source_rank"] = 1 + df_da = pd.concat([df_da, df_da_ext], ignore_index=True) + df_da = df_da.sort_values(["time", "source_rank"]) + df_da = df_da.drop_duplicates(subset=["time"], keep="first") + df_da = df_da.drop(columns=["source_rank"]) old_dagstr = "" taxes_l = 0 taxes_t = 0