Let me know if you have any questions or feedback and I'll do my best to respond to them.
- Dagster for data pipeline
- Parquet files for data format
- Cloudflare R2 for data storage
- DuckDB for data processing
- NodeJS for API
- NextJS + React for the app
- MapLibre for the base map functionality
- Deck.gl for the complex map layer (eg. wind particles)
- WeatherLayers (+ GFS) for the wind particles
- Cloudflare as a caching layer
The main difficulty was learning about the underlying data and industry nuances. I've been working on this map the past few years and I'm still learning new things that force me to change my approach. It's an incredibly complex domain and it's part of the reason I made this, to try and make this complex data a bit more digestible.
It would be amazing if this was extensible and wind farms and solar farms outside of GB could contribute their data as well.
It’s also added grid boundaries and cabling recently - and I think there’s also wind roses planned?
My one UI wish - the energy sources filters are currently displayed as options on the map - rather than controls for the entire UI - which meant I missed them entirely at first. Would be great if they could be given more prominence.
That's a great point about the controls. Originally the map only showed wind farms and as I've expanded the features the control system (and really the entire UX) hasn't kept up. I need to sit down and have a proper think about how best to design the controls and documentation.
Would be nice to have a definition of 'Turn-up' and 'Curtailment' in that FAQ for us non grid savvy visitors.
Works great though, well done.
Cool!
Two suggestions/questions if you may:
1) Would be good to see how many MW each boundary can handle, not just %? Btw, I can't see the number for the south east england boundary.
2) Great job on the battery info. I'm seeing some battery storage is curtailed. How is that possible? Please don't tell me that we are paying batteries to _not_ export :/?
1) Absolutely agree. The current approach for the boundaries is a quick hack until I can implement something more sophisticated. Safe to say an update is already in the works that adds a MW value and more insight into the state of each boundary (and is also more accurate in general)
2) "Please don't tell me that we are paying batteries to _not_ export" – it's actually the opposite, the batteries paid to not export (at least today). You can dig into this yourself via the Detailed System Prices dataset [0] and looking at one of the batteries on the sell stack (eg. KILSB-5)
Unfortunately you've got at least one negative wrong in this sentence and I'm still confused, and the linked dataset is currently blank?
Sorry for complaining, this is a great website!
The simplest answer I can give is that assets place bids and a volume of energy that they are willing to turn down if the system operator needs to. Those bids are either positive or negative in value and this depends a lot on the type of asset, for example wind assets usually bid negative (ie. we pay them to turn down) while gas assets usually bid positive (ie. they pay us to turn down). The reason for that is a lot to do with complexities of the market and also the cost of running that assets, the cost of fuels, etc.
So actually the battery operator is paying _the grid_ to turn down output from whatever was previously agreed (because they think they'd get more money for it later?).
And this shows as curtailment on the map?
Let me know if I'm directionally right here. If I am, it would be good to see 'bad curtailment' vs 'good curtailment' (i assume if bids are negative/positive?)
In terms of stack I have a self-hosted Dagster [1] data pipeline that periodically dumps the data onto Cloudflare R2 as parquet files. I then have a self-hosted NodeJS API that uses DuckDB to crunch the raw data and output everything you see on the map.
[0] Mostly from https://bmrs.elexon.co.uk/ [1] https://dagster.io/
Been following your projects and writing since the Mozilla days, always impressed!
Would show that off and zoom into them when selecting them from the list view!