I saw a claim that DeepSeek had piggybacked off of some aspect of training that ChatGPT had done, and so that cost needed to be included when evaluating DeepSeek.

This training part of LLMs is still mostly Greek to me, so if anyone could explain that claim as true or false and the reasons why, I’d appreciate it

Well that was a failed response opps. I am just cautious because while transformers get the random guessing right you can get the right answer statistically but fail on accuracy improvement long term. Clearly this model does better than the current model but extending it to be even better seems basically intractable besides throw more data at it but what if it derived the wrong model you simply cannot actually know

> My question is "what happens if you scale up to attain the same levels of accuracy throughout? Will it still be as efficient?"

I've done some work in this area, and the answer is probably 'more efficient, but not quite as spectacularly efficient.'

In a crude, back-of-the-envelope sense, AI-NWP models run about three orders of magnitude faster than notionally equivalent physics based NWP models. Those three orders of magnitude divide approximately evenly between three factors:

1. AI-NWP models produce much sparser outputs compared to physics-based models. That means fewer variables and levels, but also coarser timesteps. If a model needs to run 10x as often to produce an output every 30m rather than every 6h, that's an order of magnitude right there.

2. AI-NWP models are "GPU native," while physics-based models emphatically aren't. Hypothetically running physics-based models on GPUs would gain most of an order of magnitude back.

3. AI-NWP models have fantastic levels of numerical intensity compared to physics-based NWP models since the former are "matrix-matrix multiplications all the way down." Traditional NWP models perform relatively little work per grid point in comparison, which puts them on the wrong (badly memory-bandwidth limited) side of the roofline plots.

I'd expect a full-throated AI-NWP model to give up most of the gains from #1 (to have dense outputs), and dedicated work on physics-based NWP might close the gap on #2. However, that last point seems much more durable to me.

There are multiple efforts and a good number of VC working on AI DA system. DA is fundamentally a hand-crafted optimization process just like NN. I once reimplemented an EnKF in pytorch and it works amazingly fast. But our observations are so dirty and sparse. ECMWF tuned their system so well. NOAA definitely has potential being even better, but no hope any soon future IMHO.

Not all transformers are LLMs.

"gen" stands for "generative". If you read the GenCast paper they call it a generative AI - IIRC it's an autoregressive GNN plus a diffusion model.

Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.

I used GAN to mean graph attention network in my comment, which is how the GraphCast paper defines transformers. https://arxiv.org/pdf/2212.12794

I was looking at this part in particular:

> And while Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes them analogous to GNNs (e.g., graph attention networks [49]).

Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.

I suspect the names of those perpetrating this kind of destruction will become synonymous with ignorance and intellectual cowardice.

I can see it now

    The following snippet highlights the algorithm used to determine <thing>
    ```fortran
    .....

> Are there any other benefits? Like is there a reason to believe it could be more accurate than a physics model with some error bars?

Surprisingly, the leading AI-NWP forecasts are more accurate than their traditional counterparts, even at large scales and long lead times (i.e. the 5-day forecast).

The reason for this is not at all obvious, to the point I'd call it an open question in the literature. Large-scale atmospheric dynamics are a well-studied domain, so physics-based models essentially have to be getting "the big stuff" right. It's reasonable to think that AI-NWP models are doing a better job at sub-grid parameterizations and local forcings because those are the 'gaps' in traditional NWP, but going from "improved modelling of turbulence over urban and forest areas" (as a hypothetical example) to "improvements in 10,000 km-scale atmospheric circulation 5 days later" isn't as certain.

Machine learning _has_ been widely used in weather forecasting, but in a different way than these models. Going back to the 1970's, you never just take the output of a numerical weather model and call it a forecast. We know that limitations in the models' resolution and representation of physical processes lead to huge biases and missed details that cause the forecast to disagree with real world observations. So a standard technique has been to post-process model outputs, calibrating them for station observations where available. You don't need super complex ML to really dramatically improve the quality or skill of the forecast in this manner; typically multiple linear regressions with some degree of feature selection and other criteria will capture most of the variance, especially when you pool observation stations together.

Reading the article could have helped with this.

That is another viable path, no more no less.