Post

Conversation

Andrej Karpathy
@karpathy
DeepSeek (Chinese AI co) making it look easy today with an open weights release of a frontier-grade LLM trained on a joke of a budget (2048 GPUs for 2 months, $6M). For reference, this level of capability is supposed to require clusters of closer to 16K GPUs, the ones being brought up today are more around 100K GPUs. E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). If the model also passes vibe checks (e.g. LLM arena rankings are ongoing, my few quick tests went well so far) it will be a highly impressive display of research and engineering under resource constraints. Does this mean you don't need large GPU clusters for frontier LLMs? No but you have to ensure that you're not wasteful with what you have, and this looks like a nice demonstration that there's still a lot to get through with both data and algorithms. Very nice & detailed tech report too, reading through.
Quote
DeepSeek
@deepseek_ai
🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n
Show more
GIF
Image
2.6M
Views
David Watson 🥑
Post your reply

Emad
@EMostaque
When you have 100k GPUs everything looks like a nail, but when you have 2k you really have to focus Lots of custom things in the technical report as they needed the improvements given lack of relative compute availability Still feel 14 tr tokens is a lot vs what may be needed
49K
Maynard Handley 🟦
@handleym99
Didn't "we" already know much of this. eg - Learn from simple to complex (TinyStories paper) - Learn from quality sources (Textbooks are all you need paper) - Don't moosh together random text (Apple's Dataset Decomposition: Pretrain LLMs with Variable Sequence Lengths) - I
Show more
3.9K
James Darpinian
@modeless
Software expands to fill the available resources. If you want more efficient software, build it on less powerful hardware. AI training runs are no exception!
16K
shreyshahi
@shreyshahi
DeepSeek showed us what we should have known, you need both talent and constraints to be really creative. Throwing money at the problem works but leads to inefficiencies at every single layer.
18K
Sebastian Raschka
@rasbt
> E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). Super interesting! And DeepSeek was trained in H800’s which are probably also a tad (or noticeably?) slower than Meta’s H100’s.
40K
Justin
@AlwaysUhhJustin
I'm trying to understand the implications here. Could the labs with huge budgets just steal all the algorithms used and then scale up the data cleaning & training process and produce a fantastic model?
8.3K
David Shapiro ⏩
@DaveShapi
USSR had to do a lot of the same stuff, keeping up with the US on shoestring budgets. Constraints are beautiful. Necessity is the mother of invention, and constraints are the father of creativity.
20K
sarah guo // conviction
@saranormous
Still reading, a lot here - is it right to say 1) dualpipe/comms optimizations and 2) fine grained MOE/load balancing strategy for expert utilization are the biggest drivers of compute efficiency? with FP8 as enabler (reducing mem footprint, efficient comms, bigger batches)
43K
bloop
@optimistbloop
What does this mean? Is it possible the improvements they made to offset the constraints of a much smaller model can also be applied to bigger models.. and can we expect similar ~11x capability improvements at 100k clusters?
10K
$SPENCE
@spencience
i just had a vision of a world where chinese AI co's making frontier-grade LLMs for pennies on the dollar, this is our future, we'll have the models and the means to do something about the problems facing us, let the efficiency begin
2.6K
Jasper 🤘🌪️
@zjasper666
I was told by the
@deepseek_ai
team that they only have less than 100 people including all functionality teams. It's impressed that with a small amount of talents and compute, they can deliver the SOTA open model 🔥
7.9K
Atlas3D
@Orwelian84
Feynman was right when he said there's plenty of room at the bottom - it doesn't just apply to physics.
7K
OG Whippersnapper
@RobertDobalina7
My working hypothesis is that most of the GPUs are going to the government, something like a Manhattan Project for AI, and the for-profits are scraping by with the leftovers lest they be regulated out of existence.
2.9K
Secular Christmas Robot
@jwatte
This probably also means significant extra gains are still to be had by continuing training! It's a very chonky model though. Takes a full 8 GPU node to host!
2.2K
Yu Yang
@Yang_ML_Estate
The limit supply of newest GPU pushed the limit of engineers , to use the sources more wisely. That’s how top notch engineers work with the physical limit!
6.1K
Huntley
@Rockstox
Quite possible that a lot of their 'efficiency' wins are coming from the the fact that there are a number of frontier models that you can train off of now.
817
wh
@nrehiew_
the engineering is absurd
Quote
wh
@nrehiew_
How to train a 670B parameter model. Let's talk about the DeepSeek v3 report + some comparisons with what Meta did with Llama 405B
Image
9.9K
JuanPa
@Juanchacra
Is the US sure they want to “Keep China out of the AI race”? Feels to me that we will be racing….from behind.
8.3K
Square profile picture
Reown
@reown_
Ad
AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.
Build for free
From reown.com
50M
Vaibhav (VB) Srivastav
@reach_vb
Scarcity breeds innovation
Quote
Vaibhav (VB) Srivastav
@reach_vb
Scarcity breeds Innovation - cost to build a frontier model - 5.5 Million USD In a way, it's the maximum it'd be (Note: H800s have ~2x slower chip-to-chip data transfer) This cost, will only go down further and further as we continue to find newer walls to scale! x.com/reach_vb/statu…
Show more
Image
15K
Hot Aisle
@HotAisle
Quote
Anthonix
@zealandic1
We are now down to 160 GPU hours for speed running SOTA evals in the 100-200M param smol model class (~31x less compute!) 🚀 Thanks to @KoszarskyB, @leloykun, @Grad62304977, @YouJiacheng et al. for their work that helped to make this possible, and to @HotAisle for their great x.com/karpathy/statu…
Show more
Image
308
Radi
@X0Radi
How did they do it? Would 10x more compute have made much of a difference for V3?
558
LΞOZC
@leozc
Thank you for the long-overdue attention to this matter. We all hope that the world becomes fairer and that these remarkable researchers receive the necessary equipment to conduct their work. However, it is likely that their company will face sanctions very soon, as they are
Show more
2.2K
Pseudo Prophet
@PseudoProphet
You yourself Sir, have done an excellent job and taken us all to a respectable level of GPT-2 (124M) on a micro budget from the scratch. Hopefully we will see a GPT-3 level model next. 🤞
471
NewAIWorld
@Newaiworld_
One of the biggest improvements in 2024 was the quality - price ratio! Following this trend then models like o3 will be ~for free in 2025. This is going to be a massive boost for the AI revolution 🚀
879
Square profile picture
Reown
@reown_
Ad
AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.
Build for free
From reown.com
49M
Breck Yunits
@breckyunits
I just did a DeepSeek vs Claude comparison and was suprised to see DeepSeek win:
Quote
Breck Yunits
@breckyunits
I put the new DeepSeek v3 model head-to-head versus Claude Sonnet 3.5. The winner will surprise you:
8.5K
Square profile picture
Reown
@reown_
Ad
AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.
Build for free
From reown.com
72M
Ronny Bruknapp
@RonnyBruknapp
Efficiency at its finest. Wonder if this marks a shift toward more lean, cost-effective AI development. Thoughts on future implications for smaller players?
2.2K
PwnFunction
@PwnFunction
of course frontier LLMs don't need large GPU clusters, just look at me, all i consume is a pizza a day and i don't even hallucinate..
2.1K
Square profile picture
Reown
@reown_
Ad
AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.
Build for free
From reown.com
50M
Bitcoin Duke
@bitcoinduke
It's always like that. When you don't have an unlimited budget, you need to focus and squeeze out every little performance gain. Impressive leap forward ngl
405
Bobby van Gilder
@mindnotnull
In my experience, companies tend to throw hardware at performance problems rather than refactoring or rethinking, reducing functionality. It’s easier, and many “coders” lack a solid foundation in mathematics and little to no critical/analytical thinking prowess. Being efficient
Show more
5.2K
Aryan Agal
@aryanagxl
Deepseeks team is an ultra talented cohort of exquants Quants are famous for squeezing out every little performance gain They just did it again, only in a different domain. High IQ individuals are just a blessing to the world
5.6K
Justin Trugman
@justin_trugman
The potential impact of
@deepseek_ai
’s new model on Nvidia is huge. If you can train a model comparable to one requiring 16K GPUs using only 2K GPUs, it reduces the competitive advantage of investing in massive training clusters.
1.5K
Ilya Chetviorkin
@chetvil
With data distilled from a good model (e.g gpt-4) it’s a lot easier and feasible to train strong model on budget. This is a basic distillation.
173
Sebastian
@createX56
Theres 3 components: talent, data, and compute. They have pretty much any data they want to access to. They probably dont have as much an advantage in human talent and compute as America. Maybe they have decent talent to be equivalent, and compute will catch up. Data
Show more
3.4K
gary 🕸️³
@garyfung
Mighty impressive. Makes American AI labs look bad
Quote
gary 🕸️³
@garyfung
Replying to @giffmana
Yet, if it’s more cracked at coding tasks than both gpt4 and Claude. And with such “cheap” and fast training. That’s actually even more impressive Literally using competitor model as source of synthetic data to beat them at their own game
Show more
206
baba
@_daBaba
Some speculate that it was trained on the output of frontier models. What are your thoughts on that?
241
Square profile picture
OnChain AI
@TheOnChainAI
Resource efficiency is impressive. OnChain AI similarly optimizes operations with blockchain-based recording. The forefront of cost-effective intelligence applications.
1.7K
Human being 📐
@simplehumanhaha
Does this mean they have a much better algorithm for churning through large amount of training data while keeping compute usage as efficient as possible?
74