20VC: Who Wins the Inference Market | Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute | The Real Limits of Post-Training and RL with Erik Bernhardsson, Modal CEO

NVIDIA

Across the shows: 50 speakers in other episodes, 40 bullish and 15 bearish calls, past 30 days

• Erik Bernhardsson · ModalBullish

  Nvidia's real moat is its software ecosystem, which keeps its position dominant despite rival chips.

  Bernhardsson says CUDA itself is overstated and annoying to work with. The real moat is the ecosystem, including kernel optimizations, Torch, and open-source engines like vLLM and SGLang, which all support Nvidia first. He has predicted for fifteen years that a non-Nvidia accelerator would take over, and has always been wrong. He thinks Nvidia will keep growing and shipping strong chips. Rivals such as Cerebras, Etched, Trainium, TPUs and AMD may also grow, but he thinks they are still quite behind.

  “I don't think CUDA itself is the moat... But I do think all the NVIDIA, the ecosystem is... So like everything is NVIDIA first today. And that's probably going to continue.”at 24:18

CoreWeave

Across the shows: 58 speakers in other episodes, 15 bullish and 56 bearish calls, past 30 days

• Erik Bernhardsson · ModalBullish

  CoreWeave has executed the neocloud financing model exceptionally well.

  Bernhardsson says building a neocloud quickly is mainly a financial problem. The sequence is to lock in a long-term customer contract, raise money against it, buy the chips, deliver them, and repay the debt over several years. He has the most respect for the financial engineering that neoclouds do well. He says CoreWeave has done this exceptionally well. He says Nebius has also executed extremely well, and does not say which is the better business.

  “the thing that I have the most respect for is actually more on the financial engineering side... Corviva [CoreWeave] has done it exceptionally well.”at 26:28

OpenRouter

• Erik Bernhardsson · ModalBullish

  OpenRouter should keep going rather than sell; its growth and revenue are strong despite many rivals.

  Bernhardsson says he invested in OpenRouter personally and it had a great outcome, though he thinks it could have done even better standalone. Asked to make the case for rejecting an acquisition offer, he cites its growth rate, its revenue and its strong brand. He acknowledges about 500 competitors, but says the space is growing so fast and has so much revenue that he would have told them to keep going.

  “there's 500 other competitors, but the space is growing so fast and there's so much revenue I would have told them to keep going.”at 50:57

Turbopuffer

• Erik Bernhardsson · ModalBullish

  Vector databases could become a very large category, and Turbopuffer has room to expand into adjacent areas.

  Bernhardsson praises Turbopuffer's founder and says the company is somewhat specialised, doing vector databases. He thinks the category could be very large. If he ran the company, he would pursue many adjacent opportunities in databases, data warehouses and other query systems, so he sees a long runway.

  “It could be a large category. ... If I ran that company, I would have 500 different ideas of how to expand into many adjacent things around databases, data warehouses”at 54:50

AI infrastructure & data centers

• Erik Bernhardsson · Modal

  Inference will be an enormous market, like a new cloud, but raw token sales alone have no switching costs.

  Bernhardsson expects inference to become an enormously large market, comparable to the new cloud. He argues that inference alone is not enough. Providers need to bundle post-training, sandboxes, agentic tooling, and low-latency and batch inference. Selling only tokens for dollars creates no switching cost and becomes a race to the bottom, while a broad product suite gives customers a set of tools to build on and is hard to copy.

  “I think inference is gonna be an enormously large market. It's like the new cloud, basically.”at 13:46

• Erik Bernhardsson · Modal

  The inference market will fragment into many specialized sub-markets as it grows 10-100x larger.

  Bernhardsson argues the inference market grew far beyond what anyone expected, possibly 10-100x larger. At that size, it will split into many products, such as low-latency and batch inference, fine-tuning and post-training, and specialized models. He points to Jeff's classification and regression strengths as an example. A market worth a billion dollars has few sub-markets, but one worth tens or hundreds of billions does not. He still expects one provider to host many of these products.

  “...prediction about the inference market is gonna split into a lot of different things because the market got so big... It's like 10 x or a 100 x bigger than I ever thought.”at 16:58

• Erik Bernhardsson · Modal

  GPU prices are high from a supply shortage and may stay high for years before eventually falling.

  Bernhardsson says GPU prices are up because of a supply shortage, and buyers have to pay more to get the same capacity. He expects the shortage and high prices to last one, two or even five years. He is unsure of the exact timing. He agrees with Jensen Huang that supply will eventually catch up with demand and hardware prices always fall in the long run. Nvidia is shipping large volumes each month, but a higher price still gets you the chips.

  “Right now, we're in a very... There's a supply shortage, and that means that the prices have gone up. They may continue to go up for another year, two years, five years. I don't know. But eventually, they will come down.”at 22:02

• Erik Bernhardsson · Modal

  Compute demand is so large that the industry cannot overbuild data centers over the next five years.

  Bernhardsson argues demand for compute is huge and latent, with Jevons paradox at work. He says data centers cannot be built fast enough, so an overbuild is unlikely in the next five years. He adds that the real risk is a lag between spending and revenue. Some companies are using large amounts of GPUs without making money yet, and that gap could eventually become a constraint.

  “I don't think you can overbuild in the next five years. Probably not.”at 22:54

• Erik Bernhardsson · Modal

  Neoclouds moving up into inference will struggle to excel at both their core business and a new market.

  Bernhardsson says some top neoclouds, including Nebius, are trying to move up the stack into inference. He is personally skeptical. He thinks it is very hard to be good at a core business and move into a different market at the same time. He points to the complexity of Modal's own infrastructure, which took five years to build, as evidence that replicating it is hard.

  “I'm personally somewhat skeptical. I think it's, like, very hard to be good at, like, your core business and also trying to move into different market at the same time.”at 25:50

• Erik Bernhardsson · Modal

  Demand for compute and AI applications will be enormous, and all of them will need somewhere to run.

  Bernhardsson argues that the market for compute is one of the largest opportunities ever, and that the demand for compute and AI applications will be enormous. Each new AI application needs a place to run, and he hopes many of them run their own models. This underpins his view that Modal can grow into a generational company.

  “I think the demand for compute, the demand for AI applications, it's gonna be enormous, and they're all gonna need a place to run.”at 40:02

Open-source AI models

• Erik Bernhardsson · Modal

  Open-source models moving to licensing will capture more value and fund better training, and always matter.

  Bernhardsson says there will always be great open-source models. Open-source labs are moving to licensing models, which helps them capture more of the value they create. That should give them more funding to train better models. He also says open source will remain important because people want to fine-tune models and control parameters for low latency or high throughput. He compares it to Windows versus Linux.

  “Let's also remember, like, open source models, they're all moving into licensing models. So, they're getting better at capturing the value of those models. And I think that means they're gonna have more funding to train better models.”at 27:47

• Erik Bernhardsson · Modal

  Europe struggles to raise and aggregate capital, so it is unlikely to produce a leading open model.

  Bernhardsson says Europe has a lot of human capital, but it struggles to accumulate large amounts of capital and to bring talent together at one company. China has been good at accumulating talent, and the US is very good at raising large sums to train large models. He does not know whether Europe has either. He therefore doubts Europe will produce a competitive open-source model, though he hopes it does.

  “I think both of those things are things that Europe struggles with... I don't know if Europe has the... Either of those.”at 28:36

• Erik Bernhardsson · Modal

  Open-source inference will not plateau; it and frontier inference will each grow 5-20x over the next year.

  Harry Stebbings cited Jason Lampkin's view that open-source LLM inference usage has peaked, with frontier providers pricing cheaply and US enterprises avoiding Chinese models. Bernhardsson rejects the fixed-pie framing and argues both open-source and frontier inference will grow five to twenty times over the next twelve months and beyond. He supports this with Modal's experience: the team initially thought open-source inference would be a commodity race to the bottom, then underestimated its scale until DeepSeek came out, after which Modal grew its team from one or two people to about 30 and is now among the top three providers.

  “I think that's a little bit of a fixed pie size mental... I do think both spaces will grow five x, 10 x, 20 x over the next twelve months and beyond.”at 38:13

AI pricing & unit economics

• Erik Bernhardsson · Modal

  AI businesses will not earn classic software margins because each layer of the stack takes a cut.

  Bernhardsson says almost no company today has the 90%-plus software margins that Silicon Valley once had. The reason is the layered vendor stack: applications, models, infrastructure, cloud and Nvidia each take a cut. Each layer runs at roughly 20-30% gross margin, so a dollar from the customer is split across many vendors. Inference providers can still make a lot of money because the market is large. He thinks Modal could reach 40-60% margins long term but will never reach software margins.

  “Basically almost no company today has software margins anymore... the 90% software margin businesses are gone... we're never going to get to software margins.”at 15:43

• Erik Bernhardsson · Modal

  Coding-agent spend will keep rising as efficiency gains unlock new use cases, not fall.

  Harry Stebbings asked whether spend on Claude Code will fall as efficiency improves. Bernhardsson says it will just eat more of engineers' workflows. He argues that every time something becomes more efficient or powerful, it unlocks new use cases and demand rises, as has happened throughout the last fifty years of computing and software.

  “it's just gonna eat more... you make something more efficient, you make something more powerful, it just unlocks like five new use cases, and then demand just goes up in the end.”at 45:16

Frontier AI labs & model race

• Erik Bernhardsson · Modal

  Post-training and RL may not extend to long-horizon tasks like engineering work with long feedback loops.

  Bernhardsson says post-training works well where feedback is finite and fast, such as coding (where code can run in sandboxes), math with Lean, cybersecurity, and possibly law with evals. Many labs run Modal sandboxes for this. He is unsure it extends beyond that. Tasks with much longer feedback loops, like the work of engineers on a longer time scale, may hit limitations in reinforcement learning. He hedges, noting he has been wrong for a long time.

  “But like, I don't know if it extends beyond that. Like, I don't know. Like there's a lot of tasks out there that are like have way longer feedback loops that I don't know if like post training quite yet are is within reach.”at 42:01

• Erik Bernhardsson · Modal

  Companies selling RL environments and data to frontier labs are making a lot of money today.

  Bernhardsson says if he were 20 and wanted to make a lot of money, he would start an environment company and sell data to frontier labs, since there is so much money in that space. He has angel-invested in many such startups and considers them incredible businesses today. He doubts they will make a lot of money in the very long run, but says today they can have extremely good cash flow selling these data sets, and that the best data providers are moving into deployment and evaluation to expand margins.

  “If I was 20, and I just wanted to make a lot of money, I would go out and start an environment company and just sell data to Frontier Labs. There is so much money to be made in that space.”at 45:51

Enterprise AI adoption & AI software

• Erik Bernhardsson · Modal

  Owning proprietary models suits some use cases but not all; latency, cost and performance often matter more.

  For some use cases, fine-tuning an open-source model gives good performance, and you know the base model will not change, unlike frontier labs which have sometimes changed models under customers. But Bernhardsson says owning your intelligence is only one of many considerations. Latency, cost, performance, hackability and a platform of adjacent services matter too, and he does not think it makes sense for every company to own its own intelligence. Modal customers such as Suno, Runway and Typeset run proprietary models on the platform.

  “For some stuff, it's good to own your own intelligence. For other stuff, it's not... I don't think it makes sense for everyone to own their own intelligence.”at 47:41

• Erik Bernhardsson · Modal

  Model routing can be a standalone business today, but long term it likely becomes part of a broader platform.

  Asked whether model routing is defensible or will be commoditised, Bernhardsson says a successful standalone routing business is possible today and these are good businesses. But in the very long term he thinks routing has to be part of a more comprehensive platform, because customers want evals, fine-tuning, and control over latency and throughput. He sees this as true of AI tools generally.

  “I think it's possible to build a successful business in model routing. I think you can do that today as a standalone product. I wonder if long term it has to be part of a more comprehensive platform.”at 50:10

AI & jobs

• Erik Bernhardsson · Modal

  Software engineers will be fine for the next decade or two because productivity gains increase software demand.

  Bernhardsson has coded for thirty years through changes in languages, cloud and frameworks, and each time programmers became more efficient, demand for software rose. He argues the developer's job is not translating English into code but understanding requirements and acting as the interface between humans, and that this role keeps moving up the stack, from assembly to agents. He therefore expects software engineers to be fine for the next ten to twenty years.

  “I'm extremely bullish on software engineering. ... every single time we make programmers more efficient, it just increases demand for software.”at 43:07

AI bubble & tech valuations

• Erik Bernhardsson · Modal

  He is almost certain AI-driven growth will be followed by a 20-30% stock market pullback, but remains bullish.

  Bernhardsson says AI is growing so fast that at some point there will be a pullback of 20 to 30% in the stock market, and everyone will freak out and say it is the end of the world. He is almost certain it will happen. He is not worried by it, because he believes the long-term opportunity is so large that growth will keep going after the pullback.

  “there's gonna be a pullback, you know, 20, 30% in the stock market, everyone's gonna freak out... I'm almost certain it's gonna happen, but, like, I'm so bullish”at 51:54

Venture capital & startups

• Erik Bernhardsson · Modal

  He would avoid companies dependent on continual equity infusions, which are vulnerable to a VC pullback.

  Asked what is most overheated in AI, Bernhardsson says he does not think anything is overheated. He is nervous about companies dependent on a continual infusion of equity capital. If there is a venture capital pullback, those companies will struggle to raise more money, so he would not invest in them. He advises founders to raise every round as if it were their last, with a real path to commercialisation and revenue.

  “one of the things that I'm a little nervous about is companies that are dependent on continual infusion of equity capital. I would not invest in them.”at 55:41