Are LLMs About To Hit A Wall? | Commentary

Available to WrapPRO members

Throwing more data, compute, and energy at the problem may run its course. What’s next?

AI
(AI-generated image by Midjourney/Big Technology)

Each new generation of large language model (LLM) consumes a staggering amount of resources. 

Meta, for instance, trained its new Llama 3 models with about 10 times more data and 100 times more compute than Llama 2. Amid a chip shortage, it used two 24,000 GPU clusters, with each chip running around the price of a luxury car. It employed so much data in its AI work, it considered buying the publishing house Simon & Schuster to find more. 

Afterward, even its executives wondered aloud if the pace was sustainable.

“It is unclear whether we need to continue scaling or whether we need more innovation on post-training,” Ahmad Al-Dahle, Meta’s VP of GenAI, told me in an interview last week.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.