With everyone’s eyes focused on ChatGPT, MidJourney, and other generative AI tools, I don’t see many people talking about what is happening “under the hood” of the new AI wave we are in.
In this article, I will dive deeper into what powers all AI systems; the processors.
One company is dominating the AI market
Have you heard about NVIDIA?
If you have, you may know it as it is the market leader of graphic cards for the gaming industry.
But it is also the market leader in processors used by computers in the creative and scientific industries like media and entertainment, architecture, engineering and construction, automotive, scientific research, manufacturing design, and more.
What many do not know is that NVIDIA is the global leader in artificial intelligence hardware and software.
In fact, about 70 percent of the top 500 supercomputers in the world are powered by Nvidia GPU processors and their AI programming language CUDA.
And it is the supercomputers of the world that power the various AIs you have gotten to know recently.
Let’s unpack a few of the terms before moving on to the meat of this article.
- A GPU – Graphics Processing Unit – is a specialized processor for handling complex calculations in rendering images, videos, and animations. It is often used with a traditional CPU to improve performance.
- A CPU – Central Processing Unit – is the primary processor in a computer that performs most of the processing and can be thought of as the “brain” of the computer. It can be single-core or multi-core. The leading players here are Intel and AMD (so you may have heard the term “AMD computer” or “Intel inside”). Apple released its own processors a few years ago.
- A Supercomputer is a powerful computer that can perform complex calculations at a very high speed. They are characterized by high computational power, large memory capacity, and advanced technologies. They are also very expensive (we will get back to this below).
It is hard to explain very technical terms in simple terms, but bear with me, while I give it a try…
We will get to the more “businessy stuff” shortly.
What is the difference between a CPU and a GPU, and what is their significance?
A CPU and a GPU are both computer processors, but they are designed for different tasks.
A CPU is like the “brain” of the computer, it is responsible for running the basic functions of the computer, like opening and closing programs, running the operating system, executing applications, and more. It is optimized for sequential processing and has a small number of cores.
A GPU is like the “muscles” of the computer. They are more efficient at handling large amounts of data in parallel (more cores), making them ideal for tasks such as machine learning, scientific computing, graphics, and gaming.
The GPU is located (conceptually) closer to the user. This means that the images and videos are displayed faster for the user as the data must “travel” shorter distances to be displayed.
Cats, AI chips, and world domination
There is a story floating around the internet that NVIDIA pivoted from games and graphics hardware to dominate AI chips due to former Google computer scientist Andrew Ng trying to find cat videos on Youtube using a neural network (or an AI if you want).
The story goes that Andrew Ng was working on a project at Google X lab, building a neural network (AI) that could learn independently.
The AI was supposed to learn how to pick out cats and human faces from the 10 million YouTube videos the AI was trained with.
But AIs require an enormous amount of processing power to work and get trained, and Ng was using thousands of CPUs to run his neural network.
During a breakfast meeting in 2010 with Bill Dally, now chief scientist at NVIDIA, Andrew told him about his project and all the CPUs he was using.
Bill said to Andrew that he thinks a few GPUs would be able to do a better job than the thousands of CPUs.
With the help of Bryan Catanzaro at NVIDIA, they made it happen.
With just 12 GPUs, they managed to perform the same amount of work faster and more efficiently than the CPUs at Google, thanks to the superiority of GPUs in parallel processing.
This is how NVIDIA fell into the AI industry by luck.
But the fact is that it was a genius strategic move on their part long before 2010 – a path they were on long before a Google researcher tried to identify cats in videos on Youtube.
Although gaming still is the largest revenue driver for NVIDIA, data centers (AI basically) are closing in as the fastest-growing business unit.
Other units doing well are automotive and professional visualization.
Nvidia GPUs are still the most sought-after processors in the slowing crypto market (Nvidia GPUs are particularly popular with data miners mining for Bitcoin).
Nvidia GPUs are being deployed by almost all the biggest cloud providers: AWS (Amazon), Google, Alibaba, and Azure (Microsoft).
In 2019, the latest numbers I have found says that 97.4 percent of AI accelerators in these data centers were powered by Nvidia GPUs.
A FREE weekly newsletter about AI and its applications and implications on business for non-technical business professionals and managers.
The cost of AI
I am sure that all of you are familiar with ChatGPT by now.
OpenAI, the developer of ChatGPT and the “brain” behind it, GPT-3 (GPT 3.5) runs on Microsoft’s cloud computing platform Azure, which again is powered by NVIDIA GPUs.
By the time you read this, OpenAI has started rolling out ChatGPT Pro, a paid version of ChatGPT with more power and options, which is also faster than the free version we all have come to love.
The price of ChatGPT Pro is $42/month (a nerdy hat tip to Douglas Adams’ Hitchhiker’s guide to the galaxy).
But another thing that is not discussed a lot is the cost and power required to run an AI.
It is estimated that it costs between $5-8 million to compute OpenAI’s GPT-3 on its 175 billion parameters, the variables that make up the models.
Every time it is updated with more parameters, it must run again, making it very costly.
It is rumored that the next version of GPT, GPT-4, will have 100 trillion parameters (something Sam Altman of OpenAI has denied).
Whatever the number is, we know it will be in the trillions, meaning the cost of teaching the AI will be skyrocketing – and so will the need for computational powers (GPUs) to do the training.
The GPU landscape and the future
Although Nvidia is the king of AI chips today, the race is on to challenge them.
AMD dominates GPU in game consoles like Xbox, Playstation, and Nintendo. They are working hard towards growing hardware and GPUs for data centers and supercomputers after acquiring Xilinx.
Intel is not sitting still, though they seem left behind in this race. Still, they have bought two AI chip startups, Nervana and Habana Labs.
Google has started making its chips.
Amazon bought Annapurna Labs in 2016 and now develop its own chips called Inferentia.
Baido has Kunlun, and Qualcomm has Cloud AI 100.
And then there are startups like Graphcore, SambaNova, Cerebras, Mythic AI, and Blai.
The race to position itself for the future is definitely on, and some suggest that Nvidias “monopoly” is coming to an end.
With an expectation of 40% growth of the market towards 2030, from US$ 119.78 billion in 2022 to US$ 1,597.1 billion, I am guessing that there is room for more players without Nvidia “losing” anything (except maybe % market share, but not necessarily $$ turnover).
Sustainability and AI
There are voices concerned with energy consumption by AI and how this will affect the climate crises.
Danish researchers have calculated that the energy required to train GPT-3 could have the carbon footprint of driving 700,000 km/435,000 mi.
Researchers at the University of Massachusetts Amherst have estimated that training a single AI model can emit as much carbon as five cars in their lifetimes. And that goes for only one training run.
Meta (Facebook) is one of the many companies exploring AI’s environmental impact. If you like scientific reports, you can read a report from Meta here.
As mentioned above, it costs millions of dollars just to train one AI at one time (like GPT-3).
Many companies are running AI training models, and it is not unlikely we have 1000s of AI training models running daily.
Although it is not disclosed, some smart people have estimated that one training run of GPT-3 requires about 936 MWh.
And remember, that is 175 billion parameters.
Imagine what energy Google’s new AI – Pathways Language Model – with 540 billion parameters will cost to run – once…
It is said that China’s social scoring system AI (WuDao 2.0), the one they use to control their people, is the largest in the world with 1,75 trillion parameters.
Imagine the computation power and energy needed to run WuDao 2.0 once…
And I am sure they run it regularly.
It seems the world’s energy needs are snowballing.
Now that we are moving towards “a greener world,” how does it all fit in with the change towards EVs and crypto/digital money like Bitcoin or government CBDCs?
Those are all enormous energy consumers…
A few points on supercomputers and AI researchers
Did you know that there is a rush to hire AI developers and researchers around the world?
I am sure you can imagine there is.
Back in 2016, Nvidia did not have a supercomputer. Google and Facebook did and were vacuuming the market for the best AI researchers.
But what does the best of the best researchers wants?
They want to work with the best tools – in this case, the most powerful supercomputers on the planet.
So Nvidia started developing their own supercomputers in America to win over top talent.
While the world was locked down binging in Netflix in 2020-2021, Nvida built a new supercomputer in Oxford, UK, costing about $50 million, named Cambridge-1.
This is the largest supercomputer in the UK and the 30th largest globally.
Today NVIDIA owns many supercomputers, including Selene (7x larger than Cambridge-1), which until recently was the largest privately owned Supercomputer in the world.
If it is one thing the top AI talents want more than their fat salaries, it is to work on the largest and best supercomputers available.
Until recently, this meant that if you are the best of the best in the world of AI, your choice was basically between NVIDIA, US Government (two supercomputers), Japan (one), and China (two).
I just checked the recent numbers, and we now have to add the EU (two supercomputers) and IBM (two NVIDIA Supercomputers) to that list.
What about China, Taiwan, and war
Here is a question I have been pondering.
Chip designers largely outsource manufacturing to Taiwan.
TSMC is the largest chip manufacturer in the world.
It is where NVIDIA have theirs made.
Intel has its foundries in Taiwan.
Given the recent year of geopolitical issues, including China getting more aggressive about “taking back” Taiwan.
What happens if China invades Taiwan?
I am not going into geopolitics here, but wars in recent years have always been about resources, mostly oil.
Taiwan, a close ally of the USA, is the “heart and blood” of the American tech industry (which is the largest in the world).
Without a free Taiwan, the worlds leading tech companies would suffer greatly.
I have read somewhere that someone predicts that the next big war might be over AI resources – ie, data chips…
Where is AI going
I find it interesting to take a peek below the surface of what is going on in AI.
I understand why ChatGPT and the other AI advances get the majority of the coverage – it is revolutionary and easy to see and understand.
However, I want to end this all with a small thought experiment and a question:
Where would AI be today if we did not have a pandemic that disrupted global trade and production for 2-3 years?
Moore’s law states that the number of transistors on a microchip doubles every two years. The law claims that we can expect the speed and capability of our computers to increase every two years.
If the pandemic did not happen, would AI already be twice as big and twice as advanced as it already is?
Thomas Sorheim
I am the creator of the Practical AI newsletter and The Future Handbook website. I write about all things AI and try hard to make it all understandable for non-technical people.A FREE weekly newsletter about AI and its applications and implications on business for non-technical business professionals and managers.