Overview
As I sat watching Jensen Huang’s keynote at Nvidia’s recent GTC, I was struck once again by how this annual event has evolved from a graphics card showcase into something far more consequential for global markets. Given Nvidia’s central position in the AI ecosystem—effectively becoming the picks and shovels supplier for most of the AI gold rush—Huang’s presentations have become essential viewing for understanding where artificial intelligence is heading. The roadmaps he unfurls don’t just reveal Nvidia’s strategic bets; they effectively chart the trajectory of an entire industry that is reshaping the technological landscape.
This post provides a comprehensive overview of Huang’s keynote, tracing AI’s evolution from perception to physical intelligence and detailing Nvidia’s ambitious plans for “AI factories” and next-generation hardware like the Blackwell Ultra and Rubin architectures. Scanning the online reaction, I couldn’t help but notice the growing tensions in Nvidia’s business model. While datacenter revenues soar, consumer frustrations mount over chronic GPU shortages and pricing that many consider prohibitive. Meanwhile, emerging hardware solutions like Cerebras are showing they can deliver swift responses for models requiring advanced reasoning. For all of Nividia’s technological brilliance and market dominance, these challenges highlight that even AI’s most crucial enabler must navigate the complex balance between serving enterprise customers driving the AI revolution and maintaining goodwill among its broader user base.
Moreover, the proliferation of improved toolchains for model post-training and customization across competing hardware platforms signals a potential shift in the competitive landscape, one where Nvidia’s commanding lead may face unprecedented challenges from specialized alternatives optimized for post-deployment AI workflows.
GTC 2025: Jensen Huang’s Keynote
Tracing the Growth of Intelligent Systems
AI’s Phases: From Vision to Physical Intelligence
- AI has evolved through distinct developmental phases, starting with perception (computer vision, speech recognition)
- Generative AI followed, transforming computing from simple data retrieval to dynamic content creation
- We’re now seeing agentic AI emerge with advanced reasoning capabilities
- Physical AI represents the next frontier, teaching robots to understand friction, object permanence, and other physical concepts
Each phase unlocks new capabilities while driving exponential increases in computational demands
Core Hurdles on the AI Development Path
Advancing through each new phase of artificial intelligence necessitates overcoming three core challenges. Firstly, there is the data problem, which involves providing AI systems with sufficient and relevant training experiences to learn effectively. Secondly, we face the challenge of training methodologies, requiring the creation of techniques that eliminate human-in-the-loop bottlenecks, thereby enabling super-human learning rates. Finally, the development of scaling laws is crucial, focusing on algorithms that exhibit favorable scaling properties, where the addition of more computational resources consistently translates into demonstrably smarter and more capable AI systems. These fundamental challenges become increasingly critical and complex with each evolutionary phase of AI development.
Architecting Tomorrow’s AI-Centric Data Centers
The AI Factory Paradigm Shift
- AI factories represent specialized facilities purpose-built for generative computing, not just data storage.
- They’re engineered specifically to generate tokens that manifest as text, images, videos, and research outputs.
- Purpose-built infrastructure includes specialized hardware, software solutions like NVIDIA Dynamo, and advanced cooling.
- These facilities directly impact critical business metrics: quality of service, revenue generation, and profitability.
The fundamental difference: optimization for intensive generative computing vs. traditional retrieval-based operations.
Digital Twin Workflows for Efficient AI Factory Deployment
- NVIDIA pioneers digital twins to design and optimize AI factories before physical construction begins.
- Engineers create virtual replicas of entire data centers within Omniverse Blueprint.
- Teams collaborate on integrated 3D layouts of DGX SuperPODs, cooling systems, power infrastructure, and networking.
- Real-time simulation enables optimization of Total Cost of Ownership (TCO) and Power Usage Effectiveness (PUE).
- Benefits extend beyond design to construction planning, future upgrades, and error prevention.
This approach accelerates deployment timelines while ensuring robust, optimized infrastructure.
Orchestrating Generative Workloads with Dynamo
- NVIDIA Dynamo functions as the essential operating system for AI factories.
- The open-source software orchestrates complex AI workloads across disaggregated resources.
- Key capabilities include workload management, support for various types of parallelism (pipeline, tensor).
- Intelligently optimizes resource distribution between the “prefill” phase (context processing) and “decode” phase (token generation).
- Dramatically improves resource utilization and operational efficiency across diverse, demanding workloads.
Optimizing AI Factories: Throughput, Latency, and Power
Unpacking the Complexity of Reasoning Workloads
- Reasoning AI models require substantially larger computational capacity than predecessors.
- They employ complex, step-by-step problem-solving approaches like chain-of-thought reasoning.
- Unlike direct-answer models, reasoning AI builds sequential steps with each building on the last.
- This process generates significantly more tokens—potentially two orders of magnitude greater.
- To maintain responsiveness, these models need processing speeds approximately 10x faster.
- The combination dramatically increases overall computational requirements.
Measuring Throughput and Latency Under Power Constraints
- AI factory performance balances two competing priorities under power constraints:
- Token rate (tokens per second) reflects overall throughput capacity
- Latency represents user-perceived response time, requiring high tokens per second per query.
- The ultimate efficiency metric: tokens per second per megawatt.
- This metric directly impacts economic viability as power consumption is a major operational cost
- Success depends on effectively balancing throughput and latency while maximizing efficiency.
Balancing User Experience with System Scalability
- Companies must balance quality of service (intelligence and responsiveness) with system throughput.
- Quality of service is measured by tokens generated per second per user, directly affecting satisfaction.
- Throughput represents total tokens processed per second, influencing efficiency and cost-effectiveness
- The ideal approach: push this frontier outward to achieve both dimensions simultaneously.
- Success means creating systems that deliver superior user experience while maintaining economic viability.
Driving Enterprise Transformation with Advanced AI Training
Reinforcement Learning and Synthetic Data at Scale
- Reinforcement learning (RL) overcomes human data limitations, enabling super-human learning rates.
- Synthetic data generation complements RL by providing vast training datasets.
- AI systems can generate trillions of tokens through simulated problem-solving environments.
- This combination enables continuous model improvement beyond human capabilities.
- AI agents learn through trial and error, refining strategies based on rewards received.
- Addresses the challenge of creating training datasets at scales impossible with human-created data.
Shifting to AI-Powered Knowledge Retrieval
Artificial intelligence is driving a profound transformation across enterprise computing, fundamentally altering data access paradigms and operational workflows. Future data access within enterprises is projected to shift away from traditional retrieval-based systems towards AI-driven question answering. In this new paradigm, employees will be able to ask questions in natural language, and AI systems will intelligently provide answers by leveraging the vast knowledge base of the organization. Looking ahead to 2024, it is anticipated that all software engineers will be AI-assisted, with AI agents becoming integral to the digital workforce. This integration of AI agents will not only augment the capabilities of software engineers but also drive a broader reinvention of the entire computing stack within enterprise environments, impacting everything from hardware infrastructure to application development methodologies. This transformation will necessitate the adoption of new workflows, infrastructure requirements, and human-computer interaction paradigms across businesses, as enterprises adapt to leverage the power of AI in their daily operations.
HALO: End-to-End Assurance for AI-Driven Vehicles
NVIDIA places paramount importance on automotive safety in AI applications, exemplified by their comprehensive DRIVE HALO platform. This platform embodies a multi-layered safety approach, encompassing every aspect of development, from silicon design to software implementation, algorithms, and methodologies. NVIDIA’s safety philosophy is built upon core safety principles, including diversity, monitoring, transparency, and explainability, which are deeply integrated into the development process. To ensure the highest levels of rigor and validation, every line of code within the DRIVE HALO platform, totaling over 7 million lines, undergoes safety assessment by independent third parties. This comprehensive and meticulous approach ensures the development of safer autonomous vehicle systems through rigorous validation, testing, and adherence to stringent safety principles throughout the entire development lifecycle.
Partnering for Next-Generation Vehicle Autonomy
- NVIDIA collaborates with automotive leaders like GM to accelerate autonomous driving technology.
- Provides end-to-end AI platform with three key computing elements:
-
- Training computers for AI model development.
- Simulation computers for virtual testing.
- In-vehicle computers to power autonomous systems.
-
- Offers flexible collaboration models from data center-only solutions to fully integrated systems.
- Empowers partners to accelerate development across AI-powered manufacturing, operations, and vehicle systems.
Autonomous Agents and the Next Frontier in Robotics
Autonomy and Intelligence: Hallmarks of Agentic AI
- Agentic AI represents a pivotal advancement toward more autonomous, capable systems.
- Defined by agency: ability to perceive environments, understand context, reason, plan, and execute actions.
- Distinguished by tool utilization capabilities and processing of multimodal information (text, images, audio).
- Moves beyond pattern-matching to reasoning through complex tasks requiring step-by-step thinking
- Enables more sophisticated applications through strategic planning and intelligent tool use.
Bridging the Digital-Physical Divide in Robotics
- Physical AI focuses on enabling AI systems to interact with and understand the physical world.
- Bridges the gap between digital reasoning and real-world action for robotics and autonomous systems.
- Integrates understanding of fundamental concepts like friction, inertia, and object permanence.
- Distinguished by focus on physical interaction rather than purely digital applications.
- Poised to usher in a new era of robotics applicable to real-world tasks and industries.
Hardware, Software, and Agents: NVIDIA’s Triple Focus
- NVIDIA’s vision encompasses advancements across hardware, software, and application domains.
- Hardware roadmap progresses from Blackwell to Blackwell Ultra, then to Rubin and Rubin Ultra architectures.
- Software development focuses on scaling AI factory operations with tools like Dynamo.
- Application focus expands to increasingly sophisticated agentic and physical AI systems.
- Aims to transform industries from manufacturing to healthcare through pervasive AI integration.
- Shares long-term roadmaps to help customers confidently plan and invest in AI infrastructure.
NVIDIA’s Open-Source, High-Speed Foundation Models
- NVIDIA expands beyond hardware with the Llama Nemotron AI model family
- Models enhanced through post-training techniques for improved math, coding, and reasoning capabilities.
- 20% more accurate than the base Llama models with 5x faster inference speed, reducing operational costs.
- Available in three sizes: Nano (edge/PC), Super (single GPU), and Ultra (multi-GPU deployments).
- Datasets, tools, and optimization techniques made publicly available to maintain openness.
- Major partners include Microsoft, SAP, Accenture, Atlassian, Box, and ServiceNow.
NVIDIA’s Hardware Evolution: From Blackwell to Rubin and Beyond
Democratizing GPU Power Across Domains
- NVIDIA extends accelerated computing beyond AI through domain-specific acceleration libraries.
- Enables domain experts to leverage GPU power without extensive code rewriting or specialized AI expertise.
- Growing library portfolio includes:
- cuNumeric for numerical computing
- PARABRICKS for gene sequencing
- MONAI for medical imaging
- Earth-2 for weather prediction
- PITHA for semiconductor manufacturing
- Aerial for 5G optimization
- cuOpt for numerical optimization
- CSD solvers for electronic design
This approach enables breakthroughs across diverse industries and scientific disciplines.
Radical Density, Efficiency, and Liquid Cooling
- Blackwell architecture represents a 50,000x speed increase compared to the first CUDA GPU.
- Design philosophy prioritizes extreme scale-up and efficiency before horizontal scaling.
- Key innovations include disaggregated design separating compute nodes from NVLink switches.
- Liquid cooling enables higher density and significantly improved energy efficiency.
- Delivers 25-40x faster performance than Hopper for reasoning workloads at the same power consumption.
- A single rack houses 600,000 components and delivers one exaflops of computing power.
Looking Ahead: Blackwell Ultra, Rubin, and Rubin Ultra
- Blackwell Ultra scheduled for second half of 2024, featuring 1.5x increase in FLOPS and doubled memory bandwidth
- Vera Rubin arriving second half of 2026 with new GPU architecture, CPU, NVLink 144, and HBM4 memory
- Rubin Ultra coming second half of 2027 with NVLink 576 and 15 exaflops of computing power
- Architecture following Rubin named after physicist Richard Feynman, expected in 2028
The Debut of Vera: NVIDIA’s Custom AI CPU
- Vera Rubin introduces NVIDIA’s first custom-designed CPU based on Olympus core architecture.
- Delivers twofold performance increase compared to CPUs in Grace Blackwell chips.
- Represents shift from commercial Arm-based designs to custom CPU optimized for AI workloads.
- Combined system projected to achieve 50 petaflops inference performance, more than doubling Blackwell’s 20 petaflops.
- Features generous high-bandwidth memory capacity of up to 288 GB for larger, complex models.
FP4 Quantization: Massive Gains in Throughput
The adoption of four-bit floating point precision, known as FP4, in AI models offers substantial gains in both performance and efficiency. FP4 facilitates model quantization, a technique that significantly lowers energy consumption while largely maintaining model accuracy. This is especially beneficial in power-sensitive data center environments, allowing for increased computational throughput within the same energy footprint.
- Four-bit floating point precision (FP4) offers substantial performance and efficiency gains
- Dramatically reduces model footprint from 1-2 bytes to just 4 bits per parameter
- Enables higher deployment density options:
- Deploy models on a single GPU that previously required multiple GPUs
- Deploy models twice the size using the same hardware resources
- Particularly advantageous for popular mid-sized enterprise models (24-70 billion parameters).
- Benefits common enterprise applications like retrieval-augmented generation and customer service chatbots
Support our work by subscribing to our newsletter📩
