CUDA Explained: How a GPU Runs Thousands of Threads at Once

Diagram of a CUDA kernel launch: a grid split into blocks of GPU threads running in parallel on an NVIDIA GPU

CUDA programming is how you turn an NVIDIA graphics card into a general-purpose math engine: you write C or C++, and thousands of GPU threads run it at the same time. That's the engine under every fast AI answer you've ever received.

This guide follows the video step by step. You'll see what CUDA is and why it had to exist. You'll see how kernels, threads, blocks and grids fit together, and how data moves between CPU and GPU. Then come the places CUDA runs, from PyTorch to physics. Last are the three mistakes that quietly slow down GPU code.

TL;DR
  • CUDA (Compute Unified Device Architecture) is NVIDIA's platform for using GPUs beyond graphics.
  • You write a kernel, which is one function, and thousands of threads run it in parallel, each on its own data.
  • Threads group into blocks, and blocks group into a grid. A bigger GPU simply runs more blocks at once.
  • Every job is a round trip: copy data in, launch the kernel, copy results back.
  • Speed killers: needless memory transfers, warp divergence and scattered memory reads.
  1. 0:00Intro
  2. 0:22Every AI answer needs massive parallelism
  3. 1:00CUDA: general-purpose computing on GPUs
  4. 1:43Few strong cores vs thousands of small
  5. 2:252007: C/C++ instead of shader hacks
  6. 3:02Write a kernel, run it everywhere
  7. 3:45Threads → blocks → grids
  8. 4:26Copy in, launch, copy back
  9. 5:10CUDA runs wherever heavy math lives
  10. 5:40What PyTorch really does on GPU
  11. 6:21Ignoring memory transfer costs
  12. 6:56Branching inside a warp
  13. 7:34Uncoalesced memory access
  14. 8:07CUDA in six lines
  15. 8:31Think parallel: CUDA unlocked the GPU

What is CUDA, and why does AI depend on it?

Every ChatGPT reply is a mountain of tiny calculations that have to happen at the same moment. That speed doesn't come from one blazing chip. It comes from thousands of small GPU cores, each doing a sliver of the math in parallel. That raises a practical question: what tells all those cores what to do? On NVIDIA hardware, the answer is CUDA.

CUDA stands for Compute Unified Device Architecture. Each word has a job. Compute means it does math. Unified means one programming model. Device is the GPU. Architecture is the whole platform. You write C or C++, the CUDA toolkit compiles it, and the result launches on an NVIDIA GPU. The toolkit is the bridge between ordinary code and thousands of cores.

GPUs were built to shade pixels. CUDA lets them crunch any math you like. Without it, those cores sit idle for anything that isn't graphics. With it, they act as one giant math engine. That shift is why your graphics card can also run a neural network.

Why CUDA exists: few strong cores vs thousands of small

CPUs and GPUs are built for opposite jobs. A CPU chases speed on one task. It has a few powerful cores that excel at complex, branching logic with low latency. Think of a handful of professors. A GPU chases volume. It has thousands of simpler cores that apply the same operation to lots of data. Think of a stadium of students, each doing one sum.

For years, that stadium was locked behind graphics APIs. To do general math on a GPU, you had to disguise your data as pixels and textures and push it through the graphics pipeline. It was a huge engine with no steering wheel. Then, in 2007, NVIDIA launched CUDA. It was a real programming model built for compute, not for drawing triangles.

The language mattered as much as the hardware. CUDA is C and C++ with a few extra keywords. There are no fake textures and no shader tricks. Ordinary programmers could suddenly write code for the GPU directly. The fastest hardware in the box finally became programmable.

  • CPU: a few powerful cores, complex branching logic, low latency per task
  • GPU: thousands of simpler cores, same operation on lots of data, massive throughput
  • Before 2007: general GPU math meant disguising data as pixels
  • After CUDA: plain C/C++ with a few extensions

How does a CUDA kernel work?

Everything in CUDA starts with a kernel. A kernel is one function that thousands of threads run in parallel, each on its own piece of data. In this example, the __global__ keyword marks the function as a GPU kernel. Each thread works out its index i from its block and thread IDs. Then it adds one pair of numbers. The last line launches the kernel across many blocks of threads.

Here's the clever part. Every thread runs the identical function, but its index points it at a different element. It's like a thousand cashiers, each serving one customer. On a CPU, you'd write a loop over a huge array. In CUDA, the loop becomes the threads. You get parallelism without writing a single loop.

  • Same code, own data: every thread gets a different index
  • No loop needed: the threads replace the loop
__global__ void add(float* a, float* b) {
  int i = blockIdx.x * blockDim.x
        + threadIdx.x;
  a[i] += b[i];
}
add<<<blocks, threads>>>(a, b);

Threads, blocks and grids: how CUDA organises work

Thousands of threads need structure, and CUDA uses a simple hierarchy. At the top is the grid, which is the whole kernel launch. The grid splits into blocks, such as Block 0 and Block 1, and each block runs independently. Inside a block, the hardware runs threads in groups of 32 called warps. At the bottom, one thread handles one item.

Each level exists for a reason. Threads inside one block share fast on-chip memory and can sync up, like a team working at one table. Blocks, by contrast, don't depend on each other. That independence is what lets CUDA scale. A bigger GPU simply runs more blocks at once, so the same code gets faster on better hardware.

  • Grid: the entire kernel launch
  • Block: an independent group of threads that can share fast memory
  • Warp: 32 threads the hardware runs together
  • Thread: one worker handling one item

Copy in, launch, copy back: the CPU–GPU round trip

Here's the catch: the GPU has its own memory. The CPU is called the host, and the GPU is called the device. Each keeps its own memory, so data has to travel between them. That makes every CUDA job a round trip. The host stays in charge, scheduling work and moving data, while the device does the heavy math.

Lock these four steps in now, because the first big mistake lives here. Copies across this gap are never free. A kernel can be perfectly written and still lose time if its data spends too long in transit. Fast code plans its data movement as carefully as its math.

  • cudaMemcpy: the host copies input to the device
  • Launch the kernel, and thousands of threads fire
  • The GPU reports that the kernel is done
  • cudaMemcpy: the results come back to the host

Where is CUDA used? From PyTorch to physics

The rule of thumb is simple: any job with lots of identical math is a CUDA job. Deep learning uses it to train and run AI models. Simulation uses it for science like weather and physics. Crypto uses it to hash at scale. Video uses it to process frames. PyTorch runs on it under the hood. Every one of these splits huge data into tiny identical tasks.

Take one line of PyTorch: y = x @ w, with tensors on the GPU. Python calls torch.matmul. The tensors are already in VRAM, so no copy is needed. PyTorch launches a kernel as a grid of blocks. Threads compute tiles of the output, and the result y lands on the GPU. You never see any of it, because PyTorch calls optimised CUDA kernels for you.

That's what using the GPU in a deep learning framework actually means. Large AI models, like the ones behind ChatGPT, repeat this same pattern layer after layer. Understand CUDA, and you understand the engine underneath modern AI.

Mistake 1: Ignoring memory transfer costs

This one bites almost everyone. Copying data to the GPU can take longer than the math you wanted to speed up. Data starts in CPU RAM, squeezes across the PCIe bus and lands in GPU memory. That bus is far slower than the GPU's own memory, which makes it the bottleneck. A brilliant kernel can't rescue a program stuck waiting on the road between memories.

The fix is planning, not cleverness. Load data once and keep every step on the GPU, instead of sending it back to the CPU after each operation. When you must move data, one big copy beats a pile of tiny ones. When you profile, measure your transfers, not just your kernel time.

  • Keep data resident on the GPU between steps
  • Batch the work: fewer, bigger transfers win
  • Measure transfer time alongside kernel time

Mistake 2: Warp divergence from branching in a warp

The second trap hides in a plain if statement. The 32 threads in a warp move together on one shared instruction stream. In this example, even threads want doA and odd threads want doB. The warp can't split in two. So it runs A, then runs B, and half the threads sit idle each time.

That's warp divergence: the branches execute one after the other. A two-way split like this can cost you up to half your speed. The cure is to branch by warp. Arrange your data so whole warps take the same path, or replace the branch with math. Keep neighbouring threads on the same route. Uniform code is fast code.

  • Divergence makes a warp run both paths, one after the other
  • Keep whole warps on the same branch, or replace branches with math
# 32 threads in one warp
if (threadIdx.x % 2 == 0)
  doA();   # even threads
else
  doB();   # odd threads
# warp runs A, then B

Mistake 3: Uncoalesced memory access

The last trap is how threads read memory. The golden rule: neighbouring threads should read neighbouring addresses. Picture threads 0 to 3. In the coalesced pattern, each reads ai, meaning a0, a1, a2 and a3 side by side. The hardware merges those reads into one memory trip. In the uncoalesced pattern, each reads ai*32, meaning a0, a32, a64 and a96. That takes four trips.

Stride kills speed. The arithmetic is identical in both cases, and only the access pattern changes. Lay out your data so thread i touches element i, and you get the same math with far fewer memory trips. When a kernel is slower than it should be, check how its threads read memory before blaming the math.

The bottom line

  • CUDA is C and C++ for NVIDIA GPUs, launched in 2007 to replace shader hacks.
  • A kernel is one function that thousands of threads run, each using its own index.
  • Threads form blocks and blocks form grids; independent blocks let code scale to bigger GPUs.
  • Every job is copy in, launch, copy back, so plan data movement as carefully as the math.
  • CUDA powers deep learning, simulation, crypto, video and PyTorch.
  • To stay fast, avoid needless transfers, warp divergence and scattered reads.

Quick-fire questions

Is CUDA a programming language?

CUDA is NVIDIA's platform and programming model for running general-purpose code on GPUs. You write it in C and C++ with a few extra keywords, such as __global__ to mark a kernel. The CUDA toolkit compiles that code and launches it on the GPU.

When was CUDA released?

NVIDIA launched CUDA in 2007. Before then, doing general math on a GPU meant disguising data as pixels and textures and working through the graphics pipeline.

What is the difference between a thread, a warp, a block and a grid?

A thread handles one item, and the hardware runs threads in warps of 32. Threads group into blocks, which can share fast on-chip memory and sync up. All the blocks together form the grid, which is the entire kernel launch.

Do I need to write CUDA code to use a GPU with PyTorch?

No. When your tensors are on the GPU, a line like y = x @ w makes PyTorch call optimised CUDA kernels for you. Knowing how CUDA works still helps you understand what's happening and why some code runs slowly.

Why is my CUDA code slower than I expected?

A common cause is memory transfers. Copying data across the PCIe bus can take longer than the math itself. Warp divergence from branches and uncoalesced memory reads are the other usual suspects.

Does CUDA work on any GPU?

CUDA is NVIDIA's platform, and it targets NVIDIA GPUs. Your code is compiled by the CUDA toolkit and launched on NVIDIA hardware.

Watch the full video on YouTube →

Souy Soeng

Souy Soeng

Hi there 👋, I’m Soeng Souy (StarCode Kh)
-------------------------------------------
🌱 I’m currently creating a sample Laravel and React Vue Livewire
👯 I’m looking to collaborate on open-source PHP & JavaScript projects
💬 Ask me about Laravel, MySQL, or Flutter
⚡ Fun fact: I love turning ☕️ into code!

Post a Comment

CAN FEEDBACK
Ad