NVIDIA NVLink and NVSwitch Enhance Large Language Model Inference

Large
language
models
(LLMs)
are
expanding
rapidly,
necessitating
increased
computational
power
for
processing
inference
requests.
To
meet
real-time
latency
requirements
and
serve
a
growing
number
of
users,
multi-GPU
computing
is
essential,
according
to
the

NVIDIA
Technical
Blog.

Benefits
of
Multi-GPU
Computing

Even
if
a
large
model
fits
within
a
single
state-of-the-art
GPU’s
memory,
the
rate
at
which
tokens
are
generated
depends
on
the
total
compute
power
available.
Combining
the
capabilities
of
multiple
cutting-edge
GPUs
makes
real-time
user
experiences
possible.
Techniques
like
tensor
parallelism
(TP)
allow
for
fast
processing
of
inference
requests,
optimizing
both
user
experience
and
cost
by
carefully
selecting
the
number
of
GPUs
for
each
model.

Multi-GPU
Inference:
Communication-Intensive

Multi-GPU
TP
inference
involves
splitting
each
model
layer’s
calculations
across
multiple
GPUs.
The
GPUs
must
communicate
extensively,
sharing
results
to
proceed
with
the
next
model
layer.
This
communication
is
critical
as
Tensor
Cores
often
remain
idle
waiting
for
data.
For
instance,
a
single
query
to
Llama
3.1
70B
may
require
up
to
20
GB
of
data
transfer
per
GPU,
highlighting
the
need
for
a
high-bandwidth
interconnect.

NVSwitch:
Key
for
Fast
Multi-GPU
LLM
Inference

Effective
multi-GPU
scaling
requires
GPUs
with
excellent
per-GPU
interconnect
bandwidth
and
fast
connectivity.
The
NVIDIA
Hopper
Architecture
GPUs,
equipped
with
fourth-generation
NVLink,
can
communicate
at
900
GB/s.
When
combined
with
NVSwitch,
every
GPU
in
a
server
can
communicate
at
this
speed
simultaneously,
ensuring
non-blocking
communication.
Systems
like
NVIDIA
HGX
H100
and
H200,
featuring
multiple
NVSwitch
chips,
provide
significant
bandwidth,
enhancing
overall
performance.

Performance
Comparisons

Without
NVSwitch,
GPUs
must
split
bandwidth
into
multiple
point-to-point
connections,
reducing
communication
speed
as
more
GPUs
are
involved.
For
example,
a
point-to-point
architecture
provides
only
128
GB/s
of
bandwidth
for
two
GPUs,
whereas
NVSwitch
offers
900
GB/s.
This
difference
substantially
impacts
overall
inference
throughput
and
user
experience.
Tables
in
the
original
blog
illustrate
the
bandwidth
and
throughput
benefits
of
NVSwitch
over
point-to-point
connections.

Future
Innovations

NVIDIA
continues
to
innovate
with
NVLink
and
NVSwitch
technologies
to
push
real-time
inference
performance
boundaries.
The
upcoming
NVIDIA
Blackwell
architecture
will
feature
fifth-generation
NVLink,
doubling
speeds
to
1,800
GB/s.
Additionally,
new
NVSwitch
chips
and
NVLink
switch
trays
will
enable
larger
NVLink
domains,
further
enhancing
performance
for
trillion-parameter
models.

The
NVIDIA
GB200
NVL72
system,
connecting
36
NVIDIA
Grace
CPUs
and
72
NVIDIA
Blackwell
GPUs,
exemplifies
these
advancements.
This
system
allows
all
72
GPUs
to
function
as
a
single
unit,
achieving
30x
faster
real-time
trillion-parameter
inference
compared
to
previous
generations.

Image
source:
Shutterstock