Decoding AI Performance: Analyzing TOPS and Tokens on NVIDIA RTX PCs

The
era
of
the
AI
PC
is
here,
powered
by
NVIDIA
RTX
and
GeForce
RTX
technologies.
This
shift
brings
a
new
way
to
evaluate
performance
for
AI-accelerated
tasks,
introducing
metrics
that
can
be
daunting
to
decipher
when
choosing
between
desktops
and
laptops,
according
to
the

NVIDIA
Blog.

Coming
Out
on
TOPS

The
first
baseline
is
TOPS,
or
trillions
of
operations
per
second.
This
metric
is
akin
to
an
engine’s
horsepower
rating,
with
higher
numbers
indicating
better
performance.
For
instance,
the
Copilot+
PC
lineup
by
Microsoft
includes
neural
processing
units
(NPUs)
capable
of
performing
upwards
of
40
TOPS,
sufficient
for
light
AI-assisted
tasks.
However,
NVIDIA
RTX
and
GeForce
RTX
GPUs
deliver
unprecedented
performance,
with
the
GeForce
RTX
4090
GPU
offering
more
than
1,300
TOPS,
essential
for
demanding
generative
AI
tasks,
such
as
AI-assisted
digital
content
creation
and
querying
large
language
models
(LLMs).

Insert
Tokens
to
Play

LLM
performance
is
measured
in
the
number
of
tokens
generated
by
the
model.
Tokens
can
be
words,
punctuation,
or
whitespace.
AI
performance
can
be
quantified
in
“tokens
per
second.”
Another
crucial
factor
is
batch
size,
the
number
of
inputs
processed
simultaneously.
Larger
batch
sizes
enhance
performance
but
require
more
memory.
RTX
GPUs
excel
in
this
area
due
to
their
substantial
video
random
access
memory
(VRAM),
Tensor
Cores,
and
TensorRT-LLM
software.

GeForce
RTX
GPUs
offer
up
to
24GB
of
high-speed
VRAM,
and
NVIDIA
RTX
GPUs
up
to
48GB,
enabling
higher
batch
sizes
and
larger
models.
Tensor
Cores,
dedicated
AI
accelerators,
significantly
speed
up
operations
required
for
deep
learning
and
generative
AI
models.
Applications
using
the
NVIDIA
TensorRT
software
development
kit
(SDK)
can
unlock
maximum
performance
on
over
100
million
Windows
PCs
and
workstations
powered
by
RTX
GPUs.

Text-to-Image,
Faster
Than
Ever

Measuring
image
generation
speed
is
another
way
to
evaluate
performance.
Stable
Diffusion,
a
popular
image-based
AI
model,
allows
users
to
convert
text
descriptions
into
complex
visual
representations.
With
RTX
GPUs,
these
results
can
be
generated
faster
than
on
CPUs
or
NPUs.
Performance
is
further
enhanced
using
the
TensorRT
extension
for
the
Automatic1111
interface,
enabling
RTX
users
to
generate
images
from
prompts
up
to
2x
faster
with
the
SDXL
Base
checkpoint.

ComfyUI,
another
popular
Stable
Diffusion
interface,
recently
added
TensorRT
acceleration,
allowing
RTX
users
to
generate
images
from
prompts
up
to
60%
faster
and
convert
these
images
to
videos
up
to
70%
faster.
The
new
UL
Procyon
AI
Image
Generation
benchmark
shows
a
50%
speedup
on
a
GeForce
RTX
4080
SUPER
GPU
compared
to
the
fastest
non-TensorRT
implementation.

TensorRT
acceleration
will
soon
be
available
for
Stable
Diffusion
3,
Stability
AI’s
new
text-to-image
model,
boosting
performance
by
50%.
The
TensorRT-Model
Optimizer
further
accelerates
performance,
resulting
in
a
70%
speedup
and
a
50%
reduction
in
memory
consumption.

The
true
test
of
these
advancements
is
in
real-world
use
cases.
Users
can
refine
image
generation
by
tweaking
prompts
significantly
faster
on
RTX
GPUs,
taking
seconds
per
iteration
compared
to
minutes
on
other
systems.
This
speed
and
security
are
achieved
with
everything
running
locally
on
an
RTX-powered
PC
or
workstation.

The
Results
Are
in
and
Open
Sourced

The
AI
researchers
behind
Jan.ai
recently
integrated
TensorRT-LLM
into
their
local
chatbot
app
and
benchmarked
these
optimizations.
They
found
that
TensorRT
is
“30-70%
faster
than
llama.cpp
on
the
same
hardware”
and
more
efficient
on
consecutive
processing
runs.
The
team’s
methodology
is
open
for
others
to
measure
generative
AI
performance
for
themselves.

From
gaming
to
generative
AI,
speed
is
crucial.
TOPS,
images
per
second,
tokens
per
second,
and
batch
size
are
all
vital
metrics
in
determining
performance.

Image
source:
Shutterstock

.
.
.

Tags