NVIDIA’s NV-Embed Model Achieves Top Spot on MTEB Leaderboard
NVIDIA’s
latest
embedding
model,
NV-Embed,
has
set
a
new
record
for
embedding
accuracy
with
a
score
of
69.32
on
the
Massive
Text
Embedding
Benchmark
(MTEB),
which
encompasses
56
diverse
embedding
tasks,
according
to
NVIDIA
Technical
Blog.
Understanding
the
Metrics
for
Embedding
Models
Embedding
models
are
evaluated
using
several
metrics,
with
Normalized
Discounted
Cumulative
Gain
(NDCG)
and
Recall
being
the
most
significant.
NDCG
is
a
rank-aware
metric
that
measures
the
relevance
and
order
of
retrieved
information,
while
Recall
is
a
rank-agnostic
metric
that
measures
the
percentage
of
relevant
results
retrieved.
Most
benchmarks
report
NDCG@10,
but
for
enterprise-grade
retrieval-augmented
generation
(RAG)
pipelines,
Recall@5
is
often
recommended.
What
are
MTEB
and
BEIR?
The
MTEB
benchmark
covers
56
tasks,
including
retrieval,
classification,
re-ranking,
clustering,
and
summarization.
BEIR
focuses
on
retrieval
tasks
and
adds
complexity
with
different
question
types
and
domains,
such
as
fact-checking
and
biomedical
questions.
MTEB
is
largely
a
superset
of
BEIR,
making
it
a
comprehensive
benchmark
for
evaluating
embedding
models.
NV-Embed’s
performance
on
MTEB
has
been
exceptional,
achieving
an
NDCG@10
score
of
69.32,
the
highest
among
all
models
tested.
This
performance
is
attributed
to
several
key
improvements
in
the
model’s
architecture
and
training
process.
Key
Improvements
in
NV-Embed
-
Latent
Attention
Layer:
This
new
layer
simplifies
the
process
of
combining
the
mathematical
representation
(embeddings)
of
a
series
of
words,
improving
the
model’s
efficiency
and
accuracy. -
Two-Stage
Learning
Process:
The
first
stage
uses
in-batch
negative
and
hard
negative
pairs
for
contrastive
learning,
while
the
second
stage
blends
data
from
non-retrieval
tasks
for
further
training,
enhancing
the
model’s
robustness.
These
advancements
make
NV-Embed
a
powerful
tool
for
enterprise
retrieval
workloads,
although
its
effectiveness
depends
on
the
nature
and
domain
of
the
data
being
used.
Prototyping
with
NV-Embed
NV-Embed
is
available
through
NVIDIA’s
API
catalog,
allowing
organizations
to
integrate
this
high-performing
model
into
their
data
processing
pipelines.
Additionally,
the
NVIDIA
NeMo
Retriever
collection
of
microservices
enables
seamless
connection
of
custom
models
to
diverse
business
data,
delivering
highly
accurate
responses.
For
further
details,
visit
the
NVIDIA
Technical
Blog.
Image
source:
Shutterstock
.
.
.
Comments are closed.