Enhancing LLM Tool-Calling Performance with Few-Shot Prompting

LangChain
has
recently
unveiled
the
results
of
its
experiments
aimed
at
enhancing
the
performance
of
large
language
models
(LLMs)
in
tool-calling
tasks
through
few-shot
prompting.
According
to
the

LangChain
Blog,
the
experiments
demonstrate
that
few-shot
prompting
significantly
improves
model
accuracy,
particularly
for
complex
tasks.

Few-Shot
Prompting:
A
Game
Changer

Few-shot
prompting
involves
including
example
model
inputs
and
desired
outputs
in
the
model
prompt.
Research,
including
a
study
referenced
by
LangChain,
has
shown
that
this
technique
can
drastically
enhance
model
performance
across
a
broad
spectrum
of
tasks.
However,
there
are
numerous
ways
to
construct
few-shot
prompts,
and
few
established
best
practices
exist.

LangChain’s
experiments
were
conducted
on
two
datasets:

Query
Analysis
and

Multiverse
Math.
The
Query
Analysis
dataset
involves
invoking
different
search
indexes
based
on
user
queries,
while
the
Multiverse
Math
dataset
tests
function
calling
in
a
more
complex,
agentic
workflow.
The
experiments
benchmarked
multiple
OpenAI
and
Anthropic
models,
experimenting
with
various
methods
of
providing
few-shot
examples
to
the
models.

Constructing
the
Few-Shot
Dataset

The
few-shot
dataset
for
the
Multiverse
Math
task
was
created
manually
and
contained
13
datapoints.
Different
few-shot
techniques
were
employed
to
evaluate
their
effectiveness:

• Zero-shot:
  Only
  a
  basic
  system
  prompt
  and
  the
  question
  were
  provided
  to
  the
  model.
• Few-shot-static-msgs,
  k=3:
  Three
  fixed
  examples
  were
  passed
  as
  messages
  between
  the
  system
  prompt
  and
  the
  human
  question.
• Few-shot-dynamic-msgs,
  k=3:
  Three
  dynamically
  selected
  examples
  were
  passed
  as
  messages
  based
  on
  semantic
  similarity
  between
  the
  current
  and
  example
  questions.
• Few-shot-str,
  k=13:
  All
  thirteen
  examples
  were
  converted
  into
  one
  long
  string
  appended
  to
  the
  system
  prompt.
• Few-shot-msgs,
  k=13:
  All
  thirteen
  examples
  were
  passed
  as
  messages
  between
  the
  system
  prompt
  and
  the
  human
  question.

Results
and
Insights

The
results
revealed
several
key
trends:

• Few-shot
  prompting
  significantly
  improves
  performance
  across
  the
  board.
  For
  instance,
  Claude
  3
  Sonnet’s
  performance
  increased
  from
  16%
  using
  zero-shot
  to
  52%
  with
  three
  semantically
  similar
  examples
  as
  messages.
• Using
  semantically
  similar
  examples
  as
  messages
  yields
  better
  results
  than
  using
  static
  examples
  or
  strings.
• The
  Claude
  models
  benefit
  more
  from
  few-shot
  prompting
  than
  the
  GPT
  models.

An
example
question
that
initially
received
an
incorrect
answer
without
few-shot
prompting
was
corrected
after
few-shot
prompting,
demonstrating
the
technique’s
effectiveness.

Future
Directions

The
study
opens
several
avenues
for
future
exploration:

1. Comparing
   the
   impact
   of
   inserting
   negative
   few-shot
   examples
   (wrong
   answers)
   versus
   positive
   ones.
2. Identifying
   the
   best
   methods
   for
   semantic
   search
   retrieval
   of
   few-shot
   examples.
3. Determining
   the
   optimal
   number
   of
   few-shot
   examples
   for
   the
   best
   performance-cost
   trade-off.
4. Evaluating
   whether
   trajectories
   that
   include
   initial
   errors
   and
   subsequent
   corrections
   are
   more
   beneficial
   than
   those
   that
   are
   correct
   on
   the
   first
   pass.

LangChain
invites
further
benchmarking
and
ideas
for
future
evaluations
to
continue
advancing
the
field.

Image
source:
Shutterstock