Dune Enhances API Capabilities with DuckDB Integration

Dune
has
announced
substantial
enhancements
to
its
API
through
the
integration
of
DuckDB,
a
move
that
promises
to
address
user
requests
for
pagination,
larger
query
results,
and
additional
functionalities
such
as
filtering
and
sorting,
according
to

Dune.com.

Motivation
and
Context

The
improvements
stemmed
from
user
feedback
highlighting
the
need
for
pagination
and
the
ability
to
handle
larger
query
results,
which
were
previously
capped
at
1GB.
Dune’s
original
architecture
was
designed
primarily
for
visualizations
on
dashboards,
which
did
not
necessitate
large
datasets
or
pagination.
This
limitation
prompted
the
development
team
to
prioritize
these
features
towards
the
end
of
2023.

Expanding
Use
Cases

To
better
serve
the
developer
community,
Dune
adopted
a
holistic
approach,
focusing
on
real-world
use
cases
rather
than
specific
feature
requests.
This
strategy
involved
creating
user
stories
that
detailed
various
needs,
such
as
searching
and
filtering
large
datasets,
feeding
data
into
charts,
and
performing
data
science
on
crypto
transactions.
These
insights
guided
the
development
of
a
more
flexible
and
robust
API.

DuneSQL
and
Query
Results

All
data
on
Dune
is
queryable
using
DuneSQL,
which
employs
Trino,
a
distributed
query
engine.
While
DuneSQL
is
powerful,
it
posed
challenges
for
the
new
requirements,
such
as
low-latency
responses
and
cost-effective
execution.
This
led
the
team
to
seek
alternative
solutions.

Adoption
of
DuckDB

After
evaluating
various
options,
Dune
chose
DuckDB
for
its
ability
to
support
JSON
and
Parquet
data
formats,
high-performance
SQL
engine,
flexibility,
and
ease
of
use.
DuckDB
allowed
Dune
to
load
and
cache
query
results
quickly,
enabling
API
requests
to
be
served
within
sub-100
millisecond
response
times.
This
integration
significantly
improved
the
API’s
performance
and
cost-efficiency.

New
Features
and
APIs

The
integration
of
DuckDB
has
enabled
several
new
features
for
the
Dune
API:

• 
  Pagination:
  Retrieve
  data
  in
  manageable
  chunks.
• 
  Filtering:
  Apply
  filters
  based
  on
  specific
  columns
  and
  criteria.
• 
  Sorting:
  Organize
  query
  results
  in
  a
  specified
  order.
• 
  Sampling:
  Retrieve
  a
  uniform
  sample
  of
  the
  dataset
  for
  efficient
  analysis.

Moreover,
these
features
integrate
seamlessly
with
other
Dune
functionalities,
such
as
the
Query
Scheduler
and
Materialized
Views,
enhancing
the
overall
developer
experience.

Conclusion

Dune’s
integration
of
DuckDB
represents
a
significant
step
forward
in
addressing
user
needs
and
expanding
the
capabilities
of
its
API.
This
strategic
move
not
only
resolves
immediate
feature
requests
but
also
sets
the
stage
for
future
enhancements,
making
DuneAPI
a
more
powerful
tool
for
developers.

For
more
details,
visit
Dune’s

API
documentation.

Image
source:
Shutterstock