Sui Indexing Framework Enhances Onchain Data Access

The
Sui
Indexing
Framework
offers
customizable
access
to
Sui’s
onchain
data
through
a
powerful
data
ingestion
framework,
according
to

The
Sui
Blog.
This
framework
allows
the
collection
of
both
raw
onchain
data
and
derived
datasets
by
various
software,
whether
operating
onchain
or
offchain.

Leveraging
this
framework
enables
developers
to
create
customizable
data
feeds
that
can
build
software
and
products
responsive
to
onchain
events.
The
Sui
Indexing
Framework
addresses
the
limitations
of
traditional
blockchain
data
structures,
which
are
often
not
optimized
for
random
data
access
across
their
entire
history.
Customizable
data
feeds
built
with
this
framework
empower
developers
to
harness
onchain
data
more
effectively
for
real-time
analytics
and
responsive
applications.

The
Power
of
Onchain
Data
Feeds

For
instance,
a
musician
could
use
NFTs
to
distribute
music
to
fans.
By
creating
a
non-transferrable
NFT
collection,
each
NFT
could
grant
automatic
access
to
an
audio
file
stored
in
an
offchain
database
upon
minting.
A
custom
indexer
created
with
the
Sui
Indexing
Framework
could
track
the
minting
transactions
associated
with
these
specific
NFTs,
enabling
a
separate
offchain
service
to
perform
actions
like
transferring
audio
files,
triggered
by
events
monitored
through
the
custom
indexer.

The
framework
is
particularly
useful
for
those
seeking
a
leaner
Full
node
setup.
Without
an
indexing
solution,
Full
nodes
typically
retain
the
history
of
every
transaction.
A
custom
indexer
can
be
created
using
the
Sui
Indexing
Framework,
which
feeds
checkpoint
data
to
be
stored
separately
from
the
Full
node,
allowing
for
more
efficient
infrastructure
setups
as
Full
nodes
can
be
aggressively
pruned.

Additionally,
the
Sui
Indexing
Framework
is
crucial
for
the
development
of
onchain
data
dashboards.
While
a
data
analytics
platform
requires
many
elements,
this
framework
is
foundational
for
data
ingestion
that
these
apps
rely
on.

How
It
Works

Data
ingestion
with
the
Sui
Indexing
Framework
begins
with
subscribing
to
the
checkpoint
stream
from
Sui
to
receive
the
most
recent
data.
The
simplest
approach
is
to
subscribe
to
a
remote
store
of
checkpoint
data,
such
as
those
provided
by
Mysten
Labs:

• Testnet
  –
  https://checkpoints.testnet.sui.io
• Mainnet
  –
  https://checkpoints.mainnet.sui.io

A
worker
function
must
be
created
to
process
the
checkpoint
data.
The
main
application
then
calls
the
worker
function
whenever
it
detects
an
event
in
the
remote
store.

use async_trait::async_trait;
use sui_data_ingestion_core::{setup_single_workflow, Worker};
use sui_types::full_checkpoint_content::CheckpointData; struct CustomWorker; #[async_trait]
impl Worker for CustomWorker { async fn process_checkpoint(&self, checkpoint: CheckpointData) -> Result<()> { println!("processing checkpoint {}", checkpoint.checkpoint_summary.sequence_number); // custom processing logic ... Ok(()) }
} #[tokio::main]
async fn main() -> Result<()> { let (executor, term_sender) = setup_single_workflow( CustomWorker, "https://checkpoints.mainnet.sui.io".to_string(), 0, /* initial checkpoint number */ 5, /* concurrency */ None, /* extra reader options */ ).await?; executor.await?; Ok(())
}

For
those
operating
their
own
Full
node,
they
can
create
their
own
checkpoint
stream
by
adding
the
following

checkpoint-executor-config
information
to
the
Full
node
configuration
file:

checkpoint-executor-config: data-ingestion-dir: <path to a local directory>

Once
configured,
the
Full
node
dumps
checkpoint
data
into
a
local
directory,
and
the
indexer
daemon
listens
for
checkpoint
events
and
processes
the
data
as
new
checkpoints
arrive.
The
checkpoint
data
returned
is
a

CheckpointData
struct,
familiar
to
current
apps.
The
indexer
can
then
process
the
data
in
the
same
manner
as
hosted
subscriptions.

The
Sui
Indexing
Framework
supports
both
pull-based
and
push-based
processing
methods,
offering
developers
the
flexibility
to
choose
between
straightforward
implementation
or
reduced
latency.
This
versatility
is
crucial
for
applications
prioritizing
real-time
data
access
and
responsiveness.

Dive
Deeper

Whether
creating
apps
that
respond
to
real-time
blockchain
events
or
managing
general
data
and
infrastructure,
the
Sui
Indexing
Framework
offers
the
flexibility
and
reliability
needed.
For
detailed
implementation
guidance,
explore
the

Sui
Custom
Indexer
documentation.
To
see
the
framework
in
action,
explore
the
specialized
indexing
pipelines
used
by
Mysten
Labs,
SuiNS,
and
the
Sui
Bridge.

Image
source:
Shutterstock