Anyscale Introduces Multi-Tenant Serve Applications with Containerized Runtime Environments

In
a
recent
update,
Anyscale,
a
leading
AI
application
platform,
announced
the
introduction
of
multi-tenant
serve
applications
utilizing
runtime
environments
as
containers.
This
development
aims
to
enhance
resource
management
and
operational
efficiency,
according
to

Anyscale.

Advancements
in
Multi-Application
Support

In
an
enlightening
conversation
between
Sam
Chan,
Technical
Program
Manager
at
Anyscale,
and
Cindy
Zhang,
the
two
discussed
the
advancements
and
challenges
of
multi-application
serve
clusters
with
different
dependencies.
Multi-application
support
allows
different
applications
to
run
on
the
same
cluster,
each
using
the
same
runtime
environments
as
containers.
This
approach
helps
manage
resources
more
effectively
and
reduces
operational
complexity,
enabling
independent
upgrades
for
different
applications.

Zhang
highlighted
the
previous
limitations,
where
users
had
to
bundle
all
model
dependencies
into
one
large
Docker
image,
leading
to
bloated
images
and
mixed
dependencies.
This
was
particularly
challenging
for
customers
with
multiple
research
teams
working
on
separate
models.
The
new
feature
allows
each
team
to
deploy
their
code
in
its
own
container,
offering
cleaner
isolation
and
easier
maintenance.

The
Role
of
Runtime
Environments
as
Containers

The
new
feature,
“runtime
environments
as
containers,”
permits
specifying
a
different
Docker
image
for
each
application.
When
Ray
needs
to
start
a
replica
for
an
app,
it
will
initiate
a
container
from
that
app’s
image
and
run
the
worker
process
inside.
This
ensures
clean
isolation
between
applications
and
enhances
the
efficiency
of
resource
sharing.

Zhang
explained
that
this
feature
unlocks
Ray’s
multi-tenancy
capabilities,
allowing
multiple
applications
to
share
resources
more
efficiently
on
the
same
cluster.
For
instance,
eight
applications
can
be
squeezed
onto
a
single
large
VM
with
eight
GPUs,
each
Ray
Serve
application
configured
to
use
one
GPU.
This
granular
utilization
of
GPU
capacity
minimizes
underutilized
resources
and
simplifies
operational
management
by
maintaining
a
single
Ray
cluster.

Technical
Implementation
and
Challenges

Under
the
hood,
Ray
is
integrated
with
Podman
to
pull
images
and
spin
up
containers.
When
a
new
Ray
Worker
needs
to
start,
Ray
calls
out
to
Podman
to
orchestrate
the
pull
of
the
relevant
image
and
spin
up
the
container.
Ray
then
orchestrates
the
running
of
the
Ray
Worker
code
inside
that
container.

However,
the
feature
is
still
experimental.
Zhang
cautioned
that
there
might
be
startup
delays
the
first
time
an
image
needs
to
be
pulled,
and
the
feature
hasn’t
been
tested
at
a
large
scale.
Additionally,
other
runtime
environment
fields,
such
as
Python
environments
or
working
directories,
are
not
currently
supported
with
container
runtime
environments.

Future
Plans

Looking
ahead,
Anyscale
plans
to
refine
the
user
experience
around
combining
containers
with
other
runtime
environment
fields,
such
as
specific
environment
variables
for
each
application.
They
are
actively
gathering
user
feedback
to
determine
which
fields
to
include
and
are
planning
more
scalability
testing.

For
those
interested
in
exploring
this
new
feature,
Anyscale
provides
a
detailed
guide
to
get
started
with
multiple
Ray
Serve
applications
and
runtime
environments
as
containers.

Image
source:
Shutterstock