Anthropic Expands AI Model Safety Bug Bounty Program

The
rapid
advancement
of
artificial
intelligence
(AI)
model
capabilities
necessitates
equally
swift
progress
in
safety
protocols.
According
to

Anthropic,
the
company
is
expanding
its
bug
bounty
program
to
introduce
a
new
initiative
aimed
at
finding
flaws
in
the
mitigations
designed
to
prevent
misuse
of
their
models.

Bug
bounty
programs
are
essential
in
fortifying
the
security
and
safety
of
technological
systems.
Anthropic’s
new
initiative
focuses
on
identifying
and
mitigating
universal
jailbreak
attacks,
which
are
exploits
that
could
consistently
bypass
AI
safety
guardrails
across
various
sectors.
This
initiative
targets
high-risk
domains
such
as
chemical,
biological,
radiological,
and
nuclear
(CBRN)
safety,
as
well
as
cybersecurity.

Our
Approach

To
date,
Anthropic
has
operated
an
invite-only
bug
bounty
program
in
collaboration
with

HackerOne,
rewarding
researchers
for
identifying
model
safety
issues
in
publicly
released
AI
models.
The
newly
announced
bug
bounty
initiative
aims
to
test
Anthropic’s
next-generation
AI
safety
mitigation
system,
which
has
not
yet
been
publicly
deployed.
Key
features
of
the
program
include:

• 
  Early
  Access:
  Participants
  will
  receive
  early
  access
  to
  test
  the
  latest
  safety
  mitigation
  system
  before
  its
  public
  deployment.
  They
  will
  be
  challenged
  to
  identify
  potential
  vulnerabilities
  or
  ways
  to
  circumvent
  safety
  measures
  in
  a
  controlled
  environment.
• 
  Program
  Scope:
  Anthropic
  offers
  bounty
  rewards
  of
  up
  to
  $15,000
  for
  novel,
  universal
  jailbreak
  attacks
  that
  could
  expose
  vulnerabilities
  in
  critical,
  high-risk
  domains
  such
  as
  CBRN
  and
  cybersecurity.
  A
  universal
  jailbreak
  is
  a
  type
  of
  vulnerability
  allowing
  consistent
  bypassing
  of
  AI
  safety
  measures
  across
  a
  wide
  range
  of
  topics.
  Detailed
  instructions
  and
  feedback
  will
  be
  provided
  to
  program
  participants.

Get
Involved

This
model
safety
bug
bounty
initiative
will
initially
be
invite-only,
conducted
in
partnership
with
HackerOne.
While
starting
as
invite-only,
Anthropic
plans
to
broaden
the
initiative
in
the
future.
This
initial
phase
aims
to
refine
processes
and
provide
timely,
constructive
feedback
to
submissions.
Experienced
AI
security
researchers
or
those
with
expertise
in
identifying
jailbreaks
in
language
models
are
encouraged
to
apply
for
an
invitation
through
the

application
form
by
Friday,
August
16.
Selected
applicants
will
be
contacted
in
the
fall.

In
the
meantime,
Anthropic
actively
seeks
reports
on
model
safety
concerns
to
improve
current
systems.
Potential
safety
issues
can
be
reported
to
[email protected]
with
sufficient
details
for
replication.
More
information
can
be
found
in
the
company’s

Responsible
Disclosure
Policy.

This
initiative
aligns
with
commitments
Anthropic
has
signed
with
other
AI
companies
for
responsible
AI
development,
such
as
the

Voluntary
AI
Commitments
announced
by
the
White
House
and
the

Code
of
Conduct
for
Organizations
Developing
Advanced
AI
Systems
developed
through
the
G7
Hiroshima
Process.
The
goal
is
to
accelerate
progress
in
mitigating
universal
jailbreaks
and
strengthen
AI
safety
in
high-risk
areas.
Experts
in
this
field
are
encouraged
to
join
this
crucial
effort
to
ensure
that
as
AI
capabilities
advance,
safety
measures
keep
pace.

Image
source:
Shutterstock