Researchers Leverage GitHub Data to Assess ChatGPT’s Impact on Software Development

Economic
researchers
are
harnessing
the
power
of
GitHub’s
Innovation
Graph
to
measure
the
impact
of
generative
AI
tools,
particularly
ChatGPT,
on
software
development
activities.
This
investigation,
detailed
in
an
interview
published
by
the

GitHub
Blog,
reveals
how
causal
inference
techniques
are
applied
to
assess
the
influence
of
AI
on
coding
practices.

Analyzing
ChatGPT’s
Influence

Alexander
Quispe,
a
junior
researcher
at
the
World
Bank,
and
Rodrigo
Grijalba,
a
data
scientist
specializing
in
causal
inference,
have
conducted
an
in-depth
analysis
of
the
GitHub
Innovation
Graph
data.
Their
study
focuses
on
the
effects
of
ChatGPT
on
software
development
velocity.
According
to
their
findings,
the
introduction
of
ChatGPT
has:

• Significantly
  increased
  the
  number
  of
  Git
  pushes
  per
  100,000
  inhabitants
  in
  various
  countries.
• Shown
  a
  positive,
  albeit
  not
  statistically
  significant,
  correlation
  with
  the
  number
  of
  repositories
  and
  developers
  per
  100,000
  inhabitants.
• Enhanced
  developer
  engagement,
  especially
  in
  high-level
  programming
  languages
  like
  Python
  and
  JavaScript.

The
results
suggest
that
ChatGPT
primarily
accelerates
existing
development
processes
rather
than
increasing
the
number
of
developers
or
projects.

Research
Methodology

The
researchers
employed
various
comparative
methods
for
panel
data,
including
synthetic
difference
in
differences
(SDID),
to
estimate
the
average
treatment
effect
of
ChatGPT’s
availability.
Quispe
explained
that
these
methods
help
to
compare
treated
and
untreated
groups,
thereby
estimating
the
effect
of
ChatGPT
on
software
development
activities.

Grijalba
highlighted
the
utility
of
GitHub’s
Innovation
Graph
data,
which
provided
country-
and
language-level
aggregated
data,
facilitating
the
creation
of
control
and
treatment
groups.
This
allowed
for
detailed
analysis
by
programming
language,
revealing
significant
increases
in
developer
activity
for
languages
like
Python,
JavaScript,
and
TypeScript.

Challenges
and
Future
Directions

One
challenge
noted
by
Quispe
involves
the
potential
use
of
VPNs
to
bypass
ChatGPT
restrictions
in
certain
countries,
which
could
affect
the
study’s
control
group
validity.
However,
existing
studies
suggest
that
such
barriers
still
significantly
hinder
widespread
adoption.

Looking
ahead,
Quispe
aims
to
conduct
similar
analyses
using
administrative
data
at
the
software
developer
level
to
compare
productivity
increases
among
those
with
access
to
AI
tools
like
GitHub
Copilot.
This
future
research
could
provide
deeper
insights
into
the
impact
of
AI-assisted
development
tools
on
individual
productivity
and
software
practices.

Implications
for
Policymakers
and
Developers

The
study’s
findings
indicate
that
AI
tools
like
ChatGPT
and
GitHub
Copilot
will
likely
become
standard
in
software
engineering.
Policymakers
should
consider
supporting
the
integration
of
these
tools
to
enhance
productivity
and
foster
economic
growth.
Developers
are
encouraged
to
embrace
AI
tools
to
boost
efficiency
and
focus
on
more
complex
aspects
of
software
engineering.

Personal
Insights
from
Researchers

Both
Quispe
and
Grijalba
shared
their
journeys
into
the
intersection
of
economics,
data
science,
and
software
development.
Quispe
emphasized
the
importance
of
mastering
algorithms,
linear
algebra,
and
version
control,
while
Grijalba
highlighted
the
value
of
immersion
and
intuition
in
learning.
They
both
acknowledged
the
transformative
impact
of
generative
AI
tools
on
their
work,
particularly
in
accelerating
code
translation
and
enhancing
productivity.

For
those
starting
in
software
engineering
or
research,
the
researchers
recommend
focusing
on
foundational
skills
and
staying
abreast
of
advancements
in
AI
and
causal
inference
techniques.
They
also
suggested
valuable
learning
resources,
including

Introductory
Econometrics:
A
Modern
Approach
by
Jeffrey
M.
Wooldridge
and

Applied
Causal
Inference
Powered
by
ML
and
AI
by
Chernozhukov
et
al.

Their
ongoing
work
and
collaboration
underscore
the
potential
of
AI
tools
to
revolutionize
software
development
and
economic
research.

Image
source:
Shutterstock