Can AI coding agents do Test-Driven Development (TDD)?

Great Pods

Can AI coding agents do Test-Driven Development (TDD)?

Key Points

Test-driven development (TDD) follows three phases: red (write failing test), green (write minimal code to pass), and refactor (clean up code)
AI coding agents typically don't naturally follow TDD - they prefer big bang approaches over iterative cycles
Standard prompting techniques to encourage TDD have mixed results and agents often give up after a few attempts
TDD Guard is a library that enforces TDD principles through guardrails that cannot be bypassed
The experiment used eShop on Web reference application, a .NET application with domain-driven design concepts
The test feature was "splitting a basket" - separating expensive items (>$100) from cheap items into two baskets
Without TDD Guard: Claude Code completed the feature in 4 minutes but used "test first" development, not true TDD
With TDD Guard: The process took 10 minutes but enforced proper TDD methodology through blocking violations
TDD Guard works by using hooks that trigger before file writes, calling an internal AI judge to validate TDD compliance
The library uses its own LLM instance as a judge to determine if TDD principles are being followed
TDD Guard supports multiple languages including Python, TypeScript, Golang, and .NET
The tool requires hook support in the AI coding assistant and language-specific test framework integration
TDD Guard successfully caught violations like writing multiple tests at once instead of one at a time
The library enforced outside-in development approach, flagging bottom-up implementation attempts
Code quality with TDD Guard was somewhat higher due to preventing unnecessary method invention
Trade-offs include increased token consumption and slower execution due to AI judge validation
TDD Guard maintains state through shared folders containing file updates and test outcomes
The library is the best available tool for enforcing TDD with AI coding agents, despite performance costs

Full Transcript

Language: en
Hey folks, can AI coding agents do
test-driven development? That is what
this video is all about. So what exactly
is test-driven development or TDD? It's
a process that many experienced
engineers use to avoid goldplating to
stay laser focused on the problem at
hand and not like invent irrelevant code
that won't be used. Uh it it follows
three phases called red, green, and
And the red phase is where you write a
failing test. one single failing test uh
that describes the new functionality you
want to implement. And the second phase
then is the green phase where you write
production code to have that test
succeed. You write like the simplest
test possible or the simplest code
possible to make it work. And the third
phase is called refactor where you clean
up after yourself, remove duplication,
stuff like that. And only then you
proceed to the next test. So it's a
really iterative short feedback cycle.
It's a really iterative process.
And if you are have been using AI coding
agents for a while, you might have
noticed that they don't work this way.
They prefer to work using like bigger
steps. They do like big bang things from
nothing to all the code and tests
available.
And a lot of engineers have been trying
to get AI coding agents to actually
follow testdriven development. And the
most commonly used way to do this is to
use prompting techniques like the father
of testdriven development, Ken Beck, who
wrote the literal book on it. He has a
system prompt online. I'll link it in
the show notes below. Um that really
strongly encourages u coding agents to
use testdriven development. But I have
had mixed results with that myself.
These agents uh their instruction
following especially for test-driven
development is only average at best.
like they gave it their best effort, but
they give up after after a couple times.
So, no great success. And that's when I
started to look around the internet for
stronger guard rails to really enforce
the test-driven development process. And
that is when I found TDD guard. So, is
it worth looking at? Is it worth using
today? We'll figure that out. So, let's
dive in.
So, let's first go over the experiments
we will be running and then dive in,
shall we? So um again I am utilizing the
eshop on web reference application. It's
a somewhat large net application with
like good practices and it's a good
starting point for experiments like
this. So that's what we will be using
and let's quickly take a look at the
It has some domain driven design
concepts like application services and
aggregates stuff like that. uh not super
important for the experiment. But uh
like in the big picture, we have
application services and they have like
use cases or scenarios where you can add
an item to a basket or delete a basket
or uh transfer a basket from an
anonymous user to another user, stuff
like that. And it's almost always the
same. We look up data in the database
using repositories. Then we delegate to
the aggregate for like business logic
stuff like adding items as you can see
here. And then we do like persistence
operations and that's it. Uh and maybe
another interesting aspect is the tests.
I like my tests to talk to the system on
the application service level. So the
basket service that we just saw and I
like using fluent APIs for test data
like test data builders is which is what
you can see here. We create new baskets.
we add them to like a a stop a fake
repository that is in memory and then we
exercise the system under test and we
validate. So this is the way I like to
write code myself and this is what we
will be using for the experiment. Now
let's take a look at we are what we are
going to do like with a prompt
suggesting TDD and with the TDD guard
library really enforcing it. So let's
take a look at the desired feature we're
going to build and that feature is
called splitting a basket. I came up
with this myself. It's like a totally
fixed fixious use case, but I want cloud
code to implement splitting a basket.
Splitting a basket results in two
different baskets. And the original
basket has like only cheap items that
cost like $100 or less. And the new
basket will have all the expensive
items. That's the general gist of the
feature. And we're going to strongly
suggest to use test-driven development
as you can see here.
And we're going to point it at relevant
code like the tests we just discussed
and the service code we just saw. And I
also when I'm programming with AI coding
assistance, I like to like give method
signatures in the code or in the prompt
itself. Uh doesn't really matter, but I
I like to give it like a couple of
breadcrumbs so it knows uh what I'm
looking for. And then of course in every
story there should be some acceptance
tests or some acceptance criteria. And
here I list three one three different
ones like only splitting a cheap or a
exclusively expensive basket should
result in no change because there's no
different buckets. And then the final
one is the real uh logic where if you
have a mix of expensive and cheap items,
we should be able to split a basket into
two separate ones. And then I give it
some help um on how to run unit tests.
And in the case of TDD guard, as you can
see here, uh we have to do a bit of
black magic, but that's just to help
cloud code along. So that's the feature
we will be bidding build sorry once uh
with just this prompt and once with a
DDG guard library uh in place and we'll
see um how these two approaches compare
later on. So let's go.
All right. So now let's first try the
best effort approach. No extra tools, no
SCP servers, nothing. just vanilla cloud
code and a carefully crafted prompt
where we really really try to nudge it
to write unit tests.
Uh yeah, and see how far we get. So let
me put on the timer
so we have an idea how long it takes and
we have no changes. All tests are
passing and we have a
clear cloth code. At least now we do. So
let's take a look at the results. So
it'll implement the feature following
TDD principles or so it says. Let's take
a look at the to-do list here. Let's
take it's going to take a look at the
existing implementation. Then it's going
to write all the failing unit test. So
this is not test driven development.
This is something
test first. Sure, but not test driven.
The iterative loop is not here. And then
it's going to run the unit tests to
verify the implementation. So that
sounds like a good plan or a good enough
plan at least.
Okay, it's come up with some unit tests.
So, let's take a look at those first.
It's not compiling. Of course, it has
not implemented any ST methods and it's
figuring that out now. As you can see,
it tried to run the tests, but it's not
compiling.
And it treats that as a red phase
apparently. So, now it's going to
implement all the production code in one
big bang. In the meantime, let's look at
these tests.
uh we see our acceptance criteria
returned. So it's the criteria we
This is the interesting case where we do
an actual split
and it even added a new case like if we
cannot find the original basket then uh
yeah okay so that makes sense.
Okay. So the test looks good enough. It
already changed the aggregate the
basket. So let's take take a look at
that. It added a split method
where it looks up expensive items
and if we don't have a split to do it
returns null. So yeah, that's what I
would have expected from the method
signature. So this looks good. Creates a
new basket and adds all the expensive
items. So this looks
correct. I would not do this myself. I
would remove all the expensive items not
do the query again but okay that's a
small detail we could fix that in post
and now it's working on the service
logic we are running our unit tests
there I was waiting on my basket service
diff but there it is uh so let's take a
look at the split method on the service
we look up the original basket we return
not found we have our threshold old
hardcoded here we delegate to the
not sure about this update the sync but
if that is how this codebase handles
updates yes it is so yeah that's correct
uh don't like this optional basket thing
why would there be a nullable basket
that is not what was in our prompt right
no that was not was in what was in our
prompt so it invented a method signature
so there's a couple of problems with
this uh or minor details. I would not
call this problems. Uh but it
implemented the feature in let's say 4
minutes and it did not do test-driven
development. It did test first
development.
All right. Now let's try that again but
this time let's enforce TDD using the
TDD guard library. So I reset my code
and I configured TDD guard properly.
More on how to do that later in the
video. But let's first showcase it. So
I'm going to grab my prompt and start a
timer so we can compare a bit and I
reset my codebase to the before state.
And let's fire up this prompt. So it's
going to build a plan. Let's take a look
at the to-dos.
Look at the code. Write all tests.
Implement uh the aggregate first. So
there's a couple of interesting things
here. It's going to uh write all the
tests again. So it's the same as a
previous attempt which is not strict
testdriven development that you're
supposed to write one test at a time and
iterate. So TDD guard will flag this and
will block it. And there's a second uh
interesting observation. It's working
bottom up. So it's starting from the
aggregates and entities and then working
up to the services. And I configured DDD
guard to work outside in. So it should
also detect that we're changing like
leave nodes in the codebase first
instead of working top down outside in.
So um as you can see we're running hooks
when we're trying to edit code or when
cloud code is trying to edit code and
this is how TDD guard works before it
allows file rights like you're seeing
here it's calling a hook and this hook
is running the TDD guard logic and as
you can see it detected the first
problem. We're trying to add four unit
tests at a time when we should be
focusing on them one at a time. So now
instead of strongly suggesting to
through prompting that we should follow
TDD which doesn't work as you can see
we're now enforcing testdriven
development by uh this library. So cloth
code now has no option to not follow
these instructions.
And as you can see the green uh traffic
light here, the first test was has been
allowed to enter the system. So we
should see uh changes and now it's
running uh the tests. Now we're in the
the red phase. We're trying to run the
tests and see that they failed.
And let's take a quick look at the test
that has been generated. It's the first
acceptance test, the one with the cheap
items only. And as you can see, it's not
compiling. So we're in a deep red red,
it's not even compiling. So the next
step would be to implement ST methods.
Uh so the test compiles but fails with
an assertion error.
And that's what's happening right now.
But as you can see, it's going to try
bottom up while I prefer top down. So
this should also get flagged as a TDD
violation. It should start working from
the service and then go down level to
the aggregates and the the basket code.
Yeah. So it gets flagged correctly.
We're working uh bottom up and we prefer
outside in top down. So now it gets
nudged again. It's implementing split
stubs in the aggregate. So that's a bit
too soon. But uh it's allowed because
it's really small and I allowed for some
leeway to not doom loop too much. And
now we're going to get to the real uh
entry point of the system. the service
which uh does the lookups in a database
and delegates to the aggregates.
So it's a ST method. So this was allowed
and now the code should compile at least
and fail with a relevant uh assertion
error or actually the test should pass I
guess for the simple case we're we're
doing right now.
So that's the first test case down. Now
we should be in a refactor phase.
Uh it's trying to do like uh the tests
are green and now it's trying to add in
the full implementation but TDD guard
should flag this again. It should write
a new test that forces this behavior to
be added.
Yeah. So it's flagging again. We're
trying to add a lot of code without a
failing test.
So it's focusing on the next test now.
This is exactly how I would do it.
when I was uh if I was coding this by
hand. And there we go. A second test
with expensive items has been added,
which uh is a bad test because it'll uh
pass by default.
Yeah. So, it's going to add a third test
that should
fail, which is the the interesting
acceptance test, the one with both cheap
and expensive items in a basket. So, it
was allowed to add this test. Now we can
which is the mixture case. Yeah. So all
is proceeding as expected. Uh that's a
bad test. It's not asserting
that much or too little to my taste.
It's only counting the number of baskets
in the system. Not validating whether
have enough correct line items.
And I'm hitting my CL code limit.
Interesting.
So, we're in the basket service. We're
looking up the basket and we're
splitting.
Yeah, that seems like a
decent implementation of the service
All right. And the aggregate logic has
been implemented as well,
this looks correct, but we have no test
that is validating
the correctness. So that's an
interesting uh loophole that cloud code
was able to find.
Oh, now it detected that uh
uh it like wrote a bad test by itself.
So that's cool.
Now it's trying to update that test I
was not happy about.
Okay. And
yeah, that looks better. Uh, it's a
basket with two items. So, yeah, that
all looks correct. And now we're running
the tests again. They should be green.
So, yeah, clot is done. And it took 10
minutes, which is longer. So, now let's
go to some conclusions. All right. Now,
let's look under the covers and see how
exactly this library works. So, first
things first, you have your regular AI
coding assistant. I use cloud code, but
it doesn't matter which one you use as
long as it has hook support. And I had
to configure uh a hook that whenever a
class code is trying to do a write on a
file, so adding a new test, adding new
production code or refactoring code,
changing code, then this hook should
trigger and the the hook will only like
call the TDD guard library. It's a npm
package. Uh so that's the first thing
you need to have in place. You need to
download TDD guard uh the mpm package
and you need to install some hooks in
cloud code. Now there's a third
ingredient uh and that is specific to
the language you're programming in. I do
a lot of work in .NET. So I wrote my own
uh .NET components for TDD guard but
there's already support for Python uh
TypeScript I think Golang and now or
soon at least .NET as well. Uh and
that's a
swap in or a drop in replacement for the
.NET test framework. So that's how you
run unit tests in And this uh runs those
unit tests but also like writes the test
outcomes to a specific file in a
specific format that TDD guard
understands. And then there's this
shared folder in the cloud directory
where all this where all the file
updates get written to and where all the
test outputs get written to. So there's
a bit of state in there uh that allows
TDD guard to work. And now for the magic
the really cool part. uh how do you
figure out if you have like file edits
and test results, how do you figure out
whether or not you're following the TDD
process? Because it's hard to encode
that into actual code, right? It's hard
to really codified rules. Well, guess
what? AI is pretty good at fuzzy
problems like this. So TDD guard uses
internally its own LLM, its own coding
agent or it its own AI agent and it
internally it spins up a new instance on
cloud code that acts as a judge. So for
every change, for every time this hook
triggers, it looks at the current test
outcomes. It looks at the current file
changes that are being requested and it
starts a new AI that will act as a judge
to whether to verify whether or not like
the TDD process is being follow
followed. So the TDD guard library has a
lot of prompts that get shot to this or
fired to this uh judge. So it gets
programmed to judge accordingly. So
let's maybe take a quick look at those
uh prompts.
Uh this is uh one of those prompts.
Sorry for the light mode. uh but it
states like these are the core
principles of TDD and you recognize the
red phase, green phase, refactor phase.
So this programs clot code or any other
LLM to be like um judge for whether or
not the TDD process is being followed.
And then when this judge like is done it
spits out some JSON and that gets
returned to the hook so to the original
cloth code instance and that then gets
blocked or can proceed with work. So
it's a really nifty uh design really
cool use case of AI judges. So what did
I think about TDD guard? Well, first
things first uh I love the philosophy of
this tool. I see a lot of people
struggle and try to enforce or try to
get uh clot code to follow test-driven
development by using prompting
techniques and yeah they all fail. So
this is the first tool I found that
really enforces it and puts in
guardrails that cannot be bypassed and I
love it. But there's a trade-off. While
I think that the output the code quality
is of a
somewhat large higher standard because
uh we're forced we're forcing a cloud
code to not freewheel and not like
invent methods that are not necessary.
So the output is slightly better than
not using these tools but uh the library
uh just because it calls an AI judge in
the background uh it eats tokens. every
file edit will call an LLM to like
validate whether or not we're following
TDD and also because of that it's slower
because there's like a separate
validation going on. So for me the
trade-off is not worth it. But if you
are looking at a way to enforce TDD,
this TDD guard library is the best I
found out there. So please take a look
at it. So have you been using
test-driven development in the past and
have you found a way to make it work
with AI coding agents? Please let me
know in the comments below and I'll see
you next time. Bye-bye.

← All Summaries

Watch on YouTube