Test First vs Last: Mere-exposure effect

How the sequence of writing tests affects our understanding of what should be built.

This is the latest issue of my newsletter series, ‘Test First vs Last’, where we’ll explore how the sequence of writing tests affects our minds. The full context of this series can be found in the introduction post.

---

My product manager reached out to me and told me that what I shipped was not what he had in mind. “What are you doing, Wisen? It seems like you didn’t read the ticket. I have written a lot there!”

The day before

Let’s transport ourselves to the day when I pick up the ticket. I picked up the ticket and looked at it:

Title: Play next song if current song fails
Description: [… Lengthy description …]

Simple, I thought. This is just a try-catch issue, and I don’t have to read the description. I quickly checked the code I’ve worked with before, yes, there’s a missing try-catch. I wrote the following code, added the test, and shipped it.

try {
  playSong()
} catch (e) {
  playNext()
}

Once the feature is shipped, users on a slow network keep moving to the next songs in their playlists indefinitely. I introduced a bug. I later discovered a separate error handling where the change should have been made. (Yes, pull requests, desk checks, and all the other processes could have caught this, but that’s not the point of this essay).

I didn’t read the ticket thoroughly, and I didn’t investigate the code. Rather than deliberately trying to understand what this ticket was all about, I jumped to a conclusion on what needed to be shipped based on what I was familiar with.

This situation is a classic example of how the mere-exposure effect affects a developer, leading us to prefer things just because they’re familiar. Let me explain quickly what this cognitive bias is about and how writing tests first can counteract this bias.

Mistaking familiarity for understanding

I’ll give you one simple way to know if you have experienced the mere-exposure effect in knowledge work. Think about the time when you have finished reading a non-fiction book. You nod along when you read it, thinking that you understand what you’re reading.

You later meet a friend and have a conversation.

“I just finished reading that book, and it’s so interesting.”
”Oh great. What have you learnt from it?"
”Hmmm… Err… Ahh…”

You were taken aback. You don’t seem to remember anything. You uttered two sentences. What else? Nothing. You thought you understood this book well.

You go back home, re-read the book, and you nod along again. You’re familiar with the topic, you feel like you know it, but you don’t. You mistake familiarity for understanding.

Just like reading a book, in software development, it's also possible to mistake familiarity with understanding.

Elaboration tests your understanding

The best way to distinguish familiarity and understanding is elaboration. Elaboration takes effort. This is why you were taken aback when your friend asked about that book. Your mental resources worked hard to answer what you learned from that book.

This is the process of elaboration. It’s slow and takes time, but it helps you test if you truly understand. Verbal conversation is one way to elaborate. Writing is another form of elaboration.

In software, writing tests is a good way to test your understanding.

Writing tests is a form of elaboration

Let’s replay the day I picked up the ticket, but this time I’m writing the test first.

I picked up the ticket. I thought, okay, just a try-catch issue. Easy. I start writing the test.

it("plays the next song when ...

When what? What kind of error should I throw in the test? Hang on, why am I writing a new test file? Maybe there’s a test that has already been written elsewhere? Is this the right place to test this thing?

I paused briefly because I had to elaborate on what I was about to test. This led me to re-read the ticket again and double-check if this was the right place to make a change. I broke my familiarity barrier and began seeking more understanding.

Conclusion

When writing tests last, we are more prone to the mere-exposure effect. This means we tend to build something and make changes in places we’re familiar with. Elaboration helps us reduce this effect of familiarity, leading us to ship better software.

Writing test first helps us elaborate, but elaboration itself is not foolproof. For example, you may go through the process of summarising a book you’ve read as a form of elaboration, but it doesn’t necessarily mean you will reach the right conclusion.

While the test-first approach is not a foolproof solution, resequencing our steps is a cheap way to mitigate this bias, making us less prone to the mere-exposure effect.

Yes, writing test helps us elaborate, but when I wrote the test last at the top of this essay, I didn’t have that brief pause of questioning what I was doing. Why does this happen? There’s another bias at play. I’ll cover that in the next post of this series.