How to become a Production Blocker in a CI Theatre
A ladder from Junior Production Blocker to Principal Production Blocker
I expelled a big sigh. I’ve been asked to review a pull request, and it’s large—over 2000 lines of code added, representing more than a week’s worth of work. Gathering my willpower, I opened it up but quickly felt overwhelmed. Maybe I’ll try again in an hour, or perhaps tomorrow.
Is the delay in reviews always about the size of the pull requests? Maybe, maybe not. Just the other day, a colleague cried, “I submitted a small pull request, and it still takes 3 days to be reviewed!” Meanwhile, the work in progress started piling up.
These are frustrating experiences. We know what good practices should look like—Continuous Integration (CI). One key aspect of CI is that engineers should commit to the mainline at least daily. Delays in reviews, like the examples above, are counterexamples of CI.
While describing what good looks like is important, it’s equally crucial to define what we should avoid. We have a term for the CI counterexamples: CI theatre, where a team assumes they’re practising CI just because they have set up a CI server, ignoring the frequency of commits to mainline.
I think the term CI theatre is insufficient for us to shift our behaviours, as it doesn’t describe the issues caused by such behaviours. When engineers submit large pull requests or delay reviews, they often don’t realise the impact. They’re blocking production. Value to customers is blocked. I want to have a clearer way to describe this, especially on the individual level.
I suggest the term Production Blocker, an individual who plays well in a CI theatre.
When I procrastinate on reviewing large pull requests, I ask myself, “Do I want to be a Production Blocker? No, let’s review it.”
This idea shouldn’t be used to point fingers but used as a personal anti-goal instead. Anti-goals are a powerful concept that applies the idea of inversion to goal setting. I’ve seen this idea being used by Jack Dorsey with his ‘won’t do list’1. My colleague recently shared TypeScript Design’s ‘Non-Goals’2, which are similar.
To better understand the anti-goals of a Production Blocker, I’ve described them in a ladder format:
Junior Production Blocker
As a reviewer:
Don’t bother checking your inbox for review requests. Your sole purpose is to code; leave the inboxes to the seniors.
Approve everything without reading the code to ensure potential issues go straight to production.
As a changer:
Merge your code on broken mainline builds. Once approved, don’t check the build status—just merge. As long as your code is in the mainline, your work is done.
Don’t review your own pull request. Let others discover your typo, temporary logs, or unfinished work.
Senior Production Blocker
As a reviewer:
Comment on every minor style issue and nitpick relentlessly. Don’t allow merging until all trivial issues are resolved.
If your teammates ask if they can merge first and resolve later, firmly say no.
Delete review request emails because your current work is far too important to be interrupted.
As a changer:
Do not add any explanation, context, or screenshot to your pull requests, your code is self-explanatory.
Submit only large pull requests. Big tickets need big pull requests. The larger the better.
Never revert your changes if they break the mainline. Spend a week investigating the root cause instead.
Lead Production Blocker
As a reviewer:
Be the only one who can approve pull requests, no one else is qualified. Make others wait, even if you’re on holiday.
Provide overly detailed feedback. If the changers don’t understand the feedback, close the pull request, and complete the change by yourself.
As a changer:
Focus solely on what you’re working on, and reject every collaboration attempt from your teammates.
Reject all review feedback from your teammates. They are not mature enough to understand your superior code.
Principal Production Blocker
As a Principal, you transcend the roles of reviewer or changer. Your mission is to block the root of all production by taking it to the psychological level, you block people’s minds. Production Blocking3 is the psychological term used to describe the tendency of one person to block other people from offering ideas, that’s your ultimate goal.
In a group discussion, always be the first one to speak and make sure you win by getting the last word4. If the discussion lasts an hour, talk for the entire hour, and block production. Suck all the oxygen in the room. When people interrupt, you’ll say “I’m not finished!” This way, no pull requests are even possible to be submitted.
What else would you add to be a successful Production Blocker?
I like the notion but I tend to think about this as a social problem rather than a personal problem. I.E it’s more about how the team collaborates than how each individual behaves. And also about incentive structures that support that behaviour (which then makes it not just about the team but about what’s around the team). In my experience we don’t get good at CI until we create highly collaborative teams with incentives driving that collaboration. But using this we can get better.
Now, I have a new motto: Don't be a Production Blocker, be a Production Block Buster!