Categories
Architecture

Monolith or Services: Choosing at an Early Stage

Flat illustration of one large solid cube beside the same volume split into smaller connected cubes, representing the monolith versus services choice

I’ve watched a four-person team spend two months building a service mesh for a product that had eleven total users. I’ve also watched a team stay on a single unstructured monolith long after every deploy meant hoping the checkout code didn’t break the reporting code, because both lived in the same process with no boundaries at all. The right answer for an early-stage team is almost never at either extreme, and it’s rarely the one that’s currently trendy on engineering Twitter.

Start with a monolith – the argument is mostly settled

For a new product with a small team, a single deployable application is the right default. You don’t have the traffic that justifies independent scaling, you don’t have the team size that justifies independent deploy schedules, and you don’t yet know where your actual service boundaries should be – that only becomes clear once real usage patterns emerge. Martin Fowler’s “MonolithFirst” essay makes this case directly: most successful microservice systems started as monoliths that were split once the boundaries were understood, not designed as services from day one (martinfowler.com/bliki/MonolithFirst.html).

The real cost you’re avoiding

Services aren’t free even when they’re the right call eventually – each one adds a network boundary, a separate deploy pipeline, its own on-call surface, and a distributed systems problem (partial failure, retries, eventual consistency) that a function call inside one process never has. Fowler’s later piece on microservice trade-offs is a useful antidote to the assumption that services are simply the more “correct” architecture (martinfowler.com/articles/microservice-trade-offs.html). At five engineers, paying that cost before you need it means paying it instead of building product.

A monolith doesn’t mean a mess

The mistake that makes “monolith” a dirty word isn’t the single deployable – it’s the lack of internal boundaries inside it. A modular monolith keeps one deploy unit but enforces separation between domains in code: billing doesn’t reach into the internals of the user module, it calls a defined interface, the same way it would call a service over the network, just without the network.

src/
  billing/
    api.py          # public interface other modules call
    internals.py    # not imported outside this folder
  users/
    api.py
    internals.py
  shipping/
    api.py
    internals.py

This structure costs almost nothing to set up compared to actual services, and it does most of the work that people actually want from “microservices” – forcing explicit boundaries, making ownership clear, keeping one team’s changes from silently breaking another’s. It also makes the eventual split, if you need one, mechanical instead of archaeological: the module already has a defined interface and its own tables.

The middle path: extract one service on purpose

Splitting doesn’t have to mean an all-or-nothing rewrite into a services architecture. It’s common, and often correct, to pull exactly one piece out of an otherwise single-deployable monolith when it has a genuinely different profile – a video transcoding worker, an email-sending pipeline, a scheduled report generator – things that benefit from independent scaling or a different language, while the rest of the product stays as one deployable. This gets you the specific benefit you actually need without paying the coordination cost of a fully distributed system for the parts of the app that don’t need it.

Data ownership matters more than the deploy boundary

The part of this decision that’s hardest to undo isn’t the deploy topology, it’s the data model. A modular monolith where every module owns its own tables and never queries another module’s tables directly can be split into real services later with a manageable amount of work. A monolith where every module freely joins across the whole schema has to solve that problem first, before a service split is even possible – untangling shared tables after years of ad hoc joins is a much bigger project than extracting the code. If you take one thing from a modular-monolith approach, make it this.

Signals that it’s actually time to split

A few concrete signs are worth watching for, instead of splitting on a schedule or a feeling: one part of the system needs to scale independently and is forcing you to over-provision the whole app to compensate; deploys are getting risky because unrelated teams’ changes are landing in the same release; or a specific domain has a genuinely different operational profile – different language, different compliance requirements, different uptime target – that a shared deploy can’t serve well. A team size mismatch, more than roughly two teams working in one codebase, is an organizational signal worth taking seriously too, independent of the technical one.

What not to do

Don’t split along guessed boundaries before you’ve felt real pain at those boundaries – the seams you’d predict on day one are rarely the ones that matter once actual usage shows up. And don’t treat “microservices” as a hiring or credibility signal; AWS’s own introduction to the pattern is honest that it solves organizational and scaling problems, not code quality problems, and it introduces new ones of its own (aws.amazon.com/microservices).

The pragmatic path for a small team: one deployable, real module boundaries enforced in code from day one, and a genuine trigger – not a vibe – before you pay the cost of splitting anything out.

Categories
Teams

Onboarding a New Engineer in a Week

Flat illustration of a seven-tile pathway leading from a laptop to a flag with a small figure walking along it, representing onboarding an engineer in a week

The usual failure mode for onboarding on a small team isn’t a bad plan – it’s no plan. Someone hands the new hire a laptop, adds them to Slack, and says “ping me if you get stuck,” which quietly means the new person spends their first three days reverse-engineering tribal knowledge that lives in nobody’s head but everyone’s memory. A week of deliberate onboarding gets someone shipping real code faster than a month of osmosis.

Day one: a working environment, nothing else

The only goal for day one is a running local environment and a merged pull request, even a trivial one – a typo fix, a small copy change, anything that goes through your real CI and deploy pipeline. This proves the setup works end to end and gives the new hire a concrete, low-stakes win on day one instead of a week of “still configuring my machine.”

If getting a new machine running takes more than an hour, that’s worth fixing independent of onboarding – it’s costing every new hire the same tax, and it’s usually a sign your setup relies on undocumented steps. The Joel Test’s line about building in one step is two decades old and still the right bar: someone new should be able to check out the repo and be running with a handful of commands, not a wiki page of manual steps (joelonsoftware.com – the joel test).

Days two and three: a real ticket, closely watched

Pick a small, well-scoped, genuinely useful ticket – not a “starter task” everyone knows is throwaway, and not something ambiguous enough to need three days of requirements clarification. Pair on it for the first hour, then let them drive with you one Slack message away. The point isn’t the ticket itself, it’s forcing contact with the actual codebase, the actual review process, and the actual deploy path while someone’s available to unblock quickly.

# A reasonable first-week ticket looks like:
- Touches 1-2 files, not the whole codebase
- Has a clear, testable definition of done
- Doesn't require design decisions or cross-team coordination
- Would take an experienced engineer under half a day

Write the map down before they need it

A short doc that answers “where does X live” for the five or six things people ask about most – how the data model is organized, where background jobs run, how deploys work, who owns what – saves hours of interruption per new hire, and it only has to be written once. If you don’t have this, the new hire’s onboarding week becomes a series of interruptions for whoever’s closest, which is a cost you’re still paying every time someone joins.

Assign a specific person, not “the team”

“Ask anyone if you have questions” means asking no one, because nobody wants to be the person constantly interrupting five different people. Assign one buddy for the first two weeks whose explicit job includes answering questions, even ones that feel too basic to ask. Rotate who does this across hires so it doesn’t become one person’s permanent burden. A lightweight CODEOWNERS file also helps here, since it routes the new hire’s early PRs to the right reviewer automatically instead of them guessing who owns what.

Resist the urge to front-load documentation

Dumping the entire wiki on someone before they’ve touched the codebase produces the opposite of the intended effect – none of it sticks because there’s no context to hang it on yet. Point to documentation exactly when it becomes relevant: the deploy doc when they’re about to deploy something, the data model doc when their first ticket touches the data model. Ten pages read at the moment they matter beat a hundred pages read cold on day one.

Ask them what was confusing, while it’s still fresh

By the end of week one, the new hire is the only person on the team who still remembers what it’s like to not know how anything works – that perspective is gone within a month, replaced by the same blind spots everyone else has. Ask directly: what took longer than it should have, what documentation was missing or wrong, what question did you not want to ask. Fix the concrete ones before the next hire starts. This is the cheapest source of onboarding improvements you’ll ever get, and it disappears if you wait.

Give environment parity a real check

A surprising amount of “it works on my machine” friction during onboarding traces back to local environments quietly drifting from production – different config defaults, different service versions, dependencies installed in a different order months apart. The twelve-factor app’s argument for keeping dev, staging, and production as similar as possible is worth revisiting through this lens: a new hire’s local setup is the newest test of whether that parity actually holds (12factor.net/dev-prod-parity).

By the end of the week

A realistic bar for day five: one merged PR beyond the day-one trivial one, a written map they’ve read and can navigate from, and a specific person they know to ask instead of guessing who’s least busy. That’s not full productivity – nobody hits that in a week – but it’s enough that the second week starts with real work instead of more discovery.

Categories
Practices

Talking About Technical Debt: When to Fix It, When to Let It Ride

Flat illustration of a stack of building blocks with a few cracked ones and a balance scale weighing a block against a clock, representing technical debt trade-offs

Every codebase past its first few months has technical debt, and every team has at least one engineer who wants to stop and fix it now, and one who wants to keep shipping features and deal with it later. Both are right some of the time. The actual skill isn’t picking a side – it’s telling the difference between debt that’s quietly costing you every sprint and debt that’s genuinely fine to leave alone.

The metaphor is more useful than the argument

Ward Cunningham coined “technical debt” to describe a real trade-off, not a moral failing: shipping the fast, imperfect version now is sometimes the right call, as long as you intend to pay it down before the interest – the ongoing cost of working around it – outweighs what you saved. Martin Fowler’s writeup on the concept is worth reading in full, particularly the point that debt taken on deliberately and debt accumulated by accident are very different situations that deserve different responses (martinfowler.com/bliki/TechnicalDebt.html).

Sort it into a quadrant before arguing about it

Fowler later extended the idea into a simple two-axis split: deliberate versus inadvertent, and reckless versus prudent. A deliberate, prudent shortcut – “we know this doesn’t handle multi-currency yet, we’ll add it when a customer needs it” – is a normal engineering decision. Reckless, inadvertent debt – code nobody understood was a problem until it broke – is the kind that deserves a real conversation about why it happened (martinfowler.com/bliki/TechnicalDebtQuadrant.html). Naming which quadrant you’re in defuses a surprising amount of the argument, because it separates “was this the wrong call” from “is this worth fixing now.”

Ask what it’s actually costing this month

Vague debt (“this module is a mess”) never wins a prioritization argument against a customer-facing feature with a deadline. Specific, recurring cost does. Track it the boring way: every time a piece of debt slows down an unrelated task – a bug takes three hours to fix because the module has no tests, a feature takes two extra days because the data model doesn’t support it cleanly – log it against that piece of debt.

# debt-log.md entry
Area: order pricing calculation
Cost this month: 2 incidents, ~9 engineer-hours
Why: discount logic is duplicated in 3 places, changes require
     updating all three or a discount silently doesn't apply
Fix estimate: ~2 days to consolidate into one function
Decision: schedule for next sprint - cost is now exceeding fix estimate

Once the cost is visible in hours instead of vibes, “when to fix it” becomes an ordinary prioritization decision instead of a philosophical one.

Explaining it to people who don’t read code

A product manager or founder doesn’t need the implementation detail, but they do need the business consequence, and it’s on engineers to translate one into the other. “The discount logic is duplicated in three places” means nothing to someone prioritizing a roadmap. “We’ve had two pricing bugs reach customers this month because of how this is built, and the next feature in this area will take twice as long” is a sentence anyone can weigh against other priorities. Keep the debt log from the earlier example somewhere non-engineers can see it, not buried in an engineering wiki – it turns “just trust us” into a conversation with actual numbers.

Piggyback on feature work when you can

The easiest debt to pay down is the debt sitting directly in the path of a feature you already need to build. If you’re touching the pricing module for a new discount type anyway, that’s the moment to consolidate the duplicated logic, not six months later in a dedicated cleanup ticket that will keep losing to higher-priority work. This isn’t a substitute for the dedicated time mentioned below – some debt sits in code nobody’s touching for other reasons – but it’s the cheapest debt you’ll ever pay off, so take the opportunity when it appears.

Debt that’s fine to leave alone

Not all debt needs a plan. Code that’s ugly but stable, isolated, rarely touched, and not on a path anyone’s about to build on top of – leave it. The Agile Alliance’s definition of technical debt makes this point well: the debt itself isn’t the problem, the compounding interest is, and code nobody touches doesn’t compound (agilealliance.org/glossary/technical-debt). Refactoring code just because it offends you, with no plan to build on it soon, is time you could have spent on something that pays back.

Make room for it explicitly

The most durable fix isn’t a big cleanup sprint – it’s a standing habit. Reserve a fixed slice of every sprint, even 10-15%, for debt work chosen from the log above, ranked by cost. This keeps debt paydown from competing head-to-head against features in every single planning meeting, which is a fight debt usually loses until it’s already expensive.

The goal was never a debt-free codebase – that doesn’t exist on a real product with real deadlines. The goal is debt you chose on purpose, that you can see the cost of, and that you’re paying down faster than you’re taking it on.

Categories
Engineering

Feature Flags and Gradual Rollouts

Flat illustration of a row of toggle switches at different positions with user dots flowing through the active ones, representing feature flags and gradual rollouts

The first time I shipped a risky change behind a feature flag instead of a deploy, it felt like cheating. The code went to production Tuesday, nobody saw it, and I turned it on for 5% of users on Thursday after watching error rates for two days. If something had gone wrong, fixing it meant flipping a boolean, not rolling back a deploy and re-running CI. That’s the entire pitch for feature flags: they decouple deploying code from releasing it.

Two different problems, one mechanism

Feature flags get used for two distinct purposes that are worth naming separately. Release flags let you merge incomplete work to main safely – the code ships dark, off by default, and gets turned on when it’s ready. Operational flags let you control behavior in production without a deploy – a kill switch for a flaky third-party integration, or a percentage rollout for a risky change. Martin Fowler’s writeup on feature toggles is still the clearest reference on this distinction and the different lifecycles each type needs (martinfowler.com/articles/feature-toggles.html).

A minimal implementation

You don’t need a vendor product to start. A flags table with a percentage column and a simple hash-based bucketing function covers most early needs:

def is_enabled(flag_name, user_id):
    flag = flags_cache.get(flag_name)
    if not flag or not flag.enabled:
        return False
    if flag.rollout_percent >= 100:
        return True
    bucket = hash(f"{flag_name}:{user_id}") % 100
    return bucket < flag.rollout_percent

Hashing on flag name plus user ID keeps a given user consistently in or out of the rollout as the percentage climbs, instead of flipping randomly on every request. That consistency matters - users notice when a feature appears and disappears between page loads.

Rolling out gradually, on purpose

A gradual rollout is only useful if you're watching something while it happens. Pick the rollout steps in advance - 5%, 25%, 50%, 100% - and attach a metric and a time window to each step, not just a vibe. "Move to 25% after 24 hours if the error rate hasn't moved and support hasn't flagged anything" is a real gate. "Turn it up when it feels fine" is how a bad rollout reaches 100% of users before anyone notices.

Segment the early percentage toward internal users or a specific cohort when you can - your own team, then a beta group, then everyone. This catches obvious breakage before it reaches a paying customer, without needing a full staging environment that mirrors production traffic. It also means the first bug reports come from people who know how to write a useful one, rather than a confused support ticket from someone who has no idea a rollout is even happening.

Testing both sides of a flag

A flag that's only ever been exercised in the "on" state during development is a flag you haven't actually tested - in production it will spend real time in both states, often for different users simultaneously. Write tests that exercise the flag both ways, not just the new behavior. This matters more as a flag lives longer: the "off" path is old, well-worn code, but the moment a refactor touches shared logic underneath both branches, it's easy to fix the new path and silently break the old one that most users are still on.

Naming flags so they're findable

A flag named flag_2 or test_thing is useless six months later when someone's trying to figure out if it's safe to delete. Use a consistent pattern - area_feature_description, like onboarding_new_wizard_flow or search_fuzzy_matching - so anyone can guess roughly what a flag does from its name alone, without opening the code that reads it. This sounds like a small thing until your flag list has thirty entries and half of them read like variable names generated under deadline pressure, which, realistically, is exactly how most of them got created.

The debt flags accumulate

Every flag you add is a fork in your code that has to be reasoned about until it's removed. A codebase with forty stale flags, half of them at 100% for a year, is worse than no flag system at all - nobody's confident what's actually controlling behavior anymore. Treat "flag at 100% for 30 days" as a trigger to clean up the flag and delete the old code path, not a permanent state. Put an owner and a removal date on every flag when it's created, and review stale flags on a regular cadence, even a quick one once a month.

Where this fits with trunk-based development

Feature flags are what makes trunk-based development survivable for real feature work - without them, incomplete code either blocks a merge or leaks to users. The trunk-based development site has a good breakdown of how flags, small commits, and short-lived branches reinforce each other (trunkbaseddevelopment.com/feature-flags).

You don't need a dedicated flags platform to get the core benefit. A table, a hashing function, and the discipline to delete flags once they've served their purpose will get a five-person team most of the way there.

Categories
Teams

On-Call When There Are Five of You

Flat illustration of five avatar nodes arranged around a rotating dial with one node highlighted, representing an on-call rotation in a small team

Most on-call advice is written for organizations with a dedicated SRE team, a follow-the-sun rotation across three continents, and a dashboard nobody outside the ops team has ever opened. None of that applies when your entire engineering team is five people and everyone also writes features during the day. On-call at that scale needs different rules, or it turns into a tax that burns out whoever’s least willing to push back.

Rotate, even if it feels unnecessary

The instinct on a small team is to let whoever built a feature handle its incidents, since they know it best. This seems efficient and quietly turns into a problem: one person becomes the permanent safety net, gets paged constantly, and everyone else’s incident-response skills atrophy. Put a real rotation in place even at five people – a week each, or two if pages are rare. The person on call that week owns everything, not just their own code. This forces documentation and runbooks to exist, because the person responding won’t always be the author.

Write down what “page me” means

Vague escalation criteria are the fastest way to make on-call miserable. If every anomaly pages someone at 2 a.m., people learn to ignore pages, which defeats the point. Define a short list of conditions that justify a page – user-facing outage, data loss risk, authentication failures, error rate above a hard threshold – and route everything else to a ticket that gets triaged in the morning.

Page immediately:
- API error rate > 5% for 5+ minutes
- Any 5xx spike on checkout or login
- Database replica lag > 60s
- Background job queue depth growing unbounded

File a ticket, no page:
- Single failed job with automatic retry succeeding
- Non-critical third-party API degraded
- Elevated latency within SLA

Google’s SRE book has a chapter specifically on what makes on-call sustainable, and the core idea holds at any team size: a page should always be actionable and should always matter (sre.google/sre-book – being on call).

Budget for the interruption, not just the response

A page at 3 a.m. doesn’t just cost the ten minutes spent fixing the issue – it costs the rest of that person’s next day, when they’re running on four hours of sleep and shouldn’t be reviewing anything sensitive. On a small team, build this into planning explicitly: whoever was on call and got paged overnight doesn’t owe a full day of output the next morning. Treating on-call as free capacity is how you lose people.

Tooling doesn’t need to be a budget line

You don’t need an enterprise incident platform to run a five-person rotation well. A free-tier alerting tool wired to your existing monitoring, a shared calendar for who’s on call this week, and a pinned doc with escalation steps covers real needs at this scale. The expensive platforms earn their price once you have dozens of services and multiple teams sharing an escalation policy – below that, the extra configuration surface is often just another thing someone has to maintain instead of ship product.

Keep a running incident log

Even lightweight incidents deserve two sentences in a shared doc: what happened, what fixed it, what would prevent it. At five people you don’t need a formal postmortem template, but you do need a record, because the same failure mode will recur in six months and nobody will remember the fix. This log becomes the seed of your runbooks – the next person on call searches it before paging anyone else.

The habit is worth more than the format. A plain markdown file in the repo, sorted by date, with a one-line summary and a link to the fix, beats an elaborate incident-management tool that nobody updates because logging an entry takes ten clicks instead of one commit.

Make handoff a real conversation

When the rotation changes hands, spend five minutes actually talking – not just a bot message saying “rotation updated.” What’s flaky right now, what’s mid-fix, what alert fired twice this week and might fire again. Atlassian’s guide to on-call health has good language for framing this as a habit rather than a formality (atlassian.com – on-call guide).

Don’t skip the retro because the team is small

It’s tempting to treat a five-minute Slack thread as sufficient after a minor incident, and sometimes it is. But anything that paged more than one person, or took more than thirty minutes to resolve, deserves a real look at why – not to assign blame, there’s nowhere to hide blame on a team this size anyway, but because the same five people will be the ones fixing it again if the root cause never gets addressed.

On-call at five people will never look like on-call at a company with a platform team. That’s fine – the goal isn’t to import a large-company process, it’s to make sure the pager doesn’t quietly become one person’s permanent problem, and that whoever’s holding it knows exactly what deserves to wake them up.

Categories
Practices

Code Review Without Friction in a Small Team

Flat illustration of a document panel with a highlighted block of code lines under a magnifying glass and an approval checkmark, representing code review

On a five-person team, code review can go one of two ways. Either it’s a rubber stamp – “LGTM” thirty seconds after the PR opens – or it turns into a slow-motion argument about tabs versus spaces while the actual feature sits unmerged for three days. I’ve lived through both, and neither is what review is for. The goal is narrow: catch bugs before production, keep the codebase coherent, and spread knowledge of the system across more than one head. Everything else is optional.

Decide what review is actually checking

Before you can fix review friction, agree on what a reviewer is responsible for. In practice that’s usually three things: does this change do what it claims to do, will it break something else that isn’t obvious from the diff, and can the next person who touches this file understand it without asking you. Style, naming preferences, and “I would have done this differently” are not on that list. If your team keeps arguing about formatting, that’s a linter and formatter problem, not a review problem – configure Prettier or Black and stop discussing it in PRs.

Put a number on response time

The single biggest source of review friction on small teams isn’t disagreement, it’s latency. A PR that sits for two days loses context – the author has moved on to something else, and picking it back up costs more than the original review would have. Agree on a norm: reviews get a first pass within a few working hours, not “when I get to it.” Google’s engineering practices guide makes this point directly – a reviewer should respond quickly even if the response is “I don’t have time for a full review today, but here’s a quick pass” (google.github.io/eng-practices – review speed). On a team of five, this is a habit, not a policy document – it just needs one person to model it consistently.

Label feedback by severity

Most review friction comes from ambiguity about whether a comment is a blocker. “Consider renaming this” reads very differently depending on whether the author is expected to act on it before merging. Prefixing comments removes the guesswork:

blocking: this will throw if `user` is null - handle it before merge
nit: could shorten this to a ternary, up to you
question: why do we retry here but not in the sibling function?
praise: nice catch on the race condition in the original PR

This is close to what the Conventional Comments spec formalizes (conventionalcomments.org). You don’t need the full spec – just the habit of marking non-blocking suggestions as non-blocking so the author can merge without a second round-trip.

Keep PRs small on purpose

A 40-line PR gets reviewed in ten minutes. A 900-line PR gets an “LGTM” without being read, because nobody has an uninterrupted hour to actually follow it. If a feature is large, land it in a sequence of small, independently reviewable PRs – behind a feature flag if it isn’t ready to ship. This is more work for the author up front, splitting a change into logical steps, but it’s the single highest-leverage habit for making review fast and actually useful rather than theatrical.

Review your own diff before anyone else does

The fastest review is the one that never needs a second round. Before requesting review, open your own PR as if you were the reviewer – read the diff top to bottom, not your editor’s view of the whole file. This catches the leftover debug print, the unrelated formatting change that snuck in, the TODO you meant to resolve before pushing. On a small team, a self-reviewed PR routinely needs one comment instead of five, which is the difference between a same-day merge and a PR that bounces back and forth for two days.

Assign, don’t broadcast

“PR ready for review” posted to a channel with no assignee diffuses responsibility – everyone assumes someone else will pick it up. On a team of five, just assign a specific reviewer, and rotate who reviews what so knowledge doesn’t pool in one person. If someone owns a part of the codebase, a lightweight CODEOWNERS file will auto-request them without you having to remember.

Know when to skip it

Not everything needs full review. A one-line typo fix, a config value bump, a revert of a change that broke the build – these can go through with a rubber stamp or even bypass review if your team trusts the author and the change is trivially reversible. Reserve careful review for what actually carries risk: anything touching auth, billing logic, data migrations, or public API contracts. Treating every change with the same ceremony is how review turns into overhead instead of a safety net.

None of this requires new tooling or a review policy doc nobody reads. It requires the team agreeing, once, on what review is for and what “blocking” means – and then a couple of people modeling fast, specific, non-personal feedback until it’s just how the team works.

Categories
Software

Writing READMEs People Actually Read

Open document with structured placeholder sections, representing a well-written README file

Most READMEs are written for the person who wrote the code. That person already knows how the project works, what it depends on, and why it exists. They write the README as a formality and fill it with implementation details that are only interesting after you’ve already understood the project. Then a new team member opens it six months later, reads “this project implements a reactive event bus with configurable back-pressure,” and closes the tab.

A good README is written for someone who has never seen the project. It answers the questions they actually have, in the order they have them, without assuming context they don’t have. Here’s how to write one that people open twice.

Answer the “why” before the “what”

The first paragraph of a README should tell the reader what problem this project solves and why it exists. Not what it is technically – what it does for you.

Compare:

Bad: “notifier is a Node.js library that implements a pub/sub pattern with configurable delivery semantics.”

Better: “notifier sends real-time updates to connected browser clients when server-side data changes. Drop it into an Express app and replace polling with websocket push in about an hour.”

The second version tells me what I’m getting and roughly whether it’s worth reading further. The first one tells me the implementation approach, which I don’t need to know yet.

Structure: the five sections you always need

Every project README needs at minimum:

Quick start – the minimum steps to get a working instance running. This should work when copy-pasted by someone who has never heard of your project. If it takes more than ten commands, it’s not a quick start.

## Quick start

git clone https://github.com/your-org/notifier
cd notifier
cp .env.example .env
npm install
npm run dev
# open http://localhost:3000

Requirements – Node 20+, Postgres 15+, Redis. Be specific about versions. “latest Node” will cause problems when the next major version changes something.

Configuration – list every environment variable the application reads, what it does, whether it has a default, and what a valid value looks like. This is the section I refer back to most often when something isn’t working.

| Variable         | Required | Default   | Description                        |
|------------------|----------|-----------|------------------------------------|
| DATABASE_URL     | yes      | -         | Postgres connection string         |
| REDIS_URL        | yes      | -         | Redis connection string            |
| PORT             | no       | 3000      | HTTP port to listen on             |
| LOG_LEVEL        | no       | info      | debug, info, warn, error           |

Development setup – running tests, linting, database migrations, common development tasks. This is where the Makefile targets or npm scripts go.

Deployment – even one paragraph on how this gets to production. Is there a Docker image? A CI pipeline? Manual steps? The new person joining the team should not have to ask five colleagues how deployments work.

Code examples that actually run

Code examples in READMEs go stale fast. There are a few ways to fight this.

Source examples from actual test files rather than writing them inline. If the README says “see examples/basic.js“, that file is more likely to stay current because it’s part of the test suite. If the example lives only in the README, no one has a reason to update it when the API changes.

Mark language in fenced code blocks. Not just for syntax highlighting – it also signals to the reader what they’re looking at. An unmarked code block is ambiguous.

```bash
npm run build
```

```typescript
import { createNotifier } from 'notifier'
const n = createNotifier({ port: 3000 })
n.on('connect', (client) => console.log('client connected'))
```

Include the expected output for commands where it’s not obvious. “You should see something like:” followed by a truncated output block answers the implicit question “is this working?”

Keep it honest

Don’t document features that don’t work or aren’t finished. A README that says “supports clustering” when clustering is a stub function erodes trust faster than not mentioning it at all. Same goes for badges: a test coverage badge showing 94% on a project with three tests is misleading. Either keep badges accurate or remove them.

If there are known limitations, list them. “Does not support Windows” or “not recommended for more than 100 concurrent connections” saves someone from building on your project for a week before discovering it won’t work for their use case.

Keep it current

Stale documentation is worse than no documentation – it actively misleads people. A few practices that help:

Add a CI check that runs the quick-start commands in a clean environment. If the setup instructions fail in CI, they fail loudly before someone else hits them.

Date-stamp major sections that change infrequently: “Deployment (last updated 2026-03-01)”. It sets expectations and prompts periodic review.

Make the README easy to find and update. Keep it at the repo root. Make the first contribution guideline something like “if the README doesn’t match what you found, update it.”

Length and tone

A README is not documentation. Documentation is comprehensive; a README is a doorway into the project. It should be long enough to answer the first five questions a new reader has, and short enough that they’ll read the whole thing. For most projects that’s 500-1500 words – roughly what you’re reading now.

Write like you’re explaining to a smart colleague, not like you’re writing a spec. Active voice, short sentences, concrete examples. Avoid jargon unless the reader already knows it (if they’re looking at your project, they probably do).

The GitHub guide to README files is a useful baseline: docs.github.com – about readmes. And the Make a README project has good templates organized by project type: makeareadme.com.

The README is often the first thing someone reads before deciding whether to use, contribute to, or hire the person who built something. It’s worth an extra hour to get it right.

Categories
Web Development

API Versioning Without Breaking Clients

Two parallel pipeline arrows running side by side with connectors, representing API versioning

Breaking an API is one of those mistakes you only need to make once to take versioning seriously from that point forward. I made it a few years into my career: renamed a field in a JSON response because the old name was confusing, deployed it, and spent the next two hours fielding support tickets from mobile app users who couldn’t use the app until they updated. The server was fine. The clients – which I didn’t control – were not.

API versioning is the set of practices that let you evolve your API over time without silently breaking clients. Done well, it’s nearly invisible to clients that don’t need to upgrade. Done poorly, it creates a maintenance nightmare of parallel codepaths that never get cleaned up. Here’s what I’ve learned about the middle path.

What counts as a breaking change

The first step is knowing what you’re trying to avoid. Breaking changes are changes that cause existing clients to fail without modification:

– Removing a field from a response
– Renaming a field
– Changing a field’s type (string to integer, object to array)
– Removing an endpoint
– Changing the meaning of a field (e.g., changing a status field from free-text to an enum)
– Requiring a new field in a request
– Changing authentication schemes

Non-breaking changes are things existing clients can ignore: adding new optional fields to a response, adding new optional request parameters, adding new endpoints, relaxing validation (accepting more input than before).

This distinction matters because most changes you want to make are actually non-breaking, and non-breaking changes don’t require a version bump.

URL path versioning

The most common approach: embed the version in the URL path.

GET /api/v1/users/123
GET /api/v2/users/123

It’s explicit, easy to route in any framework, easy to document, and easy to test. Clients know exactly which version they’re talking to. The downside is that it couples the version to the resource URL, which feels wrong from a REST purity standpoint – the version isn’t a property of the resource. In practice, this rarely matters.

One decision you need to make: does /api/v1/ mean the version of the entire API, or the version of a specific resource? A single major version for the whole API is simpler operationally – one codebase, one set of docs. Per-resource versioning gives more granularity but is much harder to maintain.

Header versioning

An alternative: accept the version in a request header rather than the URL.

GET /api/users/123
Accept: application/vnd.myapi.v2+json

# or a custom header
GET /api/users/123
Api-Version: 2026-04-01

The date-based versioning (used by Stripe) is interesting – instead of integer versions, each version corresponds to the API as it existed on a specific date. New clients get the latest behavior; existing clients keep the behavior as of their integration date.

# Express: extract version from header and route accordingly
app.use((req, res, next) => {
  req.apiVersion = req.headers['stripe-version'] ?? DEFAULT_VERSION
  next()
})

app.get('/users/:id', (req, res) => {
  if (req.apiVersion >= '2026-01-01') {
    return res.json(formatUserV2(user))
  }
  return res.json(formatUserV1(user))
})

Header versioning keeps URLs clean but is harder to test in a browser, harder to document, and easy for clients to get wrong. I prefer URL versioning for public APIs for exactly this reason.

Sunset policies: how to actually retire old versions

Versioning only works if you’re willing to eventually retire old versions. Otherwise you end up maintaining v1 forever because someone, somewhere, is still using it. A sunset policy makes the contract explicit.

The HTTP Sunset header (RFC 8594) lets you signal that a version is going away:

HTTP/1.1 200 OK
Sunset: Sat, 31 Dec 2026 23:59:59 GMT
Deprecation: true
Link: <https://docs.example.com/migration/v2>; rel="deprecation"

Send this header in responses for deprecated versions starting six to twelve months before the sunset date. Good API clients will log or surface the warning. Also document it prominently in the developer portal and email clients who’ve called the deprecated version in the last 30 days.

The sunset timeline depends on your client mix. Public APIs with many independent third-party integrations need 12-18 months notice. Internal APIs consumed by your own teams can move faster. Mobile app APIs need to account for the app store review delay and the long tail of users who don’t update.

Implementation: keep version branching shallow

The failure mode in versioning is having the version branch deep in your business logic. When a version check shows up inside a database query or a domain model, the codebase becomes hard to reason about and hard to clean up.

Keep versioning at the serialization layer. The internal representation of your data doesn’t change; only the format you return to clients changes.

# versioned serializers in Python
class UserSerializerV1:
    def serialize(self, user):
        return {
            'id': user.id,
            'full_name': user.full_name,  # old field name
            'email': user.email,
        }

class UserSerializerV2:
    def serialize(self, user):
        return {
            'id': user.id,
            'name': {             # new structure
                'first': user.first_name,
                'last': user.last_name,
            },
            'email': user.email,
        }

def get_user_serializer(version):
    return UserSerializerV2() if version >= 2 else UserSerializerV1()

The domain logic – fetching the user, checking permissions, applying business rules – is the same for all versions. Only the output format differs. This makes old versions cheap to maintain and easy to delete.

API changelog and documentation

Versioning without documentation is incomplete. Maintain a changelog that lists every version, what changed, what’s deprecated, and when deprecated versions sunset. Stripe and Twilio both do this well – their changelogs are worth reading as examples of the standard.

The RFC on the Sunset header is short and worth reading: rfc-editor.org/rfc/rfc8594. The MDN documentation on HTTP headers covers the related Deprecation header and Link relation types: developer.mozilla.org/en-US/docs/Web/HTTP/Headers.

The meta-principle: version only when you have to, communicate changes clearly, give clients enough time to migrate, and enforce sunsets on the schedule you committed to. Most API versioning failures are not technical – they’re communication failures.

Categories
Tooling

Logging and Observability Basics for Web Applications

Abstract monitoring dashboard with line and bar charts and a magnifying glass, representing logging and observability

I spent the first two years of my career treating logs as the thing that fills up the disk. A console.log("here") sprinkled through the code, a timestamp, maybe the error message. Then I worked on a production incident where the only information I had was a spike in HTTP 500s and no useful logs, and spent three hours reading source code to figure out what had happened. After that I started treating observability as a first-class concern.

Observability is a bigger word than logging – it encompasses logs, metrics, and traces – but for most small teams starting from scratch, getting structured logging right is the highest-leverage first step. Metrics and tracing build on top of that foundation.

Structured logging, not free-text

The difference between useful logs and useless logs is usually structure. A free-text log line like:

2026-04-12 14:32:01 User 4821 completed checkout

is readable to a human but hard to query programmatically. You can’t group by user ID, filter to checkouts over a certain amount, or join with other events for that user. A structured log line:

{
  "timestamp": "2026-04-12T14:32:01.234Z",
  "level": "info",
  "event": "checkout.completed",
  "user_id": 4821,
  "order_id": "ord_9f2a8b",
  "amount_cents": 4999,
  "currency": "USD",
  "duration_ms": 143
}

can be indexed, queried, aggregated, and alerted on. Every major log aggregation service (Datadog, Grafana Loki, AWS CloudWatch Logs Insights, Elastic) treats structured JSON as a first-class citizen. The query difference is significant: finding all checkout failures over $100 for a given user is a one-liner against structured data; against free text it’s a fragile regex.

In Node.js, pino is the go-to structured logger – fast, JSON-first, low overhead. In Python, the standard logging module can be configured for JSON output, or structlog gives you a nicer API. In Go, slog (added in Go 1.21) is now the standard library option.

# Node.js with pino
import pino from 'pino'

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: { service: 'api', env: process.env.NODE_ENV },
})

logger.info({ userId: 4821, orderId: 'ord_9f2a8b', durationMs: 143 }, 'checkout.completed')

What to log and what to skip

Log at the boundaries of your system: incoming HTTP requests and their responses, outgoing calls to external services, job starts and completions, significant state transitions in your domain. Don’t log inside tight loops or at the database query level by default – it’s noise and IO overhead.

A request log should capture: timestamp, method, path, status code, response time in milliseconds, request ID (more on this shortly), user ID if authenticated, and any error. That’s enough to reconstruct what happened for almost any incident.

Things to never log: passwords, tokens, full card numbers, PII in query parameters. It sounds obvious but it’s easy to accidentally capture these when you log the full request or serialize a user object. Audit what goes into your logs before they reach a third-party service.

Correlation IDs

When a request touches multiple services or spawns background jobs, you need a way to trace a user’s action across all the log lines it produced. A correlation ID (or trace ID) is a random identifier generated at the start of a request and propagated through every log line and every downstream call that request triggers.

# Express middleware to attach a request ID
import { randomUUID } from 'crypto'

app.use((req, res, next) => {
  req.requestId = req.headers['x-request-id'] ?? randomUUID()
  res.setHeader('x-request-id', req.requestId)
  req.log = logger.child({ requestId: req.requestId, userId: req.user?.id })
  next()
})

With this in place, every log line from a request includes the same requestId. Finding all logs for a specific failed request is a single query: requestId = "abc123". When you hand off work to a background job, pass the requestId as part of the job payload so the worker logs it too.

Metrics: the signal layer

Logs tell you what happened. Metrics tell you what’s happening right now at a glance. The four golden signals (latency, traffic, errors, saturation) from the Google SRE book are a practical starting point.

For most applications, you want: HTTP request rate and error rate per endpoint, p50/p95/p99 response times, queue depth and job failure rate if you have background jobs, and basic infrastructure metrics (CPU, memory, database connection pool usage).

Prometheus + Grafana is the standard open-source stack for metrics. If you’re on a managed service, Datadog and New Relic give you metrics, logs, and traces integrated – the cost is justified once you’re spending engineering hours debugging production issues without them.

Error tracking

Unhandled exceptions need to go somewhere visible before a user reports them. Sentry is the default choice – free tier covers most small apps, the SDK integrates in a few lines, and the grouping algorithm is good at collapsing duplicate errors into a single issue.

import * as Sentry from '@sentry/node'

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 0.1,  // 10% of requests for performance monitoring
})

Wire up Sentry before you go to production. The first time you see a user-affecting bug surface in Sentry an hour before the user reports it is when you understand why.

Log aggregation

Logs written to stdout on a container are ephemeral – they disappear with the container. In a multi-instance deployment you also need logs from all instances in one place. Ship logs to a centralized service: Grafana Loki (self-hosted), Logtail, Papertrail, or Datadog Logs.

The simplest integration is usually a logging driver in your container runtime that forwards stdout to the aggregation service, rather than adding SDK code to your application. Docker’s GELF or Fluentd logging drivers, or the CloudWatch Logs driver on AWS, handle this at the infrastructure level.

The OpenTelemetry project is worth watching – it’s becoming the standard for instrumenting applications to emit logs, metrics, and traces in a vendor-neutral format: opentelemetry.io/docs. The Google SRE book’s chapter on monitoring is freely available and still the best conceptual introduction: sre.google/sre-book.

Categories
Backend

Background Jobs and Queues Explained

Flat pipeline of task cards moving along a conveyor belt, representing background job queues

If you’ve ever hit “submit” on a web form and watched the browser spin for ten seconds while the server sent an email, resized an image, and updated three external services before finally returning a response – you’ve felt the problem that background jobs solve. The fix is to take anything that doesn’t need to happen before the response and do it asynchronously, after the response has already been sent.

This sounds simple, and the basic implementation is. The subtleties come in when you start thinking about reliability, failure handling, ordering guarantees, and worker concurrency. Let’s walk through the whole picture.

The core pattern

A job queue has three parts: a producer that enqueues work, a storage layer that holds pending jobs, and one or more workers that pull jobs off the queue and execute them.

# Producer: enqueue from the web request handler
def handle_user_signup(user):
    db.create_user(user)
    queue.enqueue('send_welcome_email', user_id=user.id)
    queue.enqueue('provision_default_workspace', user_id=user.id)
    return redirect('/dashboard')  # returns immediately

# Worker: runs in a separate process
def send_welcome_email(user_id):
    user = db.get_user(user_id)
    email_client.send(
        to=user.email,
        template='welcome',
        context={'name': user.first_name}
    )

The web process enqueues two jobs and returns the redirect in milliseconds. The worker processes pick up the jobs and run them in the background. The user sees a fast response; the email and workspace setup happen seconds later.

Choosing a queue backend

Redis is the most common queue backend for small to medium applications. It’s fast, widely available as a managed service, and supported by mature libraries in every language. BullMQ (Node.js), Sidekiq (Ruby), Celery with the Redis broker (Python), and Faktory all use Redis.

Database-backed queues are worth considering if you want to avoid adding Redis to your stack. pg-boss (Node.js/Postgres) and Django’s django-db-queue store jobs in your existing database. The throughput ceiling is lower than Redis, but for most applications that’s fine – if you’re processing hundreds of jobs per minute rather than thousands per second, a database queue is simpler operationally.

For high-throughput or multi-team scenarios, dedicated message brokers like RabbitMQ or managed services like AWS SQS provide better durability guarantees, at the cost of more setup. SQS in particular is hard to beat for reliability on AWS – jobs are stored durably, delivery is at-least-once, and you pay per message rather than running infrastructure.

Retries and failure handling

Jobs fail. The network is unreliable, external APIs return errors, bugs in your job code surface. A queue system needs a retry strategy.

The standard approach is exponential backoff with a jitter. After the first failure, retry in 30 seconds. After the second, in two minutes. After the third, in ten minutes. Add a small random offset (jitter) to prevent all retrying jobs from hitting the same resource simultaneously after a recovery.

# BullMQ example: job options with retry configuration
const queue = new Queue('email', { connection: redisConfig })

await queue.add('send_welcome', { userId: user.id }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 30000,  // 30 seconds initial delay
  },
  removeOnComplete: { count: 1000 },
  removeOnFail: { count: 5000 },
})

Jobs that exhaust all retries should go to a dead letter queue (DLQ) – a separate queue that holds failed jobs for inspection. Don’t silently discard them. A DLQ is how you find out that a category of jobs has been failing for two days, diagnose why, fix the bug, and replay the affected jobs.

Idempotency

At-least-once delivery means a job might run more than once – during a worker crash mid-execution, a network partition, or a retry after a transient failure. Your job code needs to be safe to re-run.

For jobs with side effects (sending emails, charging a card, creating records), idempotency keys are the standard solution. Before performing the action, check whether it’s already been performed for this job’s ID. If it has, skip it.

def provision_workspace(job_id, user_id):
    if WorkspaceProvisionRecord.exists(job_id=job_id):
        logger.info(f"Job {job_id} already processed, skipping")
        return

    workspace = create_workspace(user_id)
    WorkspaceProvisionRecord.create(job_id=job_id, workspace_id=workspace.id)

This is especially important for anything that touches an external API. Most third-party APIs and email providers accept an idempotency key header precisely because they expect you to retry on failure.

Scheduled and cron jobs

Background job systems also handle work that needs to run on a schedule: sending weekly digests, running database cleanup, polling an external API for status updates. This is the job queue’s answer to cron.

The advantage over cron is visibility – you can see scheduled jobs in the same dashboard as your on-demand jobs, track execution history, and retry on failure. Cron silently drops failed jobs unless you’ve wired up alerting explicitly.

# Celery beat example: scheduled tasks
CELERYBEAT_SCHEDULE = {
    'send-weekly-digest': {
        'task': 'myapp.tasks.send_weekly_digest',
        'schedule': crontab(day_of_week='monday', hour=9, minute=0),
    },
    'cleanup-expired-sessions': {
        'task': 'myapp.tasks.cleanup_expired_sessions',
        'schedule': timedelta(hours=1),
    },
}

Observability for background jobs

Background job failures are invisible unless you make them visible. At minimum, instrument your workers with: job execution duration, failure rate per job type, queue depth (how many jobs are waiting), and worker concurrency utilization.

Most job libraries expose these metrics. BullMQ has a dashboard (Bull Board). Sidekiq has a built-in web UI. Celery integrates with Flower. If you’re using a custom setup, emit structured log events and aggregate them in your logging infrastructure.

The BullMQ documentation is the best reference for the Node.js/Redis side of this: docs.bullmq.io. For the broader patterns, AWS’s documentation on SQS gives a good treatment of the reliability properties you should expect from any queue system: docs.aws.amazon.com/AWSSimpleQueueService.