A migration is the only kind of engineering project that is routinely judged without evidence. A new feature either works or it does not. A new report either shows the number or it does not. But a migration replaces something that was already running, and once the old thing is switched off, the only record of how it behaved is whatever somebody happened to write down beforehand. In most of the projects we are brought in to review, that record does not exist. We have started asking for it in the first meeting, not because it is hard, but because it is the only part of the work that becomes impossible later rather than merely expensive.
The consequence is not that the migration fails. Usually it works fine. The consequence is that nobody can say whether it was worth doing, and so the conversation about it becomes a contest of impressions — the team who built it remember it as a success, the people who use it remember the fortnight when everything was slow, and both groups are describing the same project accurately. And because the disagreement cannot be settled, it gets settled by seniority instead, which is a bad way to decide whether to spend the same money again next year on the next system.
A baseline is not a performance test
The word "baseline" usually makes people think of load testing, and that is the wrong instinct. A load test tells you what a system does under conditions you invented. A baseline tells you what it actually did last Tuesday, under the load it actually had, with the data it actually held. The second is harder to fake and much more useful in an argument, because nobody can claim the conditions were unrepresentative. It also costs less: a load test needs an environment, a data set and somebody to interpret it, while a baseline mostly needs somebody willing to read files that already exist on a server you already pay for.
This distinction matters because the two produce different numbers and people confuse them freely. We have read a migration post-mortem that compared the new system’s load-test latency against the old system’s production latency and concluded the new one was four times faster. It may well have been. The document did not show it, and nobody involved noticed the substitution for six months. The lesson we took from it is that a comparison is only as trustworthy as its least carefully described half, and the half everybody describes carelessly is the one that is being replaced.
The four numbers that matter most
You do not need a comprehensive picture, and attempting one is how this task gets postponed until it is impossible. Four numbers carry most of the weight, and all four can usually be recovered from systems you already run without instrumenting anything new. Record these four and stop; a fifth number that nobody will look at is a cost, not a safeguard, and the instinct to be thorough here is what turns a one-day task into a project with its own kickoff meeting.
- The volume: how many of the thing happen per day, and what the busiest hour looks like against the average.
- The latency distribution at the point a person waits — not the average, the 50th, 95th and 99th.
- The failure rate, and separately the rate of the one failure that costs money.
- The business number the system exists to move, measured the way finance measures it.
The fourth is the one teams skip, and it is the only one anybody outside engineering will remember. A migration that halves page load time and does not move a single order is a real engineering achievement and a difficult thing to defend in a budget meeting. Knowing that in advance changes how you scope the work. It is also the number that decides whether anybody funds the next migration, so a team that records it is buying credibility for work it has not scoped yet.
Where to find them without new code
The usual objection is that the old system has no monitoring, which is often true and almost never a blocker. Systems leak evidence of their own behaviour in several places, and an afternoon of reading is usually enough to reconstruct a defensible picture of the last ninety days. The three places below are worth checking in order, because each one is cheaper to read than the last and the first two are usually enough on their own.
The web server log
Access logs hold request counts, paths, status codes and, in most default formats, the time taken to serve each request. Ninety days of them will give you four things.
- Volume.
- A shape for the day.
- The error rate by endpoint.
- A latency distribution that is good enough to argue with.
They are also the one artefact nobody thinks to delete. The one caveat is rotation: default configurations often keep a fortnight, so the first thing to check is not what the logs contain but how far back they go, and if the answer is two weeks then that is the measurement window you have.
The database’s own statistics
Postgres keeps cumulative call counts and total execution time per normalised query, and the equivalent exists in every serious engine. That table tells you which ten queries the system actually spends its life running, which is frequently not the ten the team would have guessed, and it is the single best predictor of what will hurt after the migration. Take the snapshot twice, a week apart, and keep both: the counters are cumulative since the last reset, so a single reading tells you about all of history at once rather than about the period you care about.
The invoice
Your hosting bill is a measurement instrument. It records, in money, how much compute and storage and egress the old system consumed, averaged over a month and impossible to argue with. It is also the number your finance team already trusts, which makes it unusually useful when the migration has to be justified to somebody who does not care about percentiles. Take the average of three months rather than the latest one, because a single invoice can contain a one-off transfer or a reserved-instance payment that makes the old system look either cheaper or far more expensive than it really was.
Averages hide the thing you care about
An average latency is the least informative summary of a latency distribution, because the users who are angry are all in the tail and the tail is exactly what the average erases. A system with a 200ms average can be one where everybody waits 200ms or one where ninety-five people in a hundred wait 80ms and the rest wait four seconds. Those are different systems with different problems and the same headline number. The arithmetic is unkind here: a tail that affects one request in twenty is invisible in a mean and is also, on a busy service, thousands of people a day, each of whom experiences the slow version as the only version they have ever seen.
Record three percentiles and the comparison afterwards becomes specific. We have seen a migration that improved the median by 15% and made the 99th percentile three times worse, which the team discovered from complaints rather than from their own dashboard, because their dashboard showed a mean.
| What you record | Where it comes from | What it settles later |
|---|---|---|
| Requests per day, peak hour | Access logs | Whether the new capacity is right |
| p50 / p95 / p99 latency | Access logs | Whether it got faster for everyone |
| Top ten queries by total time | Database statistics | What will hurt after the cutover |
| Error rate, and the costly error | Logs and support tickets | Whether reliability improved |
| Monthly infrastructure cost | The invoice | Whether it paid for itself |
Measure the old system, not the new one
There is a failure mode where the team instruments the new system beautifully and the old one not at all, which produces a rich dashboard with nothing to compare it to. It happens because the new system is the interesting one and because adding metrics to a codebase you are about to delete feels like waste. It is not waste; it is the entire point. The fix is to add the metrics to the old system first and the new one second, in that order, and to accept that a fortnight of data from the thing you are deleting is worth more than a year of data from the thing you are building.
If instrumenting the old system is genuinely impossible — a vendor product, a black box, a machine nobody has credentials for — then measure it from the outside. Time the operation with a stopwatch, twenty times, at a known hour, and write down the twenty numbers. That is a crude baseline and it is infinitely better than none, and it has the useful property that the same stopwatch can be used on the new system.
Pick the business number before you start
The business number is the one that decides whether the project is remembered as worthwhile, and it has to be chosen before the work starts rather than after the results are in. Choosing it afterwards is how a migration ends up being credited with a revenue rise that happened because of a price change in the same quarter. Pick one, not three: a project with three success metrics has none, because whichever one moved will be the one quoted afterwards and the other two will quietly stop being mentioned.
Write it down, with its definition, in the same document as the technical numbers, and get the person who owns that number to agree the definition in writing. This sounds bureaucratic and takes about twenty minutes. It prevents the most common and most expensive argument in the whole exercise, which is two people using the same word for two different quantities.
What a baseline costs
For most systems, recording all of this is one engineer for one day, and about half of that day is spent finding where the logs are kept. We have never seen it take a week. We have repeatedly seen the absence of it cost a quarter, because without it the only way to settle a disagreement about whether the migration worked is to do more engineering. The asymmetry is the argument: a day of work against a quarter of unresolvable disagreement is not a close call, and it is the one piece of project hygiene we have never regretted insisting on.
The comparison has to be repeatable
Recording the command alongside the number is what turns a baseline into a measurement rather than a claim. Six weeks after cutover, somebody will want to run the comparison again, and if the original number came from a query nobody saved, the new number will come from a slightly different query and the difference between them will be partly the migration and partly the query.
This is a mistake we have made ourselves, on our own infrastructure, and it is the reason we now write the command into the document as a matter of habit. The instrument has to be the same instrument, or the second measurement is measuring the instrument.
The data you migrate is also a number
Row counts belong in the baseline for a reason that only becomes obvious afterwards: they are the one measurement that proves nothing was lost. A migration that silently drops four thousand historical records will look fine on every latency chart and will be discovered eight months later by somebody running a year-on-year report. Counting the rows in each table before and after costs a single query and it is the only cheap defence against the one migration failure that cannot be fixed by rolling back.
Record the count per table, the sum of any column that represents money, and the oldest and newest timestamp in the largest table. Those four facts together make a silent truncation impossible to miss, because a truncation that preserves all four is not a truncation. The money sum is the one worth insisting on even when it feels redundant: it is the only figure in the document that somebody outside engineering can verify independently, which makes it the figure that ends the argument fastest.
There is a subtlety about when to take the count. Taken too early it will not match, because the old system keeps running and keeps writing rows; taken at the moment of cutover it is accurate but nobody has time to compute it. The workable answer is to take it twice — once a week before, to establish the shape, and once during the freeze window — and to write down the growth rate between them, so that a mismatch afterwards can be checked against what the growth rate predicts rather than argued about.
The rows nobody counts
Audit tables, soft-deleted records and anything in a status nobody uses any more are the rows most likely to be dropped, because they are the rows a migration script author reasonably decides are not worth carrying. Sometimes that decision is correct and sometimes the compliance obligation attached to those rows outlives the system by seven years. Either way it should be a decision somebody made on purpose and wrote down, rather than a side effect of a WHERE clause.
Who reads the baseline afterwards
The document has three readers and they arrive in order.
- The team doing the work, during the week after cutover, checking whether what they are seeing is normal.
- Whoever approved the budget, about a month later, asking the only question that matters to them.
- An engineer who has never met any of you, two years on, trying to understand why the system is shaped the way it is before changing it — and that reader is the reason the document goes in the repository rather than a wiki.
Writing for the third reader changes what goes in it. It means stating the obvious: which system this was, what it did, who used it, what hours it was busy. A baseline that assumes the reader was in the room is useless to the only reader who will still be reading it when it matters most, and the cost of writing those two extra sentences is about four minutes.
We keep ours as a single Markdown file next to the migration code, named for the system and dated, and we have never once regretted the duplication of having the numbers in the repository as well as in whatever dashboard produced them. Dashboards are reorganised, tools are replaced and accounts are closed; a text file in a repository that is still being deployed is the most durable storage most organisations actually have.
And it has one more use that is hard to anticipate: it is the fastest way to onboard the next person onto the system. A new engineer who reads one page of real numbers about how a system behaves under real load understands it better than one who reads the architecture diagram, because the diagram says what the parts are and the baseline says which of them the work actually goes through.
What we do in the first week
On a migration engagement the baseline document is the first deliverable, before any architecture work, and we will not quote the migration itself until it exists. That is not a process preference. It is that the scope of a migration depends on numbers we do not have yet, and quoting without them means quoting a guess and then defending it.
It has twice led us to recommend not migrating. In both cases the numbers showed the old system was comfortably inside its capacity and the real complaint was a single slow report, which was two days of work rather than two quarters. Those are the engagements we are most confident were the right advice, and neither would have happened without a day of measurement first.
- One page in the repository, not a spreadsheet on a laptop.
- The measurement window stated beside every number.
- The query or command that produced each figure.
- A business number with an agreed written definition.
- Three percentiles, never a single average.
None of this is sophisticated and that is rather the point. The reason it gets skipped is not difficulty but sequencing: it has to happen in the week when everybody is excited about the new architecture and nobody wants to spend a day reading old log files. That week is the only week it can happen in.
