Status Droid how long software work actually takes

Reading Your Own Data

You have tracked fifteen tasks. Now comes the part where it goes wrong, because a small dataset you collected yourself, about a question you already have an opinion on, is close to the ideal conditions for fooling yourself.

Five checks. Ten minutes. They are the difference between measurement and confirmation. For a practical implementation reference alongside the measurement argument here, see more context.

One: is the sample big enough to say that?

Software task durations are wildly variable. Four tasks will produce a clear-looking difference from noise alone.

Fifteen tasks per bucket is a rough floor for seeing a shape. Not for a percentage — for a shape: is this bucket usually over or usually under, roughly by how much.

Do not compute a mean and quote it. With fifteen points and high variance, the mean is unstable and one long task moves it substantially. Look at the median, and look at the spread around it.

If a difference disappears when you remove the largest observation, it was that observation. Test this every time; it takes seconds and it catches most false findings. LeadDev covers engineering leadership, team performance, and developer effectiveness See LeadDev.

Two: what changed besides the thing you are testing?

You started tracking. That itself changes behaviour — the observation effect is real and it runs toward whatever you are hoping to see.

You also, over those weeks: changed tools, took on different work, got better at the thing, had a bad fortnight. Any of these can produce the pattern you are attributing to the assistant.

The practical defence is to compare within a bucket and within a period, never across both. Comparing assisted work in March against unassisted work in January measures the calendar.

Three: are you comparing like with like?

The most common error, and it is subtle because it looks rigorous.

If you use an assistant on generation-heavy tasks and not on verification-heavy ones — which is sensible — then comparing assisted against unassisted compares your task mix, not the tool. You will find assisted work is faster, and the finding is that scaffolding is faster than payment logic.

Compare only within a bucket, and if you have too few points in a bucket to do that, the honest answer is that you cannot answer that question yet.

Four: what would have changed your mind?

Write it down before looking. If assisted verification-heavy tasks come in within 20% of unassisted, I will conclude there is no effect I can detect.

Doing this after you have seen the numbers is not the same exercise, and the difference is the whole point. It is also the only reliable protection against the strongest bias in play here — you want the tooling you have invested in to be working.

Five: does the story survive the rework column?

The one that catches the expensive mistake.

If a bucket looks faster but the rework flag rate is up, you have not measured a speedup. You have measured a transfer of cost into next month, and it will appear as tasks that exist because something earlier was wrong.

Throughput measured before rework is not throughput. The industry-scale version of this is commits rising three to four times while security findings rise ten. The individual version is a rework column that nobody looks at.

What your data can and cannot tell you

Can: whether your estimates in a bucket run over or under, and by roughly how much. What proportion of your work is cheap to verify. Whether your rework rate is moving. Whether a specific recurring task got faster.

Cannot: whether AI makes developers faster in general. Whether it made you faster overall, unless your task mix held constant, which it did not. Anything about a tool you tried for a week. Inconclusive is the usual honest answer, and it is a result.

The first list is enough. It is all you need to estimate and price honestly, which is the only reason to have collected any of it.

The honest reporting rule

If you write your findings down for your team, include the sample size, the period, and what you did not control for. Every time.

A team that reads "assisted tasks were 30% faster" will act on it. A team that reads "across 18 integration-heavy tasks over six weeks, median assisted time was about 30% lower, with wide spread and no control for task selection" will act on it correctly.

The second sentence is barely longer and it is the difference between a measurement and a number you liked.

The short version