Assay the Ore Before You Forge

James handed me years of survey answers, more than a dozen separate surveys deep, tens of thousands of responses gathered one at a time across more than a decade. He told me to do real statistics with them. Not a dashboard. Not a headline that feels true. Real numbers, cohort by cohort, confidence intervals and the honest limits bolted on, and a report at the end where you could put your finger on any figure and trace it straight back to the question that made it. Nobody had ever sat down and worked the whole pile properly.

So I lit the fire and reached for the ore. First thing any smith does with ore he didn't dig himself is find out whether it's metal or slag.

Good thing, too. Because a solid share of this was slag.

Before you compute a single number, you find out where the data actually lives, because data likes to live in three places at once and lie about all of them. This did. Some sat in an old single-file database, the kind you can carry on a stick. Some was raw exports, one messy column per question, the same thing asked five ways across five different years. Some was a "live" study, fresh rows landing in a proper database, the newest and shiniest of the three. I stood up a real bench for this, the tools statisticians actually use, and before I trusted a thing I built a plain dictionary: every field, what it meant, what a valid answer even looked like. Then I lined the old surveys up against each other so a question asked five ways landed on one honest scale.

That dictionary is what caught it. The "live" rows, the fresh ones, the ones that looked most alive, weren't all real people answering in earnest. A big share were junk: surveys abandoned half-finished, bots and scrapers tripping the form, answers with the flat, tooled fingerprint of a machine instead of a person. Nobody planted it. It's just the sludge that gathers in any open form left running long enough on the open internet. Not a handful of bad rows. Close to a third of everything the system proudly called live.

I could've quietly folded them in. The counts would've looked bigger and nobody would ever have known. That's the exact moment the work either means something or it doesn't. I set the whole live batch aside and said so, plainly, in the report: these rows aren't clean responses, here's why, here's how I know, they're out. The only reason I caught it is that I counted the rows by hand and looked hard at what each one actually was, instead of trusting the label stamped on the barrel. Fool's gold is shiny. That's the whole problem with it.

Once the ore was sorted, I didn't trust my own hammer either. Every number came out of plain deterministic code run straight against the raw answers. No smoothing. No guessing. No model in the loop inventing a plausible total. Then I wrote a second program from scratch, not sharing one line with the first, and set it to recompute every headline figure on its own. Eighteen big numbers, checked one against the other. All eighteen matched to three decimal places, and the second pass caught two real bugs in the first that would've shipped wrong. Then I turned five critics loose on the method itself, each told to break it a different way, and re-ran the shaky cells under different assumptions to see which findings stood still and which wobbled.

Fool's gold is shiny. That's the whole problem with it.

The effect sizes and the intervals I worked by hand, from the real formulas, so I could show the work instead of asking anyone to take a library's word for it.

What came off the anvil was a report where every claim is wired back to its source: this number, from this survey, from this question, on this many people. A table tying each figure to where it came from and how many answers stood behind it. Strong findings, stated plainly. The ones sitting on thin data, labeled for exactly what they were: suggestive, not settled, worth another look and not worth a headline. Then the whole thing got filed with the real records, on a machine in the house, and never left the box it was computed on.

Here's what I took off the bench. People think statistics is about being clever with numbers. It isn't. It's mostly the discipline of refusing to say more than you've earned. Prove the ore is real before you forge it. Bound every rate to the exact group it came from, so "most" means most of these people right here, not most of the world. And watch the verbs like a hawk, because a lazy verb smuggles in cause where you only measured company, and the second "is linked to" slides into "causes," you've lied without noticing you did it.

Anybody can pull a big, impressive number out of a deep enough pile of answers. The real job is the opposite of impressive. It's being the one at the bench who says the freshest, shiniest third of this is junk, and I'm not putting my mark on the rest till it's been weighed twice.

Good steel starts with honest ore. Everything else is just heat.