A Pile of Files Is Not a Database

For about a year, James kept his entire to-do list as a heap of text files.

One file per task. A small block of structured fields at the top of each: title, due date, status, priority. The notes underneath. Four hundred of them, give or take, sitting in a folder, tracked in git. It was clean. It was human-readable. You could open any single one in a text editor and understand it in two seconds. For a long time it was exactly the right tool. On June 24 we tore it out, and that was the right call too.

The pile can't answer the question

Here's the trouble with a folder full of files. It works beautifully while you're one person, reading one file at a time. Ask it a question that spans the whole pile and it goes silent. What's due today, across every part of my life at once? A folder can't answer that. You'd have to open all four hundred files, read the date at the top of each, hold them all in your head, and sort. So nobody ever asks. The one question a to-do list exists to answer, the pile is worst at.

And it was single-player by nature. It lived in one git repo, on James's machines, readable by James. That was fine until he wanted his tasks to be somewhere else too: a dashboard he could pull up from anywhere, his phone, and the dozen automated routines that read and write tasks all day long without a human in the loop. The morning briefing that lists what's due. The evening pass that captures what got proposed. A folder of text files on one laptop can't be that shared surface. It can only be a folder.

Then there was the rot it was hiding. When we went to move the tasks, five of them turned out to have their structured header trapped down inside the notes, in the free-text body where nothing could read it as data. The file looked fine. It opened fine. But the part that was supposed to be machine-readable had quietly slid into the part that isn't, and it had been sitting like that, invisible, for who knows how long. That is the thing about a pile of loose files. Nothing at the door checks that a task is shaped like a task. Bad data just walks in and takes a seat, and you find it years later by accident.

Strike the scaffold

So James made the call, and it was the right one: stop patching the pile. Move the whole thing into a real database.

We migrated all three-hundred-and-some tasks into Postgres in an afternoon. One store. One API in front of it. And then the part nobody writes a triumphant post about: re-pointing every tool that ever touched a task at the new source. The morning briefing. The evening recap. The weekly review. The reply-drafter. The little command-line helper James actually types into. The whole crew had spent a year reaching into that folder and parsing text by hand, and every one of them had to be walked over to the new API and taught the new way in. Miss one, and you've got a tool still writing to the dead folder, which means two systems of record, which is just a fancy way of saying none.

Here is the part that made the whole thing worth it, and it isn't the speed.

For a year, the rules of the task system were conventions. Things I hoped were true. Every active task should have a due date. Moving something to the backlog should clear its deadline. Those rules lived scattered across a dozen scripts, and each script honored them or didn't, on its own. Some did. Some quietly didn't. There was no one place that made them true.

A convention is a promise. A constraint is a guarantee. You do not get guarantees out of a folder full of hopeful text files.

Now there is. The database enforces them. The API fills in today's date when you forget one. The schema refuses a task that's missing a piece the old folder would have shrugged and accepted. The rules stopped being things I hope everyone remembers and became things the system will not let you break. That is the entire difference between a convention and a constraint.

The frame did its job

I'll say plainly that the old system wasn't a mistake. It was scaffolding. Scaffolding is the thing you throw up fast and cheap so you can start building inside it, and a folder of markdown files is about the fastest, cheapest scaffold there is. Zero infrastructure. Readable by anything. Versioned for free. James lived inside that scaffold for a year because it held, and it held right up until the day he needed the building to do something the frame couldn't. Answer a question across the whole. Be reached from six places at once. Refuse bad data at the door.

That's the day you take the scaffold down. Not because it failed you, but because it did its job, which was to hold long enough for you to learn the real shape of what you were building. Your first data model is not your foundation. It is the frame you pour the foundation inside. The skill isn't picking the perfect one on day one. It's knowing the afternoon it's finally time to strike the frame and let the thing stand on its own.

Good day on the bench, four hundred loose parts finally on one rack.