AI Grew Up on Git, Part 4: The Paper You Could Run
For centuries, checking a claim meant rebuilding the work from prose. Machine learning, raised on Git, learned a different habit: clone the paper, pin the commit, run it yourself
This is AI Grew Up on Git, a series on the infrastructure that raised artificial intelligence and what AI is now doing to it. Across eight parts it traces how Git and GitHub set the speed and shape of modern AI, and teaches you to read and use Git along the way.
In February 2018, Science put a name on something machine learning researchers had been muttering about for years. The headline read “Artificial intelligence faces reproducibility crisis,” and the subtitle carried the diagnosis: unpublished code and sensitivity to training conditions were making many of the field’s claims hard to verify. The hook was a conference the week before in New Orleans, where reproducibility had been on the agenda, with some teams diagnosing the problem and one presenting tools to fight it.
The evidence had come from inside the house. A group at McGill, Peter Henderson and colleagues, among them a professor named Joelle Pineau, had published a paper that year with a title that reads like a warning label: “Deep Reinforcement Learning that Matters.” What they showed was quietly devastating. Take a published reinforcement-learning algorithm. Run it on the same task, with the same settings, changing nothing but the random seeds, the arbitrary numbers that initialize a training run. The results moved. Sometimes they moved enough to flip the conclusion. Two runs of the same idea could support different papers, and the difference between them was luck.
A field that could not tell its findings from its luck had a problem, and the old publication culture was structurally unable to catch it. A paper described its method in prose. The code, if you wanted it, was “available upon request,” a phrase of great diplomatic beauty that in practice frequently meant no. You could not rerun what you could not obtain, so most claims were never rerun, and a field that measured itself by leaderboard numbers kept rewarding results that were new over results that were checkable.
What rescued machine learning, to the extent it was rescued, did not come from journals. It came from the culture this series has been tracing: the one that keeps its work in repositories.
Paper, meet repository
Machine learning had already left the journals behind in one important way. The field lived on arXiv, the preprint server, where a paper appears in days rather than surviving months of review. Speed was the point, and the same appetite for speed had made the field an early and total adopter of Git and GitHub. Through the 2010s these two cultures fused, and out of the fusion came a norm that nobody decreed and nobody owns: a serious paper ships with a repository.
The link changed what a paper was. The repository stopped being supplementary material, the dusty appendix nobody opened, and became half the publication. And inside the repository, a humble file quietly took over one of the paper’s oldest jobs. The README, the plain-text front page of a repository, became the real methods section: here is the environment, here are the dependencies, here are the commands, here is what you should see when it works. A prose methods section gestures at what was done. A README executes. One is a description of an experiment; the other is the experiment, one command away.
That difference rewired verification. For most of scientific history, checking a claim meant rebuilding the work from its description, a labor so heavy it was rarely performed. Now checking a machine-learning claim, in the best case, meant something a graduate student could do before lunch: clone the repository, check out the exact version the authors ran, and run it. Reproducibility collapsed, in the favorable cases, into a checkout. The Part 1 machinery made this trustworthy in a specific way: a commit hash is an address for a precise moment in a codebase’s history, tamper-evident and unambiguous, so “the code we ran” stopped being a claim and became a coordinate.
The norm needed one more thing: a map. With thousands of papers arriving monthly, knowing which ones had code, and which code matched which claim, was itself a research problem. Two researchers made it their side project.
