This post was first published on my site, which has the full version with every figure, the wave-by-wave credits and the complete reference list.
Open edX is the open-source platform organisations around the world use to run online courses. It is not one program but more than 150 repositories, maintained by Axim Collaborative, 2U, edly Arbisoft, OpenCraft, eduNEXT and dozens of individual contributors. They share dependency constraints, reusable CI workflows, cookiecutter templates and release calendars. In 2020, the map of how they depend on each other lived only in people's heads.
Six years later there is a public dashboard at openedx.ossvitals.org that grades all 169 repositories in the Open edX GitHub organisation on nine health metrics, every day, with every weight and threshold published. I built the hosted version. The idea, the checks and most of the upgrade work behind it belong to a longer list of people, and this post is about how one led to the other.
The problem I kept running into
The first time it bit me was 25 September 2020. A new release of importlib-metadata broke the Python 3.5 and 3.6 test matrix in every repository that still supported those versions. There was no list of which repositories those were. I knew because I had touched them. I pinned the package to 1.7.0 in every one I could remember that day, starting with django-config-models#90.
That was the pattern for the next two years: fix a failure in one repository, then go and find the same failure everywhere else from memory.
The fix was never the hard part. Knowing where to apply it was.
Fixes propagate too. In May 2022 pip 22.1 broke pip-tools, which broke the automated requirements upgrade job in every repository at once. I pinned pip<22.1 in the common constraints file in edx-lint (#258), which every repository consumes, and removed it three days later once pip-tools shipped a fix (#261). One line in one file changed what more than a hundred repositories installed that week, and nobody had a picture of that blast radius either.
Upgrades as code
The method came from Jeremy Bowman, who managed the Architecture team at edX and presented it at DjangoCon US 2021 as Herding Ponies: Coordinating and Automating Django Upgrades Across 100+ Repositories: codemods for the breaking changes, scripts that rewrite tox and CI configs, dependency support tracking, and a schedule across dozens of repositories. Arbisoft's BOM squad, which I was part of, built and ran that tooling.
A codemod is a map of the ecosystem written as code. To write one, you have to know which repositories it applies to.
The Travis CI to GitHub Actions move was mine: in October and November 2021 I converted nineteen of the twenty-four repositories in that wave. The Django 3.2, 4.2 and 5.2 waves, Python 3.12 and the frontend upgrades were shared across the team; the original post has who did what, with pull request counts for every wave.
The part that does not show up in PR counts is the tracking. Between June 2022 and June 2023 I turned the output of a pytest deprecation-warning script into 71 issues in public-engineering, one per warning, each with the fix and the owning package (#85 to #128). Behind most public pull requests was also a production deployment: merge, deploy, watch the dashboards, and when a change broke something elsewhere, roll it back with the infrastructure team. Between 2021 and 2025 I rolled back more than twenty changes. None of that is in git history, and it is why I came to care about knowing which repositories were drifting before a change shipped.
Repo Health
In March 2020 the Architecture team, starting with Manjinder Singh, built pytest-repo-health and edx-repo-health: pytest plugins that clone every repository in the organisation and record facts about each one, such as which Python and Django versions it declares and whether it has a README.
It began as an internal edX tool. The checks were open source, but the job ran on edX infrastructure and fed an internal Google Sheet. By early 2023 the nightly job ran on Jenkins. I rewrote it as a script (#349), moved it into a reusable GitHub Actions workflow in the organisation's .github repository (#52), and retired the Jenkins job (jenkins-job-dsl#1692). Any repository or working group could now run the full scan by calling one workflow.
Jeremy also built the first dashboard, a Streamlit grid over the SQLite output (#374, #466). It ran on a laptop from a downloaded file. In July 2024 Muhammad Soban Javed and I presented the result at the Open edX Conference in Stellenbosch (recording). The talk was well received, and then not much happened.
A CSV in a repository is not a tool anyone opens in the morning.
What changed in 2026
Two things made it urgent again. Bitergia, the analytics vendor behind the community's contribution statistics, shut down its Open edX instance in March 2026, and in April the Maintenance Working Group asked for a replacement. And the pipeline had quietly broken: I moved the job to the working group's new repository (wg-maintenance#3), and then the data stopped flowing, partly because the job's output PRs needed a human to merge them and nobody did.
So the rebuild had one design goal the original never had: everything public. No private results store, no internal spreadsheet, no token with organisation-wide scope. Anyone in the community should be able to read, run or reproduce it from public data.
Some of the most useful work was fixing what the checks had been getting wrong:
- Ownership had never been recorded. The ownership check read each repository's
catalog-info.yaml, but a default argument stopped pytest from injecting the repository path, so every owner column had been empty since the check was written (#718). - Bots were counted as reviewers. The PR response time metric excluded logins ending in
[bot]. GitHub's GraphQL API returns Dependabot, Codecov and Copilot as bare logins, so the filter never matched and automation counted as a human response. With bots excluded properly, response times rose on most repositories (#722). - The CI check measured the wrong thing. It reported whether GitHub Actions was configured, true for 167 of 168 repositories, not whether CI passed. A default-branch CI state column fixes that (#719), and the scoring page discloses the limitation until the score switches over.
The dashboard
OSSvitals is the layer on top, and its design is deliberately boring:
- A small workflow in wg-maintenance calls the shared repo health workflow every morning. It clones every repository, runs the checks and commits a CSV.
- A workflow in the dashboard repository reads that CSV, computes scores and commits the result as JSON.
- The site is a static build that bundles those pre-computed files at deploy time.
There are no API calls at runtime, so the dashboard never hits a rate limit, never leaks a token, and can be rebuilt by anyone from public data.
Scoring is nine metrics, each normalised to 0 to 100, weighted into a composite and mapped to a letter grade. Commit recency, PR response time, PR closure ratio, release frequency and contributor absence factor cover activity. README quality, CI status, metadata compliance and dependency update tooling cover structure. Missing data scores 50, not zero, so a stable, finished library is not punished for being finished. Every weight, threshold and parse rule lives in one YAML file, and the How Scoring Works page is generated from it.
The view the working group actually uses is At Risk (#7). It joins ownership with activity, so you can see which production repositories are owned by nobody, or by a single person, and are getting worse. The Upgrades page tracks upgrade jobs, migration waves and redundant bot PRs (#11, #13).
The Upgrades page, wave tab, tracking one migration across 106 repositories. Snapshot of 5 October 2026.
On 5 October 2026, 126 of the 169 repositories graded A or B and none graded F.
What the data showed
The first thing a tool like this does is tell you things you did not want to know. The automated requirements upgrade bot had opened 7,206 pull requests across 80 repositories between June 2024 and September 2026. 3,174 were merged and 3,985 were closed without merging, with 47 still open. More than half of the bot's output was superseded by the next week's PR before anyone looked. Nobody had measured that, because nobody had the data in one place.
The history also showed what the community does well. In two weeks of May 2026, a single campaign pinned GitHub Actions to full commit SHAs in 123 repositories, with 119 merged.
The ecosystem can move fast when someone draws the map first.
And several original checks had stopped carrying information. A check that returns the same value for 99 percent of repositories is not a health signal; it is a historical record. The dashboard now flags those for the working group to retire (#23).
What I took from it
- Make the system legible before you automate it. Every codemod, bulk migration and dashboard view started with a list of which repositories were affected. The tooling was only ever a way to stop keeping that list in my head.
- Automate at the scale of the ecosystem, not the ticket. Fixing one repository is boring. Understanding why 150 repositories fail in correlated ways, and building something so they stop, is the work I will stay up for.
- Stewardship is recognised differently from output. Core Contributor status in Open edX is given for looking after the commons, not for pull request counts. The pull requests were how I learned where the commons was.
The dashboard is live at openedx.ossvitals.org, the source is at UsamaSadiq/OSSvitals, and the checks are at openedx/edx-repo-health. If your grade surprises you, the scoring page will tell you exactly why, and I would like to hear if it is wrong.
The full version, with the timeline figure, the per-wave credits and every reference, is on my site. OSSvitals is an open-source community project, not affiliated with or endorsed by Axim Collaborative. Open edX is a registered trademark of Axim Collaborative.
