Why "it works on my machine" isn't enough for open-source QA

When you build an open-source testing framework, you reach a point where the code is done, the tests are passing, and the features are implemented.

For a lot of developers, that's the end of the line. "It works on my machine. I'm done."

But for a framework to be truly valuable to a client, that is only the halfway point. The code existing is not the same thing as the code being reliable.

I've been building Praman, an open-source QA engine in Python. The test suite just crossed 910 passing tests. The API automation, browser sessions, configuration, and reporting are all implemented. But as I look at my roadmap, I realize the hardest work is still ahead: proving it actually works in the real world.

Here is why building a framework for real clients requires a different mindset.

When you build a feature, it's easy to mark it as "done" in your head. But a QA framework requires independent validation.

If I build a wheel and test it locally, that proves the wheel builds. It doesn't prove that a developer in a different environment can install it, import it, and run a test without it breaking. To prove that, I have to build the wheel, spin up a completely isolated Python 3.12 environment, install the wheel from scratch, and run an external client smoke test.

Only then is the packaging "done."

Phase 5 of my roadmap is AI-assisted testing. It's the most exciting part. I want to build LLM integration, failure analysis, and suggested fixes.

But I am deliberately not starting it yet.

If I introduce an AI agent that analyzes failures, but my JSON failure reports are inconsistent, the AI will give me bad analysis. If I add LLM code generation, but my isolated installation tests aren't passing, the AI will generate tests that don't run.

You cannot build advanced AI features on top of a shaky foundation. The execution, reporting, and evidence systems have to be bulletproof first.

I created a "Foundation Release Candidate" checklist for the next milestone. It doesn't just ask "does the code exist?" It asks:

• Does the latest wheel build and install independently?
• Do the external client tests pass?
• Are failure screenshots and redaction verified?
• Does the latest hosted GitHub Actions run succeed?

If any of these are false, the foundation isn't ready.

A passing test suite is a great milestone. It means you have 910 examples of your code doing the right thing. But a reliable framework is not just about the tests you write. It's about isolated client installation, clean CI pipelines, and documenting exactly what has been verified.

If you are building a tool for other developers, don't just ask if it works on your machine. Ask if it works on theirs.

You can see the current progress and the roadmap on the repository: https://github.com/pranromumu/praman