Check case coverage before comparing two simulation solvers

If each solver exports its own results table, the rows can cover different subsets of the test plan. Compare each export to the same planned case IDs before calculating aggregate errors.

This plain Python example uses synthetic inputs. One solver omitted an export for case-03. The other exported a row marked as failed.

planned = {"case-01", "case-02", "case-03"} exports = { "A": {"case-01": "ok", "case-02": "ok"}, "B": {"case-01": "ok", "case-02": "ok", "case-03": "failed"}, } for name, rows in exports.items(): missing = sorted(planned - rows.keys()) failed = sorted(case for case in planned & rows.keys() if rows[case] == "failed") print(f"{name}: missing={missing}; failed={failed}") paired_ok = sorted( case for case in planned if all(rows.get(case) == "ok" for rows in exports.values()) ) print(f"paired successful cases: {paired_ok}") 

A: missing=['case-03']; failed=[] B: missing=[]; failed=['case-03'] paired successful cases: ['case-01', 'case-02'] 

An absent export could mean a failed job, a skipped job, or a collection error. Check the job log before assigning a cause. If you report a paired error comparison, name the two successful cases and keep the coverage table beside it. Preserve the original case identifiers and check for duplicate or unplanned IDs before turning a real export into a dictionary.

I work on RigorMed, an AI research workspace for bringing research material, analysis, and reviewable outputs into one project.