STATERA Library · Volume 46 · Chapter 08 · EN
Source: https://github.com/Statera-Library/01_Vol_46_Causality_and_Scientific_AI/blob/main/Chapter_08_Experiments_and_Benchmarks/Chapter_08_Experiments_and_Benchmarks_EN.md

Volume 46. Causality and Scientific AI

Chapter 08. Experiments and Benchmarks

08.01. Causal Benchmarks

Causal benchmarks provide standardized environments for evaluating whether algorithms can estimate treatment effects, recover causal relationships, and reason about interventions under controlled conditions. Unlike ordinary predictive benchmarks, they must address the fundamental difficulty that factual and counterfactual outcomes cannot normally be observed simultaneously. Benchmark design therefore determines how ground truth, treatment assignment, covariates, outcomes, and uncertainty can be evaluated in a reproducible manner.

A useful causal benchmark should separate predictive accuracy from causal estimation quality. A model may predict observed outcomes accurately while estimating intervention effects incorrectly because it relies on confounding correlations. Benchmark datasets therefore typically provide treatment indicators, observed outcomes, pretreatment covariates, and some mechanism for evaluating treatment effects. The strongest settings provide known or simulated counterfactual ground truth against which causal estimates can be directly compared.

IHDP, the Infant Health and Development Program dataset, is widely used as a benchmark for estimating heterogeneous treatment effects. It originates from a real study involving infants and contains realistic pretreatment covariates describing characteristics associated with children and their families. In causal machine learning benchmarks, these real covariates are commonly combined with simulated outcome mechanisms, creating a semi-synthetic environment where treatment-effect ground truth can be controlled.

The semi-synthetic nature of IHDP is important because real covariate distributions preserve dependencies and complexity that purely artificial datasets may fail to reproduce. At the same time, simulated potential outcomes make it possible to know quantities that would otherwise remain counterfactual and unobservable. This combination allows researchers to evaluate causal estimators under realistic covariate structure while retaining objective knowledge of the treatment effects used for benchmarking.

IHDP is particularly useful for studying individual and heterogeneous treatment effects. Instead of asking only whether a treatment has a positive average effect, models must estimate how the effect varies across individuals with different covariates. This makes the benchmark suitable for comparing meta-learners, representation-learning methods, Bayesian causal models, neural causal estimators, and other approaches designed to estimate conditional average treatment effects.

A common challenge in IHDP-style evaluation is treatment imbalance and selection bias. Treated and untreated populations can differ substantially in their covariate distributions, forcing models to distinguish genuine treatment effects from differences caused by treatment assignment. This makes representation balancing, propensity modeling, regularization, and uncertainty estimation relevant components of evaluation rather than merely measuring outcome prediction performance.

Performance on IHDP can be assessed through errors in treatment-effect estimation. Metrics may compare estimated individual treatment effects with known simulated effects or evaluate errors in average treatment effects. Precision in estimation of heterogeneous effects, commonly expressed through PEHE-related measures, is particularly useful because it evaluates whether a model captures variation in treatment response rather than only reproducing population-level averages.

However, IHDP should not be treated as a complete representation of real causal inference. Its outcome mechanism is simulated, and repeated benchmark configurations may reflect assumptions that favor particular model families. Strong performance therefore demonstrates effectiveness under a specific semi-synthetic causal setting rather than universal causal validity. Evaluation should ideally combine IHDP with benchmarks that contain different treatment mechanisms, dimensionality, overlap, noise, and confounding patterns.

The Twins benchmark provides another widely used environment for counterfactual and treatment-effect learning. It is based on observational information concerning twin births and exploits the unusual structure of paired individuals with closely related biological and contextual characteristics. Causal benchmarking variants construct treatment and outcome settings from these data so that algorithms can be tested under realistic covariate distributions and challenging treatment-assignment conditions.

Twins data are attractive because paired observations provide a structure that is uncommon in standard observational datasets. The similarity between twins can help construct benchmark scenarios in which outcomes under alternative conditions can be approximated or generated with more informative relationships than arbitrary synthetic samples. This makes the dataset useful for evaluating counterfactual prediction methods, treatment-effect estimators, and representation-learning approaches.

In common causal machine learning uses of Twins, researchers define treatment variables and potential outcomes according to a benchmark construction rather than assuming that the raw dataset directly reveals both potential outcomes for every individual. This distinction is essential. The value of Twins lies in the benchmark design built around real-world covariates and outcomes, while causal ground truth depends on the assumptions and transformations used to construct the evaluation task.

Twins can be used to test how models behave under treatment imbalance, nonlinear relationships, high-dimensional covariates, and varying degrees of overlap between treatment groups. Algorithms that rely heavily on interpolation may perform poorly when treated and control populations occupy different regions of the covariate space. The benchmark can therefore expose whether causal representations support reasonable generalization beyond the most densely observed combinations.

Counterfactual prediction is a central evaluation objective in Twins-based tasks. Given observed characteristics and one realized condition, a model attempts to estimate what would have occurred under the alternative condition. Errors can then be aggregated into treatment-effect metrics. This evaluation emphasizes the core causal problem of reasoning about unobserved alternatives rather than simply predicting the outcome that actually occurred.

Both IHDP and Twins demonstrate the value of semi-synthetic benchmarking. Real-world covariates preserve correlations, sparsity, imbalance, and population structure, while controlled causal mechanisms provide evaluable treatment effects. This compromise is often more informative than either fully observational data, where causal truth is unknown, or simple synthetic data, where the statistical structure may be unrealistically clean.

Synthetic causal benchmarks provide the highest level of control over the data-generating process. Researchers explicitly define causal graphs, structural equations, noise distributions, intervention mechanisms, treatment assignment, and potential outcomes. Because every component of the causal system is known, algorithms can be evaluated not only on treatment effects but also on graph discovery, edge orientation, mechanism recovery, counterfactual prediction, and intervention response.

A synthetic benchmark can begin with a structural causal model in which variables are connected through a directed acyclic graph. Each variable is generated from its causal parents and an exogenous noise term. Researchers can then sample observational data, apply interventions such as do(X=x), generate alternative outcomes, and retain the complete ground-truth graph. This produces a controlled laboratory for studying causal algorithms.

Graph complexity can be varied systematically in synthetic environments. Benchmarks may change the number of variables, graph density, causal depth, parent count, nonlinear relationships, interaction terms, and noise levels. Algorithms can therefore be evaluated as causal systems become progressively more difficult. Such controlled scaling is especially useful for understanding computational limitations and identifying where particular causal discovery methods begin to fail.

Confounding can also be manipulated explicitly. Synthetic generators can introduce observed confounders, hidden confounders, mediators, colliders, selection effects, and measurement noise. By varying the strength of these mechanisms, researchers can determine whether an algorithm remains accurate when assumptions are gradually violated. This is difficult with real datasets because the true magnitude and structure of hidden confounding are rarely known.

Treatment overlap is another controllable benchmark dimension. Researchers can generate environments with strong overlap, where similar individuals appear in both treatment groups, and weak overlap, where treatment assignment becomes highly dependent on covariates. Weak overlap forces estimators to extrapolate and increases uncertainty. Evaluating across this spectrum reveals whether a method remains appropriately cautious when causal effects are weakly supported by observed comparisons.

Synthetic benchmarks are particularly powerful for evaluating causal discovery because the true graph is available. Estimated graphs can be compared with ground truth using structural Hamming distance, adjacency precision and recall, orientation accuracy, or intervention-based metrics. Evaluation can distinguish whether a model merely identifies dependencies or correctly recovers causal directions and mechanisms needed for intervention reasoning.

Temporal synthetic benchmarks extend causal evaluation to dynamic systems. Variables evolve across time, causal effects may have delays, and feedback can occur between states and actions. Researchers can generate known temporal mechanisms and evaluate lag discovery, state-transition modeling, intervention propagation, and long-horizon counterfactual prediction. These settings are directly relevant to robotics, autonomous systems, industrial processes, and causal world models.

Multimodal synthetic benchmarks can further connect causal variables to images, language, audio, point clouds, or other high-dimensional observations. A latent causal process determines underlying objects and events, while rendering or generative mechanisms produce sensory observations. Algorithms must infer causal factors from perception before reasoning about interventions, making these benchmarks useful for causal representation learning and Physical AI.

Synthetic environments also allow mechanism shifts to be introduced intentionally. A structural equation can change between training and testing while unrelated mechanisms remain stable. This evaluates whether a model can identify which part of the causal system changed and adapt selectively rather than retraining everything. Such benchmarks provide a direct test of modularity, invariance, transfer, and causal generalization.

No single causal benchmark is sufficient for reliable comparison. IHDP emphasizes heterogeneous treatment-effect estimation using realistic covariates and simulated outcomes, while Twins offers another real-data-derived environment for counterfactual and treatment-effect evaluation. Synthetic benchmarks provide complete control over causal structure and permit systematic stress testing across assumptions that cannot be manipulated reliably in observational datasets.

A strong benchmark suite should therefore evaluate algorithms across complementary conditions rather than ranking them using one dataset. Models can first be tested in synthetic environments where causal truth is completely known, then in semi-synthetic environments such as IHDP and Twins where realistic data structure introduces additional complexity. Finally, controlled real-world experiments can determine whether improvements observed in benchmarks transfer to operational systems.

Benchmark protocols should preserve reproducibility through fixed data-generation settings, documented treatment mechanisms, standardized train-validation-test partitions, repeated random seeds, and clearly defined metrics. Reporting should include both average performance and variation across replications because causal estimation can be sensitive to sampling, overlap, initialization, and treatment imbalance. Uncertainty in benchmark performance is itself important evidence about algorithm robustness.

For Physical AI, the same benchmark philosophy can be extended from treatment datasets to interactive environments. Synthetic robotic worlds can provide exact dynamics and intervention ground truth, semi-synthetic environments can combine real sensor recordings with controlled causal modifications, and physical testbeds can provide repeated interventions. Evaluation can then measure whether learned causal mechanisms improve planning, adaptation, diagnosis, and safety.

The purpose of IHDP, Twins, and synthetic causal benchmarks is ultimately not to produce a single leaderboard score but to expose different dimensions of causal competence. Effective evaluation should reveal whether an AI system estimates intervention effects accurately, recognizes heterogeneous responses, handles confounding and weak overlap, represents uncertainty, recovers mechanisms, and generalizes when environments change. Together, these benchmarks provide progressively richer evidence that a model has learned relationships useful for causal reasoning rather than correlations that happen to predict observed data.

08.02. Evaluation Methods

Evaluation methods for causal AI determine whether a model has learned relationships that support reliable intervention, counterfactual reasoning, and decision making rather than merely reproducing statistical associations. Because causal ground truth is often partially or completely unobservable, evaluation cannot depend on a single accuracy score. Instead, multiple complementary methods are required to assess treatment effects, causal structure, mechanisms, robustness, uncertainty, and downstream utility.

Treatment-effect evaluation measures how accurately a model estimates the consequences of interventions. When true potential outcomes are known in synthetic or semi-synthetic benchmarks, estimated effects can be compared directly with ground truth. Evaluation may target the Average Treatment Effect, Conditional Average Treatment Effect, or Individual Treatment Effect, depending on whether the objective concerns population-level, subgroup-specific, or individual responses.

Error in Average Treatment Effect provides a simple measure of population-level causal estimation. The estimated difference between treatment and control outcomes is compared with the known or experimentally established effect. Although useful, this metric can hide substantial errors in heterogeneous responses because positive and negative individual errors may cancel. It should therefore be combined with measures that examine treatment effects at finer levels.

Precision in Estimation of Heterogeneous Effect, commonly represented through PEHE-related metrics, evaluates whether individual or conditional treatment effects are estimated correctly. It compares predicted differences between potential outcomes with known treatment-effect differences across samples. This is especially important for personalized interventions, adaptive policies, medical decision systems, and AI agents whose optimal actions depend on context.

Counterfactual prediction error evaluates the model\'s ability to estimate outcomes under actions or conditions that were not actually observed. Metrics such as RMSE or MAE can be used when counterfactual ground truth is available from simulation, semi-synthetic construction, or repeated controlled experiments. Good factual prediction alone is insufficient because a model may reproduce observed outcomes while generating inaccurate alternative futures.

Policy evaluation translates causal estimation quality into decision quality. Instead of asking only whether treatment effects are numerically accurate, it evaluates whether a policy constructed from those estimates selects useful interventions. Policy value, expected reward, success rate, or regret can quantify whether causal reasoning improves decisions. This is particularly important when small estimation errors can change which action is ultimately selected.

Regret measures the difference between the outcome produced by the model\'s chosen intervention and the outcome that would have been achieved by an optimal intervention. Low regret indicates that causal errors have limited practical consequences, while high regret reveals that apparently reasonable estimates lead to poor decisions. Regret therefore provides an important bridge between causal inference metrics and operational performance.

Causal discovery requires structural evaluation when the true graph is available. Estimated graphs can be compared with ground-truth causal structures using Structural Hamming Distance, adjacency precision and recall, F1 score, or related measures. These metrics quantify missing edges, extra edges, and incorrect orientations. Structural evaluation is especially valuable in synthetic benchmarks where the complete data-generating graph can be explicitly controlled.

Edge-orientation accuracy should be distinguished from adjacency accuracy. A model may correctly detect that two variables are related while reversing the direction of influence. Such an error can have limited impact on observational prediction but severe consequences for intervention reasoning. Evaluation should therefore report whether relationships were merely detected and whether their causal directions were correctly identified.

Interventional graph evaluation can provide stronger evidence than structural similarity alone. Two graphs may differ in topology yet produce similar predictions for relevant interventions, while structurally similar graphs can behave differently after intervention. Evaluators can therefore apply standardized interventions and compare predicted post-intervention distributions with ground truth, directly testing whether learned structure supports the intended causal use.

Mechanism recovery evaluates the functional relationships associated with causal edges. Even when graph topology is correct, inaccurate structural equations can produce incorrect intervention outcomes. Evaluators can perturb parent variables and compare predicted child responses with known mechanisms. This separates the ability to discover which variables influence one another from the ability to model how those influences actually operate.

Calibration evaluates whether uncertainty estimates correspond to observed causal errors. Confidence intervals should contain true treatment effects at approximately their intended coverage rate, while posterior distributions should widen when evidence is weak or treatment overlap is poor. Calibration is essential because causal decisions often involve regions where direct comparisons are limited and models must communicate uncertainty rather than provide unjustified certainty.

Overlap-aware evaluation examines performance across different levels of treatment support. When treated and control populations occupy similar covariate regions, causal estimation is relatively well supported. Under weak overlap, models must extrapolate and uncertainty should increase. Reporting performance as overlap decreases reveals whether an estimator remains reliable and appropriately cautious when observational evidence becomes sparse.

Confounding robustness evaluates how estimates respond to variables that influence both treatment and outcome. Synthetic benchmarks can systematically increase confounding strength, while semi-synthetic studies can manipulate treatment-assignment mechanisms. Performance degradation across these settings reveals how strongly a method depends on ideal assumptions and whether adjustment, balancing, or representation-learning techniques successfully reduce confounding bias.

Sensitivity analysis extends robustness evaluation to assumptions that cannot be verified directly. Evaluators can vary hidden-confounding strength, adjustment sets, graph constraints, noise assumptions, model classes, and measurement error. If causal conclusions change dramatically under small plausible modifications, they should be considered fragile. Stable conclusions across reasonable assumptions provide stronger evidence for practical use.

Environmental invariance tests whether learned mechanisms remain valid across different domains. Models can be trained in one set of environments and evaluated in others with different populations, sensors, sites, workloads, object distributions, or operating conditions. If the underlying causal mechanism remains unchanged, a robust causal model should preserve intervention accuracy despite changes in superficial statistical correlations.

Mechanism-shift evaluation deliberately changes part of the causal system. Instead of testing only ordinary distribution shift, one structural equation, causal edge, or physical relationship can be modified while other mechanisms remain unchanged. Evaluation then measures whether the model detects the changed mechanism, preserves stable components, and adapts selectively. This directly assesses modularity and causal generalization.

Temporal evaluation is required when causal effects unfold over time. Metrics can assess causal lag identification, transition accuracy, intervention-response trajectories, impulse responses, accumulated effects, and long-horizon counterfactual predictions. A model that identifies the correct causal variables but predicts effects at the wrong time may still produce poor planning or control decisions in dynamic systems.

Repeated-intervention evaluation provides stronger evidence for temporal and dynamic causal models. The same or comparable interventions can be executed under controlled initial conditions, allowing predicted trajectories to be compared with actual trajectories. Repetition also reveals variance caused by stochastic dynamics and helps distinguish systematic model errors from ordinary environmental randomness.

Multimodal causal evaluation examines whether causal conclusions remain consistent across sensing modalities. Vision, LiDAR, audio, force, proprioception, and language may provide different observations of the same underlying event. Evaluators can remove, corrupt, delay, or replace selected modalities and measure whether the model preserves a coherent causal state rather than changing conclusions because one statistical input channel has shifted.

Ablation studies identify which components contribute to causal performance. Researchers can remove causal regularization, representation balancing, graph constraints, intervention data, uncertainty modeling, or domain knowledge and measure the resulting degradation. Ablation does not by itself prove causal correctness, but it clarifies whether claimed architectural innovations actually contribute to intervention estimation, robustness, or generalization.

Cross-validation requires special care in causal evaluation. Randomly splitting individual observations may leak environmental, temporal, subject-specific, or treatment-related information across training and testing sets. Stronger protocols can split by environment, individual, intervention, time period, site, or mechanism. Such partitions provide a more realistic test of whether causal knowledge transfers beyond conditions encountered during training.

Repeated trials and multiple random seeds are necessary because causal estimators can be sensitive to treatment imbalance, initialization, sample selection, and stochastic optimization. Reporting only the best run can substantially exaggerate performance. Evaluation should report mean results, variance, confidence intervals, and failure cases across repeated experiments to characterize both expected performance and reliability.

Statistical hypothesis testing can determine whether observed performance differences between causal methods are likely to reflect systematic improvement rather than sampling variation. However, statistical significance should be interpreted alongside effect size and operational relevance. A small improvement on a large benchmark may be statistically significant while providing little practical benefit for intervention selection or deployment.

Simulation-based evaluation is especially useful when physical interventions are expensive or dangerous. Synthetic environments, digital twins, and causal world models can expose systems to rare failures, extreme conditions, sensor degradation, and unusual action combinations. Simulation allows thousands of controlled interventions to be executed, providing broad coverage before a causal model is tested on real hardware.

Real-world experimental evaluation remains essential because simulation may omit important mechanisms. Randomized trials, A/B tests, laboratory experiments, controlled robotic trials, and staged deployments can compare predicted causal effects with actual intervention outcomes. Agreement between simulation, benchmark, and physical experiments provides stronger evidence than success within any single evaluation environment.

For Physical AI, closed-loop evaluation should measure the entire sequence from perception to causal inference, planning, action, and observed consequence. Metrics may include task success, trajectory error, intervention effectiveness, energy consumption, recovery behavior, safety violations, adaptation speed, and decision regret. This reveals whether causal knowledge actually improves autonomous behavior rather than merely producing accurate offline estimates.

Safety evaluation should specifically examine conditions where causal errors could generate hazardous interventions. Rare events, actuator faults, sensor failures, unstable states, unknown environments, and conflicting observations should be included. A trustworthy system should increase uncertainty, abstain, request additional evidence, or activate a safe fallback when causal assumptions are unsupported instead of confidently extrapolating beyond validated conditions.

No individual evaluation method can establish complete causal competence. Treatment-effect metrics measure estimation, graph metrics measure structure, counterfactual metrics test alternative outcomes, robustness tests challenge assumptions, calibration evaluates uncertainty, and policy metrics measure decision consequences. Their combination provides a multidimensional view of whether an AI system has learned causal knowledge that remains useful beyond its training data.

A mature evaluation framework therefore progresses from controlled synthetic ground truth through semi-synthetic benchmarks, simulation, controlled experiments, and finally real-world closed-loop operation. Each stage removes some artificial assumptions while introducing greater uncertainty and complexity. Together, these evaluation methods provide increasingly strong evidence that causal AI can reason about interventions, adapt to changing environments, and support reliable decisions in operational systems.

08.03. Reproducibility [w/Code]

Reproducibility in causal AI means that another researcher, engineer, or operational team can reconstruct a causal analysis and obtain results that are consistent with the original findings. This requires more than preserving trained model weights. The complete chain from data generation and preprocessing to causal assumptions, graph construction, estimation, evaluation, and intervention decisions must be documented and recoverable.

Reproducibility is especially important in causal inference because results depend on assumptions that may not be directly visible in the final model. Choices about confounders, adjustment sets, treatment definitions, temporal ordering, graph constraints, and identification strategies can substantially change estimated causal effects. Two teams using the same dataset may therefore reach different conclusions if these assumptions are not explicitly recorded.

Data reproducibility begins with preserving the exact datasets used in an experiment. Dataset versions, train-validation-test partitions, inclusion and exclusion rules, missing-value handling, normalization, feature construction, and sampling procedures should be recorded. When data cannot be redistributed because of privacy or licensing restrictions, metadata and deterministic preparation procedures should still make the original experimental conditions reconstructable.

Data provenance describes where observations originated and how they were produced. For causal analysis, provenance should distinguish observational records from interventional or experimental records because they provide different types of evidence. Treatment assignment procedures, measurement instruments, sensor configurations, collection environments, timestamps, population definitions, and known selection mechanisms should accompany the data whenever they affect causal interpretation.

Variable definitions must also be reproducible. A variable called treatment, exposure, state, reward, or outcome can have different meanings across experiments. Each variable should therefore have a documented semantic definition, unit, valid range, temporal reference, source, and transformation history. Stable variable dictionaries prevent apparently identical experiments from silently using different interpretations of the causal system.

Causal graph reproducibility requires preserving the exact graph used for analysis. Nodes, directed edges, latent variables, forbidden edges, required edges, temporal constraints, and domain-knowledge assumptions should be versioned. If the graph was learned automatically, the discovery algorithm, hyperparameters, independence tests, scoring functions, initialization, and random seeds should also be recorded so that the discovered structure can be reconstructed.

Structural causal models require additional documentation because graph topology alone does not define the complete causal mechanism. Structural equations, functional forms, noise distributions, parameter values, intervention semantics, and assumptions about exogenous variables should be preserved. This allows researchers to reproduce not only which variables influence one another but also how interventions propagate through the modeled system.

Identification procedures must be documented independently from estimation. Before estimating a causal quantity, an analysis determines whether that quantity can be identified from available data and assumptions. Backdoor adjustment, front-door adjustment, instrumental variables, propensity-based approaches, mediation assumptions, or other identification strategies should therefore be explicitly recorded rather than being hidden inside implementation code.

Estimator reproducibility requires preserving model architecture, optimization settings, hyperparameters, regularization, initialization, stopping criteria, and software implementation. Neural causal models can be particularly sensitive to random initialization and stochastic optimization. Recording one final checkpoint is insufficient if the training process cannot be recreated or if different runs produce substantially different treatment-effect estimates.

Randomness should be controlled and reported rather than eliminated conceptually. Random seeds can influence dataset partitions, simulated interventions, initialization, minibatch ordering, treatment assignment, and optimization. Repeating experiments across multiple seeds provides a better characterization of model behavior. Reproducible research should therefore report distributions of results rather than presenting one favorable random run as representative.

Computational environments are another source of variation. Library versions, compiler settings, operating systems, GPU architectures, numerical precision, hardware accelerators, and parallelization strategies can influence results. Environment specifications, dependency lock files, containers, or reproducible build configurations help ensure that an experiment can be recreated after the original software environment has changed.

Code versioning should connect every experimental result with the exact implementation that generated it. Source code commits, configuration files, data versions, model checkpoints, and evaluation outputs should share persistent identifiers. A result should ideally be traceable from a published table or deployment report back through the evaluation pipeline to the exact training configuration and source code revision.

Experiment tracking provides the operational infrastructure for this traceability. Each run can record dataset versions, graph versions, model parameters, metrics, artifacts, execution time, random seeds, hardware, and software dependencies. For causal AI, experiment trackers should additionally record treatment definitions, estimands, adjustment sets, causal assumptions, intervention specifications, and uncertainty methods.

Evaluation reproducibility requires standardized metrics and clearly defined protocols. Metrics such as Average Treatment Effect error, PEHE, counterfactual prediction error, Structural Hamming Distance, graph precision and recall, policy value, regret, and calibration should be computed using documented implementations. Small differences in metric definitions or aggregation procedures can otherwise produce misleading comparisons between causal methods.

Benchmark reproducibility depends on preserving data-generation procedures and benchmark configurations. Synthetic benchmarks should publish causal graphs, structural equations, noise distributions, sample sizes, intervention procedures, and generation seeds. Semi-synthetic benchmarks such as IHDP or Twins should document how real covariates are combined with simulated treatment or outcome mechanisms so that benchmark instances can be reconstructed consistently.

Repeated experiments are essential for distinguishing reproducibility from accidental numerical agreement. A method should be executed across multiple samples, seeds, environments, and treatment assignments when possible. Mean performance, variance, confidence intervals, and failure cases provide a more reliable description than a single number. High variability itself is important evidence about the stability of a causal method.

Statistical reproducibility concerns whether reported conclusions remain supported when an experiment is repeated. Confidence intervals, hypothesis tests, effect sizes, and uncertainty estimates should be generated through documented procedures. Researchers should distinguish a failure to reproduce an exact numerical value from a failure to reproduce the substantive causal conclusion, since stochastic methods may vary numerically while supporting the same interpretation.

Replicability extends the concept by asking whether an independent implementation can recover similar conclusions. Reproducibility may use the original code and data, whereas replication can involve independently written software, newly collected data, or another environment. Strong causal evidence should ideally survive both processes, demonstrating that findings are not artifacts of one implementation or one particular dataset.

Intervention reproducibility is especially important because causal claims concern what happens when something is changed. Experimental protocols should define intervention targets, values, duration, timing, initialization conditions, control groups, and measurement procedures. Without precise intervention specifications, two experiments described using the same treatment label may actually implement different causal manipulations and produce incomparable outcomes.

Temporal systems require synchronized records of state and intervention timing. Sampling rates, clock synchronization, latency, filtering, event ordering, and delay assumptions can alter inferred causal relationships. This is particularly important in robotics, industrial control, and autonomous systems where milliseconds or seconds can determine whether one event is interpreted as the cause or consequence of another.

Simulation reproducibility requires preserving simulator versions, environment configurations, physics parameters, object models, randomization ranges, initial states, and intervention scripts. Digital twins and learned world models should also be versioned because updates can change predicted intervention outcomes. A causal algorithm evaluated against two different simulator versions should not be treated as having experienced identical experimental conditions.

For Physical AI, hardware configuration becomes part of the reproducibility record. Robot mass, payload, wheel or leg configuration, actuator parameters, sensor placement, calibration, battery condition, controller firmware, compute hardware, and environmental characteristics may affect causal mechanisms. Software alone cannot reproduce an experiment if the physical configuration that generated the observations is unknown.

Closed-loop experiments require preserving the complete dependency chain. Perception models generate state estimates, causal models predict intervention effects, planners select actions, controllers execute them, and sensors observe the consequences. The version of every component should be recorded because changing even one module can alter the data distribution and resulting behavior of the entire autonomous system.

Reproducibility also supports debugging and incident analysis. When a deployed causal system produces an unexpected intervention outcome, engineers need to reconstruct what information was available, which graph and model were active, what assumptions were applied, which action was selected, and what software and hardware configurations were operating. Reproducible records transform unexplained failures into analyzable engineering events.

Artifact management provides a practical mechanism for preserving these dependencies. Datasets, causal graphs, model checkpoints, experiment configurations, evaluation reports, intervention logs, containers, and deployment manifests can be stored as immutable or versioned artifacts. Relationships among artifacts should be recorded so that a causal result can be reconstructed as a dependency graph rather than as an isolated model file.

Automation can reduce human error in reproducibility workflows. Pipelines can automatically capture source commits, dependency versions, seeds, configurations, data hashes, graph versions, metrics, and generated artifacts whenever an experiment runs. Automated validation can then rerun selected reference experiments and compare results against expected ranges before changes are accepted into a production causal AI system.

Documentation remains necessary even with extensive automation because many causal assumptions are conceptual rather than computational. Researchers should explain why variables were included or excluded, why certain edges were constrained, which confounders were assumed observable, and where identification assumptions may fail. Machine-readable configurations preserve implementation details, while human-readable documentation preserves the reasoning behind them.

A mature reproducibility framework therefore connects data lineage, causal assumptions, graphs, estimators, software environments, experiments, evaluation metrics, deployment versions, and intervention outcomes into one traceable record. The objective is not to guarantee that every run produces identical floating-point values, but to ensure that the same evidence and assumptions lead to scientifically and operationally consistent causal conclusions.

Reproducibility ultimately provides the foundation for trustworthy cumulative progress in causal AI. Without it, improvements cannot be distinguished reliably from changes in data, assumptions, software, or experimental procedure. With reproducible pipelines, causal models can be compared, audited, replicated, debugged, and safely evolved, allowing causal knowledge to become a durable engineering asset rather than a result that exists only within one experiment.

08.04. Model Comparison

Model comparison in causal AI is fundamentally different from ordinary predictive model comparison because the best predictive model is not necessarily the best causal model. A system may achieve excellent accuracy on observed outcomes while estimating intervention effects incorrectly. Comparison must therefore examine whether competing models recover causal effects, mechanisms, counterfactual outcomes, uncertainty, and decision consequences under comparable experimental conditions.

A rigorous comparison begins by defining the causal objective before selecting models. Depending on the application, the target may be Average Treatment Effect, Conditional Average Treatment Effect, Individual Treatment Effect, causal graph recovery, intervention prediction, counterfactual reasoning, or policy optimization. Models should be compared according to the same estimand and assumptions rather than using metrics that represent different causal questions.

Baseline selection is essential for meaningful evaluation. Simple statistical estimators, regression adjustment, propensity-score methods, matching, inverse probability weighting, doubly robust estimators, and conventional machine learning models can provide reference points for more sophisticated causal approaches. A complex neural causal model should demonstrate measurable benefits over simpler methods rather than being judged only against other high-capacity architectures.

Causal meta-learners provide an important model family for treatment-effect comparison. S-learners model treatment together with covariates in a single outcome function, while T-learners construct separate models for treated and control populations. X-learners and related approaches attempt to improve heterogeneous effect estimation, particularly when treatment groups are imbalanced. Their relative performance depends strongly on sample size, overlap, and treatment assignment.

Tree-based causal methods offer another useful comparison class. Causal trees and causal forests partition populations according to differences in treatment response rather than ordinary predictive error alone. They can capture nonlinear heterogeneous effects while retaining a degree of interpretability. Comparison with neural methods helps determine whether high-dimensional representation learning provides advantages beyond structured partitioning of covariate space.

Representation-learning methods attempt to construct latent spaces in which treated and control populations become more comparable while retaining information relevant to outcomes. Balancing representations can reduce bias caused by covariate imbalance, but excessive balancing may remove variables important for treatment-effect heterogeneity. Model comparison should therefore evaluate both factual prediction and causal effect estimation rather than assuming that stronger representation alignment is always beneficial.

Deep causal models can capture nonlinear relationships and high-dimensional observations that are difficult for conventional estimators. Neural architectures may jointly learn representations, outcome models, propensity mechanisms, or latent confounders. Their flexibility can improve performance in complex environments, but it also introduces sensitivity to architecture, initialization, regularization, hyperparameters, and dataset size, making repeated evaluation especially important.

Bayesian causal models provide a different tradeoff by explicitly representing uncertainty over parameters, functions, or causal structures. They can be valuable when datasets are small or intervention decisions require calibrated uncertainty. Comparison with deterministic models should therefore consider not only point-estimation accuracy but also posterior calibration, interval coverage, robustness under limited evidence, and the quality of decisions made under uncertainty.

Causal discovery models require comparison at both structural and functional levels. Constraint-based approaches use conditional independence relationships, score-based methods search for structures that optimize graph criteria, and functional causal models exploit assumptions about data-generating mechanisms. More recent differentiable and neural methods formulate graph discovery as continuous optimization. No single family dominates across all assumptions and data regimes.

Structural Hamming Distance, adjacency precision and recall, and edge-orientation accuracy provide common measures for comparing discovered graphs when ground truth is available. However, structural similarity alone does not establish equivalent causal behavior. Models should also be compared using standardized interventions to determine whether different learned structures generate correct post-intervention distributions and downstream decisions.

Treatment-effect models should be compared using multiple levels of aggregation. Average Treatment Effect error evaluates population-level accuracy, while PEHE-related metrics evaluate heterogeneous effects across individuals. Counterfactual prediction error provides another perspective when potential outcomes are known. A model that performs well on average effects may still fail badly when treatment responses differ substantially across individuals or contexts.

Policy value provides an operational comparison between causal models. Each model can be used to construct a policy that selects interventions according to estimated effects, and the resulting expected outcome can then be evaluated. This comparison is often more meaningful than small differences in treatment-effect error because it directly measures whether one model leads to better decisions than another.

Decision regret complements policy value by measuring the difference between outcomes produced by model-selected actions and those produced by optimal actions. Two models with similar estimation error may have very different regret if their errors occur near important decision boundaries. Comparing regret therefore reveals whether causal inaccuracies actually alter actions in ways that matter...