Transformer Is Inherently a Causal Learner

What if the foundation models we use every day are secretly learning the causal structure of the world? We show that transformers trained for prediction are inherently causal learners — their gradient sensitivities reveal the true cause-and-effect relationships in data, without any explicit causal training objectives.

The big picture

Causal discovery — figuring out what causes what — is fundamental to science. Traditional methods require specialized algorithms with strong assumptions. Meanwhile, transformers have become the backbone of modern AI, excelling at prediction tasks across domains.

Our key insight: these two worlds are deeply connected. When a transformer learns to predict the future from the past, it must implicitly learn which past variables actually matter for each prediction. This is exactly what causal discovery aims to find.

From prediction to causation

Consider a $p$-variate time series $X_t = (X_{1,t}, \dots, X_{p,t})$ and a lag window $L \ge 1$. Each variable follows

$$X_{i,t} = f_i\!\left(\mathrm{Pa}(i,t),\, U_t,\, N_{i,t}\right)$$

where $\mathrm{Pa}(i,t)$ are the lagged parents, $U_t$ are unobserved processes and $N_{i,t}$ are mutually independent noises.

• A1 — Conditional exogeneity. Latent confounders are allowed as long as they don’t create spurious dependencies.
• A2 — No instantaneous effects. All parents occur at lags $\ell \ge 1$.
• A3 — Lag-window coverage. The chosen $L$ includes all true parents.
• A4 — Faithfulness. The distribution is faithful to the causal graph.

Theorem · causal identifiability via prediction. Under A1–A4 and regularity conditions, the lagged causal graph $G^{*}$ is uniquely identifiable via the score gradient energy: an edge $j \to i$ at lag $\ell$ exists if and only if

$$H^{\ell}_{j,i} := \mathbb{E}\!\left[\left(\partial_{x_{j,t-\ell}} \log p(X_{i,t} \mid X_{<t})\right)^{2}\right] > 0.$$

Unlike classical Granger causality, which tests mean prediction, the score gradient energy captures influence on the entire conditional distribution. Gradient sensitivity = causal relevance.

Why transformers are natural causal learners

Alignment with identifiability. Causal masking enforces temporal precedence (A2); the window L bounds maximum lag (A3); autoregressive training naturally fits conditional distributions.

Scalable sparsity. Finite capacity and weight decay compress observations into generalizable parameters; softmax attention induces competitive selection among candidates; multi-head context supports selecting complementary parents.

Contextual parameters. Attention matrices are input-conditioned, adapting to heterogeneity and non-stationarity — different contexts induce distinct dependency patterns, enabling a mixture-of-graphs view.

Gradient-based extraction. Layer-wise Relevance Propagation computes how much each past variable contributes to each prediction; we aggregate and threshold to recover the causal graph.

Results

We evaluate decoder-only transformers against PCMCI, DYNOTEARS, VAR-LiNGAM, NTS-NOTEARS, TCDF and Granger causality tests. The transformer recovers lagged parents accurately and consistently across settings, reaching 0.62 average F1 — nearly doubling the best baseline (DYNOTEARS at 0.37) — and stays robust as dynamics and dimension grow, without sensitive hyperparameter tuning.

F1 across regimes. (A) Mean F1 over all experiments. (B) High-dimensional: F1 against number of nodes. (C) Long-range dependencies: F1 against maximum lag. (D) Nonlinearity: F1 against functional form. (E) Non-stationarity: F1 against number of domains. DOT = decoder-only transformer.

Nonlinearity. Classical methods are efficient when relationships are simple and linear, and that is where they remain competitive. The interesting behaviour is what happens as data accumulates: the transformer keeps improving while the baselines flatten out.

Nonlinear dependencies. Traditional methods hold up in simple linear cases, but the transformer improves consistently as the sample size grows.

Non-stationarity. This is the sharper contrast. Several classical methods become computationally intractable as data grows, so more data actively hurts them. The transformer shows the opposite behaviour — accuracy improves with sample size, which is the property that makes scale a strategy rather than a cost.

Non-stationary dependencies. Where other methods become intractable with more data, the transformer exhibits scaling: accuracy rises with sample size.

Robustness, and its limit. Performance is stable across noise types and variances. It degrades in one specific way: as the number of latent confounders grows, spurious links start to appear — the failure mode the assumptions predicted.

Noise and latent variables. Stable across noise types and variances, but spurious edges appear as latent confounders increase.

When the assumptions are violated. The assumptions above are not always satisfiable, so it matters what can be done when they fail. L-PCMCI post-processing recovers latent confounders, PCMCI+ refines instantaneous effects, and supplying domain indices improves data efficiency — the failures are addressable rather than fatal.

Handling violated assumptions. L-PCMCI post-processing recovers latent confounders; PCMCI+ refines instantaneous effects; domain indices improve data efficiency.

Why attention is not enough. A natural guess is to read the causal graph straight off the attention matrices. That works in shallow models and breaks down as depth increases, since attention distributes across layers. Gradient attribution consistently wins — which is why the method is built on relevance propagation rather than on attention weights.

Attention versus gradient attribution. Raw attention works in shallow models but fails with depth; LRP-based attribution wins consistently.

Knowing which edges to trust. True edges show high mean relevance rankings and low variance across runs, which gives a pragmatic way to surface the reliable part of a recovered graph rather than presenting every edge with equal confidence.

Uncertainty. True edges combine high mean relevance ranking with low variance — a usable signal for which edges to trust.

Additional analysis

To validate that gradient attributions truly capture causal relationships, we compare them with intervention-based effects — the gold standard in causal inference — by intervening on inputs by one standard deviation and measuring the average effect on outputs. The strong correlation confirms that the transformer’s internal representation genuinely captures causal structure, not just statistical association.

Intervention effects vs. gradient attributions. Intervening on an input by one standard deviation produces an average output change that tracks the relevance score almost linearly — attribution is measuring causal influence, not correlation.

We also ask whether existing time-series foundation models can discover structure zero-shot. Forecasting accuracy is reasonable but structure recovery is suboptimal; finetuning on domain-specific data significantly improves both, and adding node embeddings — even randomly initialized — helps the model distinguish variables.

Zero-shot structure recovery. Off-the-shelf time-series foundation models forecast reasonably but recover structure poorly — until light domain finetuning.

Implications

For causal discovery. A new paradigm: instead of hand-crafting discovery algorithms, leverage the representation learning of foundation models. The transformer becomes a universal causal structure extractor that scales with data.

For foundation models. Gradient attributions reveal learned dependencies (interpretability); hallucinations may arise when insufficient data prevents accurate structure learning; causal priors such as sparsity and modularity could guide better architectures.