Four external signals explained almost nothing about AI recommendation frequency. One internal signal explained 61.4% of the variation in brand position across two separate months.

When discussing AI recommendations, most conversations focus on external signals.

Brand authority. Website quality. Public visibility. Third-party mentions.

The assumption is that improving these signals should make an AI system more likely to recommend a brand.

At Atom Foundry, we wanted to investigate a different angle.

Instead of looking only at what happens outside the model, we started examining the model's own recommendation behavior and how consistently it carries from one measurement period to another.

The results raised an interesting question about how stable AI recommendations really are.

Four external signals showed almost no explanatory power

We compared recommendation frequency against four external signals.

The results were:

External signal

Explained variation

Store quality - 0.7%

Public fame - 1.2%

Web traces - 0.2%

Intent match, within set - 1.2%

None of these signals explained much of the variation in recommendation frequency in this analysis.

This does not mean that external signals never matter. It means that the particular variables measured here had very limited explanatory power under this experimental design.

There are also two methodological details worth disclosing.

The search API rejected quoted phrase queries on its free tier, so web searches had to run without quotation marks. Both comparison groups had the same limitation, but absolute web-trace counts should be treated as approximate.

Intent match was measured within the existing set of stores, rather than across every possible store.

We wanted to understand what happened when we compared the model with its own previous behavior.

June predicted July surprisingly well

We compared 1,082 brand-intent pairs across June and July, using the same 50 buying intents in both periods.

The two periods came from separate collection runs and different data tables. The underlying model had also changed between the periods.

We measured how strongly June recommendation behavior predicted July behavior using Pearson correlation.

The results were:

Position to position: R² = 61.4%, r = 0.784

Frequency to frequency: R² = 54.4%, r = 0.737

Position to frequency: R² = 31.9%, r = -0.565

The strongest relationship was between a brand's position in June and its position in July.

Recommendation frequency also showed a strong relationship across the two periods, although it was somewhat weaker.

But position and frequency were not interchangeable.

The relationship between June position and July frequency was substantially weaker and negative.

This distinction matters because a brand being mentioned frequently does not necessarily mean it will be ranked highly.

Recommendation frequency and recommendation position capture different aspects of model behavior.

This was not the same measurement correlated with itself

We wanted to be particularly careful about circularity.

A measurement compared with itself can produce a misleadingly strong result.

That is not what happened here.

June and July came from different collection runs and were stored in different tables. The underlying model also changed between the periods.

The same 50 buying intents were used in both periods, but the measurements themselves were independent.

If June still predicts July under these conditions, the measured recommendation behavior appears more stable than the pipeline used to measure it.

This does not tell us exactly why the relationship exists, but it makes the stability finding more interesting.

The web signal was almost flat

We also examined the web-traces comparison in more detail.

The core group had an average of 8.1 distinct domains, compared with 7.9 in the rest of the group.

The presence of the brand's own website was 58% in the core group and 44% in the rest.

Review-site mentions averaged 2.7 in the core group.

Yet the correlation between web traces and recommendation frequency in the comparison group was only r = 0.046, with R² = 0.2%.

Again, this is not evidence that the web is irrelevant to AI recommendations.

It means that the particular web-traces variables measured here did not explain recommendation frequency in a meaningful way.

What happens to top recommendations over 15 days?

We then looked at another part of the problem.

How much do the top recommendations actually change when we repeat the same measurement at different points in time?

We compared four intervals, from the same sweep to 15 days apart.

The percentage of top picks that changed was:

Same sweep: 43.3%

1 day apart: 44.6%

14 days apart: 44.7%

15 days apart: 44.4%

The baseline is important.

Even when the same sweep is compared with itself, 43.3% of top picks differ. This gives us a reference point for the variation already present in the measurement.

Across the following 15 days, the measured change stayed close to that baseline.

The difference between the baseline and the 15-day comparison was only 1.1 percentage points.

This does not mean that individual recommendations stayed identical. A substantial share of top picks differed even in the baseline comparison.

It means that the additional time between measurements did not produce a meaningful increase in observed changes under this experimental design.

What this means for AI Commerce Intelligence

When we put these findings together, a different picture starts to emerge.

External signals are important areas to investigate, but measuring them alone does not tell us how an AI system actually behaves.

Recommendation position and frequency can carry a strong relationship across separate measurement periods, even when the underlying model changes.

And the amount of change in top recommendations over 15 days can remain close to the variation already present in the measurement itself.

For us, this changes what needs to be measured.

A single scan tells us what an AI system recommended at one particular moment.

Repeated measurements help us understand whether a brand's position is stable, whether its recommendation frequency is changing, and whether an observed change is larger than the noise in the measurement.

This is why we believe AI Commerce Intelligence needs to move beyond simple visibility checks.

We need to measure recommendation position, frequency, consistency and change over time.

We also need to understand what actually makes a model change its mind.

At Atom Foundry, we are building our research and tools around this idea.

We do not want to tell brands what should influence AI recommendations based on assumptions alone.

We want to measure what happens, challenge our own interpretation, try to disprove our findings and build products around results that survive that process.

Data first. Then analysis. Then challenge the result. Then build.

The most interesting question is no longer just whether an AI system recommends a brand.

It is whether we can understand the behavior behind that recommendation, measure how persistent it is and eventually identify what causes it to change.

That is what we are investigating.

Read the full research study

https://atomfoundry.dev/research/the-model-predicts-itself

Atom Foundry

AI Commerce Intelligence research into how AI systems discover, evaluate and recommend e-commerce brands.