August 05, 2025

Happy Tuesday, today’s smart brevity count is: 589 words, 4 ½ mins. Edited by Nils Koppelmann.

Today's theme: The Experimentation Maturity Gap

In today's newsletter, we will go over:

1. Strategic: Why Team Maturity ≠ Organizational Maturity​
2. Tactical: From IC to Experimentation Org
3. Your new job in experimentation?
4. Upcoming events in 2025 & 2026

Excited? Read on.

1 big thing: Why Team Maturity ≠ Organizational Maturity

Strategic Insights for Experimentation Leaders

You can have sophisticated testing teams and an organizationally immature experimentation program. Companies like Netflix run 10,000+ experiments annually. Your organization probably runs fewer than 5 per month despite having skilled data scientists and expensive tools.

Why this disconnect exists: Team-level maturity and organizational maturity develop independently. Most companies optimize for one without the other.

The Two Misalignment Patterns

Pattern 1: Advanced Teams, Immature Organization
​Marketing runs rigorous A/B tests. Product has solid statistical practices. Growth experiments systematically. But there's no shared learning, experiments conflict, and insights are siloed.

Pattern 2: Mature Vision, Immature Execution​
Leadership mandates "experiment everything" and allocates sizeable budgets. But teams lack statistical literacy operational experience, run underpowered tests, and draw invalid conclusions from results.

Why this kills programs

The plateau effect: Organizations get stuck at 5-10 experiments per month. Data science becomes a permanent bottleneck instead of a capability multiplier.

The real cost: You're not just running fewer experiments, but you're learning slower than competitors who've solved this alignment problem.

The strategic insight

Bottom line: The experimentation maturity gap is a strategic capability problem that determines who learns faster.

Further reading: The Experimentation Gap by Davis Treybig

2. From Individual Contributor to Experimentation Organization

Your roadmap for building aligned experimentation maturity

Where Are You Now?

Individual Contributor: You run experiments but work mostly in isolation. Results stay within your team.

Cross-Functional Influence: Other teams seek your experimentation advice. Some informal knowledge sharing happens.

Organizational Integration: Leadership regularly discusses experiment results in strategy meetings. Experimentation capability survives team changes.

How to Progress to the Next Level

From Individual to Cross-Functional Influence

3 key moves:

• Make your work visible - Share experiment results in broader team meetings, not just your immediate team
• Create templates others can use - Document your experiment process so non-experts can follow it
• Teach one person from another team - Help them design and run their first proper experiment

You'll know you've made it when someone outside your team asks for your experimentation advice.

From Cross-Functional to Organizational Integration

3 key moves:

• Connect experiments to business strategy - Present quarterly summaries of experiment learnings to leadership
• Build systematic knowledge sharing - Create regular cross-team experiment sharebacks or documentation space
• Map experiments to business priorities - Show how current testing aligns with company goals

You'll know you've made it when: Leadership starts citing experiment results in strategic discussions

From Organizational Integration to "Full Maturity"

3 key moves:

• Embed experimentation in standard processes - Make it part of how major decisions get made by default
• Build systems that outlast individuals - Create documentation and processes that work when people leave
• Scale capability, not dependency - Train others to run their own experiments rather than coming to you

You'll know you've made it when: Experimentation happens automatically without you pushing it

The one thing that accelerates everything

Document your experiments beyond just results. Capture hypotheses, why things failed, unexpected learnings, and what worked across different contexts.

This transforms individual experiments into institutional knowledge.

Take action now:

1. Assess what level you're at currently.
2. Pick one of the three moves for the next level.
3. Act on it this week.

3. Your new job in experimentation?

This week’s featured roles:

• ​Product Manager, Subscriptions at Dow Jones (New York, USA)
• ​Engineering Manager – Ads Experimentation Platform at Reddit (USA)
• ​Senior Optimization Manager, CRO Strategy at Launch Potato (Jersey City, USA)
• ​Conversion Rate Optimisation (CRO) Specialist at Smile White (Leeds, United Kingdom)
• ​CRO and Web Developer at Virtuagym (Spain)

Couldn't find a job that's interesting to you? Check out more here.

5. World class events

Upcoming conferences in in 2025:

• 13th-17th October 2025, Experiment Nation Connects, Online
• 20th-22nd October 2025, EXL 2025 in Austin TX, USA
• 21st-23rd November 2025, Conversion Hotel in Texel, Netherlands

Find more events over at lu.ma/experimentation-events.

That's a wrap – until next week.

– Nils

P.S.:

If you enjoyed this, please consider sharing with a friend. If a friend sent you this, get the next newsletter by signing up here.

Also, for more resources and content follow me on LinkedIn.