September 16, 2025

Happy Tuesday, today’s smart brevity count is: 871 words, 6.5 mins. Edited by Nils Koppelmann.

Today's theme: Building and Sustaining Experimentation Systems

In today's newsletter, we will go over:

1. Strategic and tactical guide on how to build and sustain experimentation systems
2. Your new job in experimentation?
3. Upcoming events in 2025

Excited? Read on.

1 big thing: Building and Sustaining Experimentation Systems: A Strategic and Tactical Guide

Strategic Insights for Experimentation Leaders

Strategic Insight: Align Experimentation with Business Goals and Customer Problems

To build a robust experimentation system, start by aligning your experiments with your business goals and customer problems. A goal tree map is a powerful tool to break down high-level objectives (e.g., increasing revenue) into actionable metrics. This map connects business unit metrics, tactical metrics, and behavior metrics, creating a clear path for experimentation to influence outcomes.

But goals alone aren’t enough. Customer problems are the driving force behind meaningful experiments. Use first-party data—reviews, support tickets, and even Reddit discussions—to identify what your customers are struggling with. Prioritize these problems based on their potential impact and connect them to your goal tree metrics. This creates strategic opportunities: hypotheses that, when tested, solve customer problems while driving business outcomes.

For example, if customers complain about not being able to find the right product, an experiment could test new navigation or search features. By solving this problem, you not only improve the customer experience but also potentially boost conversion rates.

Tactical Tip: Build a Feedback Loop for Continuous Learning

A strong experimentation system thrives on a feedback loop. Start with customer insights, transform them into ideas, and design experiments with clear hypotheses. From setup to analysis, every step should feed back into the system, creating a cycle of continuous learning.

Here’s how to operationalize this:

1. Capture Insights: Use tools like surveys or session recordings to gather customer feedback.
2. Prioritize Problems: Rank customer issues based on their impact and alignment with business goals.
3. Design Experiments: Test solutions ensuring your experiments are statistically sound and visually consistent.
4. Analyze and Learn: After running the experiment, document the results and insights in a knowledge repository. This helps you avoid repeating mistakes and builds institutional knowledge.

Strategic Insight: Embrace a Culture of Experimentation

Experimentation isn’t just a process—it’s a mindset. Smaller brands can compete with larger ones by excelling at experimentation. The key is to reduce the cost per experiment so you can run more tests without breaking the bank.

But cost isn’t the only factor. A culture of experimentation is critical. This means embracing failure as a learning opportunity. As Julia Cameron writes in The Artist’s Way, “It is impossible to get better and look good at the same time. Give your self permission to be a beginner.” or in other words: give your team permission to experiment, fail, and learn.

Tactical Tip: Measure Success Beyond Revenue

While revenue for many companies is the ultimate goal, it’s not the only metric to track. In the early stages, focus on adoption: Are more people running experiments? Are you testing a wider variety of ideas? Over time, shift to quality metrics: Are your experiments statistically valid? Are they solving real customer problems?

Finally, track impact metrics like changes in customer support requests, improvements in user feedback, or increases in key conversion points along the customer journey.

Strategic Insight: Data Quality is Non-Negotiable

Your experimentation system is only as good as the data it’s built on. Flawed or inconsistent data leads to flawed decisions. Think of it like driving a car: if every intersection gives you conflicting directions, you’ll never reach your destination.

Invest in data quality from the start. Ensure your tracking is reliable, your metrics are consistent, and your data points are actionable. Over time, expand the data you collect to open up new experimentation opportunities.

Tactical Tip: Choose the Right Tools for Your Needs

The tools you use depend on the experiments you run. At a minimum, you’ll need:

• Tracking Tools: To capture user behavior.
• Experimentation Platforms: To manage and run tests.
• Knowledge Repositories: To document experiments and learnings.

For example, if you’re (just) testing landing pages, a redirect-based tool might suffice. For more complex experiments, opt for a full-feature platform.

Final Thought: Start Small, Iterate Often

Building an experimentation system isn’t about perfection—it’s about progress. Start small, focus on learning, and iterate often. Over time, you’ll create a system that drives business growth while solving real customer problems.

What’s one experiment you’re running this week? Let me know!

Also: I'm about to launch ConversionLab – a knowledge repository for experimentation teams. Go check it out.​

2. Your new job in experimentation?

This week’s featured roles:

• ​Experimentation Delivery Manager at Creative CX (London, United Kingdom)
• ​Marketing CRO Developer at Mindbody (Kuala Lumpur, Malaysia)
• ​Director of Web Strategy & CRO at Eventbrite (USA)
• ​Senior Software Engineer, Message Optimization & Experimentation at Attentive (San Francisco, USA)
• ​CRO Specialist at PM Digital Design (United Kingdom)

Couldn't find a job that's interesting to you? Check out more here.

3. World class events

Upcoming conferences in in 2025:

• 13th-17th October 2025, Experiment Nation Connects, Online
• 20th-22nd October 2025, EXL 2025 in Austin TX, USA
• 21st-23rd November 2025, Conversion Hotel in Texel, Netherlands
• 9th December 2025, Experimentation Elite in London, UK

Find more events over at lu.ma/experimentation-events.

That's a wrap – until next week.

– Nils

P.S.:

If you enjoyed this, please consider sharing with a friend. If a friend sent you this, get the next newsletter by signing up here.

Also, for more resources and content follow me on LinkedIn.