Why Your Experimentation Program Isn't Driving Growth, And What Actually Fixes It

Most marketing teams are testing something right now. Landing pages, headlines, creative variants, button colors. Some organizations are running hundreds of experiments a year. And yet a striking number of those programs never produce the growth they were sold on.

The instinct is to blame execution, not enough tests, not enough traffic, the wrong tool. But according to Shamir Duverseau, the problem usually starts earlier than that: with what teams expect experimentation to do in the first place, and with how little they understand about the psychology of the person on the other end of the test.

The instinct is to blame execution — not enough tests, not enough traffic, the wrong tool. But according to Shamir Duverseau, the problem usually starts earlier than that: with what teams expect experimentation to do in the first place, and with how little they understand about the psychology of the person on the other end of the test.

In this episode of Performance Delivered, host Steffen Horst is joined by Shamir Duverseau, co-founder and Managing Director at Smart Panda Labs, a technical marketing agency focused on enterprise B2C brands. Over the past 25 years, Shamir has worked across travel, entertainment, and technology, partnering with brands including Southwest Airlines, The Walt Disney Company, and NBCUniversal, and his background spans product management, digital strategy, UX design, web development, testing, and analytics  a combination that gives him an unusually holistic view of the customer journey.

Today his work centers on helping high-consideration brands rethink the post-click experience, where purchase decisions are actually made. Together they dig into the gap between testing activity and business impact, why uncertainty rather than price or competition is the marketer's real enemy, and what a healthy experimentation culture looks like when marketing, product, and IT are actually aligned.

Experimentation is a decision-making framework, not a conversion-rate machine

The most common reason programs disappoint is that leadership walked in expecting the wrong graph. "Essentially they expect to see some graph where they'll see the conversion rate of the website kind of slowly climb up over time," Shamir says. "And frankly, that's not something you're ever going to see. That's not a realistic expectation."

What you should expect instead is compounding organizational judgment. Over time, a company that experiments consistently makes better UX decisions, better product decisions, and better strategic decisions — because those decisions are grounded in feedback from actual customers rather than internal opinion. The payoff shows up across the business, not as a single line on a single dashboard.

Uncertainty is the marketer's greatest enemy

Marketers like to imagine a clean path: see the ad, click the ad, scan the page, add to cart, enter a card, done. "In a perfect world that would happen," Shamir says. "But we live in far from a perfect world. So that actually pretty much never happens. Maybe on Amazon."

In high-consideration purchases, buyers move through a hierarchy of unspoken questions instead. Is this relevant to me? Do I trust what I'm seeing? Do I have any motivation to move forward? Am I properly oriented to what I'm looking for? Each one is a barrier, and each barrier is a form of uncertainty. "Marketers often underestimate the power of uncertainty to freeze a consumer in their place," he says — "more than any competitor, more than any price point." The job of the post-click experience is to remove uncertainty one step at a time so the next step feels safe.

Optimize for decisions, not just clicks

Clicks, bounce rate, and time on page all have a place — but only in context. "I can create an ad to get someone to click," Shamir points out. "Tell me that you know you're giving away a thousand dollars, I'll get a lot of clicks. But those clicks aren't going to convert when they find out you're not actually giving away a thousand dollars."

Once you accept that the goal is a decision, the quality of the click starts to matter as much as the click itself. The same logic applies to softer metrics: high time on page can mean the content is engaging, or it can mean the content is confusing and someone is hunting for an answer they can't find. Either reading is plausible from the number alone — which is exactly why the number alone isn't enough.

Five witnesses to the same accident

Shamir's favorite analogy for analysis: a police officer arrives at a traffic accident and interviews five witnesses. He'll get five different accounts. Only by overlapping them does an accurate picture emerge. "But if he just focuses on the account of one person, he's probably going to go down the wrong path."

Web analytics is one witness. Session recordings and heat maps are another. Survey responses, on-page feedback, chat logs, and customer support calls are more. Put them together and you can run a genuine thematic analysis: themes lead to insights, insights lead to recommendations, and recommendations lead to actions — a test to run, a change to make, or a gap in what you're measuring. That loop, not any single test, is what produces learning big enough to move the business.

Qualitative feedback is where the real hypotheses come from

Ideas that arrive because someone woke up on the right side of the bed are fine. They're just not a program. A real experimentation practice is fed by a standing framework for collecting both quantitative and qualitative feedback — and much of the qualitative material already exists inside the organization, unread. Chat logs and support calls are happening anyway. A single exit question — "I saw you left today, do you mind saying why?" — costs almost nothing.

The patterns that surface there are often embarrassingly fixable. A brand discovers it never explains its return policy, or that its cancellation window is buried three clicks deep, and that the missing information was quietly costing trust at exactly the moment buyers needed reassurance. As Shamir puts it, marketers are "way too close to our products and services to be able to anticipate how people feel."

AI is an accelerator for experts, not a substitute for expertise

AI meaningfully speeds up thematic analysis of metrics and open-ended feedback. But Shamir draws a sharp line: "AI is really good at making someone who knows what they're doing better at it. It's not very good at making someone who doesn't know what they're doing and giving them a new skill."

An analyst reading an AI summary can spot what's missing, what's misweighted, and what simply looks wrong. Someone without that background can't tell the difference between output that's correct and output that's merely plausible. He's also skeptical of the loudest claims about displacement, noting that much of that narrative originates with AI companies themselves, while enterprise adoption moves slowly and methodically.

Alignment takes a North Star, and treating internal partners like customers

Experimentation touches marketing, product, engineering, and analytics, and every one of those groups is measured on something different. "As humans, we kind of do what's in our own best self-interest," Shamir notes. Which is why an executive sponsor matters: someone who can say that these separate OKRs all roll up to one company-level goal, and name where experimentation fits.

The second half is on the marketer leading the charge. You already invest in understanding your external customer's goals and frustrations; do the same for your IT, product, and data engineering partners. "They're not just roles, they're people." Understanding what they're accountable for makes collaboration faster and makes it obvious to them why the program is worth their time.

Getting started is the real bottleneck, and a losing test can be the best sell

There's endless debate about centralizing versus democratizing experimentation, centers of excellence, and how to scale. Shamir's experience is that most organizations never get far enough to have that problem. Plenty have the tools to gather data and run tests but no process around them; a program starts, runs one or two experiments, and quietly peters out.

His preferred way in is a quick win — including a loss. He describes running an experiment on a change someone had been lobbying for internally, which turned out to lose money. "So now not only do you know not to make that change, but the time and cost wasn't spent putting that change in the roadmap, expending resources on it. Right there, we just saved you money." Reframing experimentation as something that saves dollars and reallocates roadmap resources — not only as something that generates incremental revenue — is often what gets a stalled program off the ground.

Where to find Shamir

Shamir posts regularly about experimentation, the post-click experience, and cross-team collaboration on LinkedIn, and you can learn more about his agency at smartpandalabs.com.

Episode Transcript

‍Steffen Horst (00:12) Welcome back to Performance Delivered, Insider Secrets for Marketing Success, the podcast where we explore what's actually driving growth and performance in today's complex marketing landscape. I'm your host, Steffen Horst, and today we're diving into a topic that sits at the intersection of data, psychology, and real business impact. Why experimentation programs often fail to drive meaningful growth and what it actually takes to fix them. Most teams today are running tests.

Steffen Horst (00:38) They are optimizing landing pages, tweaking headlines, testing creatives. But despite all that activity, many experimentation programs struggle to produce real, sustained growth. So what's missing? The answer often isn't more testing, it's a deeper understanding of how people actually make decisions. Joining me today is Shamir Duverseau. Shamir is the co-founder and managing director at Smart Panda Labs, a technical marketing agency focused on enterprise B2C brands.

Steffen Horst (01:04) Over the past 25 years, he's worked across industries including travel, entertainment, and technology, partnering with brands like Southwest Airlines, the Walt Disney Company, and NBCUniversal. His background spans product management, digital strategy, UX design, web development, testing, and analytics, giving him a uniquely holistic view of the customer journey. Today, he focuses on helping high-consideration brands rethink the post-click experience.

Steffen Horst (01:30) Where decisions are actually made. Shamir, welcome to Performance Delivered. It's great to have you here again.

Shamir Duverseau (01:37) Thank you so much for having me on. I'm excited about this one, Steffen.

Steffen Horst (01:39) Now, Shamir, let's start by grounding the conversation. A lot of organizations today are running dozens and sometimes hundreds of experiments, but despite that effort, many still struggle to see meaningful growth. From your perspective, why do so many experimentation programs fail to produce real business impact, even when teams are actively testing?

Shamir Duverseau (01:59) Yeah, I really think a lot of that has to do with what perspective they go into experimentation with. So what are they hoping to get out of it? A lot of times it can be very, very basic, very surface expectations. So essentially they expect to see some graph where they'll see the conversion rate of the website kind of slowly climb up over time.

Shamir Duverseau (02:18) And frankly, that's not something you're ever going to see. That's not a realistic expectation. Really, experimentation is much more about learning than it is about anything else. It's a decision-making framework, and it's enabling organizations to make better decisions over time, which will holistically have a better impact on the business. So you're not gonna see that one.

Shamir Duverseau (02:39) Graph, that one line consistently kind of going up and to the right the way that you'd like to. But what you will see over time is that the organization culturally is just making better decisions, better user experience decisions, better product decisions, better strategic decisions because they're basing it on experimentation and using that learning that they're getting, that feedback that they're getting from customers, whether they're internal customers or external customers, to make those decisions.

Steffen Horst (03:03) Okay. So one of the most interesting aspects of your work is how you connect experimentation with human behavior. When we think about high consideration purchases, whether it's travel, financial products, or major services, the decision process is really not linear. What's actually happening psychologically during that customer journey and how should marketers think about it differently?

Shamir Duverseau (03:26) Yeah, I think we like to think as marketers that someone goes to Instagram, they see an ad, they click on that ad, they go to a landing page or a PDP, they do a little reading, a little scanning, add it to their cart, check out, enter their credit card information, boom, they make the purchase. And in a perfect world that would happen, but we live in far from a perfect world. So that actually pretty much never happens. Maybe on Amazon. That's about it.

Shamir Duverseau (03:52) But that's also because it's a pretty low consideration purchase that you're just kind of going through the process. But what actually happens in most purchases, and in particular, as you mentioned, in higher consideration purchases, is a number of kind of psychological barriers that you're having to go through. There's this decision hierarchy that you're kind of working through without even realizing it, but we all do it. Just determining is this relevant to me? Do I trust what I'm seeing? Do I have any motivation to move forward?

Shamir Duverseau (04:19) Am I properly oriented to what I'm looking for? Right. So all these kinds of things are happening that are helping you work through the uncertainty that you often face in terms of making a decision. And marketers often underestimate the power of uncertainty to freeze a consumer in their place. And more than anything else, more than any competitor, more than any price point.

Shamir Duverseau (04:41) Uncertainty is the marketer's greatest enemy. So you're trying to create certainty. You're trying to make this person comfortable in taking the next step and the next step and the next step in the decision and ultimately make that purchase. And there's a lot of things that end up going into that. But first you kind of have to recognize what your true enemy is in order to be able to kind of create that weapon to be able to combat it.

Steffen Horst (05:02) Now a lot of optimization efforts still focus heavily on surface level metrics. Clicks, bounce rate, time on page, but those aren't always translating into decisions. How should marketing leaders think about the difference between optimizing for clicks versus optimizing for decisions?

Shamir Duverseau (05:19) Yeah, it really depends on where is that person kind of in their journey and what is it you're trying to get them to do? What's the next best action, right? So there may be times where you're trying to optimize for clicks because, for example, maybe you're very early on, you're top of funnel, you're trying to get a person's attention. And ultimately if that person doesn't click on that Instagram ad, if they don't click on that paid search result.

Shamir Duverseau (05:41) Then there's no movement forward. So you've got to get them to click. But you can't look at that click in a vacuum because I can create an ad to get someone to click. You know, tell me that you're giving away a thousand dollars. I'll get a lot of clicks, right? But those clicks aren't gonna convert when they find out you're not actually giving away a thousand dollars, right? So each of those kind of micro metrics, those vanity metrics, they have their place in kind of understanding what the person is doing, but

Shamir Duverseau (06:06) without looking at that holistically, without kind of backing away and having the full context of ultimately what is the entire journey, what is it you want the person to do? It's easy to get lost in those vanity kind of micro metrics and you're not focused on really, okay, what are the milestones I need to get them to? How am I getting this person to ultimately add this item to their cart, to actually check out, to actually go to the conversion process or to begin filling out the form or

Shamir Duverseau (06:30) whatever it happens to be that you're trying to — whatever action you're trying to drive online. So it's a matter of really looking at things more contextually, but not ever losing sight of what the ultimate prize is, which is to get that person to hit submit on that form, to hit checkout, to hit purchase, to enter their information. And everything ultimately is a means to those ends, right? To be able to move people through those milestones in the process. So when you recognize that, now all of a sudden the click matters, but

Shamir Duverseau (06:57) the quality of the click all of a sudden matters a great deal as well. So that changes your perspective on how you're trying to optimize for the click because now you want not just the action, but you want a quality action to take place where the person feels comfortable in saying, okay, I made this click. Yeah. Okay. This is what I thought I would see. This is relevant to me. And the person feels comfortable. And that barrier, that level of uncertainty has been released. Now what's the next step? Well, can I even trust where I am? Is this a trustworthy website or landing page? Okay, great.

Shamir Duverseau (07:25) You've removed another barrier of uncertainty. Now what's the next step? Right. And how do you continue that engagement in a positive sense? So, as you mentioned, some of those metrics can be very deceiving. Something like time on site — that can tell you that the content's very engaging and people are kind of digging into the landing page or the PDP. It could also tell you the content's very confusing and they're spending a lot of time searching around, trying to find the right thing, and they just can't find it. So you've got to recognize that in the proper context and say, all right, how am I getting them past the barrier of uncertainty?

Steffen Horst (07:35) Mm-hmm.

Shamir Duverseau (07:52) Surfacing the information they need in order to move to the next step in the process.

Steffen Horst (07:56) So is it about connecting different metrics to get a clearer picture and then also using some of these metrics as earlier signals? Because if the end signal is I want more sales, right? And I just focus on okay, how do I get more sales? I might miss something on the way to getting more sales because, as you just talked about it, someone might get lost on the site because the path to sale is not clear enough, right? So if I just focus on do I get more sales?

Steffen Horst (08:23) I might miss something on the way to that.

Shamir Duverseau (08:25) Absolutely. So you know you think about like maybe a traffic accident took place and a police officer shows up at the site of the traffic accident and he says, "What happened?" And he's gonna ask five different people what happened and they're gonna give him five different accounts.

Shamir Duverseau (08:39) Now there will be things that will overlap in those accounts. And when he gets all five of those accounts, then he'll probably be able to create a somewhat accurate picture of actually what happened. But if he just focuses on the account of one person, he's probably gonna go down the wrong path in terms of what actually took place in terms of the truth of the accident, right? So it's the same thing as we look at people on the web, we tend to focus on one thing. We tend to say, well, let's look at the web analytics.

Shamir Duverseau (09:04) And how many people went to this page, and then how many people went to this page, how many people clicked on this? Okay, that's what happened. Well, that's only telling part of the story, right? So when we look at the web analytics, we look at the experience analytics, we look at session recordings or heat maps, we look perhaps at feedback and survey data. Well, now all of a sudden we're starting to speak to different witnesses, right? We're looking at that data point from different perspectives, and then we're able to analyze that.

Shamir Duverseau (09:28) Put that information together, and now we kind of have a sense of okay, now we have a picture of what's truly happening contextually. When we have that picture, now we're able to understand, all right, now I'm able to see more clearly this was the particular challenge. What caused the accident? It was, you know, the cat that ran across the street. That triggered this person on the bike to swerve and they tripped and they — right. So now we're able to put together kind of the story of actually what's taking place.

Steffen Horst (09:35) Mm-hmm.

Steffen Horst (09:46) Mm-hmm.

Shamir Duverseau (09:54) And now we're able to say, okay, this is what I need to address, right? Or this is what I need to not so much give so much information or so much attention to, rather. So that's the importance of what analysts need to look at. It's really a story of having to look at everything, multiple data sources, doing this thematic analysis of what's taking place to create the story. From that thematic analysis will come meaningful insights. From those insights will come recommendations, from those recommendations will come actions of

Shamir Duverseau (10:20) this is what we should test or this is what we should change or perhaps here's a gap in what we're measuring. And it's that process that's ultimately gonna lead to the kind of learnings that are gonna have that larger impact in the business.

Steffen Horst (10:30) Is software and AI these days helping to aid finding these stories because it can go through data sets so much quicker and can kind of build connections much faster than a human?

Shamir Duverseau (10:44) Absolutely. Yeah, I mean AI is really speeding up the process significantly and being able to do that. The real warning with AI, as really with anything, is AI is really good at making someone who knows what they're doing better at it. It's not very good at making someone who doesn't know what they're doing and giving them a new skill, right? So it's one thing for someone who is an analyst, for someone who is a UX strategist.

Shamir Duverseau (11:08) To kind of take information and insights from AI and say, okay, AI, analyze all these metrics or analyze all this customer feedback and kind of tell me what the themes are. That's great because when they get that information back, that strategist or that analyst is able to look at that and they're able to put that in the right proper context. They're able to say, that doesn't seem quite right, or they're able to

Shamir Duverseau (11:30) add this other piece of information that perhaps the AI didn't take into account, maybe you lost that, you know, you lost track of it, whatever the case is. And that context makes a good decision. But if kind of Joe Schmoe comes in and Joe Schmoe is, you know, I don't know, in finance or something, and they say, well, okay, well, just look at this data and tell me what's wrong with this page, not having that skill set behind them, not having that context.

Shamir Duverseau (11:51) It's not as easy for them to discern what are the gaps in the AI. It just seems like it's correct. It seems like it's right enough. And we're just going to move forward and kind of go with that. So, as with all technology, AI does a great job at being an accelerator to what you're doing. It does not do a great job at replacing what you're doing, contrary to what the powers that be say. And it's funny, I heard an interview the other day making a great point about AI, and it really made the point that.

Shamir Duverseau (12:14) When you hear people say AI is gonna replace jobs, right? It's gonna make it so you don't have to hire as many people, all those kinds of things — the people who are kind of putting those stories out are the AI companies who are trying to drive up their valuations, right? That's where that's coming from. That's not coming from real users. Really, when it comes down to it, when it comes to large enterprise organizations, AI adoption is pretty slow because businesses move slow and they move methodically and because the AI capabilities are starting to level out a bit, at least at the moment.

Steffen Horst (12:24) Sure. Of course. Yeah.

Shamir Duverseau (12:42) Right. So we're not quite seeing all the hype that we've been hearing, which is why again it's important for AI to be used as a tool by the expert and not as a replacement for those experts.

Steffen Horst (12:50) Yeah.

Steffen Horst (12:53) Yeah. It's interesting and we could probably now pivot this conversation into a completely different direction. But I'm of the exact same opinion because I can put myself in a race car and I can drive it probably around a racetrack. The question might be how good? But I will never be able to push it to the max and get the most out of it. And it's kind of the same thing with AI. If I don't understand what AI is able to do, if I don't —

Shamir Duverseau (12:58) Yeah, we could. I kind of went on a tangent there. Sorry about that.

Steffen Horst (13:19) If I can't call BS on the output and further fine-tune things, the AI is not going to help me. You know, I might get 60, maybe 70% out of the AI, but I won't get this great output from it. So yeah. Anyway, let's go back to talking about the topic today. But so in complex buying journeys, customers often hesitate.

Shamir Duverseau (13:34) Exactly.

Shamir Duverseau (13:38) Yeah.

Steffen Horst (13:44) Even when the offer is strong, that hesitation — that's where many conversions actually get lost. What are the most common psychological barriers that prevent customers from moving forward and how can teams start to identify them?

Shamir Duverseau (13:59) Yeah, I mean there are so many, depending on the nature of it. Trust is a huge factor. Security is a huge factor, stimulation. So those are three kind of big things that block what's happening. So trust, do I feel strong enough about this brand that I can spend those dollars and feel like I'm gonna get exactly what's being promised here. Security certainly is important and of course that

Shamir Duverseau (14:23) factor becomes more important as the purchase price goes up, right? As the stakes begin to go up. And stimulation, right? So it's one thing to have the idea of spending a large amount of money to maybe buy a new car or to go out on a vacation. But it's another thing to actually say, all right, you know, charge my credit card $5,000, $10,000. All right, I'm gonna take out a loan for $50,000, right? Actually doing it.

Shamir Duverseau (14:42) requires some semblance of like, okay, I need to feel like I'm comfortable kind of going over this hump and actually having that charge and taking that responsibility on. So the idea of it and the actual action are often two very different things in our minds. And crossing over from one into the other becomes a real issue and challenge for many people. So to answer your other question, how do you figure out where those barriers are? Well

Shamir Duverseau (15:09) a lot of times it's where in the experience you're seeing that drop off and hesitation. That's certainly a pretty good indicator. I'm a big believer certainly in analytics, but I'm also a huge believer in gathering intelligence. So asking questions, looking at page feedback, looking at chats.

Shamir Duverseau (15:23) Chat logs, looking at customer support calls, right? It's when those people are interacting and the kind of questions they're asking, those kinds of things give you a lot of information. Being able to inject just a couple of questions when someone maybe exits the process and say, hey, I saw you left today. You know, do you mind saying why?

Shamir Duverseau (15:39) Doing those kinds of things and getting that real raw input and feedback from customers is invaluable because in that you'll tend to find patterns of, wow, you know, we don't talk anywhere about what our return policy is. Right. So we're not getting people's trust because, you know, they think I'm gonna spend this money and then I'm gonna be stuck. And if I don't like it or, you know, if there's an issue or challenge or my life changes, I won't be able to cancel my plans and I've lost out on my

Shamir Duverseau (16:05) you know, my deposit or I've lost out of my five thousand dollars because they didn't know that, hey, you have until seven days before to cancel or you have a 30 day money back guarantee or whatever the kind of things are, right? So those kinds of things usually come up in those more

Shamir Duverseau (16:19) qualitative measures that we put into place. And a lot of that again is just happening by nature in things like chat logs or customer support calls that are just by nature happening anyway. And picking up on those patterns will help us to say this is the information I need to surface in the process that people perhaps aren't finding as easily as we thought they should or would, or perhaps we're not even surfacing at all, or we're not surfacing at the right time in the process to be able to deal with those particular barriers when people have those things on their mind.

Steffen Horst (16:28) Yeah.

Steffen Horst (16:45) So it sounds like you're going way beyond kind of what the normal CRO service offering usually includes. It's kind of where we're developing tests and we're looking at, you know, which elements on the website we're going to change, which messaging, imagery, or not on a very basic level, et cetera. It sounds like you go beyond that part, really also looking into user interviews and

Steffen Horst (17:11) those data sources to get a better understanding of what works and what doesn't work.

Shamir Duverseau (17:15) Yeah, absolutely. I mean, anyone who's like, you know, serious about experimentation — I mean, there are certainly ideas, good ideas people have. People, you know, wake up on the right side of the bed one morning and have a good idea, and that's great. You know, awesome. Congratulations. But if you want to build a real framework and you really want to build a culture around testing and experimentation, that really needs to come from having a framework around gathering consistent feedback from users. Again, both qualitative and quantitative, both kinds of feedback. So we —

Shamir Duverseau (17:43) getting great data from analytics, from Google Analytics or from Amplitude or whatever, from a Contentsquare or a FullStory. You're getting some great insights there. But then what are you also doing on the research front to gather some surveys, to look at those logs, to look at those conversations, and use the combination of those two to begin again, feeding that loop of research leading to ideation, leading to test, leading to analysis. Then again, starting that loop again. So now we've made this change. Well.

Shamir Duverseau (18:09) The nature of a change is going to now generate different kinds of feedback and different kinds of questions. So it becomes a repetitive loop in and of itself, but you've got to set up the right framework to be able to get that flywheel kind of going and turning. And if you in fact are generating those ideas from actual research and feedback and not just kind of from someone's idea, you're going to see just in general better learnings from that. Because now you're not —

Shamir Duverseau (18:33) you're getting kind of out of your own head and the own biases that we all as marketers have when it comes to our own products and services. We think we know so well what people think of our products and services, and almost invariably we're horribly wrong. We just — we're just

Steffen Horst (18:36) Sure.

Shamir Duverseau (18:45) we're terribly bad at it. We're way too close to our products and services to be able to anticipate how people feel. So getting that flywheel in pace and getting that constant feedback is gonna be critically important to being able to set up and establish that flywheel and framework to be able to get those good testing results.

Steffen Horst (19:00) Yeah.

Steffen Horst (19:01) Yeah, makes sense. Now if experimentation is meant to uncover what works, then understanding those barriers becomes critical. But many testing programs don't incorporate that layer.

Steffen Horst (19:11) This also brings up a broader organizational change. Experimentation doesn't live in a vacuum. It touches marketing, product, engineering, and analytics. What does a healthy experimentation culture actually look like when marketing, product, and IT are aligned around improving the customer journey?

Shamir Duverseau (19:27) That's a good question. You know, collaboration, orchestrating stakeholders is incredibly important in order to get anything actually accomplished for the sake of the customer. And that comes down to kind of two big things. So, first of all, from an organizational standpoint, from a business standpoint, what are we aligning to? So what is the ultimate goal here? Because every department is gonna have their own objectives, goals, OKRs, right? All those kinds of things.

Shamir Duverseau (19:51) IT is being measured on one thing, marketing is being measured on another, the product team is being measured on another. And ultimately, as humans, we kind of do what's in our own best self-interest. So we're worried about the measures that affect us and affect our department and what am I going to be reviewed on? And the other departments are doing the same thing. So in order to really have an effective program, you need some executive sponsor to kind of come in and say, yes, you have these different OKRs, you have these different goals, but they all roll up to what we're trying to do as a company here.

Shamir Duverseau (20:18) And here's how experimentation fits into that and feeds into that. So now everyone can say, okay, so this is a North Star that we're all looking at. We're all looking in the same direction. This is what we're trying to do, right? So from a business standpoint, that's super important. But then you've got to realize that, especially as a marketer, if you're the one kind of leading this charge, you've also got an internal customer that you need to serve. And just like you need to have insight about your

Steffen Horst (20:40) Mm-hmm.

Shamir Duverseau (20:43) external customer and really kind of understand them and their behavior and what they're looking for and what they want. Well, it would behoove you as a marketer to also understand that from your IT partner and from your product partner and from your data engineering partner. Because they're not just roles, they're people and they have personality. So understanding that is going to help you deal with them and coordinate and collaborate with them better, kind of on their level in terms of what's important to them, so that all of a sudden the time that you're spending working through this becomes more efficient. So now

Steffen Horst (20:53) Yeah.

Shamir Duverseau (21:10) you've got the same North Star, you're treating them like a customer. So you're trying to look at things from their perspective and kind of meet their needs, understand their goals and their challenges. And now you're kind of working it from both ends. And by doing that, you're slowly bringing things into alignment. And now everyone's kind of moving in the same direction because they see how it's affecting them personally. They see how it's affecting their job. They can see why they should therefore be invested in moving an experimentation program further.

Shamir Duverseau (21:36) And that creates that culture of collaboration that you need in order to really have that learning begin to flow throughout the organization.

Steffen Horst (21:42) Yeah. Now before we come to the end of today's podcast episode, what are the biggest mistakes organizations make when trying to scale experimentation and how can they avoid falling into those traps?

Shamir Duverseau (21:55) Yeah, that's a difficult question because every organization is so different. And there are certainly different schools of thought around experimentation of how to either centralize it or democratize it. Are we giving it to different product managers on different teams? Are we having one central team do it? Are we having a center of excellence?

Shamir Duverseau (22:12) And each of those ways of doing it really has its pros and cons. A lot of it depends on the organization. It depends on the goals and what you're trying to accomplish, and then weighing those goals and what you're trying to accomplish against those pros and cons, and then determining, okay, for our team and for our organization and what we're trying to do, this seems to be the best way to end up tackling this. Honestly, what we see more than anything else in the space that we operate in is the

Shamir Duverseau (22:37) difficulty and challenges of people even getting started. So we have many, many, many organizations that we've worked with, clients that we've come across, we had conversations with, where they have the tools in place

Shamir Duverseau (22:49) to be able to gather data, to gather research, to actually run experimentation, but they have no processes in place. And every time it kind of sort of starts up, it kind of really falters and maybe they run one or two and then that kind of peters off over time. So the challenge that we usually find is just getting started, getting over the hump of being able just to get started and get the process going, trying to get maybe some quick wins up front to kind of show the value in it. And again,

Steffen Horst (23:15) Mm-hmm.

Shamir Duverseau (23:16) looking at it from the lens of making a decision. Because an experiment that wins is great. An experiment that loses is almost just as if not more valuable oftentimes. So there's been times we've gotten organizations into experimentation by launching an experiment that ends up losing because it was maybe, hey, you know, so and so has wanted to do this for a long time. They really want to make this change in the website. So then we run it and it's like, actually if you would have made that change, you would have lost money. That would have made things worse on the website.

Steffen Horst (23:25) Mm-hmm.

Steffen Horst (23:36) Mm-hmm.

Shamir Duverseau (23:42) So now do you know, not only not to make that change, but the time and cost wasn't spent putting that change in the roadmap, expending resources on it, right? Actually making the change. So right there, we just saved you money, right? Maybe we didn't make you incremental dollars, but we saved you money, which is just as valuable, right? To the ultimate bottom line of the organization. So going in and trying to get those wins, going in and trying to — and by understanding what

Shamir Duverseau (24:06) the power of experimentation can be just from a learning standpoint and how we can both create incremental dollars, how we can save dollars, how we can help to prioritize resources, production resources. Looking at it from kind of all those different angles will help an organization to say, all right, well, let's at least get started. Let's start to get some of those quicker wins that we can get by better prioritizing projects, by better allocating costs across different resources.

Steffen Horst (24:27) Yeah.

Shamir Duverseau (24:30) And then we can begin to pick up speed. So I'd rather get started and then get to the point where they have the problem of, okay, how do we organize this? How do we really scale this? How do we really grow this? Because they want to do it and now they want to figure it out. That's a better problem to have than well, we can't even get this off the ground. And

Shamir Duverseau (24:45) the problem is now you're not learning anything. You're just kind of doing what you've always done. And that's not gonna help the business to be able to grow. Cause every single piece of data and research we've seen says that organizations that have that culture of experimentation — with their challenges and problems — but those organizations always grow faster than the ones that don't because they're learning.

Steffen Horst (25:02) Because their decisions are based on data in the end, right? And not just the gut feel like, today, as you said, you get up in the morning, it's like, we should change the colour of our website to red because I feel like, you know, makes sense. Shamir, thank you so much for joining me on the Performance Delivered podcast and sharing your perspective on our today's topic. It's clear that experimentation alone isn't enough. Real growth comes from understanding how people think, how data is cited, and designing experiences that help them

Shamir Duverseau (25:13) Exactly.

Steffen Horst (25:28) move forward with confidence, basically. For listeners who want to learn more about you, your work, and want to connect with you, where's the best place to reach you?

Shamir Duverseau (25:37) Yeah, they can certainly please connect with me, follow me on LinkedIn where I post about this all the time. Post not just about experimentation and the post-click experience, but about the importance of collaboration and working with the other teams to be able to move the business forward and accomplish those marketing goals. And of course you can always check us out on smartpandalabs.com. But feel free to look me up on LinkedIn. I promise you I'm the only Shamir Duverseau on there. I'll be easy to find and we'll be able to connect.

Steffen Horst (26:02) Okay, perfect. As always, we'll leave that information in the show notes. Thanks everyone for tuning in. If you enjoyed this episode of Performance Delivered, please subscribe and leave us a review on iTunes or your favorite podcast platform. To learn more about Symphonic Digital, visit us at symphonicdigital.com