Design Thinking In Customer Experience

Why Every Product Manager Needs A/B Testing 🚀 Imagine cooking up a recipe for the perfect product feature. Would you trust your instincts blindly, or would you test different ingredients to get the best taste? That’s where A/B testing comes in. It’s the secret sauce that helps Product Managers make data-driven decisions with confidence. Here’s everything you need to know to master A/B testing: ❓ What is A/B Testing❓ A/B testing is the process of comparing two or more versions of a product to determine which one performs better. The versions might differ in small ways - a new button design, a revamped landing page, or an updated pricing structure but the impact on user behaviour can be monumental. This method helps you validate assumptions, optimize user experiences, and ensure every product decision adds value. ⚙️ How to Conduct a Successful A/B Test? ⚙️ 🔹 Set Clear Goals Ask yourself what are you trying to improve? It could be anything from conversion rates to user satisfaction. Your goal is your North Star. 🔹 Choose the Right Metrics Metrics like click-through rates (CTR), time spent on a page, or purchase frequency will guide you in evaluating success. 🔹 Hypothesize Frame your test with a simple prediction. Example: “I believe changing the CTA button color from blue to green will increase clicks by 15%.” 🔹 Design Your Experiment Define your control group (current version) and treatment group (variant to test), ensuring a large enough sample size for reliable results. Run the test for a sufficient duration to capture meaningful patterns and user behaviour. 🔹 Analyze & Implement Use tools like Google Optimize or Optimizely to analyze results and determine statistical significance. Roll out the winning variant confidently, or refine your hypothesis for future iterations if results are inconclusive. ♻️ Four Types of A/B Tests Every PM Should Know ♻️ 1️⃣ Feature Testing: Validate hypotheses for new features pre-launch. 2️⃣ Live Testing: Fine-tune existing features already in the wild. 3️⃣ Trapdoor Testing: Redirect traffic between variants dynamically. 4️⃣ Multi-Armed Bandit (MAB): Let machine learning allocate traffic to better-performing variants in real-time. ❌ Common Pitfalls to Avoid ❌ 1️⃣ Testing trivial changes that won’t move the needle. 2️⃣ Ignoring sample size requirements—small audiences lead to inaccurate conclusions. 3️⃣ Treating A/B testing as a one-off exercise. Optimization is an ongoing journey. What’s been your most surprising A/B testing discovery? Let’s discuss in the comments!👇 Ready to embark on an exhilarating journey into the heart of product management? I’ve recently launched a cohort that is focused on teaching end-to-end product management as well as providing career placement opportunities! 🧠 Fill in the form in the comments to register your interest in the cohort and I’ll reach out to you with further details. ✍️ #ProductManagement #ABTesting #PMTools #ContinuousOptimization

The idea that A/B tests are autonomous deployment decisions is disingenuous. You are ignoring the most important reality, which is that someone is 𝘢𝘭𝘸𝘢𝘺𝘴 making a decision. How are they making that decision? Experimentation is mostly seen as the final stage in a linear process. You conduct some research, conceive an idea, and then test it. In this way of viewing experimentation, the test IS the decision. If it wins, then deploy it without further consideration (and forecast your exact revenue for a year). If it loses, then definitely do not implement it and move on to something else. However, the idea that an A/B test represents a decision is a dangerous illusion: > There are many potential issues with experiments that you may never observe in the data. You can never be completely certain that the result you see is correct, regardless of what 'significance' indicates. The outcome is just a piece of data and information, not a definitive proof of anything and certainly not a decision. > 'Data' is only one way of assessing the benefits and outcomes of the change; there are many other factors you may need to take into account, such as broader customer experience, brand perception, alignment with wider strategic initiatives, etc. These aspects cannot be reduced to simple metrics and data, and if you ignore them you risk damaging your business. > Experimentation is not just a binary way of deciding whether to deploy something; it is a way to test theories that might help support strategy or give rise to bigger ideas. A test is a way to learn something about customer behaviour and develop theories based on that behaviour. By limiting it to just a deployment decision, you lose this potential value. Experimentation is just one form of research among others, all of which should be used in parallel to support the entire process of innovation. More importantly, recognise that YOU are making the decisions, not data. In that case, how are you making decisions? What is your PROCESS & SYSTEM for making effective and efficient decisions? #ecommerce #retail #digitalexperience #cro #experimentation

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

We’ve all heard of audience personas. But what if you could look beyond demographics and see how a persona thinks, behaves, and buys in real time? That’s exactly what I did today using Momentro, diving into the “Coffee Lovers” persona while comparing Barista Coffee Company Limited and t-Lounge by Dilmah but instead of focusing on search or content, I went deeper into behaviour. ☕ The “Coffee Lovers” Persona in Sri Lanka. 📌 Behavioural Trends: Actively follow slow living, café culture, and minimalism creators on YouTube. Prefer review led content over ads. Blend indulgence with wellness interested in both high-end desserts and clean living. 📌 Influencer Signals: Gravitate towards authentic, often micro-influencers who feel like trusted voices. Example: I checked out Alison Wijemanne who popped up in the F&B influencer space. Momentro provided me her category strength (food, beverage & travel) her brand history, her sentiment index (largely green = safe bet for partnerships) and some of the brands she has worked with in the past too. 📌 Brand Affinities: Engage with Barista, Dilmah T-Lounge, Java Lounge (Pvt) Ltd, Peppermint Cafe, Ibsons Choice Cafe, and even Starbucks — suggesting they blend local pride with global taste. 📌 Pain Points & Opportunities of coffee lovers in Sri Lanka: Tired of copy-paste content Seek genuine café experiences and behind-the-scenes narratives Want to feel spoken to, not marketed at For Content Teams: This is a Gold Mine. Most content teams are briefed with assumptions: “Target millennials,” “Make it Gen Z-friendly,” “Do something trendy.” But with Momentro, your creative team gets the nuance: -What this persona wants to hear -What frustrates them -What formats they consume -What tone feels authentic vs performative -Build campaigns based on what this persona already consumes -Choose influencers that align with their behavioural identity -Tailor content formats (YouTube > Facebook, micro > macro) No more content roulette. You build stories rooted in reality, pain points, motivations, peer influence, and preferred channels. Suddenly, your next campaign isn’t just more relevant. It’s more wanted! #marketing #influencermarketing #personaanalysis #microinfluencers #momentro

Why Buyer Personas Are Often Useless (Unless You Do Them Right) Buyer personas. Every marketer talks about them, but how many of us actually use them to drive real results? Too often, buyer personas are treated as an exercise in check-box marketing: Create a template, fill in some basic demographics, and call it a day. But this is a recipe for wasting time and burning calories. The real power of buyer personas lies in the depth of insight they provide about the emotional, psychological, and behavioral triggers of your target audience. When done right, personas become your roadmap for everything—product decisions, messaging, marketing strategies, and sales enablement. But when done wrong, they’re useless. So, how should you approach Buyer Personas? 1. Go Beyond Demographics It’s easy to create personas based on age, job title, and income. But that’s not what actually drives a purchase decision. You need to understand why your customers buy your product—what pain points are they solving, what motivates them, and what stands in their way. 2. Focus on Behavior and Needs Instead of just a “one-size-fits-all” persona, segment by behavior and customer journey stage. Are they early-stage prospects or ready to buy? How do they interact with your product? Behavior speaks volumes. 3. Constantly Evolve Your personas shouldn’t be static! The market, technology, and customer needs evolve—so should your personas. Continuously gather feedback from your users, sales teams, and customer support. Buyer personas done right can drive growth, shape product development, and create hyper-targeted marketing strategies. But done poorly? They’re just another file on the shelf. #growthmarketing #buyerpersonas #marketing

Most UX teams have been there: standing in front of a wall of sticky notes, surrounded by user quotes and caffeine, trying to decide if “Goal Oriented Greg” and “Curious Carla” are genuinely different people or just the same imaginary user with better handwriting. Persona discovery sessions like this often feel productive, the colors, the discussions, the post-its forming patterns, but deep down, we know something is off... The process is usually more art than science, more consensus building than discovery. It produces personas that sound nice in presentations but rarely hold up when real users start behaving unpredictably. Good news?! There is a more rigorous way to approach this, one that turns persona creation from a creative exercise into an analytical process grounded in evidence. Instead of guessing who your users are, you can identify them empirically by examining their real behaviors, motivations, and characteristics across your datasets. This is where clustering analysis becomes invaluable, allowing your data to uncover the story of your users on its own. Clustering uses statistical algorithms to uncover patterns and similarities across multiple dimensions of user data, revealing natural groups that exist beneath the surface. These are not personas invented in a meeting; they are personas discovered in the data. Here is how it works in practice. You begin by gathering rich, multidimensional data, including behavioral metrics. After cleaning and preparing your data, you apply a clustering algorithm such as K Means, Hierarchical Clustering, or Gaussian Mixture Models. These methods analyze the combined patterns across all features and group users who are statistically similar into clusters. Each cluster represents a group of people who share distinctive traits, perhaps they are highly efficient but disengaged, or slower but deeply curious. From there, you interpret and label these clusters in human terms. The data gives you the structure, and your UX insight gives it meaning. You might visualize the results, examine which variables most differentiate each group, and build out personas that reflect the real diversity within your audience. These personas are no longer fictional composites; they are data backed archetypes that show how meaningful subgroups actually behave, think, and feel. The benefits are substantial. Clustering eliminates much of the bias that comes from relying on small samples or internal intuition. It exposes hidden user types that might never emerge from interviews alone, such as a quiet but influential group of users whose needs are consistently overlooked. It also creates alignment across teams because the evidence is transparent and reproducible. When you present personas derived from clustering, you can trace every insight back to data, not opinion. #PersonaDiscovery #UXResearch #DataDrivenDesign #CustomerSegmentation #ProductStrategy #UserExperience #QuantitativeUX

Classic A/B testing relies on SUTVA (Stable Unit Treatment Value Assumption), which assumes one user’s decision doesn’t influence another’s. But what if your product is a social network, marketplace, or delivery service? Imagine you’ve improved the post-ranking algorithm on LinkedIn. Users in Group A (new algorithm) creates more content now. But this content spreads to Group B (old algorithm), distorting the results due to network effects. Here are two main ways to tackle this: 1. 𝐂𝐥𝐮𝐬𝐭𝐞𝐫𝐢𝐧𝐠-𝐛𝐚𝐬𝐞𝐝 𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐬: Randomize groups of users (clusters) instead of individual users. For social networks, the most popular approach is to define clusters based on interaction frequency — those who engage more often stay together in one cluster. 2. 𝐒𝐰𝐢𝐭𝐜𝐡𝐛𝐚𝐜𝐤 𝐭𝐞𝐬𝐭𝐬: In this approach, everyone in the network receives the same treatment at any given time. Over time, we flip between test and control groups, compare metrics, and evaluate the impact. This is especially useful for location-based services (e.g., taxis or delivery). Even if you’re not working with a product that has potential network effects, understanding these methods will help you in future interviews!

A 6% revenue lift. 99% statistical significance. Ship it. It couldn't go wrong, could it? 🫣 In 2016, I was leading a product analytics team at Credit Karma. We ran an A/B test for a personal loans redesign. The results looked fantastic: - 𝗔𝗽𝗽𝗿𝗼𝘃𝗮𝗹𝘀 𝘄𝗲𝗿𝗲 𝘂𝗽 (good for users). - 𝗥𝗲𝘃𝗲𝗻𝘂𝗲 𝘄𝗮𝘀 𝘂𝗽 𝟲% (good for business). - 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝗰𝗲: 𝟵𝟵%. We should have ramped it up to 100% of users and closed out the test. However, we couldn't roll it out immediately due to other constraints. Over the next few weeks, I watched that 6% revenue lift drift down to 3%. It was still positive. It was still 99% significant. But the downward trend didn't sit right with me. I dug into the segments and found the reality: 𝗨𝘀𝗲𝗿𝘀 𝗻𝗲𝘄 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: +10% revenue. 𝗨𝘀𝗲𝗿𝘀 𝗿𝗲𝘁𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: -5% revenue. The aggregate number was positive only because the traffic was initially heavy with people seeing the design for the first time. Over time, as those people returned to the page, they fell into the negative bucket. 𝗜𝗳 𝘄𝗲 𝗵𝗮𝗱 𝘀𝗵𝗶𝗽𝗽𝗲𝗱 𝗯𝗮𝘀𝗲𝗱 𝗼𝗻 𝘁𝗵𝗲 𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲, 𝘄𝗲 𝘄𝗼𝘂𝗹𝗱 𝗵𝗮𝘃𝗲 𝗲𝘃𝗲𝗻𝘁𝘂𝗮𝗹𝗹𝘆 𝗹𝗼𝘀𝘁 𝗺𝗼𝗻𝗲𝘆. We wouldn't have even known that it was due to a negative A/B test. Because we caught this, we redesigned the experience to address the issues for the returning users before rolling it out. Don't just blindly follow A/B tests and their implied results. While I love A/B testing, you need to be very careful to understand what you are truly measuring. (we did end up fixing the experience for returning users and deploying a win-win)

For 16+ years, I've run A/B tests comparing human behaviors. Starting today, there's a third variant to test: how AI agents interact with your site. Here's what changed. Adobe Analytics found traffic from AI sources surged 1300% during last holiday season. Yet those AI visitors were 23% less likely to convert than humans (Adobe Analytics, July 2025). The gap reveals something critical: ↳ Sites optimized for humans are failing AI agents. PwC's May 2025 survey shows 79% of companies already use AI agents. Adobe found 87% of shoppers turn to AI for complex purchases, and will only have those agents complete the purchase at a fast rising rate. Your perfectly optimized checkout might be invisible to these new users. Testing methodologies are evolving. Every test now needs to track three metrics: ↳ human conversion rate ↳ human user experience ↳ agent success rate Sometimes these align beautifully... clean navigation helps everyone, and clear product data benefits both. But often you face tradeoffs. Dynamic form fields that reduce human friction? Agents can't parse them. That trendy single page checkout? Humans love it. AI agents get stuck. The winners won't pick sides. They'll optimize for both. This is dual-mode optimization... CRO and AXO (Agent Experience Optimization). Each variant gets scored on human AND agent performance. Real winners excel at both. This isn't about replacing your current testing. It's adding a critical dimension. Your next A/B test needs to become an A/B/Agent test. The third user has arrived.

The A/B Testing Safety Net for Software Development. People often ask how to “sell” A/B testing internally. Zach Flynn recently wrote that experimentation culture should be guided, not forced by “laws” [1]. Guidance alone is not enough. You must show the value, and one of the best ways to encourage a “test everything” mentality with A/B testing is to show the value of the safety net.  While most A/B content focuses on power, p-values, and finding winners, an overlooked benefit is operational: detecting egregious regressions and aborting quickly, shrinking the blast radius. When Microsoft Office for desktop moved from a 3-year release cycle to monthly releases, the key problem was turning off a bad piece of code after the client shipped.  They adopted A/B testing for its kill-switch capability—safe deployments. When A/B testing was integrated, it was an easy step to also evaluate the value of features [2]. Every (good) engineering organization runs weekly postmortem (sometimes called AAR for After Action Review) to understand the root cause of outages and severe incidents and learn how to avoid them in the future. The questions to add to the postmortem form are: - Was this change behind an A/B test? While postmortem are blameless, people quickly learn that many more outages are associated with code deployed without an A/B test, driving adoption. - If yes, what guardrail metric should be added that would catch and auto-abort a similar issue? Remember the “other tail” of A/B testing: the safety net that lets you abort bad deployments fast! [1] https://lnkd.in/geZr2CmU [2] https://lnkd.in/gCSBHeTv #ABTesting #ExperimentGuide #DevOps #postmortem