Building A Culture Of Experimentation

Should you try Google’s famous “20% time” experiment to encourage innovation? We tried this at Duolingo years ago. It didn’t work. It wasn’t enough time for people to start meaningful projects, and very few people took advantage of it because the framework was pretty vague. I knew there had to be other ways to drive innovation at the company. So, here are 3 other initiatives we’ve tried, what we’ve learned from each, and what we're going to try next. 💡 Innovation Awards: Annual recognition for those who move the needle with boundary-pushing projects. The upside: These awards make our commitment to innovation clear, and offer a well-deserved incentive to those who have done remarkable work. The downside: It’s given to individuals, but we want to incentivize team work. What’s more, it’s not necessarily a framework for coming up with the next big thing. 💻 Hackathon: This is a good framework, and lots of companies do it. Everyone (not just engineers) can take two days to collaborate on and present anything that excites them, as long as it advances our mission or addresses a key business need. The upside: Some of our biggest features grew out of hackathon projects, from the Duolingo English Test (born at our first hackathon in 2013) to our avatar builder. The downside: Other than the time/resource constraint, projects rarely align with our current priorities. The ones that take off hit the elusive combo of right time + a problem that no other team could tackle. 💥 Special Projects: Knowing that ideal equation, we started a new program for fostering innovation, playfully dubbed DARPA (Duolingo Advanced Research Project Agency). The idea: anyone can pitch an idea at any time. If they get consensus on it and if it’s not in the purview of another team, a cross-functional group is formed to bring the project to fruition. The most creative work tends to happen when a problem is not in the clear purview of a particular team; this program creates a path for bringing these kinds of interdisciplinary ideas to life. Our Duo and Lily mascot suits (featured often on our social accounts) came from this, as did our Duo plushie and the merch store. (And if this photo doesn't show why we needed to innovate for new suits, I don't know what will!) The biggest challenge: figuring out how to transition ownership of a successful project after the strike team’s work is done. 👀 What’s next? We’re working on a program that proactively identifies big picture, unassigned problems that we haven’t figured out yet and then incentivizes people to create proposals for solving them. How that will work is still to be determined, but we know there is a lot of fertile ground for it to take root. How does your company create an environment of creativity that encourages true innovation? I'm interested to hear what's worked for you, so please feel free to share in the comments! #duolingo #innovation #hackathon #creativity #bigideas

𝐓𝐡𝐞 𝐄𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 𝐒𝐡𝐞𝐞𝐭 𝐈 𝐰𝐢𝐬𝐡 𝐈 𝐡𝐚𝐝 𝐟𝐢𝐯𝐞 𝐲𝐞𝐚𝐫𝐬 𝐚𝐠𝐨 🧠🧪 Over the last weeks I started turning my notes into clean one pagers and cheat sheets: Stylish, scannable and easy to save :) And of course it makes sense to do the same for the topic that never stops creating debates in product teams: experimentation 🔬 So I put together one sheet with 9 frameworks that keep showing up in conversations when people talk about running experiments well. 🧭 Why this exists Because teams rarely struggle with the mechanics of an A/B test but rather with alignment, evidence quality, metric decisions, operating models and maturity. This sheet is meant to be a quick mental toolkit for exactly that. 🧪 A few frameworks on it 🔁 The Flywheel by Aleksander Fabijan Still one of the clearest ways to explain how infra, speed and trust compound over time. 🏛️ The RIGHT Model by Fabian Fagerholm et al. A classic in academia and a surprisingly practical way to think about what you need beyond a single experiment. 🧩 And yes I also dared to include a model I built together with Ben Labay and Lukas Vermeer 🃏. Because operating model discussions around experimentation are still massively underrated 📩 If you want it: Comment FRAMEWORKS and I’ll send you ✅ the high res version ✅ plus the source links for each framework 🛠️ One more thing This is a first version so please let me know if something is not accurate or if I’m missing a key model 🙏 #experimentation #abtesting

Don’t Fear Differences—Use Them! ** In any walk of life, differences of opinion are not just inevitable, they are valuable opportunities for growth and innovation. 6 ways to leverage intellectual friction: ✅ Encourage Open Dialogue 🗣️ → How: Create a safe space for honest opinions. → Benefit: Fosters trust and promotes diverse ideas.   ✅ Promote Critical Thinking 🧠 → How: Challenge assumptions and encourage questioning. → Benefit: Enhances problem-solving and innovation.   ✅ Value Diverse Opinions 🌍 → How: Seek out and include different perspectives. → Benefit: Leads to more comprehensive and effective solutions.   ✅ Facilitate Constructive Conflict 🤝 → How: Engage in respectful disagreements. → Benefit: Prevents negative tensions and turns conflicts into productive discussions.   ✅ Lead by Example 🏆 → How: Demonstrate openness to feedback and willingness to consider alternative viewpoints. → Benefit: Inspires your team to do the same.   ✅ Celebrate Collaborative Success 🎉 → How: Reward teams that leverage diverse ideas. → Benefit: Reinforces the value of teamwork and diversity.   💡 Be the thought leader who turns differences into strengths.   👉 How do you encourage diverse opinions? Share your strategies in the comments below ⬇️ ♻️ Share this post to inspire a culture of growth and innovation. 🔔 Follow Malay Matalia for more! Image credit: Adam Grant

Fear of failure can hold us all back, but I’ve found a simple mindset shift that helps me take the plunge without being paralysed by fear: I treat everything as an experiment. An experiment isn’t about succeeding or failing. It’s about testing a hypothesis, learning, and collecting data. So, whenever I feel the jitters about a new project, I reframe it as an experiment to take the pressure off. Here’s how it works: 1. Start with a goal What’s something you want to achieve?  Let’s say you’re aiming to start a YouTube channel. 2. Turn your goal into a hypothesis Ask yourself, “What am I curious about? What do I want to find out?” You might enjoy making videos about travel. 3. Design a simple experiment Break it down into manageable steps to test your hypothesis. Post three travel videos in the next three months. After running the experiment, check your results.  Did you enjoy making the videos? Were you consistent? If it felt natural and enjoyable, that’s a good sign that this path is worth exploring. Here’s another example if you’re thinking of starting a business: Goal: Build a successful business. Hypothesis: I can help people by building websites. Experiment: Email ten people who might benefit from having a personal website and gauge interest. If you get five positive responses, great, it’s a sign there’s demand. If not, that’s okay too. You’ve collected data to refine your approach. The key takeaway? Even if your experiment doesn’t go as planned, it’s not a failure – it’s just data. Tweak your hypothesis, adjust, and try again. Over time, these experiments help build self-awareness, which, in turn, lessens the fear of failure. You’re learning and evolving rather than “failing.” BTW, if you’re an aspiring creator who’s let fear get in the way of putting yourself out there, you might want to check out my YouTube scorecard. 👇 https://lnkd.in/e5-GhB7N

This week’s question on leadership in the age of AI: How do you build a culture where experimentation is encouraged — but mistakes are still safe to make? I love this question because it is hard to get this balance right and it is needle moving when you do get it right. Here’s my take: Learning fast >> Being right. When so much is unknown, no one gets everything right. The goal isn’t perfection, it’s progress. That means setting hypotheses, running experiments, learning fast, and pivoting when you’re wrong. But that only works if we actually celebrate being wrong. Talk about the time your hypothesis failed. Share what you learned. Normalize the miss. Because when people fear mistakes, they stop taking bold bets — and that’s how innovation dies. Three things I’ve seen work: 1. Think like scientists. Every decision is an experiment. Every opinion is a hypothesis. Every result is data. I learned this from Adam Grant — it changes how teams think, act, and learn. 2. Create experimentation pods. We did this at HubSpot to accelerate AI adoption. Small tiger teams explore use cases, test tools, and share learnings. It builds a fast feedback loop — and inspires others to try. 3. Celebrate well-run experiments. We’re great at praising results. We should be just as loud about experiments that taught us something, even when the numbers didn’t move. Sometimes your hypothesis can be wrong, but that does not mean you are wrong! The faster we learn, the faster we grow. And that starts when teams shift from being know-it-alls to learn-it-alls. How are you creating a culture of experimentation in your teams? 👇

As more companies embrace A/B testing, the bottleneck is no longer running experiments—it’s ensuring those experiments lead to trustworthy decisions. In this tech blog, the data science team at Booking.com explains how they scaled experimentation quality across the organization. Rather than enforcing rigid rules, they chose to preserve team autonomy while building the supporting systems needed to encourage better experimentation practices. The team’s solution followed a simple but thoughtful progression: process, metric, then tool. They first invested in community initiatives like Experiment Ambassadors and peer experiment reviews to build a shared experimentation culture. They then introduced an Experimentation Quality framework that evaluated every experiment across three dimensions—Design, Execution, and Decision—making experimentation quality measurable and easier to improve. Finally, they embedded those standards directly into their internal experimentation platform through features such as quality checks, power-calculation guidance, and stronger defaults that naturally guided teams toward better decisions. The goal was to maintain flexibility while making good experimentation practices easier to follow. This work highlights an important lesson: improving experimentation at scale requires more than statistical knowledge or individual discipline. Sustainable improvement comes from combining strong organizational processes, meaningful quality metrics, and tooling that reinforces good practices into the everyday workflow. When these pieces work together, teams can make more reliable decisions. #DataScience #MachineLearning #Experimentation #ABTesting #Analytics #SystemDesign #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gg8eX3Yv 

Great leadership isn’t about ensuring alignment all the time. Here is why: I recently worked with a leadership team in a global company that, at first glance, seemed to be thriving. Meetings were quick, decisions were made efficiently, and everyone was on the same page. They believed this harmony meant they were operating at peak performance. But beneath the surface, something critical was missing: 🚫 innovation. Their constant agreement was stifling progress. Without diverse ideas, challenges, or healthy debate, the team was simply recycling the same thinking, overlooking new opportunities and struggling with complex problems. It was a classic case of ‘groupthink’—where everyone falls into agreement to avoid conflict or discomfort.  👇 Here’s what I did with the team: - Diagnosed the agreement cycle & TPS - Introduced psychological safety practices - Encouraged intellectual humility - Secured mechanism for diverse input integration We started worked on inclusive decision-making practices by ensuring that every voice in the room was heard. We integrated mechanisms like structured brainstorming, anonymous idea submissions, and rotating roles of idea champions to reduce bias and prevent dominant voices from overtaking discussions. 📈 The result? Not only did their decision-making improve, but their solutions became more creative and forward-thinking. Leaders, here're the takeaways: 1️⃣ If your meetings are full of "Yes, I agree," ask yourself what you might be missing. 2️⃣ Diversity of thought is your competitive advantage. 3️⃣ Teams thrive when they feel safe enough to disagree and bold enough to innovate. This is psychological safety. P.S. Do you think your team challenges each other enough? I’d love to hear your thoughts 👇

𝐌𝐲 𝐩𝐚𝐫𝐞𝐧𝐭𝐬 𝐡𝐚𝐝 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐩𝐥𝐚𝐧𝐬 𝐟𝐨𝐫 𝐦𝐞. My father, a lawyer, wanted me to follow him into law. My older sister beat me to it. And candidly, we don’t need more lawyers, at least in my family 😊 My mother worked for Bhabha Atomic Research Centre (under the Department of Atomic Energy) for years and wanted me to become a scientist like the ones she supported and worked alongside. She’d take me to work with her, and I was fascinated by the isotope facility. So I became a Chemistry major spending half my days in a lab wearing a lab coat that permanently smelled like rotten eggs (hydrogen sulfide, for those keeping score). The one class I loved: Drugs and Dyes, where we learned to synthesize things like aspirin. Making something tangible, something useful from raw materials. That’s what hooked me. Turns out, my life’s calling wasn’t the lab. But those years, from watching scientists to running my own failed experiments, shaped a few things that I have continued to build upon. Here are my 3 “𝐋𝐞𝐬𝐬𝐨𝐧𝐬 𝐟𝐫𝐨𝐦 𝐭𝐡𝐞 𝐋𝐚𝐛” in honor of the United Nations 𝐈𝐧𝐭𝐞𝐫𝐧𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐃𝐚𝐲 𝐨𝐟 𝐖𝐨𝐦𝐞𝐧 𝐚𝐧𝐝 𝐆𝐢𝐫𝐥𝐬 𝐢𝐧 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. 1️⃣ 𝐏𝐫𝐨𝐠𝐫𝐞𝐬𝐬 𝐜𝐨𝐦𝐞𝐬 𝐟𝐫𝐨𝐦 𝐝𝐢𝐬𝐜𝐢𝐩𝐥𝐢𝐧𝐞𝐝 𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧. In chemistry, the first experiment never works. You don’t expect it to! You test, observe, adjust, and try again. Failure is failure, it's data. That's exactly how we approach transformation. Start small, learn fast, and scale what works. The best innovations rarely emerge from the first attempt. 2️⃣ 𝐒𝐭𝐚𝐲 𝐜𝐮𝐫𝐢𝐨𝐮𝐬 𝐥𝐨𝐧𝐠𝐞𝐫 𝐭𝐡𝐚𝐧 𝐲𝐨𝐮'𝐫𝐞 𝐜𝐨𝐦𝐟𝐨𝐫𝐭𝐚𝐛𝐥𝐞. The moment you assume you know the answer in the lab, you stop seeing what's actually happening. You miss the signal in the noise. As leaders navigating rapid change, curiosity is our competitive advantage. It keeps us from locking into the wrong solution too soon and opens doors we didn't know existed. Success requires a bottomless well of curiosity, even when it feels like everyone around you is demanding certainty. 3️⃣ 𝐁𝐫𝐞𝐚𝐤𝐭𝐡𝐫𝐨𝐮𝐠𝐡𝐬 𝐡𝐚𝐩𝐩𝐞𝐧 𝐭𝐡𝐫𝐨𝐮𝐠𝐡 𝐝𝐢𝐯𝐞𝐫𝐬𝐞 𝐭𝐞𝐚𝐦𝐬. In the lab, different perspectives spot different variables. Same in business. The best outcomes come from teams that see the problem from multiple angles. Diverse thinking isn’t a nice to have. It’s how you actually solve problems. To every woman and girl in STEM: stay curious, trust the process. And remember: you absolutely belong here. The future is being built and it needs your perspectives. What lessons from your early curiosity still guide you today?

✨ Four in ten Americans are stockpiling items. Walmart is selling Gucci online. Consumers are boycotting major retailers. The top 10% of households by income account for 50% of all spending. Major CPG companies are signaling price hikes. Oh, and eggs. If that sounds like a disheartening list of recent headlines, it’s not. The list represents changes in consumer behavior that will shape the rest of the year. The fun part is to lean into the change to find opportunities. If consumers are stockpiling, maybe you should offer larger sizes (it’s more sustainable anyway). If consumers are focused on price, maybe you should build your products in a new, less costly way. If the luxury market is lurching, redefine it. Experiment! Experimentation is the pathway to opportunity. And I'm not talking about experimenting in focus groups and surveys--I'm talking real-life experiments (online or otherwise). Shifting behavior creates big questions. Opinions and preferences won't get to the heart of what's driving change--real-life experiments are a fast way to learn what’s going on. Some questions to generate hypotheses for experimentation: 1. What changes are creating new problems for target customers? Are there signs of behavioral shifts? (Like, um, boycotting big retailers) 2. Can you articulate a research question to describe the opportunity? (‘Does highlighting my brand’s support of small retailers increase our sales in those channels?’) 3. Can I define independent and dependent variables? What is my prediction about those variables? (‘If we offer ad support to selected small retailers, they (and we) will see sales increases at higher rates than at small retailers we don’t offer support for.’) Experimentation benefits? Validating opportunities. Avoiding pitfalls. Moving fast.

When people say they don't trust the results of A/B tests, what they often really mean (even if they don't articulate it) is that the metric on which the test is measured is only one way of looking at things. Leaders love to give people simple metrics as objectives, and yet they also are acutely aware that business is just not that simple. If you present the outcome of an experiment to senior leaders and are talking about conversion rate increases, there are often, even for simple tests, a lot more intangible things for them to consider: • What is the impact on the brand? • Does it fit with the XYZ transformation/change project that nobody else in the business knows about yet but which is coming? • What will the new VC who might be about to invest in us think of this direction? • Will it have any effect on customer service or operational costs? • What will it do to tech debt? • Is it as important as these other roadmap projects that don't have tangible revenue/conversion percentages and yet nevertheless feel very important?    'Experimentation' ought to invest its time and energy in helping people to make decisions by synthesizing a variety of evidence types and perspectives, and to recognise that decisions are not made on the basis of single, simple metrics. #experimentation #cro #productmanagement #growth #digitalexperience #decisionmaking #growthexperimentation