Improving Predictive Accuracy

Training LLMs for spam classification: I added 14 experiments comparing different approaches: https://lnkd.in/gTNVvGcj - which token to train - which layers to train - different model sizes - LoRA - unmasking - and more! Any additional experiments you'd like to see? And here are the take aways for the table shown in the picture: 1. Training the Last vs. First Output Token (Row 1 vs. 2): Training the last output token results in substantially better performance compared to the first. This improvement is expected due to the causal self-attention mask. 2. Training the Last Transformer Block vs. Last Layer (Row 1 vs. 3): Training the entire last transformer block is also results in substantially better results than training only the last layer. 3. Training All Layers vs. Last Transformer Block (Row 1 vs. 4): Training all layers shows a modest improvement of ~2% over just training the last transformer block, but it requires almost three times longer in terms of training duration. 4. Using Larger Pretrained Models (Row 1 vs 5, and Row 1 vs. 6 and 7): Employing a 3x larger pretrained model leads to worse results. However, using a 5x larger model improves performance compared to the initial model, as was anticipated. Similarly, the 12x larger model improves the predictive performance even further. (The medium model was perhaps not well pretrained or the particular finetuning configuration works not as well for this model.) 5. Using a Model with Random Weights vs. Pretrained Weights (Row 1 vs. 8): Utilizing a model with random weights yields results that are only slightly worse by 1.3% compared to using pretrained weights. 6. Using LoRA (Low-Rank Adaptation) vs Training All Layers (Row 9 vs. 4): Keeping the model frozen and adding trainable LoRA layers (see Appendix E for details) is a viable alternative to training all model parameters and even improves the performance by 1% point. As it can be seen by the 1% lower gap between the training and validation accuracy when using LoRA, this is likely due to less overfitting. 7. Padding Input to Full Context Length vs. Longest Training Example (Row 1 vs. 10): Padding the input to the full supported context length results is significantly worse. 8. Padding vs no padding (Row 1 vs. 11 and 12): The `--no_padding` option disables the padding in the dataset, which requires training the model with a batch size of 1 since the inputs have variable lengths. This results in a better test accuracy but takes longer to train. In row 12, we additionally enable gradient accumulation with 8 steps to achieve the same batch size as in the other experiments. 9. Disabling the causal attention mask (Row 1 vs. 13): Disables the causal attention mask used in the multi-head attention module. This means all tokens can attend all other tokens. The model accuracy is slightly improved compared to the GPT model with causal mask.

You are in a Senior Machine Learning Interview at Google DeepMind. The interviewer sets a trap: "We have a 1:1000 class imbalance for fraud detection. We applied 𝘤𝘭𝘢𝘴𝘴_𝘸𝘦𝘪𝘨𝘩𝘵𝘴 to the 𝐂𝐫𝐨𝐬𝐬-𝐄𝐧𝐭𝐫𝐨𝐩𝐲 loss, but the model is still missing the hard edge cases. What do we do?" 90% of candidates walk right into the wall. Most candidates immediately suggest aggressive oversampling (𝘚𝘔𝘖𝘛𝘌) or tuning the class weights even higher (e.g., 1:5000). They think: "If the minority class is ignored, I just need to scream louder (higher weights) during backprop." ------ 𝐓𝐡𝐞 𝐑𝐞𝐚𝐥𝐢𝐭𝐲: You aren't losing because the weights are wrong. You are losing because of 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐃𝐫𝐨𝐰𝐧𝐢𝐧𝐠. Even with perfect class weights, your dataset likely contains 990,000 "easy" negatives (legitimate transactions that are obviously legit) and 1,000 "hard" positives. In standard 𝐖𝐞𝐢𝐠𝐡𝐭𝐞𝐝 𝐂𝐫𝐨𝐬𝐬-𝐄𝐧𝐭𝐫𝐨𝐩𝐲 (𝐖𝐂𝐄), the gradients from those 990,000 easy examples, even if individually small, sum up to dominate the update step. The model spends all its capacity optimizing examples it has already learned, drowning out the signal from the difficult, subtle fraud cases. ------ The Solution: 𝐓𝐡𝐞 𝐄𝐚𝐬𝐲-𝐄𝐱𝐚𝐦𝐩𝐥𝐞 𝐒𝐮𝐩𝐩𝐫𝐞𝐬𝐬𝐢𝐨𝐧 You don't need to re-balance the counts. You need to re-balance the difficulty. The solution is switching from 𝐖𝐞𝐢𝐠𝐡𝐭𝐞𝐝 𝐂𝐫𝐨𝐬𝐬-𝐄𝐧𝐭𝐫𝐨𝐩𝐲 to 𝐅𝐨𝐜𝐚𝐥 𝐋𝐨𝐬𝐬. Focal Loss adds a modulating factor (1 − pₜ)ᵞ to the standard loss equation. Here is what happens in production: - 𝘐𝘧 𝘵𝘩𝘦 𝘮𝘰𝘥𝘦𝘭 𝘪𝘴 𝘶𝘯𝘴𝘶𝘳𝘦 (𝘏𝘢𝘳𝘥 𝘌𝘹𝘢𝘮𝘱𝘭𝘦): The modulating factor stays near 1. The loss is unchanged. The model learns. - 𝘐𝘧 𝘵𝘩𝘦 𝘮𝘰𝘥𝘦𝘭 𝘪𝘴 𝘤𝘰𝘯𝘧𝘪𝘥𝘦𝘯𝘵 (𝘌𝘢𝘴𝘺 𝘌𝘹𝘢𝘮𝘱𝘭𝘦): The factor drops to near 0. The loss contribution is effectively "shut off." This forces the model to stop patting itself on the back for identifying the obvious negatives and focus 100% of its gradient descent budget on the edge cases. 𝐓𝐡𝐞 𝐀𝐧𝐬𝐰𝐞𝐫 𝐓𝐡𝐚𝐭 𝐆𝐞𝐭𝐬 𝐘𝐨𝐮 𝐇𝐢𝐫𝐞𝐝: "𝐖𝐞𝐢𝐠𝐡𝐭𝐞𝐝 𝐂𝐫𝐨𝐬𝐬-𝐄𝐧𝐭𝐫𝐨𝐩𝐲 solves for moderate imbalance (1:10) by balancing counts. 𝐅𝐨𝐜𝐚𝐥 𝐋𝐨𝐬𝐬 solves for extreme imbalance (1:1000+) by balancing hardness. In a fraud scenario, I would implement 𝐅𝐨𝐜𝐚𝐥 𝐋𝐨𝐬𝐬 with γ = 2 to down-weight the easy negatives that are currently dominating the gradient." #MachineLearning #DeepLearning #MLEngineering #AIEngineering #NeuralNetworks #ModelOptimization

In the last three months alone, over ten papers outlining novel prompting techniques were published, boosting LLMs’ performance by a substantial margin. Two weeks ago, a groundbreaking paper from Microsoft demonstrated how a well-prompted GPT-4 outperforms Google’s Med-PaLM 2, a specialized medical model, solely through sophisticated prompting techniques. Yet, while our X and LinkedIn feeds buzz with ‘secret prompting tips’, a definitive, research-backed guide aggregating these advanced prompting strategies is hard to come by. This gap prevents LLM developers and everyday users from harnessing these novel frameworks to enhance performance and achieve more accurate results. https://lnkd.in/g7_6eP6y In this AI Tidbits Deep Dive, I outline six of the best and recent prompting methods: (1) EmotionPrompt - inspired by human psychology, this method utilizes emotional stimuli in prompts to gain performance enhancements (2) Optimization by PROmpting (OPRO) - a DeepMind innovation that refines prompts automatically, surpassing human-crafted ones. This paper discovered the “Take a deep breath” instruction that improved LLMs’ performance by 9%. (3) Chain-of-Verification (CoVe) - Meta's novel four-step prompting process that drastically reduces hallucinations and improves factual accuracy (4) System 2 Attention (S2A) - also from Meta, a prompting method that filters out irrelevant details prior to querying the LLM (5) Step-Back Prompting - encouraging LLMs to abstract queries for enhanced reasoning (6) Rephrase and Respond (RaR) - UCLA's method that lets LLMs rephrase queries for better comprehension and response accuracy Understanding the spectrum of available prompting strategies and how to apply them in your app can mean the difference between a production-ready app and a nascent project with untapped potential. Full blog post https://lnkd.in/g7_6eP6y

When L'Oréal uses AI to create new hair colors based on social media trends, they're in salons within weeks. Kraft Heinz—dead last in our study—still takes months to tweak a formula. After analyzing 26 major CPG companies at IMD's Center for Future Readiness, I discovered what separates winners from losers: The most future-ready companies treat consumer data like insider trading information. BACKGROUND: CPG in 2025 is brutal. Inflation persists. Gen-Z demands sustainability without premiums. Tariffs reshape supply chains daily. McKinsey & Company identified 150+ AI use cases for CPG transformation. Only 5 of 26 companies actually execute them. THE REVELATION: Coca-Cola didn't randomly launch Topo Chico Hard Seltzer. Their AI spotted the trend through social listening while competitors debated in boardrooms. By launch, they'd secured distribution nationwide. That's not innovation. That's prediction. What separates the top 5: L'Oréal (#1): 3.5% of sales to R&D. AI analyzes preferences real-time. Virtual try-on apps. Creates products from social trends. A 110-year company with startup velocity. The Coca-Cola Company (#2): Democratized AI internally. Every manager accesses demand forecasting. They analyze weather + social sentiment + sales simultaneously. These aren't tech companies selling beauty and beverages. They're prediction machines that happen to make products. THE WINNER'S FRAMEWORK: 1. AI at scale, not in pilots Winners integrate into workflows. Losers run demos. 2. Supply chains that anticipate Real-time visibility + AI forecasting = competitive firepower 3. D2C as intelligence goldmine 73% use multiple channels. Mine every interaction. 4. Disrupt yourself first Coca-Cola launched Costa Coffee, hard seltzers. Grew. Kraft Heinz protected legacy brands. Shrank. 5. Sustainable without premium Gen-Z spending hits $12T by 2030. They demand action at everyday prices. —— The inconvenient truth: Most CPG companies treat data like reporting instead of radar. Winners don't predict trends—they're already shipping products while competitors debate. Technological patience (knowing when to scale) + organizational agility (pivoting fast) = market domination. Three years from now, every CPG company operates like L'Oréal. Or they don't operate at all. P.S. Full Future Readiness Indicator here: https://bit.ly/3YTBzbX

Power BI is dominating demand planning. This document shows how to use Power BI for demand planners: Step # 1 - Prepare Your Files ↳ Start with 3 Excel sheets: sales history, forecast, calendar table ↳ How it helps: a clean, consistent starting point ensures accurate relationships and smooth automation later Step # 2 - Power Query: Clean and Merge Data ↳ Go to Home → Transform Data ↳ How it helps: this gives you one clean dataset that can refresh itself automatically every time new data arrives Step # 3 - Data Model: Connect the Dots ↳ In Model View, drag relationships like: Forecast[SKU] → Actuals[SKU] Forecast[Date] → Calendar[Date] ↳ How it helps: this tells Power BI how data connects across tables so that your metrics and visuals update correctly when filters are applied. Step # 4- Create DAX Measures (Your KPIs) ↳ Go to Modeling → New Measure and create formulas for forecast accuracy and bias  ↳ How it helps: these KPIs refresh automatically with each data update; no manual recalculation or formula fixing required. Step # 5 - Build Visuals That Matter ↳ Start simple: Line Chart: Actual vs Forecast by Month Bar Chart: Forecast Accuracy by SKU Scatter Chart: Bias vs Accuracy per SKU KPI Cards: Forecast Accuracy %, Bias %, and FVA ↳ How it helps: instantly spot where the forecast is failing and which products or planners need attention. Step # 6 - Add Slicers (Filters) ↳ Insert slicers for region, planner name, product category, month ↳ How it helps: easily move from a company-level view to SKU-level insight. Step # 7 - Add Drillthrough Pages ↳ Create a second page called SKU-Level Details; add a Drillthrough filter on SKU ↳ How it helps: move from a summary view to detailed root cause in one click Step # 8 - Add Time Intelligence ↳ Create time-based measures such as accuracy LY, accuracy YoY change ↳ How it helps: track improvement over time year-over-year or month-over-month without rebuilding formulas Step # 9 - Automate the Refresh ↳ Under Data → Schedule Refresh, set Power BI to pull data daily or weekly from your Excel files or SQL system ↳ How it helps: your dashboard updates itself Step # 10 - Build a Forecast Evolution View ↳ Use a Line + Area Chart to show: Statistical Forecast, Adjusted Forecast, Actual ↳ How it helps: see whether planner overrides are improving or worsening forecast accuracy over time Any others to add?

I recently demoed 4 FP&A platforms that claim to effectively forecast 13-week cash flows using AI. Three of the companies are dedicated planning tools. One company is a financial reporting tool. Despite them being leaders in the FP&A space, seeing their 13-week cash flow tools left me unconvinced. ----------- What the FP&A tools got right? (1) Cash flow forecasts were generated in a flash It was remarkable to see how quickly these tools can create a direct format 13-week cash flow. It took seconds. When you're needing to update a cash flow model, taking days or weeks to refresh a rolling forecast isn't an option. (2) Cash flow forecasts were traceable Many company cash flow models are driven by lots of data. Auditing Excel formulas isn't a great use of time for Treasurers or FP&As. These tools make it easy to vouch back to the root data and explore the detail. (3) Cash flows are good enough for companies that don't have to worry Some FP&As struggle to accept that top-down forecasts may be good enough for most companies that don't have to worry much about cash flow. That's because they're flush with liquidity, have a line or credit, and aren't laser-focused or hands-on with cash. A decent forecast that isn't remarkably accurate isn't always a liability. It's can be an asset since it's a reasonable-enough snapshot in time. ----------- What the FP&A tools get wrong? (4) Forecasts use past data and trends for almost all assumptions about the future If managing cash flow for a business that's seasonal, volatile, or has cash flow issues, relying on past data and trends can be reckless and lazy. When it comes to cash flow management, relying too much on historical trends can lead to really poor assumptions. If decisions are based on those bad assumptions, you get bad decisions too. (5) Forecasts were mostly observational, not prescriptive Unless you're working with a large corporation, where operations are steady and bank accounts are full, cash flow forecasts should enable thoughtful choices. That means the model should reveal operational drivers, opportunities, and scenarios. These tools don't really allow for these basic features. They're mostly just reports and data extrapolations. (6) Forecasts didn't capture nuance In the example I show here, my cash flow model can quickly and easily incorporate actuals, weekly and monthly forecast periods. I'm able to hold back 20% of accounts payable. I can pay back the A/P at any rate and timing that I want. I can be aggressive with catch-up payments and early-payment discounts. It's what a company needs to be able to see, whether it's doing $20 million or $200 million in revenue. It's not that AI can't do cash flow forecasting and modeling. It's that it can't do it as well as you'd hope. And that's the problem. Cash flows are full of nuance. AI-driven cash flow forecasts aren't great at understanding nuance. You can learn cash flows with me live: https://lnkd.in/grQVkeyE

Modeling something like time series goes past just throwing features in a model. In the world of time series data, each observation is associated with a specific time point, and part of our goal is to harness the power of temporal dependencies. Enter autoregression and lagging -  concepts that taps into the correlation between current and past observations to make forecasts.  At its core, autoregression involves modeling a time series as a function of its previous values. The current value relies on its historical counterparts. To dive a bit deeper, we use lagged values as features to predict the next data point. For instance, in a simple autoregressive model of order 1 (AR(1)), we predict the current value based on the previous value multiplied by a coefficient. The coefficient determines the impact of the past value on the present one only one time period previous. One popular approach that can be used in conjunction with autoregression is the ARIMA (AutoRegressive Integrated Moving Average) model. ARIMA is a powerful time series forecasting method that incorporates autoregression, differencing, and moving average components. It's particularly effective for data with trends and seasonality. ARIMA can be fine-tuned with parameters like the order of autoregression, differencing, and moving average to achieve accurate predictions. When I was building ARIMAs for econometric time series forecasting, in addition to autoregression where you're lagging the whole model, I was also taught to lag the individual economic variables. If I was building a model for energy consumption of residential homes, the number of housing permits each month would be a relevant variable. Although, if there’s a ton of housing permits given in January, you won’t see the actual effect of that until later when the houses are built and people are actually consuming energy! That variable needed to be lagged by several months. Another innovative strategy to enhance time series forecasting is the use of neural networks, particularly Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) networks. RNNs and LSTMs are designed to handle sequential data like time series. They can learn complex patterns and long-term dependencies within the data, making them powerful tools for autoregressive forecasting. Neural networks are fed with past time steps as inputs to predict future values effectively. In addition to autoregression in neural networks, I also used lagging there too! When I built an hourly model to forecast electric energy consumption, I actually built 24 individual models, one for each hour, and each hour lagged on the previous one. The energy consumption and weather of the previous hour was very important in predicting what would happen in the next forecasting period. (this model was actually used for determining where they should shift electricity during peak load times). Happy forecasting!

Enterprise demand forecasting isn't getting any easier. More SKUs, more sales channels and shorter product lifecycles mean traditional forecasting approaches are finding it harder to keep up. In our latest blog, we introduce MMF Agent - a guided AI workflow built on Databricks' Many Model Forecasting framework that brings advanced, multi-model forecasting to teams without requiring deep data science expertise. By guiding users through data preparation, model evaluation and deployment, MMF Agent can reduce days of specialist setup to just hours. More importantly, it helps demand planning teams apply sophisticated forecasting techniques using the tools and talent they already have. Are you already using Many Model Forecasting? This new MMF agent is worth a look. 👇 https://lnkd.in/ehmUFfk5 Great collaboration with Ryuta Yoshimatsu, Puneet Jain, Lourdes MARTINEZ and Lucas B.. 

Why Do Multi-Agent LLM Systems “still” Fail? A new study explores why Multi Agent Systems are not significantly outperforming single-agent. The study identifies 14 failure modes multi-agent system. Multi-agent system (MAS) are agents that interact, communicate, and collaborate to achieve a shared goal, which would to be difficult or unreliable for a single agent to accomplish. Benchmark: - Selected five popular, open-source MAS (MetaGPT, ChatDev, HyperAgent, AppWorld, AG2) - Chose tasks representative of the MAS intended capabilities (Software D Development, SWE-Bench Lite, Utility Service Tasks, GSM-Plus) total of 150 tasks - Recorded the complete conversation logs, human annotators reviews, Cohen's Kappa score to ensure consistency and reliability, LLM-as-a-Judge Validation Multi Agent Failure modes: 1. Disobey Task Spec: Ignores task rules and requirements, leading to wrong output. 2. Disobey Role Spec: Agent acts outside its defined role and responsibilities. 3. Step Repetition: Unnecessarily repeats steps already completed, causing delays. 4. Loss of History: Forgets previous conversation context, causing incoherence. 5. Unaware Stop: Fails to recognize task completion, continues unnecessarily. 6. Conversation Reset: Dialogue unexpectedly restarts, losing context and progress. 7. Fail Clarify: Does not ask for needed information when unclear. 8. Task Derailment: Gradually drifts away from the intended task objective. 9. Withholding Info: Agent does not share important, relevant information. 10. Ignore Input: Disregards or insufficiently considers input from others. 11. Reasoning Mismatch: Actions do not logically follow from stated reasoning. 12. Premature Stop: Ends task too early before completion or information exchange. 13. No Verification: Lacks mechanisms to check or confirm task outcomes. 14. Incorrect Verification: Verification process is flawed, misses critical errors. How to improve Multi-Agent LLM System: 📝 Define tasks and agent roles clearly and explicitly in prompts. 🎯 Use examples in prompts to clarify expected task and role behavior. 🗣️ Design structured conversation flows to guide agent interactions. ✅ Implement self-verification steps in prompts for agents to check their reasoning. 🧩 Design modular agents with specific, well-defined roles for simpler debugging. 🔄 Redesign topology to incorporate verification roles and iterative refinement processes. 🤝 Implement cross-verification mechanisms for agents to validate each other. ❓ Design agents to proactively ask for clarification when needed. 📜 Define structured conversation patterns and termination conditions. Github: https://lnkd.in/ebmCg28d Paper: https://lnkd.in/etgsH6BH

𝐃𝐢𝐝 𝐲𝐨𝐮 𝐤𝐧𝐨𝐰 𝐋𝐋𝐌 𝐡𝐚𝐥𝐥𝐮𝐜𝐢𝐧𝐚𝐭𝐢𝐨𝐧𝐬 𝐜𝐚𝐧 𝐛𝐞 𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐝 𝐢𝐧 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞? In a recent post, I talked about why hallucinations happen in LLMs and how they affect different AI applications. While creative fields may welcome hallucinations as a way to spark out-of-the-box thinking, business use cases don’t have that flexibility. In industries like healthcare, finance, or customer support, hallucinations can’t be overlooked. Accuracy is non-negotiable, and catching unreliable LLM outputs in real-time becomes essential. So, here’s the big question: 𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐜𝐚𝐥𝐥𝐲 𝐦𝐨𝐧𝐢𝐭𝐨𝐫 𝐟𝐨𝐫 𝐬𝐨𝐦𝐞𝐭𝐡𝐢𝐧𝐠 𝐚𝐬 𝐜𝐨𝐦𝐩𝐥𝐞𝐱 𝐚𝐬 𝐡𝐚𝐥𝐥𝐮𝐜𝐢𝐧𝐚𝐭𝐢𝐨𝐧𝐬? That’s where the 𝐓𝐫𝐮𝐬𝐭𝐰𝐨𝐫𝐭𝐡𝐲 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥 (𝐓𝐋𝐌) steps in. TLM helps you detect LLM errors/hallucinations by scoring the trustworthiness of every response generated by 𝐚𝐧𝐲 LLM.  This comprehensive trustworthiness score combines factors like data-related and model-related uncertainties, giving you an automated system to ensure reliable AI applications. 🏁 The benchmarks are impressive. TLM reduces the rate of incorrect answers from OpenAI’s o1-preview model by up to 20%. For GPT-4o, that reduction goes up to 27%. On Claude 3.5 Sonnet, TLM achieves a similar 20% improvement. Here’s how TLM changes the game for LLM reliability: 1️⃣ For Chat, Q&A, and RAG applications: displaying trustworthiness scores helps your users identify which responses are unreliable, so they don’t lose faith in the AI. 2️⃣ For data processing applications (extraction, annotation, …): trustworthiness scores help your team identify and review edge-cases that the LLM may have processed incorrectly. 3️⃣ The TLM system can also select the most trustworthy response from multiple generated candidates, automatically improving the accuracy of responses from any LLM. With tools like TLM, companies can finally productionize AI systems for customer service, HR, finance, insurance, legal, medicine, and other high-stakes use cases.  Kudos to the Cleanlab team for their pioneering research to advance the reliability of AI. I am sure you want to learn more and use it yourself, so I will add reading materials in the comments!