Training Content Management Systems

Disclosing the full list of datasets used to train IBM LLMs Granite 3.0. This is true transparency - no other LLM provider shares such detailed information about their training datasets. WEB Data - FineWeb: More than 15T tokens of cleaned and deduplicated English data from CommonCrawl. - Webhose: Unstructured web content in English converted into machine-readable data. - DCLM-Baseline: A 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks. CODE - Code Pile: Sourced from publicly available datasets like GitHub Code Clean and StarCoderdata. - FineWeb-Code: Contains programming/coding-related documents filtered from the FineWeb dataset using annotation. - CodeContests: Competitive programming dataset with problems, test cases, and human solutions in multiple languages. DOMAIN - USPTO: Collection of US patents granted from 1975 to 2023. - Free Law: Public-domain legal opinions from US federal and state courts. - PubMed Central: Biomedical and life sciences papers. - EDGAR Filings: Annual reports from US publicly traded companies over 25 years. MULTILINGUAL - Multilingual Wikipedia: Data from 11 languages to support multilingual capabilities. - Multilingual Webhose: Multilingual web content converted into machine-readable data feeds. - MADLAD-12: Document-level multilingual dataset covering 12 languages. INSTRUCTIONS - Code Instructions Alpaca: Instruction-response pairs about code generation problems. - Glaive Function Calling: Dataset focused on function calling in real scenarios. ACADEMIC - peS2o: A collection of 40M open-access academic papers for pre-training. - arXiv: Scientific paper pre-prints posted to arXiv. Full author acknowledgement can be found here. - IEEE: Technical content from IEEE acquired by IBM. TECHNICAL - Wikipedia: Technical articles sourced from Wikipedia. - Library of Congress Public Domain Books: More than 140,000 public domain English books. - Directory of Open Access Books: Publicly available technical books from the Directory of Open Access Books. - Cosmopedia: Synthetic textbooks, blog posts, stories, and WikiHow articles. MATH - OpenWebMath: Mathematical text from the internet, filtered from 200B HTML files. - Algebraic-Stack: Mathematical code dataset including numerical computing and formal mathematics. - Stack Exchange: User-contributed content from the Stack Exchange network. - MetaMathQA: Dataset of rewritten mathematical questions. - StackMathQA: A curated collection of 2 million mathematical questions from Stack Exchange. - MathInstruct: Focused on chain-of-thought (CoT) and program-of-thought (PoT) rationales for mathematical reasoning. - TemplateGSM: Collection of over 7 million grade-school math problems with code and natural language solutions. BOOM!

You’re in an AI Engineer interview. Interviewer asks: How do you handle multi language prompting effectively? Most people jump to translation APIs. Strong answer goes deeper. 1. Detect language first Never assume. Identify the user’s language and script before prompting. 2. Preserve intent, not just words Literal translation often breaks tone, context, and business meaning. 3. Prompt in the user’s language when possible Models usually respond better when instructions and output language align. 4. Use English for complex reasoning, then localize output For harder logic tasks, reasoning in English + final response in target language often works better. 5. Handle mixed language inputs Real users switch languages mid sentence. Your system should too. 6. Keep terminology consistent Especially for healthcare, finance, legal, and product names. 7. Test by language, not globally Kannada, Hindi, Tamil, Japanese, Arabic, Spanish all fail differently. 8. Build fallback layers If confidence is low, ask clarifying questions instead of hallucinating. What interviewers want to hear: You understand that multilingual AI is a product problem, not just a translation problem. #AI #GenerativeAI #PromptEngineering #LLM #AIEngineer #MachineLearning #NLP #AIEngineering Follow Sneha Vijaykumar for more... 😊

Latest research from KAIST and Imperial College London introduces Zero-AVSR, an innovative framework that enables audio-visual speech recognition across languages without requiring training data in target languages. By learning language-agnostic speech representations through romanisation and leveraging LLMs, it can recognise speech even in languages never seen during training. What makes this approach interesting is the scale of language support. The team created MARC, a dataset spanning 2,916 hours of audio-visual speech across 82 languages—far beyond the 9 languages typical systems support. Their results show comparable performance to traditional multilingual systems while supporting this vastly larger language inventory. Zero-AVSR represents a significant advancement for speech tech in low-resource languages, potentially democratising access across thousands of languages without requiring extensive labelled datasets for each. The approach particularly excels when recognising languages from families similar to those in the training data, suggesting promising pathways for further expansion. Paper: https://lnkd.in/dnw_V7XK Authors: Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, Yong Man Ro #SpeechRecognition #MultilingualAI #SpeechAI

Exciting breakthrough in multilingual embedding models! A team of researchers from HIT and Tongji University have developed KaLM-Embedding, setting a new standard for models under 1B parameters. What makes this model special? It leverages cleaner, more diverse training data and introduces three game-changing techniques: 1. Persona-based synthetic data generation using QWen2-72B-Instruct, creating 550k diverse examples across 6 task types 2. Ranking consistency filtering to remove noise and improve data quality by ensuring positive examples rank within top-k matches 3. Semi-homogeneous task batching that balances negative sample hardness with false negative risks Under the hood, KaLM-Embedding uses Qwen2-0.5B as its foundation and implements Matryoshka Representation Learning for flexible dimension embedding (896 to 64 dimensions). The model excels in Chinese and English while showing strong performance across other languages. The results? KaLM-Embedding achieves state-of-the-art performance on the MTEB benchmark, outperforming larger models with scores of 64.13 for Chinese and 64.94 for English tasks. This work demonstrates how thoughtful data curation and innovative training techniques can push the boundaries of what's possible with compact models. The team has open-sourced their work for the research community.

ModernBERT goes MULTILINGUAL! One of the most requested models I've seen, The Johns Hopkins University's CLSP has trained state-of-the-art massively multilingual encoders using the ModernBERT architecture: mmBERT. Model details: - 2 model sizes: 42M non-embed (140M total) and 110M non-embed (307M total) - Uses the ModernBERT architecture, but with the Gemma2 multilingual tokenizer (so: flash attention, alternating global/local attention, unpadding/sequence packing, etc.) - Maximum sequence length of 8192 tokens, on the high end for encoders - Trained on 1833 languages using DCLM, FineWeb2, and many more sources - 3 training phases: 2.3T tokens pretraining on 60 languages, 600B tokens mid-training on 110 languages, and 100B tokens decay training on all 1833 languages. - Also uses model merging and clever transitions between the three training phases. - Both models are MIT Licensed, and the full datasets and intermediary checkpoints are also publicly released Evaluation details: - Very competitive with ModernBERT at equivalent sizes on English (GLUE, MTEB v2 English after finetuning) - Consistently outperforms equivalently sized models on all Multilingual tasks (XTREME, classification, MTEB v2 Multilingual after finetuning) - In short: beats commonly used multilingual base models like mDistilBERT, XLM-R (multilingual RoBERTa), multilingual MiniLM, etc. - Additionally: the ModernBERT-based mmBERT is much faster than the alternatives due to its architectural benefits. Easily up to 2x throughput in common scenarios. Check out the full blogpost with more details. It's super dense & gets straight to the point: https://lnkd.in/ebqTK3JS Based on these results, mmBERT should be the new go-to multilingual encoder base models at 300M and below. Do note that the mmBERT models are "base" models, i.e. they're currently only trained to perform Mask Filling. They'll need to be finetuned for downstream tasks like semantic search, classification, clustering, etc. I'm very much looking forward to seeing embedding models based on mmBERT! Great work by Marc Marone, Orion Weller, and the rest of the team at JHU!

One 600M model. 40 languages. Real-time streaming. Open weights, commercial license. NVIDIA just made the multilingual ASR decision a lot simpler. Nemotron 3.5 ASR is the multilingual successor to NVIDIA's English Nemotron ASR (40K+ downloads, top of the Artificial Analysis streaming board). Now it does 40 language-locales from a single checkpoint. Why it matters for voice agents and captioning: → One model, 40 locales — including Arabic — no per-language deployment, no model-swapping → Auto language detection — run it with target_lang=auto and one deployment transcribes mixed-language traffic, tagging each utterance. No separate language-ID component → Cache-aware FastConformer-RNNT — processes each audio frame once instead of re-chewing overlapping buffers, so ~17x the concurrent streams of buffered approaches on the same H100 → Latency you set at inference — anywhere from 80ms to 1.12s, trading speed for accuracy per use case → Punctuation and capitalization built in The open-weights part is the real unlock: NVIDIA's own blog shows a 31% relative WER drop on Greek and Bulgarian from a short fine-tune. You take the base model and adapt it to your domain, accent, or language — on your own hardware, with a commercial license. For anyone running multilingual voice on-prem, this collapses four problems — accuracy, latency, language coverage, deployment — into one model you can actually own and tune. Links in comments. ♻️ Repost if your team works on speech.

Does RLHF transfer to different languages? RLHF Can Speak Many Languages! Cohere shows that training one or multiple languages improves the performance of unseen languages and shows that online RLHF methods have stronger transfer capabilities than offline methods. 👀 Experiments: 1️⃣ Created synthetic multilingual preference dataset using ~50K English prompts from ShareGPT, translated to 22 languages. Completions were generated using Cohere's Command and Command R+ models, with Cohere May 2024 as the Reward Model. 2️⃣ Created 4 dataset Mixtures: EN-1-50K: English-only, 50K prompts; ML-5-50K: 5 languages, 10K prompts each; ML-23-50K: 23 languages, ~2.2K prompts each; ML-23-230K: 23 languages, 10K prompts each. 3️⃣ Used Aya 23 8B as Base Model (SFT) and for RLHF DPO (offline) and RLOO (online). 4️⃣ Trained on all mixtures with both methods and evaluated them using win-rates judged GPT-4-Turbo. Learnings: 🌐 Training only on English preference data leads to up to 7% performance improvements on other languages. 🌍 Training in 5 languages increased win rates in unseen languages by up to 19%. ⚡ Online (RLOO) outperforms offline (DPO) by up to 10.6% in average win-rates. 🔄 Online (RLOO) shows stronger language transfer capabilities than offline (DPO). 📈 Increasing data from 2K to 10K examples per language improves DPO but not RLOO. Paper: https://lnkd.in/eNG8FriC

🌍 I’m excited to share my largest research project to date: 𝗔𝗧𝗟𝗔𝗦 🗺️ - 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗧𝗿𝗮𝗻𝘀𝗳𝗲𝗿 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗮𝘄𝘀 𝗳𝗼𝗿 𝗠𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗣𝗿𝗲𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 This is the largest public multilingual pretraining experiment to date, laying the scientific foundation for scaling beyond English. Across 774 experiments, up to 8B parameter models, on 400+ languages, we are able to explicitly model the curse of 𝘮𝘶𝘭𝘵𝘪𝘭𝘪𝘯𝘨𝘶𝘢𝘭𝘪𝘵𝘺. Our key contributions: 🔹 𝗔𝗧𝗟𝗔𝗦 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗮𝘄𝘀 – Modeling monolingual and multilingual mixtures, we outperform prior work by large margins (often 30%+ R²) through explicit modeling of cross-lingual transfer, data repetition, and multilingual capacity limits. 🔹 𝟯𝟴×𝟯𝟴 𝗖𝗿𝗼𝘀𝘀-𝗟𝗶𝗻𝗴𝘂𝗮𝗹 𝗧𝗿𝗮𝗻𝘀𝗳𝗲𝗿 𝗠𝗮𝘁𝗿𝗶𝘅 – The most comprehensive empirical map of how language A helps or hinders language B during training (1,444 pairs). We find that shared scripts (e.g., Latin vs Cyrillic) often matter more than shared families (e.g., Indo-European vs Sino-Tibetan). 🔹 𝗗𝗲𝗰𝗼𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗖𝘂𝗿𝘀𝗲 𝗼𝗳 𝗠𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹𝗶𝘁𝘆 – We explicitly model how compute (C), model size (N), data (D), and language count (K) interact. Our fitted formulas tell practitioners how much to scale (N,D) when doubling the number of languages (K) to maintain the same performance. 🔹 𝗣𝗿𝗲𝘁𝗿𝗮𝗶𝗻 𝘃𝘀 𝗙𝗶𝗻𝗲𝘁𝘂𝗻𝗲 𝗚𝘂𝗶𝗱𝗮𝗻𝗰𝗲 – For builders training language-specific models (e.g., Greek), we identify the crossover points where—given enough compute—training from scratch surpasses starting from a multilingual checkpoint. [See the paper link in comments!] Grateful to collaborate with an amazing team, without which this work wouldn’t have been possible: Sneha Kudugunta, Niklas Muennighoff, I-Hung Hsu, Isaac Caswell, Alex 'Sandy' Pentland, Sercan Arık, Chen-Yu Lee, and Sayna Ebrahimi across Google, MIT, UW, and Stanford.

Excited to share some research I've worked on under Cohere Labs! We present LLINK, a compute efficient method to extend LLM multilinguality without changing the tokenizer or pre-training. In the same way LLaVA teaches models to learn images, we treat languages as modalities. We align a multilingual encoder with LLaMA-1B, significantly improving retrieval and conversational performance in low-resource languages. We further find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment. ArXiv Paper: https://lnkd.in/eY6ibngS Code: https://lnkd.in/eGbVhsUZ AlphaXiv: https://lnkd.in/e_YU8X2T Built with Aarush Gupta

Extending a text-to-speech model to new languages is no mean feat. You need to: 1. Find a 𝗺𝗼𝗱𝗲𝗹 𝗮𝗱𝗮𝗽𝘁𝗮𝗯𝗹𝗲 to multiple languages — Many TTS models rely on language-specific mechanisms, making adaptation challenging. 2. 𝗦𝗼𝘂𝗿𝗰𝗲 𝗵𝗶𝗴𝗵-𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗱𝗮𝘁𝗮𝘀𝗲𝘁𝘀 — Critical for achieving accurate, natural-sounding speech. 3. 𝗧𝗿𝗮𝗶𝗻 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲𝗹𝘆 — Knowing the right hyper-parameters isn’t always straightforward. I was thrilled to see that many Parler-TTS users managed to fine-tune it across diverse languages like 𝗪𝗼𝗹𝗼𝗳, 𝗞𝗼𝗿𝗲𝗮𝗻, 𝗙𝗿𝗲𝗻𝗰𝗵, 𝗮𝗻𝗱 𝗲𝘃𝗲𝗻 𝗧𝗵𝗮𝗶. Today, I’m even happier to introduce an even easier-to-use version of Parler-TTS for fine-tuning in other languages, alongside a 𝗻𝗲𝘄 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗣𝗮𝗿𝗹𝗲𝗿-𝗧𝗧𝗦 𝗺𝗼𝗱𝗲𝗹! This is a joint effort with our friends at 𝗤𝘂𝗮𝗻𝘁𝘂𝗺 𝗦𝗾𝘂𝗮𝗱𝗿𝗮 and 𝗔𝗜𝟰𝗕𝗵𝗮𝗿𝗮𝘁. What did we learn? 1/ Once the datasets are ready—which can be the toughest part—fine-tuning the model is 𝘀𝘂𝗿𝗽𝗿𝗶𝘀𝗶𝗻𝗴𝗹𝘆 𝗾𝘂𝗶𝗰𝗸 and straightforward. 2/ 𝗠𝗼𝗿𝗲 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲𝘀, 𝗯𝗲𝘁𝘁𝗲𝗿 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: Expanding the language range strengthens the model. 3/ 𝗛𝗶𝗴𝗵-𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝗻𝗻𝗼𝘁𝗮𝘁𝗶𝗼𝗻𝘀 make all the difference: Detailed data leads to better performance. 4/ The model is impressively 𝗮𝗱𝗮𝗽𝘁𝗮𝗯𝗹𝗲, even to new features like age, accent, and more. Links to try the new multilingual model are in the comments!