Algorithmic Audio Processing

Neural audio codecs have transformed how we process and generate audio, but most struggle significantly with out-of-domain content, leading to error propagation in downstream tasks. Meta's ComplexDec tackles this fundamental challenge by modelling audio in the complex spectral domain. The researchers demonstrate that information loss from temporal and dimensional compression is a key factor degrading out-of-domain robustness. What makes ComplexDec particularly impressive is its architectural approach. Unlike waveform-based codecs, it adopts a fully convolutional architecture without downsampling layers to bypass temporal compression. By coding speech in the complex spectral domain with a 150Hz frame rate and 256-dimension embedding, it maintains the same 24kbps bitrate as baseline models while dramatically reducing compression ratio. The results speak for themselves—trained on just 30 hours of reading-style VCTK corpus speech, ComplexDec shows remarkable robustness when tested on expressive speech from the EARS dataset. It achieves similar performance in both in-domain and out-of-domain scenarios, while baselines like AudioDec and ScoreDec suffer significant degradation. This breakthrough suggests that merely scaling up training data isn't enough to solve the out-of-domain robustness problem. The architecture's fundamental approach to minimising information loss proves more effective than throwing more data at the problem. Paper: https://lnkd.in/eJdaRAus Demo: https://lnkd.in/eZVMRi5G Authors: Yi-Chiao Wu, Dejan Marković, Steven Krenn, Israel D. Gebru, and Alexander Richard #NeuralAudioCodecs #Meta #AudioAI

🌟 Adaptive Filters: A Fascinating Application of Statistical Signal Processing 🌟 Adaptive filters are one of the most exciting applications of statistical and adaptive signal processing. Unlike normal FIR filters 🎛️—designed with fixed coefficients to remove specific unwanted frequencies like noise or interference—adaptive filters adapt dynamically to unknown signal characteristics while meeting specific performance metrics 📊. This is where statistical signal processing shines. ✨ 📌 One example is the Wiener filter (LMMSE), which I’ve discussed previously. Another is its gradient descent version, offering a recursive implementation of the Wiener filter. However, these optimal LMMSE estimators require knowledge of input and desired signal statistics, which isn’t always feasible in real-world scenarios. 🔑 Enter stochastic gradient descent (SGD): • A practical alternative that estimates the gradient, with the LMS adaptive filter algorithm being the simplest example. • The LMS algorithm provides an unbiased gradient estimator, allowing the solution to fluctuate around the optimal MMSE solution (hence the name least mean square). ✨ Key benefits of LMS: • 📡 Adaptability: Tracks variations in the channel and adjusts to changing conditions (e.g., in wireless communication). • ⚙️ Simplicity: A lightweight yet effective estimator. For scenarios involving sudden input signal variations, the normalized LMS (NLMS) is a robust choice, as it’s less sensitive to input signal statistics 📉. Simon Haykin’s book Adaptive Filter Theory presents an elegant proof where NLMS minimizes changes in filter weights, enabling superior performance under dynamic conditions. 🌍 Applications of adaptive filters span: • 📶 Telecommunications • ❤️ Biomedical signal processing • 🎧 Audio processing & echo cancellation • 🚫 Noise/interference cancellation • 🏗️ System identification The beauty of adaptive filters lies in their versatility: by redefining input, output, and desired signals, you can solve a variety of problems using the same foundational principles. This makes it one of the most fascinating topics in signal processing in my opinion!

MOSS-Audio-Tokenizer: The Foundation Behind Next-Gen Voice AI 🐠 I. Why Audio Tokenizers Matter Before an LLM can understand or generate speech, audio must be converted into compact tokens—just like text is converted into words or subwords. MOSS-Audio-Tokenizer is designed to be that bridge between raw audio and AI models. Instead of processing huge audio waveforms directly, it compresses audio into discrete tokens that can be modeled by transformers. II. Architecture Overview The diagram shows a fully Transformer-based audio tokenizer pipeline: 1. Audio Input Accepts speech, music, and general sounds. 2. Causal Transformer Encoder Converts audio into compact latent representations. Built entirely using Transformer blocks rather than CNN-heavy architectures. 3. RVQ-32 (Residual Vector Quantization) Compresses audio into discrete tokens. Uses 32 quantization layers for high-fidelity reconstruction. Supports variable bitrate encoding. 4. Causal Transformer Decoder Reconstructs audio from tokens. Optimized jointly with the encoder. 5. Decoder-Only LLM Learns semantic understanding from audio tokens. Enables downstream tasks like ASR, captioning, and TTS. III. Key Technical Highlights 1. Compresses 24 kHz audio to only 12.5 frames/sec. 2. Supports streaming-friendly inference. 3. Trained on roughly 3 million hours of audio. 4. Handles speech, music, and environmental sounds in a unified framework. 5. Learns semantic and acoustic information jointly. IV. Why It Is Important Traditional audio codecs focus only on reconstruction quality. MOSS-Audio-Tokenizer is different because its tokens preserve meaning, making them useful for: ASR (Speech-to-Text) TTS (Text-to-Speech) Voice Agents Audio Understanding Audio Captioning Native Audio LLMs V. Key Takeaway MOSS-Audio-Tokenizer is doing for audio what BPE tokenization did for text. Instead of feeding raw waveforms into AI systems, it converts speech, music, and sounds into compact semantic-rich tokens that transformers can efficiently process. This makes it one of the foundational building blocks for future real-time voice agents and audio-native LLMs.

Tencent just open-sourced a 7B model that takes audio in and produces audio out. End-to-end. One model. No ASR → LLM → TTS pipeline. No cascading. No error propagation between stages. Audio in. Audio out. Single unified architecture. ━━━━━━━━━━ Covo-Audio is a large audio language model that directly processes continuous audio and generates audio within one framework. Not a text model with speech bolted on. The model natively understands and generates speech — including paralinguistic cues like emotion, speaker identity, age, and tone. ━━━━━━━━━━ The architecture innovation: Hierarchical Tri-modal Interleaving Most audio LLMs either discretize speech into tokens (losing prosody) or process continuous features (losing semantic structure). Covo-Audio does both simultaneously: → Continuous acoustic features — preserves fine-grained prosodic nuances → Discrete speech tokens — maintains structured representation for generation → Natural language text — anchors semantic reasoning All three modalities are interleaved in a unified sequence. The model reasons across all of them at once. Built on Qwen2.5-7B backbone with Whisper-large-v3 as the audio encoder. ━━━━━━━━━━ The problem nobody talks about in voice AI: Intelligence-Speaker Coupling When you train an end-to-end voice model, the model's intelligence gets entangled with the speaker's voice. Want a different voice? You lose dialogue quality. Want better reasoning? You're stuck with one voice. Covo-Audio decouples them. Train intelligence and voice rendering separately. Then transfer any high-quality TTS voice with minimal data — without degrading dialogue performance. This is how you ship one model to enterprise clients who each need their own brand voice. ━━━━━━━━━━ Two variants, two use cases: → Covo-Audio-Chat — half-duplex spoken dialogue. Understanding, contextual reasoning, instruction following, empathetic responses. SOTA at 7B scale. → Covo-Audio-Chat-FD — native full-duplex. Both sides talk and listen simultaneously. Barge-in support. Backchannel signals. Low-latency turn-taking. Full-duplex is what makes voice AI feel like a real conversation — not a walkie-talkie. ━━━━━━━━━━ 🔗 HuggingFace: https://lnkd.in/g6-TDqqp 🔗 GitHub: https://lnkd.in/gsjbThrB 🔗 Paper: https://lnkd.in/g-WAgJ87 ━━━━━━━━━━ Deploying voice AI agents for enterprise call centers, IVR, or multilingual customer workflows? Need custom voice model training or full-duplex voice integration? DM me. ♻️ Repost if end-to-end audio language models replacing ASR→LLM→TTS pipelines is the future of voice AI 🔔 Follow Pasha S for drops like this daily

The latest AI that lets you edit sound by pointing at things. Meta just released SAM Audio, and it works in ways that feel impossible. You're recording a podcast in a coffee shop. There's espresso machine noise in the background. Instead of opening audio software and tweaking equalizers for 20 minutes, you just mark the time when the machine starts and tell SAM Audio "espresso machine." It isolates and removes it. Or you've got footage of a street musician. You want just the guitar, not the traffic. Click on the guitarist in the video. Done. The system extracts that specific instrument. It works three ways: - Text prompts – type "singing voice" or "train passing" - Visual selection – click the sound source in a video frame - Time marking – highlight when a sound starts, label it No waveforms. No filters. No technical knowledge required. This is the Segment Anything model applied to audio. The same AI framework that made image editing conversational is now doing it for sound. Point, describe, or mark and the system figures out the rest. It's early. The model still struggles with similar overlapping sounds (like isolating one violin in an orchestra). But the direction is clear. We're moving toward interfaces where you don't need to understand how the machine works. You just need to understand what you want

StepFun AI Releases Step-Audio-EditX: A New Open-Source 3B LLM-Grade Audio Editing Model Excelling at Expressive and Iterative Audio Editing How can speech editing become as direct and controllable as simply rewriting a line of text? StepFun AI has open sourced Step-Audio-EditX, a 3B parameter LLM based audio model that turns expressive speech editing into a token level text like operation, instead of a waveform level signal processing task. Step-Audio-EditX reuses the Step-Audio dual codebook tokenizer. Speech is mapped into two token streams, a linguistic stream at 16.7 Hz with a 1024 entry codebook, and a semantic stream at 25 Hz with a 4096 entry codebook. Tokens are interleaved with a 2 to 3 ratio. The tokenizer keeps prosody and emotion information, so it is not fully disentangled. On top of this tokenizer, the StepFun research team builds a 3B parameter audio LLM. The model is initialized from a text LLM, then trained on a blended corpus with a 1 to 1 ratio of pure text and dual codebook audio tokens in chat style prompts. The audio LLM reads text tokens, audio tokens, or both, and always generates dual codebook audio tokens as output...... Full analysis: https://lnkd.in/g9CujMWF Paper: https://lnkd.in/gcUd6Ddz Repo: https://lnkd.in/gpCXKf5m Model weights: https://lnkd.in/gkqUc83S StepFun

You are in a loud coffee shop. People are talking, the espresso machine is screaming, and yet you hold up your phone and Shazam IDs the exact song playing in three seconds. How does it actually do that? Finding one track out of 100 million while completely ignoring background chaos requires serious backend architecture. Uploading a raw audio recording to compare waveforms against a massive database is out of the question. A brute-force approach like that would melt the servers before the song even finished playing. The actual system relies heavily on signal processing and aggressive data reduction. First, the algorithm abandons the concept of audio waves entirely. It uses a Fast Fourier Transform to convert the sound from a timeline into a frequency map, basically turning audio into a 3D picture. To ignore the espresso machines and conversations, the app throws away almost all of the data. It only keeps the absolute loudest peak frequencies. These are the strongest musical notes that actually survive the background noise. This reduces a massive audio file into a sparse scatter plot of dots called a constellation map. Searching a database for millions of single dots is still way too slow. Instead, the algorithm groups the dots into pairs. It takes the frequency of one note, the frequency of another, and the exact time difference between them, and packs it all into a tiny 32-bit integer. By pairing points, it turns a heavy audio search into an incredibly fast hash table lookup. When the app queries the database, it retrieves these pairs. If the time gaps from your phone recording perfectly match the time gaps in the original track, they form a perfect diagonal line on a graph. The background noise just drops away because random noise never mathematically aligns. Shazam isn't listening to music. It is searching for mathematically aligned geometric shapes. Backend and data engineers: when building massive search systems today, do you still reach for exact hash keys like this, or have you fully migrated over to vector embeddings and approximate nearest neighbor searches? #SystemDesign #SoftwareEngineering #AudioProcessing #Algorithms #BackendArchitecture #DataEngineering