MLX-Node runs MLX models from Node.js. It includes language and multimodal inference, an HTTP server, model conversion, and GRPO/SFT training. The public APIs are TypeScript; the native layer is Rust and C++. Python is not required at runtime.
The main target is Apple Silicon with Metal. There is also an experimental CUDA backend for Linux on NVIDIA GB10 / DGX Spark.
The CLI requires Node.js 22.19 or newer. The published native binary requires macOS 26 or newer on Apple Silicon.
npm install --global @mlx-node/cli
mlx download model --model Qwen/Qwen3-0.6B
mlx serve --port 8080Models downloaded by the CLI are stored in ~/.mlx-node/models by default. Test the server with:
curl http://127.0.0.1:8080/v1/responses \
-H 'content-type: application/json' \
-d '{"model":"qwen3-0.6b","input":"Write a haiku about TypeScript."}'The CLI also includes two ways to run a coding agent against a local model:
mlx agent
mlx launch claudeRun mlx --help for the full command list. See docs/cli.md for command options and model conversion examples.
Install the language-model package directly if you do not need the CLI:
npm install @mlx-node/lmimport { homedir } from "node:os";
import { join } from "node:path";
import { loadSession } from "@mlx-node/lm";
const modelPath = join(homedir(), ".mlx-node", "models", "qwen3-0.6b");
const session = await loadSession(modelPath);
const first = await session.send("Write a haiku about TypeScript.");
console.log(first.text);
const followUp = await session.send("Make it shorter.");
console.log(followUp.text);ChatSession owns the conversation and KV cache. The same API supports regular turns, streaming, tool results, and reset:
send()sendStream()sendToolResult()reset()
See docs/models.md for loading options and model-specific behavior.
| Type | Models | Notes |
|---|---|---|
| General | Qwen3, Qwen3.5/3.6 Dense and MoE, Gemma4, LFM2, LFM2.5 | Qwen models support GRPO and SFT; others inference-only |
| Vision | Qwen3.5/3.6 VLM, Gemma4 VLM, PaddleOCR-VL, Qianfan OCR, PP-StructureV3 | General vision, OCR, and document processing |
| Audio | Qwen3-ASR | Offline, streaming, and meeting transcription |
| Embedding | Harrier | Embedding inference |
The detailed support matrix is in docs/models.md.
mlx serve scans the models directory and loads a model when it is first requested. One model is resident at a time.
mlx serve
mlx serve --port 8080 --model qwen3-0.6b
mlx serve --host 0.0.0.0 --auth-token "$(openssl rand -hex 16)"The server implements:
POST /v1/responsesPOST /v1/messagesPOST /v1/messages/count_tokensGET /v1/modelsGET /healthandGET /v1/health
Paged text models use continuous batching when the model supports it. Media turns and request-specific speculative decoding stay on ordered paths. The server also bounds admission and callback queues, propagates cancellation, and honors SSE backpressure. See docs/concurrent-inference.md for the scheduler details.
The training package contains GRPO and SFT trainers. This is a small GRPO example using the GSM8K dataset downloaded by mlx download dataset:
import { homedir } from "node:os";
import { join } from "node:path";
import { GRPOTrainer, loadLocalGsm8kDataset } from "@mlx-node/trl";
const trainer = await GRPOTrainer.create({
modelPath: join(homedir(), ".mlx-node", "models", "qwen3-0.6b"),
outputDir: "outputs/grpo",
groupSize: 4,
lossType: "grpo",
rewardFunction: async (outputs) =>
outputs.map(({ completion }) =>
completion.text.includes("correct") ? 1 : 0,
),
});
const dataset = await loadLocalGsm8kDataset("train", { limit: 100 });
await trainer.train(dataset);Available GRPO loss types are grpo, dapo, dr_grpo, and bnpo. Training supports custom and built-in rewards, gradient accumulation, checkpoint resume, and the Adam/AdamW, SGD, and RMSprop optimizers.
The repository also contains a Ratatui frontend for watching and controlling a training run:
cargo run -p mlx-tui -- \
--import '@oxc-node/core/register' \
--script ./examples/grpo/train-github-tool.ts
See docs/training.md for GRPO, SFT, datasets, checkpointing, and the TUI protocol.
| Platform | Backend | Status |
|---|---|---|
| macOS, Apple Silicon | Metal | Inference, training, and multimodal |
| Linux aarch64, NVIDIA GB10 | CUDA | Experimental, inference only |
The npm darwin-arm64 binary has a macOS 26.0 deployment target. It does not load on macOS 15 or older. The binary contains NAX kernels for M5-class GPUs; MLX enables them on macOS 26.2 or newer. A local source build works on macOS 14 or newer and can set its deployment target with MACOSX_DEPLOYMENT_TARGET.
The CUDA path has been tested with Qwen3.6 27B Dense and 35B-A3B MoE on GB10 / DGX Spark (sm_121, CUDA 13.0). It currently uses eager fallbacks and has no mlx-node-specific CUDA kernels. Training, speculative decoding, x86_64 Linux, and prebuilt CUDA binaries are not supported.
Build it on an aarch64 glibc host with CUDA 13.0 and nvcc on PATH:
yarn install --immutable
yarn build:nativePaged attention is Metal-only, so CUDA inference must use the eager path:
MLX_QWEN35_FORCE_EAGER=1 MLX_QWEN35_PAGED_OVERRIDE=0 \
oxnode examples/lm.ts Qwen3.6-27B-UD-Q4_K_XL-mlxMeasured results and the test setup are in docs/cuda-poc-benchmark.md.
The full workspace requires Node.js 22.19 or newer and Rust 1.89 or newer.
git clone --recurse-submodules https://github.com/mlx-node/mlx-node.git
cd mlx-node
yarn install --immutable
yarn buildUseful development commands:
yarn build:native
yarn build:ts
yarn test
yarn typecheck
yarn lintUse yarn build:native to build the Node addon. A direct cargo build does not produce it. More development notes are in CONTRIBUTING.md and docs/architecture.md.
@mlx-node/lm: model loading and chat sessions@mlx-node/server: HTTP inference server@mlx-node/agent: local coding agent@mlx-node/trl: GRPO and SFT training@mlx-node/vlm: vision-language models and document pipelines@mlx-node/asr: speech recognition@mlx-node/privacy: local PII detection and redaction@mlx-node/cli: CLI commands@mlx-node/core: internal native bindings