map-reduce

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Kubernetes-native platform to run massively parallel data/streaming jobs

A search engine which can hold 100 trillion lines of log data.

Efficient transducers for Julia

Fundamentals of Spark with Python (using PySpark), code examples

Parallelized Base functions

Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

Fast & furious GroupBy operations for dask.array

Data science and Big Data with Python

Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

Efficient and scalable parallelism using the message passing interface (MPI) to handle big data and highly computational problems.

Data-parallelism on CUDA using Transducers.jl and for loops (FLoops.jl)

The core parallel and shared memory library used by Hack, Flow, and Pyre

Apache Airflow-compatible workflow orchestrator, re-engineered in Go — the drop-in Airflow UI + REST API, but on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

pingcap 面试小作业

Global, unlimited persistent memory for Claude Code agents. Context-activated hints injected automatically via hooks using scatter-gather map-reduce.

Build scalable data pipelines on YTsaurus with automatic stage management, local development simulation, and more.

RedisGears python client

A lightweight modern map reduce framework brought to k8s