The idea

P2P inference for open-source LLMs

Language models run their input through a long stack of transformer layers. Language Pipes cuts that stack into segments and hands each segment to a different machine, so the memory cost is shared across a network you control. No single node needs to hold the whole model and no central server sits between your nodes. It's peer-to-peer, decentralized, and Python-native.

Why this instead of llama.cpp, exo, or vLLM? →

Why Language Pipes

Distributed · Decentralized · Standardized API

How it works

Layer models and the End model

Inference flows through a pipeline. The End model keeps the text-handling stages (tokenization, embedding, final norm and the output head) on one trusted node. The transformer layers in between are distributed across Layer models on other machines, which only ever see continuous-valued hidden-state tensors.

Each layer performs matrix multiplications between learned weights and a hidden-state tensor, then passes the result down the pipe. Splitting where the layers are hosted shares the memory cost and keeps text off every node but the one making the request.
See the job processor state machine →
Understand the threat model →

Quick start

Two nodes, two config files

Distribute Qwen/Qwen3-1.7B across two machines. Write a TOML config for each node and run it headless — no clicking through the TUI. Node 1 hosts the End Model, so prompts and responses stay on Node 1, plus enough layers to fit its memory. Node 2 hosts the rest and joins Node 1's network.

# node-1.toml — hosts the End model, so text stays here
node_id = "node-1"
job_port = 8000              # OpenAI-compatible API
end_models = ["Qwen/Qwen3-1.7B"]

[[layer_models]]
model_id = "Qwen/Qwen3-1.7B"
device = "cpu"
memory = 2

# node-2.toml — hosts the remaining layers only
node_id = "node-2"

[[layer_models]]
model_id = "Qwen/Qwen3-1.7B"
device = "cuda:0"
memory = 4

[[bootstrap_nodes]]
address = "192.168.0.1"   # node-1's LAN IP
port = 5000

Start each node from its config file:

# on machine 1
$ language-pipes -c node-1.toml run

# on machine 2
$ language-pipes -c node-2.toml run

Once both nodes have loaded their layers, call Node 1 like any OpenAI endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",  # node-1 IP + job port
    api_key="not-needed",
)

resp = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[{"role": "user",
               "content": "Write a haiku about distributed systems."}],
)
print(resp.choices[0].message.content)

Prefer a guided setup? Run language-pipes with no arguments to configure each node in the interactive TUI instead.

Support

Supported models

Built on

Reusable standalone packages

Two building blocks of Language Pipes are published as their own MIT-licensed packages on PyPI, maintained from this repo. Use them on their own, or let Language Pipes pull them in. It pins exact versions of both, so a language-pipesupgrade always installs the matching releases.

On PyPI:llm-layer-collector ·distributed-state-network