Pathway

Editor’s note: This Pathway review is based on my hands-on test of Pathway Live Data Framework 0.33.1, its Question-Answering RAG template and the serviette RAG server, in November 2026. I weighed pricing, setup, live sync, the LLM xpack, scaling, licensing and company direction against Apache Flink, Bytewax, Quix, LlamaIndex, LangChain, Unstructured, Airbyte and RisingWave.

Quick verdict: I recommend Pathway for Python teams whose RAG index or ETL pipeline must follow SharePoint, Google Drive, S3, Kafka or Postgres changes without re-indexing jobs. The trade-off: free nodes stop at 16 GB RAM and 4 cores, and exactly-once consistency is Enterprise-only.

The engine at the center of this Pathway review is a Rust core built on Differential Dataflow that recomputes only what changed. In this review, I’ll take a closer look at Pathway’s pricing, features, and licensing, so you can see exactly whether it fits your live-data pipelines or whether a stream processor like Apache Flink or a RAG framework like LlamaIndex is the better buy.

Key Takeaways

• One Rust engine runs the same Python code for batch, streaming and backfills.
• Community is free to 8 GB RAM and 4 cores per node, or 16 GB with a free Scale key.
• Built-in vector, BM25 and hybrid indexes mean no separate vector database.
• Exactly-once consistency and hot failover are Enterprise-only, at an unpublished price.
• BDH research now leads Pathway’s homepage, though 0.33.1 shipped in November 2026.

Pros and Cons

Pathway Pricing: How Much Does It Cost?

Pathway publishes three license tiers and no list prices for the Pathway Live Data Framework. Community and Scale are free to self-host, and every tier caps RAM and cores per node.

• Community (free): for developers and small production pipelines under BSL 1.1. Each node gets up to 8 GB RAM and 4 cores, with 20+ app templates, core connectors, 300+ sources through Airbyte and at-least-once consistency.
• Scale (free license key): for teams that need enterprise connectors. Each node gets up to 16 GB RAM and 4 cores, plus SharePoint, Delta Lake and Iceberg connectors, full persistence, OpenTelemetry monitoring with Grafana, SlideParser and extra templates. A GitHub or LinkedIn sign-in provides the key, and using it sends anonymized usage data to Pathway. Paid Scale keys cover larger nodes at unpublished prices.
• Enterprise (custom quote): for multi-node production, up to 24 TB RAM and 128 cores across 40 nodes. It adds exactly-once consistency, horizontal scaling, hot failover, managed hosting, a private package registry and a multi-zone SLA.

Tier	Price	Limit per node	What it adds	Support
Community	$0 under BSL 1.1	8 GB RAM, 4 cores	Core connectors, Python and SQL APIs, 20+ app templates	Discord and GitHub
Scale	$0 with free key; paid keys for larger nodes	16 GB RAM, 4 cores with free key	Enterprise connectors, full persistence, monitoring	Paid keys: tickets, 1-business-day reply
Enterprise	Custom quote	Up to 24 TB RAM and 128 cores across 40 nodes	Exactly-once, horizontal scaling, hot failover, managed services	24/7 phone support

The only published dollar figure is the AWS Marketplace annual contract for hosted Pathway services: $15,000 per unit per year, with tailored offers on request. A unit measures your yearly commitment.

Is Pathway Good Value for Money?

• License fees stay at $0 for any pipeline that fits 16 GB and 4 cores per node. A managed Apache Flink app on AWS bills $0.11 per KPU-hour in US East, plus $0.10 per GB-month of running storage and one extra KPU per application.
• RAM and model calls are the real bill. Pathway keeps the whole pipeline in memory, and embedding and answer calls go to your OpenAI or Gemini account unless you run local SentenceTransformers embeddings and an Ollama model.
• No per-document charge: Unstructured gives 10,000 free pages, then charges $0.015 a page, and LlamaParse charges $1.50 per 1,000 credits after 10,000 free credits. With Pathway, you run and patch the infrastructure yourself.
• Airbyte syncs on a schedule: Standard from $20 a month runs hourly syncs and Plus from $189 a month runs 15-minute syncs, while Pathway updates indexes as changes arrive.
• Scaling stays manual below Enterprise: non-enterprise users may not automate launching new instances, so scaling up or down needs a person.

Reviewer’s Notes: I recommend starting on Community, then adding the free Scale key the day you need SharePoint, full persistence or monitoring and can accept its usage telemetry. Open an Enterprise conversation only when one 16 GB node stops being enough or a pipeline needs exactly-once consistency.

My Experience With Pathway

Pathway installs with one command, pip install -U pathway, on Python 3.10 or newer, and only on macOS or Linux. There is no Windows wheel, so Windows teams run it in a VM, WSL or Docker.

Installing Pathway and Getting a License Key

I installed the core package first, then added the LLM tooling with pip install "pathway[xpack-llm]"; pathway[all] pulls every extra. Wheels cover x86_64 and arm64, and the pathwaycom/pathway Docker image is the container route.

For the free Scale key, I signed in with GitHub on Pathway’s get-license page (LinkedIn also works) and set the key as the PATHWAY_LICENSE_KEY environment variable; pw.set_license_key() does the same inside Python. Version 0.33.1 adds Python 3.15 for the core package only, so the ML and document extras do not support 3.15 yet.

Building My First Streaming Pipeline

I built the README’s Table API example, which runs unchanged as a batch job or a live stream:

1. I defined a pw.Schema with a value column.
2. I read a folder with pw.io.csv.read.
3. I filtered the rows and summed them with pw.reducers.sum.
4. I wrote the result with pw.io.jsonlines.write.
5. I started the pipeline with pw.run().

I ran it with python main.py, then with pathway spawn --threads 3 python main.py for three threads. The built-in monitoring dashboard shows messages per connector and latency while the pipeline runs.

Next, I passed a pw.persistence.Config with a filesystem backend to pw.run(); S3 is the other backend. Each source needs a stable unique name, so a restart resumes where the pipeline stopped.

Running the Question-Answering RAG Template

I cloned the Pathway repository and opened examples/llm-app/templates/question_answering_rag; the app templates moved there from the former llm-app repository, which is now serviette. I added OPENAI_API_KEY to .env, and the whole pipeline lives in app.yaml:

• Sources: a local data folder by default, with Google Drive and SharePoint options.
• Parsing and chunking: DoclingParser, then TokenCountSplitter.
• Retrieval: OpenAIEmbedder and a USearch KNN index.
• Answers: gpt-4.1-mini, set explicitly.

I started the app with docker compose up, and QASummaryRestServer served five endpoints on port 8000:

• /v1/retrieve
• /v1/statistics
• /v2/list_documents
• /v2/answer
• /v2/summarize

Files added, edited or deleted in the source folder flow through parsing, embedding and indexing without a restart. I then edited app.yaml to swap in a hybrid USearch KNN plus Tantivy BM25 index and to route answers to a local Mistral model through LiteLLM and Ollama, with no Python changes. A Streamlit UI folder ships with the template.

Trying serviette, the No-Code RAG Server

I installed serviette with pip install serviette on pip 23.1 or newer, exported the Pathway license key and an OpenAI key, and ran serviette demo. It opened a chat on local port 8989 over a bundled sample corpus.

Then serviette wizard generated a config.yaml, and serviette up started the indexer and the stateless server, with the REST API under /api/v1 and DuckDB as the default store. The server answers one question at a time, with no conversation memory, user accounts or hosted offering yet.

Reviewer’s Notes: I recommend always passing an explicit model to OpenAIChat. It still defaults to gpt-3.5-turbo, which OpenAI retired on October 23, 2026, so a bare OpenAIChat() call fails; the Question-Answering template already sets gpt-4.1-mini.

Live Sync: Connectors, Incremental Indexing and Consistency

Pathway’s Differential Dataflow engine recomputes only changed rows and documents, so an edited SharePoint file flows through parsing, embedding and indexing without a full rebuild. LangChain’s RecordManager also skips unchanged content by hash, but only when you trigger a re-index.

Connectors are gated by license tier:

• Community: Kafka, Redpanda, PostgreSQL, S3, filesystem, Google Drive, Google PubSub, HTTP, JSON Lines, Logstash, Slack and custom Python inputs, plus 300+ sources through Airbyte.
• Scale: SharePoint, Delta Lake, Iceberg, BigQuery, Elasticsearch and QuestDB, plus Apache Pulsar, which 0.33.0 added for Scale and Enterprise.
• Enterprise: custom messaging formats, such as MQTT for IoT.

Since 0.32.1 (July 2026), native output connectors keep these vector stores in sync with inserts, updates and deletes:

• Chroma
• Qdrant
• DuckDB
• Weaviate
• Pinecone

Version 0.33.1 also reads PostgreSQL change streams through replication slots, without Debezium and Kafka in between.

Every input carries a timestamp, every output states which input times it reflects, and Pathway emits no intermediate results; late or out-of-order data updates results as it arrives. The delivery guarantee depends on the license: at-least-once on Community and Scale, exactly-once on Enterprise only.

Amazon Managed Service for Apache Flink includes exactly-once state at $0.11 per KPU-hour in US East, a guarantee Pathway reserves for Enterprise.

Persistence combines snapshots and a write-ahead log on a filesystem or S3; Community persists to either, and full persistence is a Scale feature. Since 0.33.0, an MQTT reader with persistence and QoS 1 or 2 needs a stable client_id, a breaking change that stops restarts from silently losing messages.

Best for: Python pipelines that mix changing documents with structured streams. Skip if: exactly-once recovery is mandatory and an Enterprise contract is off the table.

The LLM xpack: Parsers, Indexes, Adaptive RAG and MCP

The LLM xpack ships parsers, splitters, embedders, indexes, rerankers and two RAG question answerers as components of the same streaming pipeline, so only changed documents are re-parsed and re-embedded. Ready templates wire these parts for Adaptive RAG, Private RAG with Mistral and Ollama, Multimodal RAG with GPT-4o and Slides AI Search.

Parsers and chunking

• DoclingParser: PDF text, tables and images; tables go through Docling OCR or a multimodal LLM with an OpenAI-compatible API.
• UnstructuredParser: five character-based chunking modes.
• Utf8Parser, PypdfParser, PaddleOCR and ImageParser: plain text, PDF and image options.
• SlideParser: a Scale feature that reads PPTX and PDF decks with a vision LLM.
• TwelveLabsVideoParser: videos from 4 seconds to 2 hours, up to 2 GB, with processing billed by TwelveLabs.
• TokenCountSplitter: token-based chunking for retrieval.

Embedders, indexes and rerankers

Component	Available options	Buying implication
Embedders	OpenAI, LiteLLM, Gemini, SentenceTransformers	Local embeddings are possible
Indexes	Brute-force KNN, USearch HNSW, Tantivy BM25, hybrid	Built-in indexes live in memory
Rerankers	LLMReranker, CrossEncoderReranker, EncoderReranker	LLMReranker uses a 1-5 scale
Chat wrappers	OpenAI, HuggingFace, Cohere, LiteLLM	LiteLLM includes Azure OpenAI and Gemini

USearch KNN is the Question-Answering template’s default index, and the hybrid index adds Tantivy BM25 keyword matching. Every index lives in RAM, so skipping a separate Qdrant or Pinecone service spends the node’s 8 GB or 16 GB allowance instead.

SentenceTransformers keeps embeddings local, with no embedding API calls; in serviette, the local-embeddings install pulls in PyTorch, about 0.8 GB for the CPU build.

Adaptive RAG and MCP

AdaptiveRAGQuestionAnswerer starts with a few retrieved documents and adds more only until it finds an answer, to cut token use; BaseRAGQuestionAnswerer is standard top-k retrieval.

The MCP server, PathwayMcp, exposes tools and a real-time document store over streamable HTTP. It needs a license key, and the free key works.

LlamaIndex and LangChain integrations let either framework use Pathway as its live retrieval layer. LlamaIndex’s IngestionPipeline also skips unchanged documents and upserts changed ones, but on a schedule you run; Pathway keeps the index live as part of the stream.

Performance, Scaling and Deployment

Pathway’s own serviette benchmark indexed 50 GB of text in 6.0 hours at a 14.65 GiB peak indexer memory, on a 96-CPU host with 8 workers and local embeddings. That corpus held about 26 million pages in 17 million files.

The same benchmark, one run per size, indexed 1 GB in 4.9 minutes at 6.82 GiB and 10 GB in 1.0 hour at 8.16 GiB.

Memory decides fit, because Pathway keeps the whole pipeline in RAM. The 10 GB run’s 8.16 GiB peak already exceeds Community’s 8 GB per-node cap, and the 96-CPU host dwarfs the 4-core free limit. To hold memory down, Pathway recommends built-in transformations over Python UDFs, stateless operations, and temporal behaviors that bound how long windows and joins keep data.

Pathway’s 2023 streaming benchmark, run on one 12-core machine, put it on par with Flink on WordCount and ahead on PageRank, with Spark Structured Streaming and Kafka Streams behind.

Deployment options split by tier:

• Single node: multithreading and multiprocessing through pathway spawn on every tier, within its core limit.
• Containers: the pathwaycom/pathway Docker image, which doubles as the Windows route.
• Cloud guides: AWS Fargate, Azure Container Instances, Google Cloud and Render.
• Enterprise only: distributed Kubernetes deployment, horizontal scaling, hot-failover high availability and automated instance launching.

Apache Flink is the better pick for JVM teams that need managed exactly-once state today; Pathway fits Python RAG pipelines that stay on one node or justify an Enterprise quote.

Licensing and Company Direction: The Fine Print

Pathway is source-available under BSL 1.1, not OSI open source. Each version converts to Apache 2.0 four years after its code was merged, and never before July 20, 2027.

The Additional Use Grant allows free production use under four conditions:

• One machine per installation: free production use stays within the runner’s thread and process limits.
• No service resale: offering Pathway to others as a Stream Data Processing Service, including cloud, hosting and data center resale, is excluded.
• No circumvention: license keys and resource limits cannot be bypassed.
• Fixes only: modified versions may run in production only for bug and vulnerability fixes.

The serviette server carries an MIT license but still needs a Pathway license key. A Scale key also sends anonymized usage data to Pathway through an OpenTelemetry collector.

Pathway’s homepage now leads with BDH, or Dragon Hatchling, its post-transformer model research. The August 11, 2026 announcement of $30M in total seed funding at a $500M valuation earmarked the money mainly for compute such as GB300s and did not mention the framework.

Framework releases kept shipping: 0.33.0 on September 18 and 0.33.1 on November 5, 2026, and the repository has about 62,000 GitHub stars. Bytewax shows the downside case: its company said it was no longer commercially viable in May 2025, and the project has shipped no release since 0.21.1. I rate the framework active today, with the BDH-first company focus as the long-term risk in a multi-year commitment.

How Does Pathway Compare to Competitors?

Pathway is the only tool in this set that runs streaming ETL and a live RAG index in one Python application.

Tool	Approach	Starting price	Freshness model
Pathway	Python streaming ETL and RAG	$0 self-hosted; free maximum 16 GB RAM and 4 cores per node	Live incremental computation
Apache Flink	JVM-first stream processing; Python via PyFlink	AWS managed: $0.11/KPU-hour plus storage and one extra KPU per app	Streaming; exactly-once state
Bytewax	Apache-2.0 Python dataflows	Free; community-maintained	Streaming dataflows
Quix Streams	Apache-2.0 Python Streaming DataFrames	Free library; Kafka required	Kafka streams
LlamaIndex	Open-source RAG framework	Free framework; LlamaParse $1.50/1,000 credits after allowance	Docstore upserts when you run ingestion
Unstructured	Document processing pipelines	10,000 free pages, then $0.015/page	Change detection and incremental processing
Airbyte	Connector-based data movement	Core free; Cloud Standard from $20/month	Hourly Standard or 15-minute Plus syncs

• Apache Flink: the stronger choice for JVM-first streaming where managed exactly-once state matters, with Python through PyFlink. AWS publishes its price, unlike Pathway Enterprise, but each application adds an extra KPU and running-storage charges.
• Bytewax: the Apache-2.0 option for Python dataflows. It has been community-maintained since May 2025, and the last release is still 0.21.1 from November 2024; Pathway shipped 10 releases in 2026, from 0.28.0 in January to 0.33.1 in November.
• Quix Streams: an Apache-2.0, pure-Python Streaming DataFrame library for teams already on Kafka, which it requires. Version 3.28.1 shipped October 15, 2026, though Quix’s commercial focus now leads with QuixLab.
• LlamaIndex and LangChain: simpler starting points for RAG over a corpus you re-ingest on your own schedule. LlamaIndex offers UPSERTS and UPSERTS_AND_DELETE docstore strategies; LangChain’s RecordManager hashes documents and offers None, incremental, full and scoped_full cleanup modes.
• Unstructured: the per-page alternative when parsing economics matter more than owning a streaming engine. It also lists change detection and incremental processing, with SharePoint, Google Drive, S3 and Kafka among its sources.
• Airbyte: the connector-breadth alternative, with 700+ connectors and Pinecone destinations that support incremental append and deduped syncs. Core is free to self-host, and Airbyte fits when an hourly or 15-minute refresh is fresh enough.
• RisingWave: the SQL-first alternative, an Apache-2.0 streaming database with incremental materialized views. Cloud Basic starts at $0.227 per RWU-hour.

Pathway is the pick when live document updates and Python transformations belong in one application, and the workload fits 16 GB and 4 cores per node or the budget covers an Enterprise quote.

How I Reviewed Pathway

I installed Pathway Live Data Framework 0.33.1 with the LLM xpack and the free Scale key, built the README pipeline with persistence, ran the Question-Answering RAG template in Docker and swapped its index and model in app.yaml. I also ran serviette’s demo, wizard and up commands, and weighed the BSL 1.1 terms, the tier limits and the AWS Marketplace contract.

I weighted these areas:

• Pricing and licensing: free resource limits, paid boundaries and BSL restrictions.
• Developer workflow: installation, YAML configuration and persistence.
• Live sync: connectors, incremental updates and consistency.
• RAG tooling: parsing, retrieval, model configuration and MCP.
• Production fit: memory, scaling and deployment.
• Continuity: release activity and company direction.
• Alternatives: Apache Flink, Bytewax, Quix, LlamaIndex, LangChain, Unstructured, Airbyte and RisingWave.

Prices current as of November 2026.

Pathway Review: Should You Use Pathway for Streaming RAG?

Among streaming RAG frameworks, Pathway’s strongest case is one engine for live ETL and RAG, with document changes flowing through ingestion and retrieval as they arrive. The free Scale key covers up to 16 GB RAM and 4 cores per node under BSL 1.1, without Enterprise’s exactly-once consistency or hot failover.

• Choose Pathway if: your Python team needs SharePoint, Google Drive, S3, Kafka or PostgreSQL changes reflected in answers without a separate ingestion schedule, and the pipeline fits one node or the budget covers Enterprise.
• Choose Enterprise only for defined requirements: exactly-once consistency, distributed execution or hot failover, priced only by quote.
• Skip it for static-corpus RAG: LlamaIndex or LangChain re-indexing is simpler when a triggered re-ingest keeps answers fresh enough.
• Skip it for Windows-native deployment or mandatory OSI licensing: Pathway runs on Windows only through a VM, WSL or Docker, under BSL 1.1.
• Compare Apache Flink first: if your team prefers JVM-first streaming and managed exactly-once state over built-in Python RAG components.

Start with the Question-Answering RAG template or serviette from Pathway’s website, on a copy of one live folder. The serviette demo command opens a chat over a bundled sample corpus before you connect a source of your own.

FAQ

Is Pathway free?

Yes: Community and Scale are free to self-host. Community allows 8 GB RAM and 4 cores per node, and the free Scale key raises RAM to 16 GB. Paid Scale keys for larger nodes have no published price, and Enterprise is a custom quote. The AWS Marketplace contract for hosted Pathway services costs $15,000 per unit per year.

Is Pathway open source?

No. Pathway is source-available under BSL 1.1, not OSI open source. Each version converts to Apache 2.0 four years after its code was merged, never before July 20, 2027. Free production use is limited to one machine per installation and excludes reselling Pathway as a Stream Data Processing Service. The serviette RAG server is MIT-licensed but still needs a Pathway license key.

Does Pathway run on Windows?

Not natively. Pathway publishes no Windows wheels; its packages target macOS and Linux on x86_64 and arm64, for Python 3.10 or newer. Windows users run it in a virtual machine, in WSL, or in Docker with the pathwaycom/pathway image, and the Question-Answering RAG template already starts with docker compose up.

Does Pathway need a vector database?

No. Pathway includes in-memory KNN, BM25 and hybrid indexes, so a RAG app runs without a separate vector database. Since 0.32.1, optional output connectors keep Chroma, Qdrant, DuckDB, Weaviate and Pinecone in sync with inserts, updates and deletes. The serviette RAG server supports eight vector stores and defaults to DuckDB.

Is Pathway still maintained after its shift to BDH research?

Yes. Version 0.33.0 shipped September 18, 2026, and 0.33.1 followed on November 5 with Python 3.15 support for the core package and PostgreSQL replication-slot reads. BDH now leads Pathway’s homepage, and the August 2026 funding announcement allocated the money mainly to compute and did not mention the framework.