- Home
- LLM Application Development Frameworks
- Pathway Live Data Framework
Pathway Live Data Framework
Palo Alto, California, USA
What it is
Pathway Live Data Framework is a Python framework for analytics and AI pipelines over data streams, including streaming ETL and retrieval-augmented generation (RAG). The same pipeline can work with static data and live streams. Its Rust engine runs Python code using multithreading and multiprocessing, while Differential Dataflow handles out-of-order data points in real time.
AI tooling includes online machine learning, RAG pipelines and vector indexes for fresh data. Pathway includes 350+ connectors, including SharePoint, and lets users implement their own. It supports Python and SQL programming APIs and a REST endpoint for serving queries, answers and real-time features. Deployments can use Docker or Kubernetes, and the framework is compatible with OpenTelemetry. It can save computation state to resume after a failure or pipeline update. The free Community tier is self-hosted and supports up to 8 GB RAM and 4 cores. Scale is also self-hosted, requires a license key and supports up to 16 GB RAM and 4 cores; it can be free or paid.
For and against
For
- One pipeline can process static data and live streams.
- Differential Dataflow handles out-of-order data in real time.
- Includes 350+ connectors and supports custom connectors.
- Computation state can be saved for recovery.
- Free Community tier supports up to 8 GB RAM and 4 cores.
Against
- Community is self-hosted and limited to 8 GB RAM and 4 cores.
- Scale requires a license key.
- Scale is supported up to 16 GB RAM and 4 cores.
The verdict
Pathway combines batch and streaming processing with AI pipeline tooling and a broad connector set. The free Community tier has defined self-hosted resource limits, while Scale requires a license key.
Who it suits
Pathway suits developers building data pipelines for analytics or AI over stored and live data. It offers Python and SQL APIs and self-hosted deployment.
Plans and prices
Community
Up to 8 GB RAM · 4 cores · self-hosted
Read 4 Oct 2026 · pathway.com
Scale
Up to 16 GB RAM · 4 cores · self-hosted
Read 4 Oct 2026 · pathway.com
Enterprise
Up to 24 TB RAM · 128 cores × 40 nodes · self-hosted or managed
Read 4 Oct 2026 · pathway.com
On the record
- AI features
- Its LLM tooling includes online ML, RAG pipelines, and vector indexes for working with fresh data.pathway.com · read 4 Oct 2026
- API
- The pricing page lists Python and SQL programming APIs and a REST API endpoint for serving queries, answers, and real-time features.pathway.com · read 4 Oct 2026
- Batch and streaming
- The same pipeline can process static data and live data streams.pathway.com · read 4 Oct 2026
- Community limits
- The Community tier is free under BSL 1.1, self-hosted, and supported up to 8 GB RAM and 4 cores.pathway.com · read 4 Oct 2026
- Deployment
- Pathway supports deployment with Docker or Kubernetes and is compatible with OpenTelemetry.pathway.com · read 4 Oct 2026
- Enterprise support
- Enterprise features listed include a multi-zone SLA and 24/7 phone support.pathway.com · read 4 Oct 2026
- Incremental computation
- Pathway uses Differential Dataflow to handle out-of-order data points in real time.pathway.com · read 4 Oct 2026
- Integrations
- The documentation says Pathway includes 350+ connectors, including SharePoint, and lets users implement their own connectors.pathway.com · read 4 Oct 2026
- License telemetry
- The licensing guide says Scale license keys send anonymized product usage data to Pathway via an OpenTelemetry collector.pathway.com · read 4 Oct 2026
- Persistence
- Pathway can save computation state to resume after a failure or pipeline update.pathway.com · read 4 Oct 2026
- Processing engine
- Its Rust engine runs Python code with multithreading and multiprocessing.pathway.com · read 4 Oct 2026
- Purpose
- Pathway is a Python data processing framework for analytics and AI pipelines over data streams, including streaming ETL and RAG pipelines.pathway.com · read 4 Oct 2026
- Scale limits
- The Scale tier is self-hosted, requires a license key, and is supported up to 16 GB RAM and 4 cores; the page says it can be free or paid.pathway.com · read 4 Oct 2026
- Support response
- The pricing page lists ticket-based business support with a one-business-day reply time for paid-tier customers.pathway.com · read 4 Oct 2026
