HPCC Systems

HPCC Systems is an open-source platform for enterprise big-data processing and data-lake management. It combines the ECL language with Thor, which handles data ingestion, transformation, linking, and indexing, and Roxie, which serves concurrent queries. ECL is declarative and modular; its compiler optimizes programs for parallel processing and compiles them into C++. The platform brings mixed-schema data together, and clusters can range from two computers to more than a thousand commodity-hardware nodes. Its cloud-native deployment runs on Kubernetes and supports Azure Kubernetes Service and Amazon Elastic Kubernetes Service. Storage options include AWS S3, Azure Blob Storage, AWS Elastic Block Store, Azure Files, and Azure Disks. Security features include encryption, service meshes, OAuth 2.0 with Azure AD support, JWT, and configurable controls for data and services. The Machine Learning Library offers ECL-accessible prediction algorithms, and integrations include Spark, Kafka, JDBC, Java APIs, and Tableau connectors. The open-source plan is free to use under Apache 2.0 and is self-hosted.

HPCC Systems combines data-lake management, parallel processing, and query serving in one open-source platform. Check the operating-system requirements and self-hosted deployment model against your environment.

It suits enterprise teams managing mixed-schema data lakes or building parallel data-processing and query workflows. The listed tutorials and training are free, with community support available through Stack Overflow.

Overview

HPCC Systems is an open-source data-lake platform for teams handling large, mixed-schema datasets across clusters. It suits organizations with the Linux and infrastructure capacity to run their own data platform; its strongest case is pairing large-scale refinement with concurrent query delivery in one system.

Its architecture joins ECL, a programming language, with Thor for data preparation and Roxie for query serving. The design gives data teams a broad processing path, but it is not a lightweight, managed option for organizations seeking to avoid operating infrastructure.

For more options, browse Data Lake Software.

Key features

Code-driven parallel processing

ECL is declarative and modular. Its compiler optimizes code for parallel processing and translates it into C++, which makes the platform a better fit for teams comfortable expressing data work in code than for users looking for a no-code workflow.

Distinct preparation and query engines

Thor handles ingestion, transformation, linking and indexing; Roxie serves high-performance concurrent queries. Separating these tasks can help teams prepare data while making query access available, rather than treating refinement and serving as a single workload.

Scale and cloud infrastructure

Clusters can grow from two computers to more than a thousand commodity-hardware nodes. The cloud-native platform runs on Kubernetes, including Azure Kubernetes Service and Amazon Elastic Kubernetes Service. Its storage plane supports AWS S3, Azure Blob Storage, AWS Elastic Block Store, Azure Files and Azure Disks. This range is useful for organizations building on those platforms, though it still leaves deployment and operations in the customer’s hands.

Security and integration

Cloud-native security includes end-to-end encryption, Linkerd or Istio service meshes, OAuth 2.0 with Azure AD support, and JWT. Configurable security managers can protect landing zones, file scopes, recordset data, workunit execution and ESP services. Official integrations include Spark, Kafka, Couchbase, Redis, Memcached, Pentaho, R, JDBC, Java APIs and Tableau connectors, giving teams several routes into existing data and analytics stacks.

Models, metadata and query access

The Machine Learning Library exposes ECL-accessible algorithms for building and testing qualitative or quantitative prediction models. HPCC Systems also provides a metadata catalog, governance controls and data sharing. ESP exposes ECL queries through XML, HTTP, SOAP and REST; deployed queries can also be called with REST/JSON. Together, these capabilities serve teams that want data preparation, governed access and query interfaces within one platform.

Pricing

The Open source plan costs nothing. It is free to use, Apache 2.0 licensed and self-hosted, so it removes a software license charge but not the work of provisioning and maintaining the environment. The published plan has no stated seat, quota, renewal or trial terms.

Free tutorials and online training, along with public Stack Overflow community support, give teams ways to get started without a paid support plan. Organizations that need a managed service or paid support arrangement should compare alternatives rather than assume those are included.

Platforms

HPCC Systems supports API, extension, Linux, macOS, self-hosted, web and Windows environments. A single-node cluster requires a current supported Linux system. The ECL IDE is available for Windows, while Apple OSX is supported for client tools only; macOS should not be mistaken for full cluster support.

Who it's for

This platform is best suited to data engineering organizations that need to combine mixed-schema data, scale processing across clusters and serve concurrent queries, and that can operate Linux or Kubernetes infrastructure. Its language, metadata catalog, governance controls and integration set favor teams that want a programmable data-lake system and have the skills to run one.

It is a weaker fit for small teams seeking a fully managed service, or for readers who need a simple interface rather than code-driven data processing. The self-hosted model and Linux requirement for a single-node cluster are meaningful operational commitments even though the software itself is free.

Pros and cons

• Pros: Thor and Roxie separate data refinement from concurrent query serving, supporting both stages in one platform.
• Pros: Cluster sizing spans two computers to more than a thousand commodity-hardware nodes, with Kubernetes deployment and several AWS and Azure storage options.
• Pros: Security controls cover multiple data and execution surfaces, while official integrations connect to widely used processing, storage and analytics tools.
• Pros: The Apache 2.0 open-source plan costs nothing, and free tutorials and training lower the barrier to learning the platform.
• Cons: Self-hosting places infrastructure operations on the customer, making it less suitable for teams wanting a managed data lake.
• Cons: ECL is central to the platform, so teams unwilling to build data workflows in a programming language may prefer a less code-oriented tool.
• Cons: Linux is required for a single-node cluster; macOS support is limited to client tools, not full cluster operation.

Alternatives

Choose AWS Lake Formation if you want to create and manage permissions through integrated AWS services; permission creation and use are free, while integrated services such as Amazon S3 carry standard usage rates.

Apache Hadoop HDFS is a free, self-hosted open-source alternative for readers comparing distributed storage projects. Apache Ozone is another free Apache License 2.0 option for readers looking at a self-hosted project with web access.

Parseable may suit teams prioritizing SQL queries, dashboards, threshold alerts, OIDC SSO, unlimited users and configurable retention in its free OSS plan. For readers comparing application performance monitoring rather than data-lake management, Tencent Cloud Application Performance Management is another option.

Unilake is worth comparing if you want a free platform licensed under AGPL 3.0 and EUPL that can run fully isolated in an environment of your choice. Cloudera Data Lake Service and HPE Ezmeral Data Fabric are paid alternatives for readers considering commercial offerings.

Verdict

Choose HPCC Systems if your organization needs an open-source platform to refine and serve large datasets at cluster scale, and has the Linux or Kubernetes expertise to operate it. Its combination of ECL, Thor and Roxie is the core reason to pick it; look elsewhere if you need managed operations or want to avoid programming data workflows.