- Home
- Distributed File System Software
- Hadoop Distributed File System
Hadoop Distributed File System
What it is
Hadoop Distributed File System (HDFS) is the primary distributed storage used by Hadoop applications. A cluster centers on a NameNode that manages filesystem metadata and DataNodes that store data. HDFS is designed for fault tolerance on commodity hardware and for high-throughput streaming access and batch processing, rather than low-latency workloads. Its design targets files from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Written in Java, Hadoop is supported on major platforms, with self-hosted deployment. Administration tools include shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. HDFS permissions share much with POSIX and can optionally use POSIX ACLs. Hadoop supports simple operating-system identity and Kerberos authentication. Configured encryption protects data at rest and in transit through encryption zones and the Hadoop Key Management Server. WebHDFS provides REST access, and listed storage integrations include Amazon S3, Azure Blob Storage, and Azure Data Lake Storage. HDFS is free under Apache License 2.0.
For and against
For
- Designed for fault tolerance on commodity hardware
- Supports files from gigabytes to terabytes
- Provides shell, web, diagnostic, and balancing tools
- Offers REST access through WebHDFS
- Free under Apache License 2.0
Against
- Designed for high throughput rather than low latency
- Deployment requires a self-hosted cluster
- Support is through volunteer-maintained project mailing lists
The verdict
HDFS is a free distributed storage system shaped for large-scale, high-throughput workloads. Its cluster architecture and administrative tools suit teams prepared to deploy and manage storage themselves.
Who it suits
HDFS suits teams building self-hosted Hadoop storage for large files and batch or streaming workloads. Its design is aimed at high throughput rather than low latency.
Plans and prices
Apache Hadoop (open source)
Apache License 2.0 · self-hosted cluster deployment
Read 30 Sep 2026 · apache.org
On the record
- Administration
- HDFS provides shell-like commands, built-in NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer.hadoop.apache.org · read 30 Sep 2026
- Architecture
- An HDFS cluster primarily consists of a NameNode managing filesystem metadata and DataNodes storing data.hadoop.apache.org · read 30 Sep 2026
- Authentication
- Hadoop supports simple operating-system identity mode and Kerberos authentication mode.hadoop.apache.org · read 30 Sep 2026
- Downloads and integrity
- Apache distributes source and binary tarballs and recommends verifying releases with GPG signatures or SHA-512 checksums.hadoop.apache.org · read 30 Sep 2026
- Encryption
- Configured HDFS encryption provides transparent end-to-end encryption for data at rest and in transit using encryption zones and the Hadoop Key Management Server.hadoop.apache.org · read 30 Sep 2026
- Fault tolerance
- HDFS is highly fault-tolerant and designed to run on low-cost commodity hardware.hadoop.apache.org · read 30 Sep 2026
- Language and platforms
- Hadoop is written in Java and supported on all major platforms.hadoop.apache.org · read 30 Sep 2026
- License
- Apache Hadoop is licensed under the Apache License 2.0.hadoop.apache.org · read 30 Sep 2026
- Permissions
- HDFS implements a permissions model sharing much of POSIX and optionally supports POSIX access control lists.hadoop.apache.org · read 30 Sep 2026
- Purpose
- HDFS is the primary distributed storage used by Hadoop applications.hadoop.apache.org · read 30 Sep 2026
- REST integration
- The WebHDFS HTTP REST API supports the complete FileSystem and FileContext interfaces for HDFS.hadoop.apache.org · read 30 Sep 2026
- Scale
- HDFS is tuned for gigabyte-to-terabyte files, hundreds of nodes per cluster, and tens of millions of files in one instance.hadoop.apache.org · read 30 Sep 2026
- Storage integrations
- The documentation lists Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible file systems.hadoop.apache.org · read 30 Sep 2026
- Support
- Apache Hadoop support is provided through project mailing lists maintained by non-paid volunteers.hadoop.apache.org · read 30 Sep 2026
- Workload fit
- HDFS is designed for streaming access and batch processing with high throughput rather than low latency.hadoop.apache.org · read 30 Sep 2026
