What it is and the problem it solves
Hadoop is an open-source framework for reliable, scalable, distributed computing on commodity hardware. It solves the problem of storing and processing massive datasets when hardware failure is expected, not exceptional.
How it works
Hadoop splits files into large blocks and distributes them across cluster nodes. It transfers packaged code to those nodes to process data in parallel. This exploits data locality: nodes process data already stored locally.
What works
HDFS reliably stores large files across many machines with replication and rack awareness. MapReduce executes parallel computations while automatically rerouting work around failed nodes. Heartbeat monitoring and failover support keep jobs running despite frequent hardware faults.
What does not
Hadoop does not handle real-time or interactive queries well. It does not simplify data engineering—users must manage partitioning, serialization, shuffle bottlenecks, and failure recovery manually. It does not abstract away distributed systems complexity.
What it changes
It changes how organisations store and process petabyte-scale logs, web crawls, and offline analytics by replacing expensive proprietary data warehouses with open-source, rack-aware clusters that self-heal from node failures.
Is it worth your time
Yes—if you need fault-tolerant, scalable batch processing on commodity hardware and can absorb the operational overhead of cluster coordination, manual tuning, and MapReduce’s rigid two-stage model.
