Explain like I'm five · SW Eng

What is a Big Data Platform?

Big data platforms and processing architecture, told as a story about a publishing house's nationwide warehouse network handling orders from everywhere.

orders pour in from across the country, millions a dayone warehouse can't hold all the order records, let alone find them

Orders pour in from across the country, millions a day.One warehouse can't hold all the order records, let alone find them.

before yesterday's orders are even sorted, new ones pour inwith one warehouse and one worker, sorting can't keep pace with orders

Cramming every order into one warehouse means it overflows and sorting slows down.With one warehouse and one worker, sorting can't keep pace with orders.

a big-data platform = data split across many warehouses, sorted by many workers at onceif one warehouse fills up, data goes to the others

A big data platform stores data across many warehouses and sorts it with many workers.If one warehouse fills up, data goes to the others.

Distributed storagedata isn't all stuffed into one warehouse
Distributed processingmany workers sorting at once
Batch, overnightcollected data gets sorted all together
Raw storage (data lake)data is kept exactly as it came in
WHEN TO SORTbatch — sort it all overnightstreaming — sort it the instant it arrivesHOW TO STORE ITdata lake — raw originals, one big warehousedata warehouse — tidied, organized shelveswhen to sort and how to store are chosen separately, to fit the use

When to sort and how to store are chosen separately, to fit the use.Streaming when results are needed right away; batch for big jobs that can wait.

add warehouses with no plan, and no one knows what's wheredata piled up untidied becomes a swamp no one can searchadding warehouses and designing them well are two different problems

Adding more warehouses doesn't automatically make sorting faster.Pile data up without tidying, and eventually nothing can be found — a swamp.

In one breath

A big data platform spreads incoming data across many warehouses (distributed storage) and sorts it with many workers at once (distributed processing).

A big data platform distributes storage (e.g. HDFS) and processing (e.g. Hadoop MapReduce, Spark) for high-volume, high-velocity, varied data. Processing splits into batch (collected and run on a schedule) and stream (handled the instant it arrives), and storage commonly pairs a data lake (raw data kept as-is) with a data warehouse (cleaned, structured data).

When grown-ups say it

Big Data Platform
The nationwide warehouse network. Spreads data across many warehouses and sorting across many workers.
Distributed Storage
Splitting the warehouse into many sites. If one site fills up, the others still have room.
Distributed Processing
Many workers sorting at the same time. Tools like Hadoop and Spark do distributed processing.
Batch Processing
Sorting it all in one overnight run. Favored for large jobs whose results can wait.
Stream Processing
Sorting the instant it arrives. Favored when results are needed right away.
Data Lake
The big warehouse that keeps raw originals. Without upkeep, it can turn into a swamp.
Data Warehouse
The tidied, organized shelf. Structured to be ready for analysis.
High Availability
Holding up even if one warehouse goes down. Covered in the SRE field, not this one. → when the spare generator takes over
Next story
Checking by Relationship, With No Answer Key →
Want another term as a picture book? Request →