Gyanrush
Start freeOpen Gyanrush
Start freeOpen Gyanrush
  1. LPU MCA Semester 1
  2. CAP470
  3. Unit 5

CAP470 · Unit 5: Cloud access, storage and file systems

Big data, MapReduce and the Hadoop framework

11 practice questions · in the ETE syllabus

Practise this topicPractise this topic

Key points

  • Hadoop = HDFS storage + MapReduce processing + YARN
  • Map emits key-value pairs; reduce combines values per key
  • Shuffle groups values by key
  • Computation moves to the data (locality)

Big data is data too large, fast or varied for a single machine, so it is stored and processed on clusters. Hadoop is the best-known open-source framework for this. Its core parts are HDFS (the distributed storage layer), MapReduce (the processing model) and YARN (cluster resource management). Hadoop runs on clusters of inexpensive commodity machines and tolerates failures by keeping several copies of data.

MapReduce has two phases:

PhaseWork
MapTurns each input record into intermediate (key, value) pairs, such as (word, 1)
ShuffleGroups all values that share a key, for example a: [3, 7]
ReduceCombines the values of each key into one result (sum, maximum, average)

An average per key is computed by reducing a total and a count and dividing. Because moving huge datasets over the network is costly, Hadoop schedules map tasks on the nodes that already store the data (data locality), moving the computation to the data.

Exam trap: the reduce phase receives values already grouped by key. It does not read the original input.

Try two questions

Question 1 · Tier 2, Understand

What does the map phase of MapReduce produce?

  1. The final sorted output file
  2. Intermediate key-value pairs
  3. A backup copy of the input data
  4. A list of failed machines in the cluster
Show the answer

B. Each map task turns input records into (key, value) pairs.

Why not A: Final output is produced by the reduce phase.

Why not C: Map does not create backups.

Why not D: Failure tracking is a separate cluster function.

Question 2 · Tier 3, Apply

Which Hadoop component manages cluster resources and schedules jobs?

  1. HDFS
  2. YARN
  3. Pig
  4. Sqoop
Show the answer

B. YARN (Yet Another Resource Negotiator) allocates resources to applications.

Why not A: HDFS stores the data.

Why not C: Pig is a data flow scripting tool.

Why not D: Sqoop moves data between Hadoop and relational databases.