4.C) Explain the YARN application framework. – 5 Marks
Answer:-
YARN Application Framework
YARN (Yet Another Resource Negotiator) is a resource management platform that manages computational resources such as CPUs, memory, and network I/O required to execute applications. It handles the scheduling of resources and allows the parallel execution of application tasks and sub-tasks.
Key Features of YARN
- Separation of Concerns: YARN separates resource management from task processing.
- Parallelism: Applications consist of tasks, each divided into sub-tasks (threads) that run in parallel across cluster nodes.
- Efficient Resource Allocation: YARN dynamically manages and allocates resources to sub-tasks within specified time intervals.
- Multi-Threading: It supports multi-threaded applications, enhancing scalability and performance.
YARN Components
- Resource Manager (RM):
- Acts as the cluster master, maintaining information on available resources and handling resource scheduling.
- Node Manager (NM):
- A slave component on each cluster node, managing node-specific resources and container execution.
- Application Master (AM):
- Manages a single application’s lifecycle, estimates resource requirements, and coordinates task execution.
- Containers:
- Basic units of execution, providing the environment to run tasks with allocated resources (CPU and memory).
- Client:
- Submits application requests to the ResourceManager and tracks progress.
Execution Model
- Application Submission: Client sends a request to the RM.
- Resource Allocation: RM assigns resources (via containers) to the Application Master (AM).
- Task Execution: Sub-tasks are executed in containers on various Node Managers.
- Monitoring: The AM tracks execution and handles failures.
YARN ensures efficient, scalable, and fault-tolerant execution of distributed applications.