A data center is a facility that houses computing resources such as servers, storage, and networking equipment. It is essential for cloud services and enterprise-level computing. Data centers are connected through high-speed interconnection networks for fast and reliable communication.
Warehouse-Scale Data-Center Design
- Large data centers house 400,000 to 1 million servers.
- Built using economies of scale to reduce unit costs.
- Microsoft has over 100 data centers worldwide.
- A small data center may have 1,000 servers, but cost per unit is higher.
- Example: Network cost in smaller DC is 7x more, storage cost is 5.7x more.
Construction Requirements
- Off-the-shelf servers with multi-core CPUs, DRAM, and disk drives.
- Example setup: 2,000 servers, each with 8 GB DRAM and 4×1 TB disks.
- First-level: Rack switches, Cluster-level: High-speed switches.
- Failures (hardware/software) are common; redundancy and replication are essential.
- Total disk storage is much higher than RAM but has lower bandwidth and higher latency.
Cooling System
- Data center rooms have raised floors to route cables and cooling air.
- Cooling is managed using CRAC (Computer Room Air Conditioning) units.
- Arrangement: Cold Aisles (cool air inlet) and Hot Aisles (exhaust).
- Cooling towers are used to pre-cool and reduce power usage.
- Efficient cooling reduces energy costs by up to 80% in newer setups.
Data-Center Interconnection Networks
Interconnection networks ensure efficient communication among thousands of servers.
Design Requirements:
- Low latency
- High bandwidth
- Low cost
- MPI (Message Passing Interface) support
- Fault tolerance
Application Traffic Support
- Network should support point-to-point and collective MPI communication.
- Needed for tasks like distributed file access, MapReduce, etc.
- High bisection bandwidth is required for parallel operations.
Network Expandability
- Must allow future growth without bottlenecks.
- Use of Fat-tree or Crossbar topologies with Ethernet switches.
- Modular design using data center containers supports scalability.
- Containers can be connected via external cabling and switches.
Fault Tolerance
- Provide redundancy in paths (multiple links) between servers.
- Avoid single point of failure.
- Must support graceful degradation and hot-swappable components.
Switch-Centric vs Server-Centric Designs
- Switch-centric: Switches manage all routing; servers remain unchanged.
- Server-centric: Servers are modified to assist in traffic routing.
Modular Data Centers in Shipping Containers
- Data centers now use truck-towable containers with blade servers.
- Example: SGI ICE Cube supports 46,080 cores or 30 PB per container.
- Easy to transport, scalable, and optimized for cooling and energy efficiency.
- Containers offer mobility, faster deployment, and reduced maintenance cost.
Container Construction
- Built with network, storage, and cooling inside the container.
- Steps: Build server ➝ rack ➝ full container ➝ testing.
- Requires weatherproofing and efficient cooling & airflow control.
- Useful in sectors like healthcare, defense, and remote cloud computing.
Interconnection of Modular Data Centers
- Containers are connected using inter-module networks like MDCube.
- BCube network inside each container handles internal communication.
- MDCube connects multiple containers using high-speed switches.
- Forms a virtual hypercube at the container level for enhanced scalability.
Data Center Management Issues
A successful data center must address these management challenges:
| Management Goal | Description |
|---|---|
| User Satisfaction | Provide consistent service for at least 30 years. |
| Controlled Info Flow | Maintain high availability and streamlined traffic. |
| Multi-user Manageability | Handle tasks like DB updates, maintenance, etc. |
| Scalability | Easily grow with demand—storage, power, I/O, cooling. |
| Reliability | Use failover, VM live migration, redundancy. |
| Low Cost | Reduce costs for both users and providers. |
| Security | Protect data from attacks, ensure data integrity. |
| Green IT | Improve energy efficiency and reduce carbon footprint. |
Marketplaces and Cost Management
- Factory-built containers reduce site-level installation work.
- Power densities exceed 1250 W/sq ft.
- Modular containers are suitable for rooftops, parking lots, etc.
- Must maintain redundancy, ensure future upgrades, and handle scalable growth.