7.a) Explain operating system security and virtual machine security.
Answer:
Operating system security
A critical function of an OS is to protect applications against a wide range of malicious attacks, e.g., unauthorized access to privileged information, tempering with executable code, and spoofing.
The elements of the mandatory OS security:
Access control: mechanisms to control the access to system objects
Authentication usage: mechanisms to authenticate a principal
Cryptographic usage policies: mechanisms used to protect the data
Commercial OS do not support a multi-layered security; only distinguish between a completely privileged security domain and a completely unprivileged one.
Trusted paths mechanisms: support user interactions with trusted software. Critical for system security; if such mechanisms do not exist, then malicious software can impersonate trusted software. Some systems provide trust paths for a few functions, such as login authentication and password changing, and allow servers to authenticate their clients.
Closed-box platforms, e.g., cellular phones, game consoles and ATM could have embedded cryptographic keys to reveal their true identity to remote systems and authenticate the software running on them.
Such facilities are not available to open-box platforms, the traditional hardware for commodity operating systems.
Commodity operating system offer low assurance. An OS is a complex software system consisting of millions of lines of code and it is vulnerable to a wide range of malicious attacks.
An OS provides weak mechanisms for applications to authenticate to one another and create a trusted path between users and applications.
An OS poorly isolates one application from another; once an application is compromised, the entire physical platform and all applications running on it can be affected. The platform security level is reduced to the security level of the most vulnerable application running on the platform.
Virtual machine security
Hybrid and hosted VMs, expose the entire system to the vulnerability of the host OS.
In a traditional VM the Virtual Machine Monitor (VMM) controls the access to the hardware and provides a stricter isolation of VMs from one another than the isolation of processes in a traditional OS.
A VMM controls the execution of privileged operations and can enforce memory isolation as well as disk and network access.
The VMMs are considerably less complex and better structured than traditional operating systems thus, in a better position to respond to security attacks.
A major challenge a VMM sees only raw data regarding the state of a guest operating system while security services typically operate at a higher logical level, e.g., at the level of a file rather than a disk block.
A secure TCB (Trusted Computing Base) is a necessary condition for security in a virtual machine environment; if the TCB is compromised then the security of the entire system is affected.
VMM-based threats
Starvation of resources and denial of service for some VMs
Probable causes:
(a) Badly configured resource limits for some VMs.
(b) A rogue VM with the capability to bypass resource limits set in VMM.
VM side-channel attacks: malicious attack on one or more VMs by a rogue VM under the same VMM.
Probable causes:
(a) lack of proper isolation of inter-VM traffic due to misconfiguration of the virtual network residing in the VMM.
(b) limitation of packet inspection devices to handle high speed traffic, e.g., video traffic.
(c) presence of VM instances built from insecure VM images, e.g., a VM image having a guest OS without the latest patches.
Buffer overflow attacks
Deployment of rogue or insecure VM: Unauthorized users may create insecure instances from images or may perform unauthorized administrative actions on existing VMs.
Probable cause: improper configuration of access controls on VM administrative tasks such as instance creation, launching, suspension, re-activation and so on.
Presence of insecure and tampered VM images in the VM image repository
Probable causes:
(a) lack of access control to the VM image repository.
(b) lack of mechanisms to verify the integrity of the images, e.g., digitally signed image.