Clouds are based on virtualization since it provides simplicity of building and using distributed environment without creating dependencies on physical resources

Types of Virtualization:

1. Native, full
2. Hardware assisted
3. Para-virtualization (such as XEN)
4. OS level, including: Containers, Jails, Chroot, Zones, etc.

**IaaS (Infrastructure as a Service) : ** Amazon EC2

PaaS (Platform as a Service): Salesforce, repl.it

SaaS (Software as a Service): Gmail

Level of available resources need to manage:

Security related topics in distributed systems

Common types of security threats

Leakage: Unauthorized access to data or service (Confidentiality)

Tampering: Unauthorized modification to data or service (Integrity)

Vandalism: Interference with normal service. (Availability)

Policy of security

Confidentiality, Integrity, and Availability together means CIA properties, which is security policy (what we want to achieve).

Common types of attacks

Eavesdropping: attacker taps into networks

Masquerading: attackers pretend to be someone else

Message tampering: attacker modifies messages

Replay attack: attacker replay old messages

Denial of service: bombard a port

Shared Memory in Distributed System

It is similar to file system in a way that different processes are sharing pages instead of sending/receiving messages. It means to design a memory that allows processes to use same code as if they were all running over the same operating system/machine.

Page has two state: read, write

For concurrent read of pages. it is OK that many processes have their own copy of page in memory to concurrently access.

System programming concepts in distributed systems

File concept review

File is a higher level of abstraction. In UNIX like system, we can say that everything is a file.

A File contains:

1. timestamps: creation, read, write, header.
2. File type
3. ownership
4. Access control list: who man access in what mode.
5. Reference count: number of directories containing this file. When 0, delete this file.

Like I said, everything is a file. So directory is just a special case of file, too. It is a file containing the meta-data about files in that directory and pointers(on disk) to those files.

We open and close a file with file descriptors with mode (r read, w write, x executable).

Scheduling is a term mentioned many times in the field of system programming. Here I post some study notes about scheduling approaches used in Distributed Computing system.

Basic Scheduling Approaches review

FIFO

FIFO, or first in first out, utilizes a queue to store the tasks that are waited to be completed based on the order of arrival.

Advantages: Simply priority scheduling using time order, easy to implement.

Disadvantages: Average completion time may be high due to Convoy effect .

STF

STF, or shortest task first, maintains a queue to store the task based on the running time of tasks in increasing order. It is a priority scheduling that uses user-provided priority.

Advantages: Average completion time is the shortest among all approaches.

Disadvantages: Starvation may occurs, meaning that if task with shorter time keeps coming, some longer time tasks will not be solved.

FIFO/STF are preferable for Batch applications.

In the trend of social media, graph is frequently mentioned as a common description of social network/friend circle/work connection in real-life. But graph can be really large and hard to store in one server. Therefore, distributed clusters/systems are here to help.

Typical graph processing

Each node is assigned with a value, in each iteration, it gather values from its immediate neighbors and does some computation using its own value and neighbor value to generate new value. After that, update its new value and send it out to neighbors.

Brought by the trend of social media, large amounts of data is needed for real-time views.

Stream processing

Stream processing is used to process large amount of data with low latency and high throughput. Some classical solution such as Mapreduce may not be efficient here since it is essentially batch processing and needs to wait for entire computation of a large dataset to complete before viewing (there is not intermediate result.).

So I choode one of the most common project from Apache called : “Storm” to investigate its inner mechanism and components.

Apache Storm

The component of storm stream processing includes entities: Tuples, Streams, Spouts, Bolts, Topologies.

Tuples

Tuple is basically an ordered list of elements.

Streams

Stream is a sequence of tuples with potentially unbounded number.

Concurrency control is about how to control multiple clients operation concurrently. Replication control is about how to handle operations when data is stored at multiple servers, with/without replication.

Concurrency Control in distributed system including pessimistic control (such as read-write lock) and optimistic control. Here we discuss the optimistic approaches:

Basic approach:

1. Check for serializability at commit.
2. If not, abort, and roll back updates it did.

This approach may causes other transaction to read dirty data, which may be aborted as well.

Multi-version approach

1. For each object, we maintain per-transaction version of it, called “tentative version”, and a final version of “committed version”.
2. “tentative version” has timestamp, sometimes we have timestamp for both read and write.
3. On read and write, find the correct version.