System Programming

" One vision, one purpose. "

Copyright © Tony's Studio 2020 - 2022

Chapter Seven - Inter Process Communication

This is where the fun begins. There are several ways to communicate between two, or more processes.

IPC Description
Pipe The most simple one, but not that powerful.
Signal This is simple, too, but not enough information.
Semaphore Sync or mutual exclusion method for inter-process, or inter-thread communication.
Message Queue a.k.a. 消息队列, including POSIX one and System V one.
Shared Memory Share common memory with other processes, the fastest IPC method.
Socket More general, can be used between different computer.

7.1 Pipe

Pipe is the basic method for two processes to communicate with each other. Anonymous pipe can only be used between parent and child, though.

An anonymous pipe can be created with pipe(), pipe[0] is in and pipe[1] is out. Parent closes pipe[0] and output to pipe[1], while child read in from pipe[0] and closes pipe[1]. pipe() should be called before fork() and fork() will duplicate file description at the same time.

1
2
#include <unistd.h>
int pipe(int pipe[2]);

Here is a comprehensive example for pipe, all error checks were removed. Err… Eratosthenes sieve for prime numbers. For more information, check this out: https://swtch.com/~rsc/thread/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <wait.h>

void sieve(int oldfd[]);
int main(int argc, char* argv[])
{
	int n = atoi(argv[1]);
	int fd[2];
	pipe(fd);
	if (fork() > 0)               // the root!
	{
		close(fd[0]);   // close in
		for (int i = 2; i <= n; i++)
			write(fd[1], &i, sizeof(int));
		close(fd[1]);   // close out, or the child won't stop reading
		waitpid(ret, NULL, 0);          // wait for its child
	}
	else
		sieve(fd);
	return 0;
}

void sieve(int oldfd[])
{
	close(oldfd[1]);        // close old write
	int prime;
   	// Read one first, to check if it's neccessary to
	// create a new process.
	if (read(oldfd[0], &prime, sizeof(int)) == 0)   // nothing to read...
		return;
	int num;
	int fd[2];                      // new fd
	pipe(fd);
	if (fork() > 0)               // parent
	{
		close(fd[0]);			// close new in as parent
		printf("%d\n", prime);	 // print prime
		// read from old
		while (read(oldfd[0], &num, sizeof(int)) == sizeof(int))
		{
			if (num % prime != 0)
				write(fd[1], &num, sizeof(int));
		}
		close(fd[1]);
		waitpid(ret, NULL, 0);  // wait for its child
	}
	else    // child
		sieve(fd);
}

7.2 Signal

7.2.1 Meet Signal

7.2.1.1 What Is Signal?

Signal: 信号是 Linux 操作系统中进程之间一种通信方式，信号传递一种信息，接收方根据该信息进行相应的动作，可用于控制信息的传递，本质是一种软中断。e.g. 当发生某种情况时通知进程进行处理。

Signal can be raised by the process itself or come from outside.

7.2.1.2 Purpose of Signal

  1. 让进程知道发生了某种事件；
  2. 据该事件执行相应的动作即执行它自己代码中的信号处理程序。

7.2.1.3 Types of Signal

Signals are defined in signal.h and can be listed out by kill -l command. Usually begin with SIG and are all macros.

7.2.1.4 Source of Signal

Signals all come from kernel, and we just ask kernel to generate or send it for us. And such requests can come in three ways:

  1. user: Ctrl-C, Ctrl-\, etc.
  2. kernel: when error encountered, or to notice certain process, e.g. Segmentation Fault (SIGSEGV), Alarm time up (SIGALRM).
  3. process: call system function kill to send a signal.

User can send signal to a process by keyboard.

keyboard signal meaning
Ctrl-C SIGINT interrupt, to terminate the process by defualt
Ctrl-\ SIGQUIT quit process
Ctrl-Z SIGTSTP to stop a process, then can be continued by sending SIGCONT

Notice that SIGTSTP is similar to SIGSTOP, but SIGSTOP can not be blocked or ignored, while SIGTSTP could.

7.2.2 Status of Signal

Delivery: 递送，当进程对信号采取动作（执行信号处理函数或忽略）时称为递送。

Pending: 信号产生和递送之间的时间间隔内称信号是未决的。

Block: 信号递送阻塞，进程可指定对某个信号采用递送阻塞，若此时信号处理为默认或者捕捉的，该信号就会处于未决的状态。

信号未决状态是指从信号产生到起作用之间的状态，信号在未决状态时已经产生，但是并没有起作用，可以在不需要该信号时阻止信号被处理。一旦信号退出未决状态，则会被立即处理。

7.2.3 Respose to Signal

There are three types of response to a signal, as follows.

  1. 缺省操作：Linux 对每种信号都规定了默认操作，如果没有特殊说明，就按照默认的方式执行。
  2. 忽略信号：对信号不做处理，假装看不见，但是有两个信号不能忽略，即 SIGKILL 和 SIGSTOP。
  3. 捕捉信号：捕捉响应的信号，进行函数处理。SIGKILL 和 SIGSTOP 也不可以捕捉。

7.2.3 Classification of Signal

7.2.3.1 By Source

同步信号：由进程的某个操作产生的信号，即信号的产生和操作同时发生。e.g. 除零错误。

异步信号：由进程外的事件引起的信号，该信号产生的时间进程不可控。e.g. 用户键盘时间。

7.2.3.2 By Handle Behavior

不可靠信号：同时有多个信号产生，且无法及时处理时，会导致信号丢失。

可靠信号：不是不可靠信号的信号，来不及处理时会排入进程信号队列。

值小于 SIGRTMIN 的信号为不可靠信号，建立在 UNIX 早期机制上，SIGRTMIN 到 SIGRTMAX 的信号为可靠信号。

7.2.3.3 Real-Time Signal

Linux 目前有 64 种信号，前 32 种为非实时信号，后 32 种为实时信号。

非实时信号都不支持排队，都不可靠。实时信号都支持排队，都可靠。

It seems that real-time signal and reliable signal are the same?

7.2.4 Handling of Signal

7.2.4.1 signal

There are three ways to handle a signal, by registering different handler functions using signal() function. It returns the previous handler on success, or -1 on error. sig is the signal type except SIGKILL and SIGSTOP. It is passed automatically by signal mechanism and is exactly what we passed in signal().

1
2
#include <signal.h>
void (*signal(int sig, void (*handler)(int)))(int);

There are two system handlers, SIG_DFL is the default handler of the system, and SIG_IGN means ignore the signal.

After handling, the handler will be reset to default, so we need to re-hook it again.

1
2
3
4
5
void handler(int sig)
{
 // Handling...
 signal(sig, handler);
}

7.2.4.2 sigaction

sigaction provides a more powerful way to handle signal, and is compatible with old method. If you do not want to keep old handler, just leave it NULL.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#include <sys/siginfo.h> // actually doesn't need
union sigval {
	int sival_int;
	void *sival_ptr;
};
typedef struct {
	// ...
	union sigval si_value;
	// ...
} siginfo_t;

#include <signal.h>
struct sigaction {
	void (*sa_handler)(int);
	sigset_t sa_mask;
	int sa_flags;
	void (*sa_sigaction)(int, siginfo_t *, void *);
};
int sigaction(int sig, const struct sigaction *new, struct sigaction *old);

sa_handler is the old style, while sa_sigaction is the new style, which to use depends on the value of sa_flags. The new style can then access siginfo_t::si_value::sival_int, while siginfo_t::si_value::si_prt is only available when in the same process or shared memory is used.

sa_flags meaning
SA_RESETHAND reset handler after handling, a.k.a. 捕鼠器模式
SA_NODEFER close auto-block when handling, allow recursive
SA_RESTART restart system call if failed
SA_SIGINFO use new style, if not assigned, old style will be used

7.2.5 Block Signal

任何时候进程都有一些信号被阻塞，这个信号的集合被称为信号挡板，系统调用 sigprocmask() 可修改这个被阻塞的信号集。sigprocmask 是一个原子操作，根据所给的信号集来修改当前被阻塞的信号集。

7.2.5.1 Signal Set

To block signal, we need to indicate which set of signal to block or unblock.

1
2
3
4
5
6
#include <signal.h>
int sigemptyset(sigset_t *setp);          // remove all signals int the set
int sigfillset(sigset_t *setp);           // add all signalas to the set
int sigaddset(sigset_t *setp, int sig);   // add a signal to the set
int sigdelset(sigset_t *setp, int sig);   // remove a signal from the set
int sigismember(sigset_t *setp, int sig); // check if a signal is in the set or not

7.2.5.2 Mask Signals

If we want, or do not want some signals to be handled, we can unblock or block them use sigprocmask(). prev will be set to previous signal set if roll back is needed. how indicates how to treat the signal set, it has three values: SIG_BLOCK, SIG_UNBLOCK and SIG_SET.

1
2
#include <signal.h>
int sigprocmask(int how, const sigset_t *sigs, sigset_t *prev);

7.2.6 Send Signal

Remember, signal can be sent by a process. There are generally three ways to send a signal: kill, raise and sigqueue. All of them requires signal.h and sys/types.h (for pid_t).

7.2.6.1 kill

Just like, ya know, the command, send a signal to target process.

1
int kill (pid_t pid, int sig);

7.2.6.2 raise

Similar to kill, but this one send signal to itself.

1
int raise(int sig);

7.2.6.3 sigqueue

This one works with [sigaction](#7.2.4.2 sigaction), it will send a signal with siginfo_t, whose si_value is set to value. And the corresponding handler should be new style, and pay attention to the value.

1
int sigqueue(pid_t pid, int sig, const union sigval value);

7.2.7 Signal Process Inheritance

By default, child process will inherit all the handlers of parent process. However, if child process then calls exec function series, the signal handlers will be reset to default.

7.2.8 Reentrant

Definition: 某个函数可被多个任务并发使用，而不会造成数据错误，则该函数具有可重入性（Reentrant） 。

信号处理函数中，避免使用不可重入函数，因为信号处理函数有可能被调用多次。若处理函数使用了不可重入函数而变成不可重入时，则必须阻塞信号，若阻塞信号，则信号有可能丢失。

可重入函数中不能使用静态变量，不能使用 malloc/free函数和标准I/O库，使用全局变量时也应小心。

7.2.9 Applications

7.2.9.1 Prevent Zombie Process

When child process exits, it will send a SIGCHLD signal to its parent. By default, this signal is ignored. So parent can register a handler to this to do its own stuff, and take care of dead children only when needed to avoid waste of time on waiting.

7.3 Semaphore

7.3.1 Synchronization and Mutual Exclusion

Before we take a glance at semaphore, let review some concepts.

Term Description
临界资源 临界资源在某一时刻只能允许一个进程使用
临界区 访问临界资源的代码段称为临界区
同步（Synchronization） 进程之间相互依赖，一个进程必须等待另一个进程
互斥（Mutual Exclusion） 进程间相互排斥的使用临界资源的现象

7.3.2 POSIX vs System V

System V 是 Unix 操作系统的标准之一；POSIX 是 IEEE 的标准，相对更新，语法更简单。

System V 在同步互斥手段方面的无竞争条件下无论何时都会陷入内核，性能稍低；POSIX 在同步互斥手段方面的无竞争条件下是不会陷入内核的，性能较高。

System V 提供了 SEM_UNDO 可以在进程意外终止时释放信号量，可靠性高；POSIX 并没有实现，可靠性稍差。

System V 更多用于进程间通信，用于线程间通信则会丧失线程的轻量优势；POSIX 进程和线程间通信同步更优。

For more information, see System V 标准 & POSIX 标准.

7.3.3 System V Semaphore

For System V semaphore, it requires sem.h.

7.3.3.1 Create Semaphore

To get a semaphore, use semget(), 

1
2
3
4
5
6
7
8
9
10
/*
**    key - semaphore key id, must be identical
**  nsems - how many semaphores to create
** semflg - behavior: IPC_CREAT, IPC_EXCL; permission: S_IRUSR | S_IWUSR (read and write)
** return - EACCES: process have no access premission
**          ENOENT: key doesn't exists
**          EINVAL: nsems less than 0, or reaches maximum
**          EEXIST: when both IPC_CREAT and IPC_EXCL are assigned and key exists
*/
int semget(key_t key, int nsems, int semflg);

For the key, we often use ftok() to get an identical one, and System V use this key_t as a name for IPC object. It will generate hash value of pathname, which must exist, and combine it with id. Usually, current directory ./ is used with id equals zero.

1
2
#include <sys/ipc.h>
key_t ftok(const char *pathname, int id);

7.3.3.2 Control Semaphore

Semaphore can be represented as follows. I wonder why it is commented from Linux kernel.

1
2
3
4
5
typedef union semun {
	int val;
	struct semid_ds* buf;
	unsigned short* array;
} sem_t;

To control semaphore, we need semctrl().

1
2
3
4
5
6
7
/*
**  semid - semaphore id get from semget()
** semnum - only valid when using semaphore set, usually 0, the first one
**    cmd - operation to the semaphore
**    ... - a sem_t value, used when operate semaphore.
*/
int semctl(int semid, int semnum, int cmd, ...);

Common value for cmd are SETVAL and IPC_RMID.

cmd meaning
SETVAL initialize semaphore, 4th parameter is required
IPC_RMID remove semaphore of semid, semnum and 4th parameter are omitted

When IPC_RMID is applied, all blocked process by the semaphore will be awaken.

7.3.3.3 Operate Semaphore

There are two types of operation, P and V. In Dutch, P stands for Passeren (Pass), and V stands for Verhoog (Increment). P will ask for resource, if not available, the current process will be forced to wait. V will release resource, to wake up waiting processes.

To operate semaphore, we need semop().

1
2
3
4
5
6
7
8
9
10
11
12
13
// Semaphore operation unit
struct sembuf {
	short sem_num;	// indicate which semaphore to change in semaphore array
	short sem_op;	// -1 for P, and 1 for V
	short sem_flag;
};

/*
** semid - semaphore id get from semget()
**  sops - operation array
** nsops - how many operations in sops
*/
int semop(int semid, struct sembuf *sops, unsigned nsops);

For sops, usually, there is only one operation, and for the operation sembuf, sem_num is the same meaning as semnum in semctl, and sem_flag is often set to SEM_UNDO, to release semaphore automatically on process exit if forget to do so.

7.3.4 Application

Semaphore can used to coordinate processes for synchronization or mutual exclusion. More than one semaphore may be used. One example is with shared memory.

1
2
3
4
5
6
7
8
9
10
+: working
-: waiting
// syncronization, execute in order, semaphore initialized to 0
// n processes need n semaphores, so they can execute in desired order
ProcessA: +++++VP----- +++++VP ...
ProcessB: P-----+++++V P-----+ ...
// mutual exclusion, compete over one resource, semaphore initialized to 1
// only one semaphore, one process may get the resource more than one time in a roll
ProcessA: P+++++VP-----+++++VP+++++VP----- ...
ProcessB: P------+++++VP------------+++++V ...

7.4 Message Queue

Just maintain a queue from sender to receiver, easy to understand, huh?

7.4.1 Feature

发送方不必等待接收方去接收该消息就可不断发送数 据，而接收方若未收到消息也无需等待。

这种方式实现了发送方和接收方之间的松耦合，发送方和接收方只需负责自己的发送和接收功能，无需等待另外一方，从而为应用程序提供了灵活性。

7.4.2 System V Message Queue

Similar to System V semaphore, Duh, both System V, come on, and it requires sys/msg.h.

7.4.2.1 Create Message Queue

Err… the same as semaphore.

1
int msgget (key_t key, int msgflg);

7.4.2.2 Control Message Queue

Err… similar to semaphore.

1
int msgctl(int msgid, int cmd, struct msgqid_ds *buf);

Listen, I don’t want to get complicated here, for now, it is only used to delete message queue, simple, huh?

1
msgctl(msgid, IPC_RMID, NULL);

7.4.2.3 Send Message

We can send a message to the message queue with msgsnd().

1
2
3
4
5
6
7
/*
**  msgid - message queue id get from getmsg()
**   msgp - message to send, any type
**   msgz - size of message
** msgflg - flag
*/
int msgsnd(int msgid, const void *msgp, size_t msgz, int msgflg);

msgp is a custom structure, it can be defined like this. And msgz is then sizeof(message_t). 

1
2
3
4
typedef struct tag_message {
	long type;               // required field
	char data[BUFFER_SIZE];  // custom field, anything is OK
} message_t;

For msgflg, usually 0 is OK, and process will be hung up until message is sent. Others are OK, of course. If set to IPC_NOWAIT, it will return -1 immediately if message queue is full, and the message remain unsent.

7.4.2.4 Receive Message

Receive message is similar to send message, and is declared as follows.

1
int msgrcv(int msgid, void *msgp, size_t msgz, long msgtype, int msgflg);

All parameters remain the same meaning as msgsnd, and msgflag, too, return -1 immediately if no message can be received when set to IPC_NOWAIT. (This doesn’t mean the message queue is empty. It depends on what message to receive, which is indicated by msgtype)

For msgtype, it indicates what message to receive. This value is corresponding the message_t::type.

msgtype meaning
0 receive the first message in queue
> 0 receive the first message with the same type
< 0 receive the first message, whose type is less or equal to |msgtype|

7.5 Shared Memory

7.5.1 Feature

与消息队列和管道通信机制相比，一个进程要向队列/管道中写入数据时，引起数据从用户地址空间向内核地址空间的一次复制，进行消息读取时也要进行一次复制。

共享内存的优点是完全省去了这些操作，是 GNU/Linux 现在可用的最快速的进程间通信机制。

7.5.2 System V Shared Memory

Ah… System V again :P. And this one requires sys/shm.h

7.5.2.1 Create Shared Memory

Err… System V style, huh? size is the total size of shared memory you want to have, and shmflg is the same as files, you can refer to semaphore.

1
int shmget(key_t key, int size, int shmflg);

7.5.2.2 Control Shared Memory

Well, all the same. Like message queue, for now, we just use it to delete shared memory. Just set cmd to IPC_RMID and leave buf as NULL.

1
int shmctl(int shmid, int cmd, struct shmid_ds *buf)

7.5.2.3 Attach Shared Memory

In the last step, we only created a shared memory, but didn’t know where it is. So before we use it, we have to attach it to an address. In fact, it is done by mapping memory from this address to the actual shared memory.

1
2
3
4
5
6
7
/*
**   shmid - shared memory id
** shmaddr - memory to attach, ususally remain NULL to allocate automatically
**  shmflg - 0 by default for read and write; SHM_RDONLY for read only
**  return - the address attached to, -1 if error.
*/
char *shmat(int shmid, const void *shmaddr, int shmflg)

Now, you can use it as your own!

Notice that child process will inherit parent shared memory when created by fork(). Shared memory will detach automatically if process ends, or child process execute exec function set.

7.5.2.4 Detach Shared Memory

When we don’t want a memory, we can simply detach it. It returns 0 when success and -1 on failure.

1
int shmdt(const void *shmaddr);

7.5.3 Aplication

It’s easy to understand the use of shared memory. However, you can easily find that memory read and write are mutual exclusive, so shared memory often accompanied by semaphore. Since shared memory is used between processes, there’s nothing to do with mutex, which is used between threads.

" Do or do not. There is no try. "

Copyright © Tony's Studio 2020 - 2022

- EOF -