Descriptor passing between processes—sendmsg() and recvmsg()

The ability to pass an open descriptor between jobs can lead to a new way of designing client/server applications.

Passing an open descriptor between jobs allows one process, typically a server, to do everything that is required to obtain the descriptor (open a file, establish a connection, wait for the accept() API to complete) and let another process, typically a worker, handle all the data transfer operations as soon as the descriptor is open. This design results in simpler logic for both the server and the worker jobs. This design also allows different types of worker jobs to be easily supported. The server can make a simple check to determine which type of worker should receive the descriptor.

Sockets provide three sets of APIs that can pass descriptors between server jobs:

spawn()
Note: spawn() is not a socket API. It is supplied as part of the i5/OS™ Process-Related APIs.
givedescriptor() and takedescriptor()
sendmsg() and recvmsg()

The spawn() API starts a new server job (often called a "child job") and gives certain descriptors to that child job. If the child job is already active, then the givedescriptor() and takedescriptor() or the sendmsg() and recvmsg() APIs need to be used.

However, the sendmsg() and recvmsg() APIs offer many advantages over spawn() and givedescriptor() and takedescriptor():

Portability

The givedescriptor() and takedescriptor() APIs are non-standard and unique to the iSeries™. If the portability of an application between iSeries and UNIX^® is an issue, you might want to use the sendmsg() and recvmsg() APIs instead.

Communication of control information

Often the worker job needs to know additional information when it receives a descriptor, such as:

What type of descriptor is it?
What should the worker job do with it?

The sendmsg() and recvmsg() APIs allow you to transfer data, which might be control information, along with the descriptor; the givedescriptor() and takedescriptor() APIs do not.

Performance

Applications that use the sendmsg() and recvmsg() APIs tend to perform slightly better than those that use the givedescriptor() and takedescriptor() APIs in three areas:

Elapsed time
CPU utilization
Scalability

The amount of performance improvement of an application depends on the extent that the application passes descriptors.

Pool of worker jobs

You might want to set up a pool of worker jobs so that a server can pass a descriptor and only one of the jobs in the pool becomes active and receives the descriptor. The sendmsg() and recvmsg() APIs can be used to accomplish this by having all of the worker jobs wait on a shared descriptor. When the server calls sendmsg(), only one of the worker jobs receives the descriptor.

Unknown worker job ID

The givedescriptor() API requires the server job to know the job identifier of the worker job. Typically the worker job obtains the job identifier and transfers it over to the server job with a data queue. The sendmsg() and recvmsg() do not require the extra overhead to create and manage this data queue.

Adaptive server design

When a server is designed using the givedescriptor() and takedescriptor(), a data queue is typically used to transfer the job identifiers from worker jobs over to the server. The server then does a socket(), bind(), listen(), and an accept(). When the accept() API is completed, the server pulls off the next available job ID from the data queue. It then passes the inbound connection to that worker job. Problems arise when many incoming connection requests occur at once and there are not enough worker jobs available. If the data queue that contains the worker job identifiers is empty, the server blocks waiting for a worker job to become available, or the server creates additional worker jobs. In many environments, neither of these alternatives are what you want because additional incoming requests might fill the listen backlog.

Servers that use sendmsg() and recvmsg() APIs to pass descriptors remain unhindered during heavy activity because they do not need to know which worker job is going to handle each incoming connection. When a server calls sendmsg(), the descriptor for the incoming connection and any control data are put into an internal queue for the AF_UNIX socket. When a worker job becomes available, it calls recvmsg() and receives the first descriptor and the control data that was in the queue.

Inactive worker job

The givedescriptor() API requires the worker job to be active while the sendmsg() API does not. The job that calls sendmsg() does not require any information about the worker job. The sendmsg() API requires only that an AF_UNIX socket connection has been set up.

An example of how the sendmsg() API can be used to pass a descriptor to a job that does not exist follows:

A server can use the socketpair() API to create a pair of AF_UNIX sockets, use the sendmsg() API to send a descriptor over one of the AF_UNIX sockets created by socketpair(), and then call spawn() to create a child job that inherits the other end of the socket pair. The child job calls recvmsg() to receive the descriptor that the server passed. The child job was not active when the server called sendmsg().

Pass more than one descriptor at a time

The givedescriptor() and takedescriptor() APIs allow only one descriptor to be passed at a time. The sendmsg() and recvmsg() APIs can be used to pass an array of descriptors.