Buffer Deadlock in Socket Application

Application protocols have to be designed with some care to avoid deadlock—that is, a
state in which each peer is blocked waiting for the other to do something. For example,
it is pretty obvious that if both client and server try to do a blocking receive immediately
after a connection is established, deadlock will result. Deadlock can also occur in less
immediate ways.

The buffers SendQ and RecvQ in the implementation have limits on their capacity.
Although the actual amount of memory they use may grow and shrink dynamically, a hard
limit is necessary to prevent all of the system’s memory from being gobbled up by a single
TCP connection under control of a misbehaving program. Because these buffers are finite,
they can fill up, and it is this fact, coupled with TCP’s flow control mechanism, that leads
to the possibility of another form of deadlock.

Once RecvQ is full, the TCP flow control mechanism kicks in and prevents the transfer
of any bytes from the sending host’s SendQ , until space becomes available in RecvQ as a
result of the receiver calling the input network stream’s Read() method. (The purpose of
the flow control mechanism is to ensure that the sender does not transmit more data than
the receiving system can handle.) A sending program can continue to call send until SendQ

is full; however, once SendQ is full, a call to out.Write() will block until space becomes
available, that is, until some bytes are transferred to the receiving socket’s RecvQ. If RecvQ
is also full, everything stops until the receiving program calls in.Read() and some bytes
are transferred to Delivered.

Let’s assume that the sizes of SendQ and RecvQ are SQS and RQS, respectively.
A write() call with a byte array of size n such that n>SQS will not return until at least
n−SQS bytes have been transferred to RecvQ at the receiving host. If n exceeds (SQS+RQS),
Write() cannot return until after the receiving program has read at least n−(SQS + RQS)
bytes from the input network stream. If the receiving program does not call Read(), a
large Send() may not complete successfully. In particular, if both ends of the connection
invoke their respective output network streams’ Write() method simultaneously with
buffers greater than SQS + RQS, deadlock will result: neither write will ever complete, and
both programs will remain blocked forever.