Date: Sun, 28 Oct 2001 00:37:09 +0100 (BST) From: =?iso-8859-1?q?A=20G=20F=20Keahan?= <agfk2000@yahoo.co.uk> To: freebsd-hackers@freebsd.org Subject: A question about multithreaded server Message-ID: <20011027233709.83232.qmail@web20809.mail.yahoo.com>
next in thread | raw e-mail | index | archive | help
I have a server built around the following model:
One "manager" thread selecting on multiple descriptors
(using kqueue() on FreeBSD, poll() on Solaris,
select() on some other Unix systems, and
WSAWaitForMultipleEvents() on Win32).
A small number (typically 2 x the number of CPUs) of
worker threads doing the actual work.
The manager thread handles accepting new connections
(the listening descriptor is also included in the
select loop), as well as reading and sending. The
read handler knows when a complete request has arrived
and dispatches it to one of the worker threads
(sleeping on a condvar), waking it up. The file
descriptors and their states (e.g. read and send
buffers) are stored in an array. If all workers are
busy, the connection's index is placed in a global run
queue (which along with its counter is protected by a
mutex). When a worker finishes processing a
connection and has nothing else to do, it is allowed
to get an item from the run queue.
The connection array has a fixed size, and connections
are not protected by any mutices, with the following
restriction: the selecting thread ("manager") is not
allowed to touch the read and send buffers of a
connection that is currently being processed by a
worker (so reading and writing is serialized: read -
work - write - read - ...)
The problem I'm having with this model is that I don't
see an efficient way of telling the manager that a
worker has finished processing a connection and that
the data is ready to be sent back to the client.
When a connection is dispatched to a worker, it is
deleted from the select queue, because we don't care
if it's ready for reading or writing until the worker
is done processing the data (and even if we cared, we
are still not allowed to read() or write() to the
connection buffers). Unfortunately, the worker thread
cannot add the connection back to the select queue,
because select() will break if the data is changed
from under it. But how can I tell select() "wake up,
the data is here" from another thread without breaking
it?
One solution is not to delete the worker connections
from the queue, but give select() a reasonably small
timeout and poll the workers for data every so and so
milliseconds. If a connection is ready for writing
but still has no data to send, ignore it and go around
the loop once again. Unfortunately, such polls steal
CPU time from the workers and lead to noticeable
delays.
Perhaps a better solution would be to add another file
descriptor (let's call it notify_fd) to the select
queue, and have the workers write to it on finishing
the processing, immediately waking up the manager
thread blocked on select. When the manager wakes up,
it checks if it was woken up by notify_fd, and if so,
goes through all the workers, adding their fds back to
the select queue if their data status is marked
"ready". Then it processes other active connections
and once again goes around the loop, but this time, if
any of the worker connections are ready for writing,
they will actually have some data to write.
The question is -- what can I use as a dummy
"notify_fd" descriptor? Another socket? A Unix
domain socket? Does this sound like a good solution?
Is there a better way to achieve the same result?
Any suggestions will be greatly appreciated.
Alex Keahan
____________________________________________________________
Nokia Game is on again.
Go to http://uk.yahoo.com/nokiagame/ and join the new
all media adventure before November 3rd.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20011027233709.83232.qmail>