From owner-freebsd-hackers Sat Oct 27 16:37:15 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from web20809.mail.yahoo.com (web20809.mail.yahoo.com [216.136.226.198]) by hub.freebsd.org (Postfix) with SMTP id 0967337B401 for ; Sat, 27 Oct 2001 16:37:10 -0700 (PDT) Message-ID: <20011027233709.83232.qmail@web20809.mail.yahoo.com> Received: from [62.90.91.30] by web20809.mail.yahoo.com via HTTP; Sun, 28 Oct 2001 00:37:09 BST Date: Sun, 28 Oct 2001 00:37:09 +0100 (BST) From: =?iso-8859-1?q?A=20G=20F=20Keahan?= Subject: A question about multithreaded server To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I have a server built around the following model: One "manager" thread selecting on multiple descriptors (using kqueue() on FreeBSD, poll() on Solaris, select() on some other Unix systems, and WSAWaitForMultipleEvents() on Win32). A small number (typically 2 x the number of CPUs) of worker threads doing the actual work. The manager thread handles accepting new connections (the listening descriptor is also included in the select loop), as well as reading and sending. The read handler knows when a complete request has arrived and dispatches it to one of the worker threads (sleeping on a condvar), waking it up. The file descriptors and their states (e.g. read and send buffers) are stored in an array. If all workers are busy, the connection's index is placed in a global run queue (which along with its counter is protected by a mutex). When a worker finishes processing a connection and has nothing else to do, it is allowed to get an item from the run queue. The connection array has a fixed size, and connections are not protected by any mutices, with the following restriction: the selecting thread ("manager") is not allowed to touch the read and send buffers of a connection that is currently being processed by a worker (so reading and writing is serialized: read - work - write - read - ...) The problem I'm having with this model is that I don't see an efficient way of telling the manager that a worker has finished processing a connection and that the data is ready to be sent back to the client. When a connection is dispatched to a worker, it is deleted from the select queue, because we don't care if it's ready for reading or writing until the worker is done processing the data (and even if we cared, we are still not allowed to read() or write() to the connection buffers). Unfortunately, the worker thread cannot add the connection back to the select queue, because select() will break if the data is changed from under it. But how can I tell select() "wake up, the data is here" from another thread without breaking it? One solution is not to delete the worker connections from the queue, but give select() a reasonably small timeout and poll the workers for data every so and so milliseconds. If a connection is ready for writing but still has no data to send, ignore it and go around the loop once again. Unfortunately, such polls steal CPU time from the workers and lead to noticeable delays. Perhaps a better solution would be to add another file descriptor (let's call it notify_fd) to the select queue, and have the workers write to it on finishing the processing, immediately waking up the manager thread blocked on select. When the manager wakes up, it checks if it was woken up by notify_fd, and if so, goes through all the workers, adding their fds back to the select queue if their data status is marked "ready". Then it processes other active connections and once again goes around the loop, but this time, if any of the worker connections are ready for writing, they will actually have some data to write. The question is -- what can I use as a dummy "notify_fd" descriptor? Another socket? A Unix domain socket? Does this sound like a good solution? Is there a better way to achieve the same result? Any suggestions will be greatly appreciated. Alex Keahan ____________________________________________________________ Nokia Game is on again. Go to http://uk.yahoo.com/nokiagame/ and join the new all media adventure before November 3rd. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message