From owner-freebsd-hackers Thu Aug 10 18: 8:44 2000 Delivered-To: freebsd-hackers@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 90CD437BBA8 for ; Thu, 10 Aug 2000 18:08:40 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e7B18ck01677; Thu, 10 Aug 2000 18:08:38 -0700 (PDT) Date: Thu, 10 Aug 2000 18:08:38 -0700 From: Alfred Perlstein To: Michael Owens Cc: hackers@FreeBSD.ORG Subject: Re: R. Stevens select() Collisions Scenario Message-ID: <20000810180838.V4854@fw.wintelcom.net> References: <39934E64.A5BEE8EE@earthlink.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.4i In-Reply-To: <39934E64.A5BEE8EE@earthlink.net>; from owensmk@earthlink.net on Thu, Aug 10, 2000 at 07:52:52PM -0500 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG * Michael Owens [000810 17:51] wrote: > Purpose: > > I am trying to write a non-blocking, preforked server, specifically to > run on FreeBSD, and have a general question as to whether or not my > strategy is sound. > > > Problem: > > In Unix Network Programming Vol. 1 (section 27.6, p.741), Stevens > mentions a scenario under a preforked server design where multiple > children are calling select() on the same descriptor, causing collisions > for the kernel to resolve. The children would look something like... > . > . > . > for( ; ; ){ > FD_SET(listfd,&rset); > Select(listfd+1,&rset,NULL,NULL,NULL); > if(FD_ISSET(listfd,&rset)==0) > err_quit("listenfd readable"); > > clilen = addrlen; > connfd = Accept(listenfd, cliaddr, &clilen); > > process(connfd); > Close(connfd); > } > . > . > . > > Hypothesis: > > If you were implement the server such that the children used > non-blocking I/O, so that select() returned immediately, would this > alleviate the problem of collisions in the kernel and its associated > overhead? Select ignores non-blocking IO, if you have a non-blocking socket and perform a select on it, you _will_ block (unless you use a zero valued timeout). > If so, would you then have to worry about mutual exclusion when select > did return a ready descriptor for accept(). For example, a child gets > switched just after calling select() but just before it makes it to > accept(). Thus, the next child would also receive the same descriptor > ready for accept()? Now only one of the two children will get to > accept() first, leaving the other blocking on accept(). Stevens mentions > a similar case at the end of the Nonblocking I/O chapter (section 15.6, > p. 422) and to avoid this he recommends 1) setting the listening > descriptor to non-blocking and 2) ignoring EWOULDBLOCK, ECONNABORTED, > etc. on accept(). Yes, that is correct and the correct thing to do. > So, if you make the listening descriptor non-blocking, and treat > select() and accept() appropriately, you should be alright in both > avoiding select collisions and there overhead, as well as avoid children > blocking due to losing possible race conditions for the listen > descriptor in the event (albiet small) of a context switch between > select() and accept(). Another way of doing this would be to have a 'master' accept process that uses fd passing (it's in your book) to hand off connections to children, this would avoid select collisions at the cost of an additional context switch. > > Apology: > > The reason I bother you with all this is that while I (think I) > understand the logic, I (know I) am at the limits of my understanding, > and might be missing some important considerations as to other goings on > in the kernel, and that there might be a better way to go about all > this. Well for one thing I wouldn't be using select, I would use FreeBSD's kqueue mechanism, it's vastly superior to select and poll, although given the choice of select vs poll I would use poll(). > > Summary: > > Ultimately, all I am seeking to do is have an efficient and scalable > server design, and to my knowledge, this would consist of a number of > preforked children who are non-blocking/multiplexing themselves. > Furthermore, each child will have a pool of worker threads which will > handle jobs sent by different clients. As long as the threads' work > doesn't entail an operation that blocks for appreciable amount of time > (so that a single thread puts the whole child to sleep), it would seem > like this might be a decent proposal: multiple clients and multiple jobs > being handled simultaneously by each child, and the number of children > can be controlled by the parent according to the load and limits of the > hardware. > > Does this seem like a good way to go about it? Yes. :) When FreeBSD gets scheduler activations you'll be able to change to a single threaded process that will have excellent performance, the scheduler activations are just around the corner. > > Thanks: you're welcome. > > Thanks. too much coffee today? :) -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message