From owner-freebsd-net@FreeBSD.ORG Sat Apr 6 18:11:55 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B6A5D523; Sat, 6 Apr 2013 18:11:55 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 8C3A29BC; Sat, 6 Apr 2013 18:11:55 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r36IBluH064656 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 6 Apr 2013 11:11:48 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r36IBlIm064655; Sat, 6 Apr 2013 11:11:47 -0700 (PDT) (envelope-from jmg) Date: Sat, 6 Apr 2013 11:11:47 -0700 From: John-Mark Gurney To: Bakul Shah Subject: Re: close(2) while accept(2) is blocked Message-ID: <20130406181147.GJ76354@funkthat.com> Mail-Followup-To: Bakul Shah , freebsd-net@freebsd.org, Carl Shapiro , Andriy Gapon , FreeBSD Hackers References: <515475C7.6010404@FreeBSD.org> <20130329235431.32D7FB82A@mail.bitblocks.com> <20130330161434.GG76354@funkthat.com> <20130330202249.F218BB82A@mail.bitblocks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130330202249.F218BB82A@mail.bitblocks.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sat, 06 Apr 2013 11:11:48 -0700 (PDT) Cc: freebsd-net@freebsd.org, Carl Shapiro , Andriy Gapon , FreeBSD Hackers X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2013 18:11:55 -0000 Bakul Shah wrote this message on Sat, Mar 30, 2013 at 13:22 -0700: > On Sat, 30 Mar 2013 09:14:34 PDT John-Mark Gurney wrote: > > > > As someone else pointed out in this thread, if a userland program > > depends upon this behavior, it has a race condition in it... > > > > Thread 1 Thread 2 Thread 3 > > enters routine to read > > enters routine to close > > calls close(3) > > open() returns 3 > > does read(3) for orignal fd > > > > How can the original threaded program ensure that thread 2 doesn't > > create a new fd in between? So even if you use a lock, this won't > > help, because as far as I know, there is no enter read and unlock > > mutex call yet... > > It is worse. Consider: > > fd = open(file,...); > read(fd, ...); > > No guarantee read() gets data from the same opened file! > Another thread could've come along, closed fd and pointed it > to another file. So nothing is safe. Might as well stop using > threads, right?! Nope, you need your threads to cooperate w/ either locks or some ownership mechanism like token passing... As long as only one thread own the fd, you won't have troubles... > We are talking about cooperative threads where you don't have > to assume the worst case. Here not being notified on a close Multiple people have said when threads cooperate without locks, but no one has given a good example where they do... The cases you list below are IMO special (and extreme) cases... We were talking about "cooperating" threads in a general purpose langage like Java where you can not depend upon your other threads doing limited amount of work... > event can complicate things. As an example, I have done > something like this in the past: A frontend process validating > TCP connections and then passing on valid TCP connections to > another process for actual service (via sendmsg() over a unix > domain). All the worker threads in service process can do a > recvmsg() on the same fd. They process whatever tcp connection > they get. Now what happens when the frontend process is > restarted for some reason? All the worker threads need to > eventually reconnect to a new unix domain posted by the new > frontend process. You can handle this multiple ways but > terminating all the blocking syscalls on the now invalid fd is > the simplest solution from a user perspective. This'd make an interesting race... Not sure if it could happen, but close(), closes the receiving fd, but another thread is in the process of receiving an fd and puts the new fd in the close of the now closed unix domain socket... I can draw a more detailed diagram if you want.. > > I decided long ago that this is only solvable by proper use of locking > > and ensuring that if you call close (the syscall), that you do not have > > any other thread that may use the fd. It's the close routine's (not > > syscall) function to make sure it locks out other threads and all other > > are out of the code path that will use the fd before it calls close.. > > If you lock before close(), you have to lock before every > other syscall on that fd. That complicates userland coding and > slows down things when this can be handled more simply in the > kernel. There is "ownership" that can be passed, such as via kqueue _ONESHOT w/ multiple threads which allows you to avoid locking and only one thread ever owns the fd... > Another usecase is where N worker threads all accept() on the > same fd. Single threading using a lock defeats any performance > gain. In this case you still need to do something special since what happens if one of the worker threads opens a file and/or listen/accept socket as part of it's work? Thread 1 Thread 2 Thread 3 about to call accept but after flag check sets flag that close is going to be called accept connect process connection close listen'd socket open socket as part of processing gets same fd calls listen on socket calls accept on socket And there we have it, a race condition... You can't always guarantee what your worker threads do, if you can, it's good, but we need to make sure that beginner programs don't get into traps like these... They can easily make the mistake of saying, well, since close kicks all my threads out, I'll just do that instead of making sure that they don't have a race condition... > > If someone could describe how this new eject a person from read could > > be done in a race safe way, then I'd say go ahead w/ it... Otherwise > > we're just moving the race around, and letting people think that they > > have solved the problem when they haven't... > > In general it just makes sense to notify everyone waiting on > something that the situation has changed and they are going to > have to wait forever. The kernel should already have the > necessary information about which threads are sleeping on a > fd. Wake them all up. On being awakened they see that the fd > is no longer valid and all return with a count of data already > read or -1 and EBADF. Doing the equivalent in userland is > complicated. > > Carl has pointed out how BSD and Linux have required a > workaround compared to Solaris and OS X (in Java and IIRC, the > Go runtime). Seems like we have a number of usecases and this > is something worth fixing. I would like to know how a general purpose high level language like Java can avoid all the races that I brought up w/o locking the fd and, as you basicly said earlier, w/o the threads cooperating... To me it doesn't seem possible... Now if someone could explain it, I'd love to be proved wrong as it'd improve my knowlege of concurent systems... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."