From owner-freebsd-current@freebsd.org Wed Jul 20 04:38:31 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04DC7B9ED9A for ; Wed, 20 Jul 2016 04:38:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E193C1777 for ; Wed, 20 Jul 2016 04:38:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id E0F80B9ED99; Wed, 20 Jul 2016 04:38:30 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E09CAB9ED98 for ; Wed, 20 Jul 2016 04:38:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B8261776 for ; Wed, 20 Jul 2016 04:38:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u6K4c95v022381 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 20 Jul 2016 07:38:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u6K4c95v022381 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u6K4c9n3022380; Wed, 20 Jul 2016 07:38:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 20 Jul 2016 07:38:09 +0300 From: Konstantin Belousov To: Hartmut Brandt Cc: current@freebsd.org Subject: Re: (boost::)asio and kqueue problem Message-ID: <20160720043809.GP38613@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 04:38:31 -0000 On Tue, Jul 19, 2016 at 05:35:59PM +0200, Hartmut Brandt wrote: > Hi, > > I'm trying to use asio (that's boost::asio without boost) to handle > listening sockets asynchronuosly. This appears not to work. There are also > some reports on the net about this problem. I was able to reproduce the > problem with a small C-programm that does the same steps as asio. The > relevant sequence of system calls is: > > kqueue() = 3 (0x3) > socket(PF_INET,SOCK_STREAM,6) = 4 (0x4) > setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4) = 0 (0x0) > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0) > setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4) = 0 (0x0) > bind(4,{ AF_INET 0.0.0.0:8080 },16) = 0 (0x0) > listen(0x4,0x80) = 0 (0x0) > ioctl(4,FIONBIO,0xffffea2c) = 0 (0x0) > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0) > kevent(3,0x0,0,0x7fffffffe5a0,32,0x0) ERR#4 'Interrupted system call' > > The problem here is that asio registers each file descriptor with > EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call). > After bringing the socket into the listening state and when async_accept() > is called it registers the socket a second time. According to the man page > this is perfectly legal and can be used to modify the registration. > > With this sequence of calls kevent() does not return when a connection is > established successfully. > > I tracked down the problem and the reason is in soo_kqfilter(). This is > called for the first EVFILT_READ registration and decides based on the > SO_ACCEPTCONN flag which filter operations to use solisten_filtops or > soread_filtops. In this case it chooses soread_filtops. > > The second EVFILT_READ registration does not call soo_kqfilter() again, > but just updates the filter from the data and fflags field so the > listening socket ends up with the wrong filter operations. > > The attached patch fixes this (kind of) by using the f_touch > operation (currently used only by the user filter). The filt_sotouch() > function changes the operation pointer in the knote when the socket is now > in the listening state. I suppose that the required locking is already > done in kqueue_register(), but I'm not sure. Asynchronous accepting now works. > > A better fix would probably be to change the operation vector on all > knotes attached to the socket in solisten(), but I fear I don't have the > necessary understanding of the locking that is required for this. > > Could somebody with enough kqueue() knowledge look whether the patch is > correct lock-wise? I find it weird that the fix still requires re-registration of the socket event to get it working after socket is marked as listen. In other words, until the re-registration is done, the events for the registered filter are lost. IMO more correct solution would be to merge the filt_solisten and filt_soread, deciding which path to take by testing the SO_ACCEPTCON flag in the f_event() op. This should also eliminate the concerns with the patching of pointer which is not supposed to be changed.