From owner-freebsd-current@freebsd.org Tue Jul 19 15:37:16 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3D3AB9D56B for ; Tue, 19 Jul 2016 15:37:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id AC5481D34 for ; Tue, 19 Jul 2016 15:37:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: by mailman.ysv.freebsd.org (Postfix) id A808CB9D56A; Tue, 19 Jul 2016 15:37:16 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7A80B9D569 for ; Tue, 19 Jul 2016 15:37:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailhost.dlr.de (mailhost.dlr.de [129.247.252.32]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mailhost.dlr.de", Issuer "DLR CA - G02" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 122361D33 for ; Tue, 19 Jul 2016 15:37:15 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from DLREXHUB01.intra.dlr.de (172.21.152.130) by mailhost.dlr.de (172.21.163.100) with Microsoft SMTP Server (TLS) id 14.3.301.0; Tue, 19 Jul 2016 17:35:59 +0200 Received: from KNOP-BEAGLE.kn.op.dlr.de (129.247.178.136) by smtp.dlr.de (172.21.152.151) with Microsoft SMTP Server (TLS) id 14.3.301.0; Tue, 19 Jul 2016 17:35:59 +0200 Date: Tue, 19 Jul 2016 17:35:59 +0200 From: Hartmut Brandt X-X-Sender: brandt_h@KNOP-BEAGLE.kn.op.dlr.de To: Subject: (boost::)asio and kqueue problem Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="1964543108-1161917916-1468942563=:30581" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jul 2016 15:37:16 -0000 --1964543108-1161917916-1468942563=:30581 Content-Type: text/plain; format=flowed; charset="US-ASCII" Hi, I'm trying to use asio (that's boost::asio without boost) to handle listening sockets asynchronuosly. This appears not to work. There are also some reports on the net about this problem. I was able to reproduce the problem with a small C-programm that does the same steps as asio. The relevant sequence of system calls is: kqueue() = 3 (0x3) socket(PF_INET,SOCK_STREAM,6) = 4 (0x4) setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4) = 0 (0x0) kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0) setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4) = 0 (0x0) bind(4,{ AF_INET 0.0.0.0:8080 },16) = 0 (0x0) listen(0x4,0x80) = 0 (0x0) ioctl(4,FIONBIO,0xffffea2c) = 0 (0x0) kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0) kevent(3,0x0,0,0x7fffffffe5a0,32,0x0) ERR#4 'Interrupted system call' The problem here is that asio registers each file descriptor with EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call). After bringing the socket into the listening state and when async_accept() is called it registers the socket a second time. According to the man page this is perfectly legal and can be used to modify the registration. With this sequence of calls kevent() does not return when a connection is established successfully. I tracked down the problem and the reason is in soo_kqfilter(). This is called for the first EVFILT_READ registration and decides based on the SO_ACCEPTCONN flag which filter operations to use solisten_filtops or soread_filtops. In this case it chooses soread_filtops. The second EVFILT_READ registration does not call soo_kqfilter() again, but just updates the filter from the data and fflags field so the listening socket ends up with the wrong filter operations. The attached patch fixes this (kind of) by using the f_touch operation (currently used only by the user filter). The filt_sotouch() function changes the operation pointer in the knote when the socket is now in the listening state. I suppose that the required locking is already done in kqueue_register(), but I'm not sure. Asynchronous accepting now works. A better fix would probably be to change the operation vector on all knotes attached to the socket in solisten(), but I fear I don't have the necessary understanding of the locking that is required for this. Could somebody with enough kqueue() knowledge look whether the patch is correct lock-wise? Regards, harti Index: kern_event.c =================================================================== --- kern_event.c (revision 302977) +++ kern_event.c (working copy) @@ -1350,8 +1350,8 @@ KQ_UNLOCK(kq); knl = kn_list_lock(kn); kn->kn_kevent.udata = kev->udata; - if (!fops->f_isfd && fops->f_touch != NULL) { - fops->f_touch(kn, kev, EVENT_REGISTER); + if (kn->kn_fop->f_touch != NULL) { + kn->kn_fop->f_touch(kn, kev, EVENT_REGISTER); } else { kn->kn_sfflags = kev->fflags; kn->kn_sdata = kev->data; Index: uipc_socket.c =================================================================== --- uipc_socket.c (revision 302977) +++ uipc_socket.c (working copy) @@ -160,6 +160,7 @@ static void filt_sowdetach(struct knote *kn); static int filt_sowrite(struct knote *kn, long hint); static int filt_solisten(struct knote *kn, long hint); +static void filt_sotouch(struct knote *kn, struct kevent *kev, u_long type); static int inline hhook_run_socket(struct socket *so, void *hctx, int32_t h_id); fo_kqfilter_t soo_kqfilter; @@ -172,6 +173,7 @@ .f_isfd = 1, .f_detach = filt_sordetach, .f_event = filt_soread, + .f_touch = filt_sotouch, }; static struct filterops sowrite_filtops = { .f_isfd = 1, @@ -3091,6 +3093,31 @@ return (0); } +static void +filt_sotouch(struct knote *kn, struct kevent *kev, u_long type) +{ + struct socket *so = kn->kn_fp->f_data; + + switch (type) { + case EVENT_REGISTER: + if (kn->kn_fop == &soread_filtops && + (so->so_options & SO_ACCEPTCONN)) + kn->kn_fop = &solisten_filtops; + + kn->kn_sfflags = kev->fflags; + kn->kn_sdata = kev->data; + break; + + case EVENT_PROCESS: + *kev = kn->kn_kevent; + break; + + default: + panic("filt_sotouch() - invalid type (%ld)", type); + break; + } +} + /* * Some routines that return EOPNOTSUPP for entry points that are not * supported by a protocol. Fill in as needed. --1964543108-1161917916-1468942563=:30581 Content-Type: text/plain; charset="US-ASCII"; name="k.c" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="k.c" I2luY2x1ZGUgPHN5cy9zb2NrZXQuaD4NCiNpbmNsdWRlIDxzeXMvdHlwZXMu aD4NCiNpbmNsdWRlIDxzeXMvZXZlbnQuaD4NCiNpbmNsdWRlIDxzeXMvZmls aW8uaD4NCg0KI2luY2x1ZGUgPHN5cy9pb2N0bC5oPg0KI2luY2x1ZGUgPG5l dGluZXQvaW4uaD4NCiNpbmNsdWRlIDxzdGRpby5oPg0KI2luY2x1ZGUgPHN0 cmluZy5oPg0KI2luY2x1ZGUgPGVyci5oPg0KDQpzdGF0aWMgdm9pZA0Kd2Fp dF9sb29wKGludCBrcSwgaW50IHNvY2spDQp7DQoJc3RydWN0IGtldmVudCBl dlszMl07DQoJc3RydWN0IHNvY2thZGRyX2luIGFkZHI7DQoJc29ja2xlbl90 IHNvY2tsZW47DQoNCglmb3IgKDs7KSB7DQoJCWludCBuZXYgPSBrZXZlbnQo a3EsIE5VTEwsIDAsIGV2LCAzMiwgTlVMTCk7DQoJCWlmIChuZXYgPCAxKQ0K CQkJZXJyKDEsICJrZXZlbnQiKTsNCgkJZm9yIChpbnQgaSA9IDA7IGkgPCBu ZXY7ICsraSkgew0KCQkJaWYgKGV2W2ldLmlkZW50ID09IHNvY2spIHsNCgkJ CQlwcmludGYoImFjY2VwdFxuIik7DQoJCQkJaW50IGZkID0gYWNjZXB0KGV2 W2ldLmlkZW50LA0KCQkJCSAgICAoc3RydWN0IHNvY2thZGRyICopJmFkZHIs ICZzb2NrbGVuKTsNCgkJCQlpZiAoZmQgPT0gLTEpDQoJCQkJCWVycigxLCAi YWNjZXB0Iik7DQoJCQl9DQoJCX0NCgl9DQp9DQoNCmludA0KbWFpbigpDQp7 DQoJc3RydWN0IHNvY2thZGRyX2luIGFkZHI7DQoNCgkvKiBvcGVuIGEgVENQ IHNvY2tldCAqLw0KCWludCBrcSA9IGtxdWV1ZSgpOw0KDQoJaW50IHNvY2sg PSBzb2NrZXQoUEZfSU5FVCwgU09DS19TVFJFQU0sIDApOw0KDQoJc3RydWN0 IGtldmVudCBldlsyXTsNCglFVl9TRVQoJmV2WzBdLCBzb2NrLCBFVkZJTFRf UkVBRCwgRVZfQUREIHwgRVZfQ0xFQVIsIDAsIDAsIE5VTEwpOw0KCUVWX1NF VCgmZXZbMV0sIHNvY2ssIEVWRklMVF9XUklURSwgRVZfQUREIHwgRVZfQ0xF QVIsIDAsIDAsIE5VTEwpOw0KDQoJaW50IG9wdCA9IDE7DQoJc2V0c29ja29w dChzb2NrLCBTT0xfU09DS0VULCBTT19OT1NJR1BJUEUsICZvcHQsIHNpemVv ZihvcHQpKTsNCg0KCWlmIChrZXZlbnQoa3EsIGV2LCAyLCBOVUxMLCAwLCBO VUxMKSA9PSAtMSkNCgkgICAgZXJyKDEsICJrZXZlbnQiKTsNCg0KCXNldHNv Y2tvcHQoc29jaywgU09MX1NPQ0tFVCwgU09fUkVVU0VBRERSLCAmb3B0LCBz aXplb2Yob3B0KSk7DQoNCgltZW1zZXQoJmFkZHIsIDAsIHNpemVvZihhZGRy KSk7DQoJYWRkci5zaW5fcG9ydCA9IGh0b25zKDEwMDAwKTsNCg0KCWJpbmQo c29jaywgKHN0cnVjdCBzb2NrYWRkciAqKSZhZGRyLCBzaXplb2YoYWRkcikp Ow0KCWxpc3Rlbihzb2NrLCAweDgwKTsNCg0KCWlvY3RsKHNvY2ssIEZJT05C SU8sICZvcHQpOw0KDQoJaWYgKGtldmVudChrcSwgZXYsIDIsIE5VTEwsIDAs IE5VTEwpID09IC0xKQ0KCQllcnIoMSwgImtldmVudCIpOw0KDQoJd2FpdF9s b29wKGtxLCBzb2NrKTsNCn0NCg== --1964543108-1161917916-1468942563=:30581--