Date: Tue, 19 Jul 2016 17:35:59 +0200 From: Hartmut Brandt <hartmut.brandt@dlr.de> To: <current@freebsd.org> Subject: (boost::)asio and kqueue problem Message-ID: <alpine.BSF.2.00.1607191024380.30581@KNOP-BEAGLE.kn.op.dlr.de>
next in thread | raw e-mail | index | archive | help
--1964543108-1161917916-1468942563=:30581
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Hi,
I'm trying to use asio (that's boost::asio without boost) to handle
listening sockets asynchronuosly. This appears not to work. There are also
some reports on the net about this problem. I was able to reproduce the
problem with a small C-programm that does the same steps as asio. The
relevant sequence of system calls is:
kqueue() = 3 (0x3)
socket(PF_INET,SOCK_STREAM,6) = 4 (0x4)
setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4) = 0 (0x0)
kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4) = 0 (0x0)
bind(4,{ AF_INET 0.0.0.0:8080 },16) = 0 (0x0)
listen(0x4,0x80) = 0 (0x0)
ioctl(4,FIONBIO,0xffffea2c) = 0 (0x0)
kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
kevent(3,0x0,0,0x7fffffffe5a0,32,0x0) ERR#4 'Interrupted system call'
The problem here is that asio registers each file descriptor with
EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call).
After bringing the socket into the listening state and when async_accept()
is called it registers the socket a second time. According to the man page
this is perfectly legal and can be used to modify the registration.
With this sequence of calls kevent() does not return when a connection is
established successfully.
I tracked down the problem and the reason is in soo_kqfilter(). This is
called for the first EVFILT_READ registration and decides based on the
SO_ACCEPTCONN flag which filter operations to use solisten_filtops or
soread_filtops. In this case it chooses soread_filtops.
The second EVFILT_READ registration does not call soo_kqfilter() again,
but just updates the filter from the data and fflags field so the
listening socket ends up with the wrong filter operations.
The attached patch fixes this (kind of) by using the f_touch
operation (currently used only by the user filter). The filt_sotouch()
function changes the operation pointer in the knote when the socket is now
in the listening state. I suppose that the required locking is already
done in kqueue_register(), but I'm not sure. Asynchronous accepting now works.
A better fix would probably be to change the operation vector on all
knotes attached to the socket in solisten(), but I fear I don't have the
necessary understanding of the locking that is required for this.
Could somebody with enough kqueue() knowledge look whether the patch is
correct lock-wise?
Regards,
harti
Index: kern_event.c
===================================================================
--- kern_event.c (revision 302977)
+++ kern_event.c (working copy)
@@ -1350,8 +1350,8 @@
KQ_UNLOCK(kq);
knl = kn_list_lock(kn);
kn->kn_kevent.udata = kev->udata;
- if (!fops->f_isfd && fops->f_touch != NULL) {
- fops->f_touch(kn, kev, EVENT_REGISTER);
+ if (kn->kn_fop->f_touch != NULL) {
+ kn->kn_fop->f_touch(kn, kev, EVENT_REGISTER);
} else {
kn->kn_sfflags = kev->fflags;
kn->kn_sdata = kev->data;
Index: uipc_socket.c
===================================================================
--- uipc_socket.c (revision 302977)
+++ uipc_socket.c (working copy)
@@ -160,6 +160,7 @@
static void filt_sowdetach(struct knote *kn);
static int filt_sowrite(struct knote *kn, long hint);
static int filt_solisten(struct knote *kn, long hint);
+static void filt_sotouch(struct knote *kn, struct kevent *kev, u_long type);
static int inline hhook_run_socket(struct socket *so, void *hctx, int32_t h_id);
fo_kqfilter_t soo_kqfilter;
@@ -172,6 +173,7 @@
.f_isfd = 1,
.f_detach = filt_sordetach,
.f_event = filt_soread,
+ .f_touch = filt_sotouch,
};
static struct filterops sowrite_filtops = {
.f_isfd = 1,
@@ -3091,6 +3093,31 @@
return (0);
}
+static void
+filt_sotouch(struct knote *kn, struct kevent *kev, u_long type)
+{
+ struct socket *so = kn->kn_fp->f_data;
+
+ switch (type) {
+ case EVENT_REGISTER:
+ if (kn->kn_fop == &soread_filtops &&
+ (so->so_options & SO_ACCEPTCONN))
+ kn->kn_fop = &solisten_filtops;
+
+ kn->kn_sfflags = kev->fflags;
+ kn->kn_sdata = kev->data;
+ break;
+
+ case EVENT_PROCESS:
+ *kev = kn->kn_kevent;
+ break;
+
+ default:
+ panic("filt_sotouch() - invalid type (%ld)", type);
+ break;
+ }
+}
+
/*
* Some routines that return EOPNOTSUPP for entry points that are not
* supported by a protocol. Fill in as needed.
--1964543108-1161917916-1468942563=:30581
Content-Type: text/plain; charset="US-ASCII"; name="k.c"
Content-Transfer-Encoding: BASE64
Content-ID: <alpine.BSF.2.00.1607191735590.30581@KNOP-BEAGLE.kn.op.dlr.de>
Content-Description:
Content-Disposition: attachment; filename="k.c"
I2luY2x1ZGUgPHN5cy9zb2NrZXQuaD4NCiNpbmNsdWRlIDxzeXMvdHlwZXMu
aD4NCiNpbmNsdWRlIDxzeXMvZXZlbnQuaD4NCiNpbmNsdWRlIDxzeXMvZmls
aW8uaD4NCg0KI2luY2x1ZGUgPHN5cy9pb2N0bC5oPg0KI2luY2x1ZGUgPG5l
dGluZXQvaW4uaD4NCiNpbmNsdWRlIDxzdGRpby5oPg0KI2luY2x1ZGUgPHN0
cmluZy5oPg0KI2luY2x1ZGUgPGVyci5oPg0KDQpzdGF0aWMgdm9pZA0Kd2Fp
dF9sb29wKGludCBrcSwgaW50IHNvY2spDQp7DQoJc3RydWN0IGtldmVudCBl
dlszMl07DQoJc3RydWN0IHNvY2thZGRyX2luIGFkZHI7DQoJc29ja2xlbl90
IHNvY2tsZW47DQoNCglmb3IgKDs7KSB7DQoJCWludCBuZXYgPSBrZXZlbnQo
a3EsIE5VTEwsIDAsIGV2LCAzMiwgTlVMTCk7DQoJCWlmIChuZXYgPCAxKQ0K
CQkJZXJyKDEsICJrZXZlbnQiKTsNCgkJZm9yIChpbnQgaSA9IDA7IGkgPCBu
ZXY7ICsraSkgew0KCQkJaWYgKGV2W2ldLmlkZW50ID09IHNvY2spIHsNCgkJ
CQlwcmludGYoImFjY2VwdFxuIik7DQoJCQkJaW50IGZkID0gYWNjZXB0KGV2
W2ldLmlkZW50LA0KCQkJCSAgICAoc3RydWN0IHNvY2thZGRyICopJmFkZHIs
ICZzb2NrbGVuKTsNCgkJCQlpZiAoZmQgPT0gLTEpDQoJCQkJCWVycigxLCAi
YWNjZXB0Iik7DQoJCQl9DQoJCX0NCgl9DQp9DQoNCmludA0KbWFpbigpDQp7
DQoJc3RydWN0IHNvY2thZGRyX2luIGFkZHI7DQoNCgkvKiBvcGVuIGEgVENQ
IHNvY2tldCAqLw0KCWludCBrcSA9IGtxdWV1ZSgpOw0KDQoJaW50IHNvY2sg
PSBzb2NrZXQoUEZfSU5FVCwgU09DS19TVFJFQU0sIDApOw0KDQoJc3RydWN0
IGtldmVudCBldlsyXTsNCglFVl9TRVQoJmV2WzBdLCBzb2NrLCBFVkZJTFRf
UkVBRCwgRVZfQUREIHwgRVZfQ0xFQVIsIDAsIDAsIE5VTEwpOw0KCUVWX1NF
VCgmZXZbMV0sIHNvY2ssIEVWRklMVF9XUklURSwgRVZfQUREIHwgRVZfQ0xF
QVIsIDAsIDAsIE5VTEwpOw0KDQoJaW50IG9wdCA9IDE7DQoJc2V0c29ja29w
dChzb2NrLCBTT0xfU09DS0VULCBTT19OT1NJR1BJUEUsICZvcHQsIHNpemVv
ZihvcHQpKTsNCg0KCWlmIChrZXZlbnQoa3EsIGV2LCAyLCBOVUxMLCAwLCBO
VUxMKSA9PSAtMSkNCgkgICAgZXJyKDEsICJrZXZlbnQiKTsNCg0KCXNldHNv
Y2tvcHQoc29jaywgU09MX1NPQ0tFVCwgU09fUkVVU0VBRERSLCAmb3B0LCBz
aXplb2Yob3B0KSk7DQoNCgltZW1zZXQoJmFkZHIsIDAsIHNpemVvZihhZGRy
KSk7DQoJYWRkci5zaW5fcG9ydCA9IGh0b25zKDEwMDAwKTsNCg0KCWJpbmQo
c29jaywgKHN0cnVjdCBzb2NrYWRkciAqKSZhZGRyLCBzaXplb2YoYWRkcikp
Ow0KCWxpc3Rlbihzb2NrLCAweDgwKTsNCg0KCWlvY3RsKHNvY2ssIEZJT05C
SU8sICZvcHQpOw0KDQoJaWYgKGtldmVudChrcSwgZXYsIDIsIE5VTEwsIDAs
IE5VTEwpID09IC0xKQ0KCQllcnIoMSwgImtldmVudCIpOw0KDQoJd2FpdF9s
b29wKGtxLCBzb2NrKTsNCn0NCg==
--1964543108-1161917916-1468942563=:30581--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1607191024380.30581>