From owner-freebsd-hackers@freebsd.org Mon Oct 3 17:50:18 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2827FAF4446 for ; Mon, 3 Oct 2016 17:50:18 +0000 (UTC) (envelope-from badger@FreeBSD.org) Received: from sasl.smtp.pobox.com (pb-smtp2.pobox.com [64.147.108.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB9B2E14 for ; Mon, 3 Oct 2016 17:50:17 +0000 (UTC) (envelope-from badger@FreeBSD.org) Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id D334E415B7; Mon, 3 Oct 2016 13:48:22 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=subject:to :references:from:cc:message-id:date:mime-version:in-reply-to :content-type; s=sasl; bh=EVP55RLT5hAfoIp1di1Kk0a9RWA=; b=ykVmTQ kPtO2K9AGzobZ5w28VC9+ZojuWmR9ADx30Sg7pm1RWMefTjoZ74d68DTw2fjOwgM gZ4OnoTW+G6395Rq31IvLdR2LGKr+37W6+mraXfViZhqKgpbqXyL/GyGL0tkkAur Tj/+c54WGnjujrGMpwRASmBvFD1ILc84RHE5A= Received: from pb-smtp2.nyi.icgroup.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id CC514415B6; Mon, 3 Oct 2016 13:48:22 -0400 (EDT) Received: from [172.31.100.239] (unknown [76.164.8.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pb-smtp2.pobox.com (Postfix) with ESMTPSA id 5F29D415B3; Mon, 3 Oct 2016 13:48:22 -0400 (EDT) Subject: Re: Kqueue and threading To: freebsd-hackers@freebsd.org References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> From: Eric Badger Cc: wjw@digiware.nl Message-ID: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> Date: Mon, 3 Oct 2016 12:48:03 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D" X-Pobox-Relay-ID: 93D2DF34-8991-11E6-950A-EAAE7A1B28F4-46178211!pb-smtp2.pobox.com X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2016 17:50:18 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D Content-Type: multipart/mixed; boundary="NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9"; protected-headers="v1" From: Eric Badger To: freebsd-hackers@freebsd.org Cc: wjw@digiware.nl Message-ID: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> Subject: Re: Kqueue and threading References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> In-Reply-To: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> --NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 10/01/2016 13:02, Willem Jan Withagen wrote: > Hi, > > Ceph uses a lot of threading, and for any part of it communication it > uses epoll(), which FreeBSD does not use. For that there was already a > EvenKqueue implementation. > > But I think I'm now running into: > The kqueue() system call creates a new kernel event queue and > returns a descriptor. The queue is not inherited by a child > created with fork(2). However, if rfork(2) is called without the > RFFDG flag, then the descriptor table is shared, which will allow > sharing of the kqueue between two processes. > > Kqueue descriptors are created and events are added, but then the > process starts other threads and expects the kqueue-id to be valid ther= e. > > However adding more events returns an error, also waiting on the ID for= > events to happen returns an error (descriptor invalid) > > Threading is done with 2 different constructions: > std::thread > and creating Workers > > Would any of these qualify with the quoted warning? and invalidate the > descriptor table? > > If not, how can I (easily) debug the reason why my descriptors go inval= id? > Sharing a kqueue between threads of a process works. Are the workers created using rfork without RFFDG as suggested in the manpage? I've never had reason to do this, but a quick test seems to indicate that it works as advertised. A normal fork closes the kqueue file descriptor. If you suspect that's what's happening, you might run "procstat -f {worker pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type) column appear (if not, they were presumably closed on the fork). Eric --NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9-- --3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGQBAEBCgB6BQJX8pnfXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzQTlFODAxM0JDQTdDOTQ1ODI1Mzc3NTk2 MkU1MDA5NjVBM0YyNEFDExxiYWRnZXJAZnJlZWJzZC5vcmcACgkQYuUAllo/JKzc cQgA3/9zl9I4TZDdn+Gljbt7w/q/g0tiDoJudqogcNaR0cIwIl91dUh5KuchcuHd EdEwsVDoWoNDlESLdGpDo4dHnifVT1ax9tQKt4pdRwFd8/eg5GYvekZa3AnWTCFX m7VDE9osJpMKROoh+moLqufmkegH9iXtcsPRhpOExtou4zX9kZ8D6znmzn1DlMRS csO5ym8qxWl6Aeh2FRqoYwnezZw3qA0bYS+Rd0W2UyA8p8VF1eEAqXqNoAyW+DQ0 U6mv1LKcOsp1aDAVlVeuqJ/sQl25c53QjuFx0rLVJSiSO7haswCy9CBEUTIZBGiW PGJJr6SycNN6SgX2cZRu9SyzWA== =Q9ht -----END PGP SIGNATURE----- --3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D--