Date: Mon, 3 Oct 2016 12:48:03 -0500 From: Eric Badger <badger@FreeBSD.org> To: freebsd-hackers@freebsd.org Cc: wjw@digiware.nl Subject: Re: Kqueue and threading Message-ID: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> In-Reply-To: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D
Content-Type: multipart/mixed; boundary="NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9";
protected-headers="v1"
From: Eric Badger <badger@FreeBSD.org>
To: freebsd-hackers@freebsd.org
Cc: wjw@digiware.nl
Message-ID: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org>
Subject: Re: Kqueue and threading
References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl>
In-Reply-To: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl>
--NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 10/01/2016 13:02, Willem Jan Withagen wrote:
> Hi,
>
> Ceph uses a lot of threading, and for any part of it communication it
> uses epoll(), which FreeBSD does not use. For that there was already a
> EvenKqueue implementation.
>
> But I think I'm now running into:
> The kqueue() system call creates a new kernel event queue and
> returns a descriptor. The queue is not inherited by a child
> created with fork(2). However, if rfork(2) is called without the
> RFFDG flag, then the descriptor table is shared, which will allow
> sharing of the kqueue between two processes.
>
> Kqueue descriptors are created and events are added, but then the
> process starts other threads and expects the kqueue-id to be valid ther=
e.
>
> However adding more events returns an error, also waiting on the ID for=
> events to happen returns an error (descriptor invalid)
>
> Threading is done with 2 different constructions:
> std::thread
> and creating Workers
>
> Would any of these qualify with the quoted warning? and invalidate the
> descriptor table?
>
> If not, how can I (easily) debug the reason why my descriptors go inval=
id?
>
Sharing a kqueue between threads of a process works. Are the workers
created using rfork without RFFDG as suggested in the manpage? I've
never had reason to do this, but a quick test seems to indicate that it
works as advertised. A normal fork closes the kqueue file descriptor. If
you suspect that's what's happening, you might run "procstat -f {worker
pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
column appear (if not, they were presumably closed on the fork).
Eric
--NJmlvC3pRPsoUetngw9qSPsiTUs965LJ9--
--3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQGQBAEBCgB6BQJX8pnfXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzQTlFODAxM0JDQTdDOTQ1ODI1Mzc3NTk2
MkU1MDA5NjVBM0YyNEFDExxiYWRnZXJAZnJlZWJzZC5vcmcACgkQYuUAllo/JKzc
cQgA3/9zl9I4TZDdn+Gljbt7w/q/g0tiDoJudqogcNaR0cIwIl91dUh5KuchcuHd
EdEwsVDoWoNDlESLdGpDo4dHnifVT1ax9tQKt4pdRwFd8/eg5GYvekZa3AnWTCFX
m7VDE9osJpMKROoh+moLqufmkegH9iXtcsPRhpOExtou4zX9kZ8D6znmzn1DlMRS
csO5ym8qxWl6Aeh2FRqoYwnezZw3qA0bYS+Rd0W2UyA8p8VF1eEAqXqNoAyW+DQ0
U6mv1LKcOsp1aDAVlVeuqJ/sQl25c53QjuFx0rLVJSiSO7haswCy9CBEUTIZBGiW
PGJJr6SycNN6SgX2cZRu9SyzWA==
=Q9ht
-----END PGP SIGNATURE-----
--3D4nSlAjJQtbUB3ovR0IQ2RN0FjcQx52D--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4209b8d4-6674-a51d-dceb-81c3ecd179c2>