Date: Tue, 4 Oct 2016 17:29:41 +0200 From: Willem Jan Withagen <wjw@digiware.nl> To: Eric Badger <badger@FreeBSD.org>, freebsd-hackers@freebsd.org Subject: Re: Kqueue and threading Message-ID: <c0e8b588-bb6a-010e-99cf-02fb3a8253f4@digiware.nl> In-Reply-To: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3-10-2016 19:48, Eric Badger wrote:
> On 10/01/2016 13:02, Willem Jan Withagen wrote:
>> Hi,
>>
>> Ceph uses a lot of threading, and for any part of it
>> communication it uses epoll(), which FreeBSD does not use. For
>> that there was already a EvenKqueue implementation.
>>
>> But I think I'm now running into: The kqueue() system call
>> creates a new kernel event queue and returns a descriptor. The
>> queue is not inherited by a child created with fork(2). However,
>> if rfork(2) is called without the RFFDG flag, then the descriptor
>> table is shared, which will allow sharing of the kqueue between
>> two processes.
>>
>> Kqueue descriptors are created and events are added, but then
>> the process starts other threads and expects the kqueue-id to be
>> valid there.
>>
>> However adding more events returns an error, also waiting on the
>> ID for events to happen returns an error (descriptor invalid)
>>
>> Threading is done with 2 different constructions: std::thread and
>> creating Workers
>>
>> Would any of these qualify with the quoted warning? and
>> invalidate the descriptor table?
>>
>> If not, how can I (easily) debug the reason why my descriptors go
>> invalid?
>>
>
> Sharing a kqueue between threads of a process works. Are the
> workers created using rfork without RFFDG as suggested in the
> manpage? I've never had reason to do this, but a quick test seems
> to indicate that it works as advertised. A normal fork closes the
> kqueue file descriptor. If you suspect that's what's happening, you
> might run "procstat -f {worker pid}" to see if file descriptors
> with a "k" (kqueue) in the "T" (type) column appear (if not, they
> were presumably closed on the fork).
Yup, I know what the man page writes. :) But now the trick question
is: how does the C++ worker/thread library function....
Some of the code actually does: std:move on a lambda-function, which
should instantiate a complete fresh environment. And that would
certainly qualify for expected closure of the kq descriptors.
OTOH it keeps regular file descriptors available in the thread.
I guess I'll have to start testing with a few simple C++ programs to
get things clear.
--WjW
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c0e8b588-bb6a-010e-99cf-02fb3a8253f4>