From owner-freebsd-hackers@freebsd.org Mon Oct 17 13:41:46 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C985DC12BBE for ; Mon, 17 Oct 2016 13:41:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 92245D31; Mon, 17 Oct 2016 13:41:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 806E72A08D; Mon, 17 Oct 2016 15:41:42 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gJAvDMcABS-w; Mon, 17 Oct 2016 15:41:41 +0200 (CEST) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id AD1892A089; Mon, 17 Oct 2016 15:41:41 +0200 (CEST) Subject: Re: Kqueue and threading To: Eric Badger , freebsd-hackers@freebsd.org References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> From: Willem Jan Withagen Message-ID: <6441cf69-f559-fe5a-fe9e-464b0f13c3b4@digiware.nl> Date: Mon, 17 Oct 2016 15:41:39 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Oct 2016 13:41:46 -0000 On 3-10-2016 19:48, Eric Badger wrote: > On 10/01/2016 13:02, Willem Jan Withagen wrote: >> Hi, >> >> Ceph uses a lot of threading, and for any part of it communication it >> uses epoll(), which FreeBSD does not use. For that there was already a >> EvenKqueue implementation. >> >> But I think I'm now running into: >> The kqueue() system call creates a new kernel event queue and >> returns a descriptor. The queue is not inherited by a child >> created with fork(2). However, if rfork(2) is called without the >> RFFDG flag, then the descriptor table is shared, which will allow >> sharing of the kqueue between two processes. >> >> Kqueue descriptors are created and events are added, but then the >> process starts other threads and expects the kqueue-id to be valid there. >> >> However adding more events returns an error, also waiting on the ID for >> events to happen returns an error (descriptor invalid) >> >> Threading is done with 2 different constructions: >> std::thread >> and creating Workers >> >> Would any of these qualify with the quoted warning? and invalidate the >> descriptor table? >> >> If not, how can I (easily) debug the reason why my descriptors go invalid? >> > > Sharing a kqueue between threads of a process works. Are the workers > created using rfork without RFFDG as suggested in the manpage? I've > never had reason to do this, but a quick test seems to indicate that it > works as advertised. A normal fork closes the kqueue file descriptor. If > you suspect that's what's happening, you might run "procstat -f {worker > pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type) > column appear (if not, they were presumably closed on the fork). Right, This has become a rather serious and painful undertaking, but it seems to work now, and all tests seem to be working oke .... But getting at the workers creation is rather impossible since that is mostly hidden in some Boost libs... Fortunate the systemcode for events is rather fleshed out into modules and there is a EventKqueue class, that hides all gory details. So there I main lists of what events are being set, and once I detect a change of thread, I reinstall the list with events. But that is obviously rather cumbersome. --WjW So what I do know