From owner-freebsd-hackers@freebsd.org Tue Oct 4 15:29:51 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10CD4AF50D3 for ; Tue, 4 Oct 2016 15:29:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF3BF78D; Tue, 4 Oct 2016 15:29:50 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 7E3A92A40E; Tue, 4 Oct 2016 17:29:46 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 26jRu5jhkAxv; Tue, 4 Oct 2016 17:29:45 +0200 (CEST) Received: from [192.168.101.51] (vpn.ecoracks.nl [176.74.240.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 7FFF32A40D; Tue, 4 Oct 2016 17:29:45 +0200 (CEST) Subject: Re: Kqueue and threading To: Eric Badger , freebsd-hackers@freebsd.org References: <111e0c35-7a4b-b6c7-ef1d-1a0d85112e61@digiware.nl> <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> From: Willem Jan Withagen Message-ID: Date: Tue, 4 Oct 2016 17:29:41 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <4209b8d4-6674-a51d-dceb-81c3ecd179c2@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Oct 2016 15:29:51 -0000 On 3-10-2016 19:48, Eric Badger wrote: > On 10/01/2016 13:02, Willem Jan Withagen wrote: >> Hi, >> >> Ceph uses a lot of threading, and for any part of it >> communication it uses epoll(), which FreeBSD does not use. For >> that there was already a EvenKqueue implementation. >> >> But I think I'm now running into: The kqueue() system call >> creates a new kernel event queue and returns a descriptor. The >> queue is not inherited by a child created with fork(2). However, >> if rfork(2) is called without the RFFDG flag, then the descriptor >> table is shared, which will allow sharing of the kqueue between >> two processes. >> >> Kqueue descriptors are created and events are added, but then >> the process starts other threads and expects the kqueue-id to be >> valid there. >> >> However adding more events returns an error, also waiting on the >> ID for events to happen returns an error (descriptor invalid) >> >> Threading is done with 2 different constructions: std::thread and >> creating Workers >> >> Would any of these qualify with the quoted warning? and >> invalidate the descriptor table? >> >> If not, how can I (easily) debug the reason why my descriptors go >> invalid? >> > > Sharing a kqueue between threads of a process works. Are the > workers created using rfork without RFFDG as suggested in the > manpage? I've never had reason to do this, but a quick test seems > to indicate that it works as advertised. A normal fork closes the > kqueue file descriptor. If you suspect that's what's happening, you > might run "procstat -f {worker pid}" to see if file descriptors > with a "k" (kqueue) in the "T" (type) column appear (if not, they > were presumably closed on the fork). Yup, I know what the man page writes. :) But now the trick question is: how does the C++ worker/thread library function.... Some of the code actually does: std:move on a lambda-function, which should instantiate a complete fresh environment. And that would certainly qualify for expected closure of the kq descriptors. OTOH it keeps regular file descriptors available in the thread. I guess I'll have to start testing with a few simple C++ programs to get things clear. --WjW