From owner-freebsd-current@freebsd.org Fri Oct 6 19:11:03 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EEBCE3F071 for ; Fri, 6 Oct 2017 19:11:03 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5304A6BAF8 for ; Fri, 6 Oct 2017 19:11:03 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 1c9b575f-aaca-11e7-a937-4f970e858fdb X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id 1c9b575f-aaca-11e7-a937-4f970e858fdb; Fri, 06 Oct 2017 19:11:10 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v96JB0KQ006513; Fri, 6 Oct 2017 13:11:00 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1507317060.86205.268.camel@freebsd.org> Subject: Re: RFC how to use kernel procs/threads efficiently From: Ian Lepore To: Rick Macklem , "freebsd-current@freebsd.org" Date: Fri, 06 Oct 2017 13:11:00 -0600 In-Reply-To: References: Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Oct 2017 19:11:03 -0000 On Fri, 2017-10-06 at 19:02 +0000, Rick Macklem wrote: > Hi, > > I have now dropped the client side of Flexible File Layout for pNFS into head > and I believe it is basically working. > Currently when talking to mirrored DS servers, it does the Write and Commit > RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed to to > completion. > > To do them concurrently, I need separate kernel processes/threads to do them. > I can think of two ways to do this: > 1 - The code that I have running in projects/pnfs-planb-server for the pNFS server >       side does a kproc_create() to create a kernel process that does the RPC and >       then krpc_exit()s. >       - This was easy to code and works. However, I am concerned that there is >         going to be excessive overheads from doing all the kproc_create()s and >         kproc_exit()s? >        Anyone know if these calls will result in large overheads? > 2 - I haven't coded this, but the other way I can think of to do this is to >       create a pool of threads (kthread_create() is sufficient in this case, I >       think?) and then hand each RPC to an available thread so it can do the RPC. >       - Other than a little more complex coding, the main issue I see with this one >         is "How many threads and when to create more/less of them.". > > Anyhow, any comments w.r.t. the merits of either of the above approaches > (or a suggestion of other ways to do this) would be appreciated, rick taskqueue(9) is an existing mechanism to enqueue functions to execute asynch using a pool of threads, but it doesn't answer the scalability questions.  In fact it may make them harder, inasmuch as I don't think there's a mechanism to dynamically adjust the number of threads after first calling taskqueue_start_threads(). -- Ian