From owner-freebsd-current@freebsd.org Sat Oct 7 20:34:09 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77EDFE41CCC for ; Sat, 7 Oct 2017 20:34:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0062.outbound.protection.outlook.com [104.47.37.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D47671AB3; Sat, 7 Oct 2017 20:34:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM (52.132.78.18) by YQXPR0101MB1013.CANPRD01.PROD.OUTLOOK.COM (52.132.78.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Sat, 7 Oct 2017 20:34:06 +0000 Received: from YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM ([fe80::7548:a72a:e054:70d5]) by YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM ([fe80::7548:a72a:e054:70d5%13]) with mapi id 15.20.0077.011; Sat, 7 Oct 2017 20:34:06 +0000 From: Rick Macklem To: Ian Lepore , "freebsd-current@freebsd.org" Subject: Re: RFC how to use kernel procs/threads efficiently Thread-Topic: RFC how to use kernel procs/threads efficiently Thread-Index: AQHTPtSLQYujf8n5KEGHc2bjAF54kqLXMAcAgAGndEo= Date: Sat, 7 Oct 2017 20:34:06 +0000 Message-ID: References: , <1507317060.86205.268.camel@freebsd.org> In-Reply-To: <1507317060.86205.268.camel@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQXPR0101MB1013; 6:c7azBMsukv4rIUXksMaUYWS/YPHbHUYdmF4mmicK4XdTi93s3MIVaZ7XnmxmgL3LAbXoLrAr8gl+/FpusOaY4C32X68iF0a48ZRJqBXpUL91AIBzo3S8ozC0OQ31tn0gE0jF1wjFl41N3lZvFlLImizKfOC+ZWvL7h2Z7kgOSjNbJZTVWisK9BFK+pWNvbE0khIywOjb5gxKXPU11aYDL7ma1lKPU8GxMeqLf3BF17URb/eJkrA5ZM+p0a8PP1S1cdE1o+z/++FGDwqiF3Z/Q2C/DlDTovW2h6uzXy+1A3nVhuxhP52M7vrNJnHltonztht/EVAHbr3EoFCkl3eOPg==; 5:QyHAGkG/4WCsbROmsCTaiqK4XcwgUXGQcvv8fa8lXKI5EV7WB5qPwJiFT5ERTToUcSv0DgrbveSsryn2dW1q+n2Op7EIO6tNRwuEzyRywCuKT3bcMlNVAXiQVsRP/QXnm+ajNHNXPDAa4wVcX/GSQQ==; 24:n7lC1cEdBVwNo94habGKSh+jp7p9UFcQks+OempSqcgT5DON5eGkS+lPtP2NgRbYaED7tECF4//zIsDuoiy867xsV8CCn3ZBY3xknPhKONA=; 7:q1v2O7zoM9nqE36gDu23JPXV8+lLKrV7l8FfrAGlm60mnSIKTO3PUVVOdqqZ31n2KJ5UUEFLYcmx5wkSoR9ZROY38PqLh2IaOBeB0aa6k3Vf5L4G5BOvOsOKCNcyMpx1DO3Xt4wdq+TAPi24XISnWVB4nsUh76vuKpLq8u3stYRuUStyEEDTySTD4q/sqyK03Dh4DFVVVecYI2+cGaXnNbRFfYOlDlHD0bkvl8qtWyI= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: fcd69c7a-efd0-4925-cc75-08d50dc2c190 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:YQXPR0101MB1013; x-ms-traffictypediagnostic: YQXPR0101MB1013: x-exchange-antispam-report-test: UriScan:(158342451672863)(5213294742642); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(3002001)(6041248)(20161123562025)(20161123555025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123564025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YQXPR0101MB1013; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YQXPR0101MB1013; x-forefront-prvs: 045315E1EE x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(189002)(377424004)(199003)(24454002)(51914003)(5660300001)(68736007)(97736004)(106356001)(86362001)(74316002)(786003)(8936002)(316002)(450100002)(81166006)(229853002)(6436002)(14454004)(76176999)(6506006)(7696004)(102836003)(110136005)(74482002)(55016002)(6246003)(25786009)(305945005)(2906002)(189998001)(101416001)(2950100002)(478600001)(8676002)(2501003)(81156014)(50986999)(53936002)(3280700002)(9686003)(3660700001)(54356999)(2900100001)(33656002)(105586002)(5250100002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQXPR0101MB1013; H:YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Oct 2017 20:34:06.4164 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB1013 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Oct 2017 20:34:09 -0000 Ian Lepore wrote: >On Fri, 2017-10-06 at 19:02 +0000, Rick Macklem wrote: >> Hi, >> >> I have now dropped the client side of Flexible File Layout for pNFS into= head >> and I believe it is basically working. >> Currently when talking to mirrored DS servers, it does the Write and Com= mit >> RPCs to the mirrors serially. This works, but is inefficient w.r.t. elap= sed to to >> completion. >> >> To do them concurrently, I need separate kernel processes/threads to do = them. >> I can think of two ways to do this: >> 1 - The code that I have running in projects/pnfs-planb-server for the p= NFS server >> side does a kproc_create() to create a kernel process that does th= e RPC and >> then krpc_exit()s. >> - This was easy to code and works. However, I am concerned that th= ere is >> going to be excessive overheads from doing all the kproc_create(= )s and >> kproc_exit()s? >> Anyone know if these calls will result in large overheads? >> 2 - I haven't coded this, but the other way I can think of to do this is= to >> create a pool of threads (kthread_create() is sufficient in this c= ase, I >> think?) and then hand each RPC to an available thread so it can do= the RPC. >> - Other than a little more complex coding, the main issue I see wi= th this one >> is "How many threads and when to create more/less of them.". >> >> Anyhow, any comments w.r.t. the merits of either of the above approaches >> (or a suggestion of other ways to do this) would be appreciated, rick > >taskqueue(9) is an existing mechanism to enqueue functions to execute >asynch using a pool of threads, but it doesn't answer the scalability >questions. In fact it may make them harder, inasmuch as I don't think >there's a mechanism to dynamically adjust the number of threads after >first calling taskqueue_start_threads(). Hmm, yes. Thanks for the pointer. I hadn't read "man taskqueue" until now. The kernel RPC doesn't use this and I suspect that it is because of what yo= u said w.r.t. dynamically adjusting the # of threads. However, it does save "hand coding" the queues for #2 and I'm lazy (plus don't believe reinventing the wheel is the best plan). I think I will try using taskqueue and just have a sysctl for #of-threads. (Actually most of the code ends up the same, because basically they all end up with a function with a single argument that does the RPC. The only difference is what call starts the RPC.) Anyone else have comments? rick=