From owner-freebsd-scsi@freebsd.org Tue Dec 22 09:37:49 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CBC3A4FD50 for ; Tue, 22 Dec 2015 09:37:49 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0079.outbound.protection.outlook.com [157.56.112.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 849D61C1B for ; Tue, 22 Dec 2015 09:37:47 +0000 (UTC) (envelope-from maxg@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox365.onmicrosoft.com; s=selector1-Mellanox-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+M0DkwYNy8UhCTpeZPzLdYd+/jGqvs30BWF0ye9VhVI=; b=NPhet2itMJEETVaJPVth/C5DlQ4/6qZy2+88a+a7kwULZxUfwQO7pOaSTrGILPShEmpm0EbYK7utTr3dNBjYFLTRGqkTxk37Mg5nbsxc2WgYYEx+0g4KKflZT86XumUgtE8bIGwAtHuAXuvZNKX+7+G0AWLDuSqFk+oQgSHFpxc= Received: from DB3PR05CA0021.eurprd05.prod.outlook.com (10.160.41.149) by DB3PR05MB428.eurprd05.prod.outlook.com (10.141.7.20) with Microsoft SMTP Server (TLS) id 15.1.361.13; Tue, 22 Dec 2015 09:37:38 +0000 Received: from DB3FFO11FD008.protection.gbl (2a01:111:f400:7e04::174) by DB3PR05CA0021.outlook.office365.com (2a01:111:e400:9428::21) with Microsoft SMTP Server (TLS) id 15.1.361.13 via Frontend Transport; Tue, 22 Dec 2015 09:37:38 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD008.mail.protection.outlook.com (10.47.216.97) with Microsoft SMTP Server (TLS) id 15.1.355.15 via Frontend Transport; Tue, 22 Dec 2015 09:37:36 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Tue, 22 Dec 2015 11:37:35 +0200 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Tue, 22 Dec 2015 11:37:35 +0200 Received: from [10.222.32.225] (10.222.32.225) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 22 Dec 2015 11:37:33 +0200 Subject: Re: splitting iovecs to bios To: Sagi Grimberg , Konstantin Belousov References: <56696E03.8050202@mellanox.com> <20151210150210.GY82577@kib.kiev.ua> <566D2C91.9050900@dev.mellanox.co.il> CC: Sagi Grimberg , , "Hans Petter Selasky" , Oren Duer From: Max Gurtovoy Message-ID: <567919D3.6050004@mellanox.com> Date: Tue, 22 Dec 2015 11:37:23 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <566D2C91.9050900@dev.mellanox.co.il> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.222.32.225] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD008; 1:XZn1UykLTb956ROePvxE2uk2/u48urzXWXaeH1S1aJt7Xd+Iwii9+7aEaU9LKAnI5L/wXKzAqBZnH1d4vVLvCeE47+n2wV+uN3nY2EJX6ZMVik6r5wh1M3TETb613W2AKEq6SRZnqPDcB6yx6Kbi/zXR+9dJjhGQ/1ewCZHCefJqogEe8W6K/OcNq+hVcXXiRzbHvRQaJsdctBP2vkyiG8QZfItqImY01latLY4fpGn3jZTtwy3gTA9Lll+ycEBdxfqHJ+Jii3eP5cHrTlqe+150V5skwB9UoVyqrdxu32EZDeLYKwvJL0xaCX19/cfV2IXPsIqnzFFXW6bpEgh64qXH1eB/BNjDoof/vkamw0LFxLFCE/c6u/kF9EvnpY5ieUcalC7usQNlXLPx4FNtjQ== X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(189002)(199003)(164054003)(99136001)(50986999)(33656002)(97736004)(80316001)(1096002)(1220700001)(64126003)(65806001)(87936001)(87266999)(50466002)(76176999)(65956001)(86362001)(83506001)(54356999)(107886002)(5004730100002)(11100500001)(586003)(3846002)(5001770100001)(4001430100002)(65816999)(6806005)(5008740100001)(23746002)(59896002)(106466001)(36756003)(77096005)(4001350100001)(47776003)(2950100001)(92566002)(230700001)(6116002)(189998001)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR05MB428; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:ErrorRetry; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB428; 2:ZQDDmaBp4lD/Ct47mdHOhrN1l/IB7f2v+yqUawOT+e8EiSD3aBEhCAGBWVUE9Y/O8gf1OBNK0/4BHI5L6iE9iAK2qL+TebRc7wYI6t0Wj8YpiC22jVv0ckmQI9aT/H0IE1V5Ksl1DEYmSrGllU353w==; 3:LuVbo2kULhfh6Y4ob40XC/Dz155qQXAJQ30V329QUtHNwOummfw+47nNvudw+3fRaoOJ0GpX9qCrZJPKI1rB/h+3hnASx+hyJA4DoU80yy4v9R9lYOtv2p0Dspz5US8A1EGRxOBbrtcpizaYCae1vQp13QpXGMXdRjlIfBaeFTKY49ByAdwYcbT7MvgaeHiI6Sp4JW04BAQMpoJzm/YhEyVPyGay23X+XZIudNkBsgX8tQiSNeRu9UyDQKcdRvoSpw9HyBaVs1JNcIeUjjkjZg==; 25:UWNRYoniKyQ/1iRU9+Xs0Qg5z4obcQujOLLlDtZrlKRVnjBa1lD+Xu+Kh5mCzz6sHQ8StoZa5aw3CdOwoTcoCEgjlWPLVV2xV17eWul8LsnRM//jfAEjLzlsJctRbWX99kemEG9PRLBkbQ1lPHYne8tWiFOQvZobWlVCojTSihUybvTois4yq7NwbRQhT/g0M0lU9+jxvgQfHzuBtD+8iounU6UajIJMIFfKMBOUXKnnDoOv+NUlNIib8ggfwXMCCn6yBO8a17jZBrmXG7Qvjw== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501001); SRVR:DB3PR05MB428; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB428; 20:+76FK+Kh5dEOEEF1KNt0vufdJ/75O2TaiMbhBso4ETv8WmgzvlRGWRDsy3XDUG0dY32zQcnAB+wq0tIdqvnZFnNRn/Pz1CZ9AbUVgxmVkI0DRdlN6xSyjuCgtfG3MIOklcKQh2wsVf/4aegIa3SXri6FlYNsudbS1ac2IHgZTmuW6qmOIPzz7HGW+j1ZoDxbYrvCEh50fcPni23hfJWquViScMVmZHnj9ghTnVzZ/o8+BmHvM80VZFFcq3leMrxJIQTTDJV3bvDxhYQ3dc7fmpHkD1eyjzBXHBsCL0+NNV7yh/hHUyzmljLensyq4ys+WciYvJ1/p6yKDy52njBh8yIx58i6PoR8xCidh4ARDHnRyv+HbBmKfvh2WXZ7H1chaw8BWDWCcnQLrupJWIpWNZpqO4+2fOE8TCFhax44UJtVroIQrHgw1nGYTfxW2oiEwASYcokyzOTABeEDeE2pIdolBDaZ55vYeCEnvoqnCeTjfJsMGQYYHpTybcMHU/Y8; 4:3DvFGR/6WPxj9x8aSii/XaWRq+v+ZdSD51OewoQzVfJTQxMENxAPzMk3KMq20e+wf4WwHbiJ0QNHVk+MXoV0w8AlBBwTyi4LHSLv+02o5ztCAfECwkgxGAIr279I4L6QBAso+TAALWxpLD2SnoJxX2NZx6NY8uqJdVReiydS6p7Gairr4PnkMJ3Z5qUj3vsiOqxqUUEpGAPDwv3eR3pUMCVXJXdGCaiLbXKwmjJ+FSTuHoJc343d3CyDaqtxtWrzoLHSPcoxsD8G7SlVe3qUPOWHiFKh2ZvmQ66ZUgvH4dQoujDY585Pi5Ey2ClG7+Yqh9LcGo2n8CjBrJMljeNtHThPQCrJaoBQ++nbMiSOxKlc2AYi7/prqrkRq8v4EflE X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(520078)(10201501046)(3002001); SRVR:DB3PR05MB428; BCL:0; PCL:0; RULEID:; SRVR:DB3PR05MB428; X-Forefront-PRVS: 0798146F16 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; DB3PR05MB428; 23:NHOrX8ad5zqGPk9mRvCRMuK4cIsChsnbdP+xJr?= =?Windows-1252?Q?f1M+tIk3lLirsYbIMyaPp+I8G8TBIZ6nK2eZxsQAD3N05TOBOg4JoefU?= =?Windows-1252?Q?Eocrt9/k5aLL0fIO2ntq4O+iXXpZpV4QSc3EJeENqD9739eYpOFPIBap?= =?Windows-1252?Q?WH1zldEGcfkZPefOvmpLdyUvKdnCkp73ivqGBWTatSE06ZJ9b5c7Ew+y?= =?Windows-1252?Q?2OEo+lSqw9zbS5COVJAGokdFXPsFfB4XyPGp3CTQfS1ynpCIxvjMZi+u?= =?Windows-1252?Q?UQepHI2ryHSZNQkvNoq0lHfAB5BLBBGciewbHY4WovGd0cEFMi+ocEL2?= =?Windows-1252?Q?wNS7nMZCOifuEwdT44XjLfgJD/89rBTPxv+5PhckniA03Kd/eCtfjsEU?= =?Windows-1252?Q?lB9n5qGQf9j4bW8jCaExLGqCjUobSR6VF7gTrR1ab8nhCtycbXiEVhpP?= =?Windows-1252?Q?Pt4/UcnrWn40jTwwLJsY7duixS2yM/AndyEAAmS9glWV2skewRVamffX?= =?Windows-1252?Q?dHykV5RKHE6Coer0G/GoX+q247NCieDfDVryfK2ZeXN/SGQNxgc6Xtbr?= =?Windows-1252?Q?oaVXUkDqNsccVn947uyXqySOzwrHKyHXSqEQzIuQvqrYZmuRMMPHKTQu?= =?Windows-1252?Q?PH1Pfm2UPCdt8ymzOfZf63/ma17is5DN58P7uz9JutHvYCp14Frc+bQN?= =?Windows-1252?Q?6jFVUMAnpj2RbO7Wndh+7864dCVDtmnqzvSOt9kW8MkGIwy+9RJU6c9W?= =?Windows-1252?Q?yP8P4XE7qh3LsYqJ9w2h174/gOSUgT8VDdijFMCkPFtFRhCwJ3dRYprd?= =?Windows-1252?Q?LWZl+jhK+7oSXsQJo9ClcBjP3Oy9nuv3PWeCHiEzSBUprNLqeggxstd/?= =?Windows-1252?Q?fIL/BVYytU8F+ZfNe6xugArT7s3ucp6yIIvfhyI9cuxF0cGIvKtWtLoK?= =?Windows-1252?Q?6k7n8Zw1XSIrYiJelCdphhtY6mRMYEQrls34e7a3/DcuAqYeHmWwlLV1?= =?Windows-1252?Q?Dkb9lSYEwmY/LcUhDR0kVmsaPxwlVe2e/riOVwoUGmD24l6orNZowJGR?= =?Windows-1252?Q?0rk9KEuWuFgyJHWd2ncIQ+TZaURki+pLzkMZddsJdJ4BDh8zz0lAGQvF?= =?Windows-1252?Q?VwBTfTPsV/PDeOGrxdicswq2AO7YpT9/4KUSdtCSvksn1U1zZTo3UbBS?= =?Windows-1252?Q?DChu7F+vCxPdsw2EdISog1kgw3ZZgi6oThOxKM2Bl5cDE2VVLWRNvsLj?= =?Windows-1252?Q?pYaVhZUaQBnQRTkXlDcGsDr9iOO1YOMHE+T880escwJmFIIKUFyRDL6X?= =?Windows-1252?Q?hK?= X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB428; 5:8C5bjWKs/DjJmy5FpMQbhoW5rvixy6YbSI3/alnKewBa9+7QxFAekWoeOJcevI1sz55wLeEAYznmloaysehIMup4pabzJ8R8XCGQV4WGay81ynJjkjhqVZ/Cy0P9v9mgORRGavJGHlZk0YlPqlmEow==; 24:MNBr2/4dj5/ZzoSjJepGgEeJQBEQ5IcCaDNFT138rcSrnoCnJfWXx54iPRsBla6y+uPy/WK36MnSRY6ugHQKMZUXvdo8w6xJ5WH03b4cbTU= X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Dec 2015 09:37:36.1489 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR05MB428 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2015 09:37:49 -0000 Hi Konstantin, > >> There might be indeed a reason, it could be that some drivers expect >> blocking to be done by the userspace. The drivers could have some >> restrictions on transfer sizes and atomicity of transfer, which would >> be broken by the unconditional merge. I cannot give you an example >> of such driver, known block-aware drivers like sa(4) only require the >> bio size to be multiple of the basic block size. > > I'm surprised to learn that the generic access layer splits IO requests > just because some block drivers cannot handle it. I'd expect that this > sort of limitation would be communicated by the drivers in the form of > device flag SI_NOMERGE. > >> OTOH, I see no issue with adding a SI_PHYSIOMERGE flag and doing the >> merges for the driver in physio(), when unmapped request has consequtive >> iov elements ending and starting at the page boundary. > > I'd say it should be the other way around, physio would always strive > to append/merge iov elements but wouldn't in case the device does not > support it. Moreover, some modern devices does not even require the page > boundary alignment you mentioned. These devices can execute IO to/from > any arbitrary scatter list of buffers. Do you know if this issue is on someone's plate ? If it doesn't, maybe we can try to advance it and start implementing some solutions. As I said earlier and as Sagi mentioned, this feature can improve the performance of modern devices. Thanks, Max. From owner-freebsd-scsi@freebsd.org Tue Dec 22 09:51:39 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FE90A4E183 for ; Tue, 22 Dec 2015 09:51:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C557E125C for ; Tue, 22 Dec 2015 09:51:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBM9pWhT001639 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 22 Dec 2015 11:51:32 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBM9pWhT001639 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBM9pVX0001638; Tue, 22 Dec 2015 11:51:31 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 22 Dec 2015 11:51:31 +0200 From: Konstantin Belousov To: Max Gurtovoy Cc: Sagi Grimberg , Sagi Grimberg , freebsd-scsi@freebsd.org, Hans Petter Selasky , Oren Duer Subject: Re: splitting iovecs to bios Message-ID: <20151222095131.GJ3625@kib.kiev.ua> References: <56696E03.8050202@mellanox.com> <20151210150210.GY82577@kib.kiev.ua> <566D2C91.9050900@dev.mellanox.co.il> <567919D3.6050004@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <567919D3.6050004@mellanox.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2015 09:51:39 -0000 On Tue, Dec 22, 2015 at 11:37:23AM +0200, Max Gurtovoy wrote: > Hi Konstantin, > > > > >> There might be indeed a reason, it could be that some drivers expect > >> blocking to be done by the userspace. The drivers could have some > >> restrictions on transfer sizes and atomicity of transfer, which would > >> be broken by the unconditional merge. I cannot give you an example > >> of such driver, known block-aware drivers like sa(4) only require the > >> bio size to be multiple of the basic block size. > > > > I'm surprised to learn that the generic access layer splits IO requests > > just because some block drivers cannot handle it. I'd expect that this > > sort of limitation would be communicated by the drivers in the form of > > device flag SI_NOMERGE. > > > >> OTOH, I see no issue with adding a SI_PHYSIOMERGE flag and doing the > >> merges for the driver in physio(), when unmapped request has consequtive > >> iov elements ending and starting at the page boundary. > > > > I'd say it should be the other way around, physio would always strive > > to append/merge iov elements but wouldn't in case the device does not > > support it. Moreover, some modern devices does not even require the page > > boundary alignment you mentioned. These devices can execute IO to/from > > any arbitrary scatter list of buffers. > > Do you know if this issue is on someone's plate ? > If it doesn't, maybe we can try to advance it and start implementing > some solutions. > As I said earlier and as Sagi mentioned, this feature can improve the > performance of modern devices. Sure, feel free to implement it. If you need a help, just ask.