From owner-freebsd-geom@freebsd.org Thu Dec 10 12:20:54 2015 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79F3C9D72D5; Thu, 10 Dec 2015 12:20:54 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0075.outbound.protection.outlook.com [157.55.234.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89C6E13BA; Thu, 10 Dec 2015 12:20:52 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from AM3PR05CA0054.eurprd05.prod.outlook.com (10.162.114.22) by DB4PR05MB541.eurprd05.prod.outlook.com (10.242.155.142) with Microsoft SMTP Server (TLS) id 15.1.355.16; Thu, 10 Dec 2015 12:20:43 +0000 Received: from AM1FFO11FD035.protection.gbl (2a01:111:f400:7e00::111) by AM3PR05CA0054.outlook.office365.com (2a01:111:e400:52b7::22) with Microsoft SMTP Server (TLS) id 15.1.355.16 via Frontend Transport; Thu, 10 Dec 2015 12:20:43 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by AM1FFO11FD035.mail.protection.outlook.com (10.174.64.224) with Microsoft SMTP Server (TLS) id 15.1.337.8 via Frontend Transport; Thu, 10 Dec 2015 12:20:41 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Thu, 10 Dec 2015 14:20:37 +0200 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Thu, 10 Dec 2015 14:20:37 +0200 Received: from [10.222.32.225] (10.222.32.225) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 10 Dec 2015 14:20:25 +0200 To: , Sagi Grimberg , Oren Duer , Hans Petter Selasky , From: Max Gurtovoy Subject: splitting iovecs to bios Message-ID: <56696E03.8050202@mellanox.com> Date: Thu, 10 Dec 2015 14:20:19 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1255"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.222.32.225] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11FD035; 1:CoVvXxpIKf3OjH7y8IX/Eck1IkCjDV1v0/VmFb+JK2/Jc9xhnjUm8YqZ8zROvw4HM2JsZEhjR+ToVnZcKO6dOz0GZFWtw3ObloUTAQNZGGgg463c+qwlpnmM1fqtkBXTZOmd2CGiMKbai2J5wXovCw7N7mqelsDS3slTD42+DR+ed3GXxZ4cG5f/X1EWIgwQ4gJTFrrzAV/Aba4WB/cr3zuh22s/YvF/UbLFZjR5rs3Iyvhh9yBFQG7G8cDIn5TCZofZHJZffwFEjeI7Rw92cKeD36d1fDZ/bTlEJW2s3S0EqGPTuj9pTPhQM+h86qbkvdWfJgyV4eZmIYs/fX19D4jvZgsopmSq9VdVQz1AbAM= X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(164054003)(53754006)(199003)(189002)(230700001)(3846002)(1220700001)(80316001)(6116002)(586003)(1096002)(47776003)(50466002)(87266999)(36756003)(50986999)(11100500001)(64126003)(450100001)(189998001)(54356999)(107886002)(5004730100002)(6806005)(86362001)(5001770100001)(99136001)(229853001)(97736004)(106466001)(87936001)(33656002)(4001350100001)(5008740100001)(65956001)(65816999)(59896002)(65806001)(77096005)(83506001)(92566002)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB4PR05MB541; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:ErrorRetry; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 2:IdrhbVgZ1A9jB3sZDS/igSOoDhQMxePw4+U6wjng+Tp+SksIgADtWdyPbeZHSvrQubUomMyPg6XoKCdkqzD5FAI931d1kkI/AGfYmecupTW4Obyko9U/8P1IkI8i6h2i62GmEysC38hVpg6g3J2n5w==; 3:Kp7lOpjYDdQaZekFhllffd3CBDz9BduDnN9uJvVRW8dk4RNPj1/O6zwdv53I0WMXg7p2IhABuVPJS5PGc9Ty8yB11c6I7JVUncUkVZQwETZ8Tml1rMGMVZeGVDVOVgzM4tv+ZQ0XOwNG3FfjgmtG9AYRUAXrVcFet1p14GXU5tBh3cebm1h0yeZkp4KRj4+7rWX3fi3TslLCLTICD2av4Z1TLkKiwl9V9ldyCigV+hMe6Z7gmwq472lzNYz/TjKD8t8wcndxe/OSM8vGarA+Vw==; 25:ovoK2V69xQWKaZiEx5CArhRV4w4I2RFMCcK+/4IK+9PXYroQe2HVV9V+qHbC68p+LmEJ9pD4RUYNQ3/hiLGJ1Z0rPK7jvXxmKQNT1U4xKhfYHPgVR0bSGWf2XlbjVOvC5+ol3KXyO7cqwkAp70WdgXACif9c0fQMsDAyjhwMQPbsUUt4yJZcOnCdZGQr54zrRsZ2SsVEQ24Yet/bHYG4qEdjt0xFjEvYPdoKdXeyDyKBCGFoGyHAqaEdIKznEv3l13zG9GBfp6OyvuPBFcTJIA== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501001); SRVR:DB4PR05MB541; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 20:ygjIBYUI3PIwhnQ0Yip09fWybUufSBQZzfuZd4MxO1gJRcF/I+zz7HnCuc4aQVFitZCK9GGh1zNpGm2FbTAze7SnrbVafctXRXZywYJG+6Rm9I3UtfNpja7cxed5qvBuQbUDBcuC1SEySQMtiFbkniLQwctnqKMeNZcqdpgwNElF7vsAriJiJe9PfcGcjPfFHl7heJ6zukBJoa0T3gLnvt8x/N4A18ioHzzIm6fiuYr2E8TGynlLJvWW3lZWtQsHmRm2PMhxjNUYl+QZ+rxBYsLDcV/4W8wbzj+VLUv2tX6niNEblYB9lp6oOfr1Dd13y5dL75Csz6HthFw63FoCrPqdpjmFdQ87HOMjxUOC17oCNh4o3ycF1THhgEnUs+cqGaPoSKILvEzz2IVEVLGLOlP98B8vEjqNSKHT6B1AuO9qvq68z7/CLZ+0bsOyw3eyBQVGXMbQOg4VDLIS8eGZXodFJJtdw7xpHV9P8MELZiYk4Ma/OFiZUkvQQ3XtdSlK; 4:KTqgpnQeAppJWLGFNti/M6wP0zst3gQlmLYKiiSOOqVTafrOGzkvhZIFl+jM3YtpMnQQUWB8R6+HdRrXN2ZvNw2eoYDGl/CXfTklN4g2NQ4lj8wPj44UWAJNoataXu71v8g5NHyfTZaPsw1X6pPikGrxMrCsUklV9KMKuXn13Dg6xgHpOENjGgAE4ornu8XC1CRD3EALwQZjuhCkmHXRO2PYHfuMYWiVpgKpji/4pLY2FJnNmAyFstNuK1KNW8c2onvUl53cCJPlEuI6b2SvlRxZxNK0AXtOUWbKMede0QQFxunLJ1SwxYLCMq2lYKsflzRFPud0HkWBVfYmcho0btlGCQheJ6qwOw69eos57F4pDC4ZsCm0pXMjvNYhzNJG X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(8121501046)(5005006)(10201501046)(3002001); SRVR:DB4PR05MB541; BCL:0; PCL:0; RULEID:; SRVR:DB4PR05MB541; X-Forefront-PRVS: 078693968A X-Microsoft-Exchange-Diagnostics: =?windows-1255?Q?1; DB4PR05MB541; 23:SueQiPHTqb9ms/thhb4aApKaF5/7hm5gm2o/R+?= =?windows-1255?Q?S2nHfNZI7XppmkUJJ7szU5tPyvUwDtdGo2m6MZwrKaR+9CKn4ALxG/Bn?= =?windows-1255?Q?YQ1K/LDkAwSRxrMLbTweRwrWve9Y70mFqXDBOECQ0ge3sLftA7o0EPFZ?= =?windows-1255?Q?v4h++poRZ7JsKNGT+RMtp+JMVK/EnfO9xMcLmaT5ZKsof7sCCGgPeoi9?= =?windows-1255?Q?fFkFhlUZXSjUlhACffndzuN9IjxahzYT4aujG4t1N95dS6KPnrp90ivg?= =?windows-1255?Q?YNX2kC8pc5yEXt6YP/gQWytqboi7cYIrOdOtWGBrBIdHbRAH1H9gAYBX?= =?windows-1255?Q?kxZgRu9pTeADhmI5DoJ2Z03fFfUDOVjmXXNVkQMhSm5D2Nytcph9Ob3W?= =?windows-1255?Q?t2ppv2hhQ2jSN+3KgBWD93FyIvdHEizxCFa/IfuyXuPnGArZYrfjiIEN?= =?windows-1255?Q?EDay3BVKIoLWSSRqVzR4zce4ueigEDihCJyxJpiIs00104JjRS/AhIgV?= =?windows-1255?Q?T8Yvkrdv4FxGhfJcYXVlsOvQQwaAqlz95OfPJaIMahiCvlVG9jWBDy2u?= =?windows-1255?Q?bx3RyvAVfxWYzowViao/u5MS9+dMUPZVrKjTn9g+TPFAZegpTFrDvTx4?= =?windows-1255?Q?+XO/iwd3w7re4AaGGpnfYJUlIs9GLG+mTQ/rtRimBn/lHeRY9tWOr0N1?= =?windows-1255?Q?EH5Z0j7BbecuFdRbt8r/mX8BqQlSPv5e5afcUFXzyW8S0hF/3BlsJZVO?= =?windows-1255?Q?T6eGiqeWrKr/tTLzNEzmcx8LzmsIpcQvbuyfcdgm7Uy1R7vzxGsgWgSJ?= =?windows-1255?Q?lilBMWa6Zk7eS9uu5OV76L75iV2ETHc9sf/8thuXKZRScIaDiB69RJC9?= =?windows-1255?Q?Z0NJRonXKo14MQNS070fd4NK78EVsgfOra2NP9QFFZK9k3cXaopWvY+/?= =?windows-1255?Q?ZjVW/gaB6vhvko3MEf7BsbLHPXQyBT0diVZOr5KFAAwvcmJM2iE5GLGE?= =?windows-1255?Q?7AXSaqicZtt8GSp/VyUPbDFMEhKSGYJ5YyHx/n+wK+Dj6We1GCzR2DDC?= =?windows-1255?Q?9vpR9k4A8d9tahvn6RYQbemeY9JWmhAPpXqoSTprh0D+pGp43vOzf1Es?= =?windows-1255?Q?AzcEJOPRYKQocygDOXhrTiJR65Min/4DLzg8hB4SwLLmdVsxuWcB9NgP?= =?windows-1255?Q?pE4LJWO6iIiv0AG3rVIhkDSHXxJkHgkqa/iuARgpDK3qjFTNCaGpEGez?= =?windows-1255?Q?mLAu3VCZ3LAmk9VQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 5:pNDsZVtdK1hjE0/1jY3ZLFjaMzmdOs9atPxpkOj9f1PtfYQWBhgeKqKo8/yzVMZeQ7clRGVP3H9NMKptM+qIvI2jBQlyRQgrORdikf+JSREuM8OD5YmdHTlDV9UaFK3GdFPajmWUr4v/R1/37a8uNA==; 24:Ji2sZTGcmR1Qaqu/e71mzOXJHQO4mf2LJUjGEZQ5lbjwxbSzcTrsD+TAobp/F6f5gGvfjSHyPSkCBXnJAL9KnkKr8urnPPMUbu86/PEkKHw= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Dec 2015 12:20:41.1852 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR05MB541 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 12:20:54 -0000 Hi all, I'm developing an iSER (iSCSI extensions for RDMA) driver for FreeBSD 11-Current and I encounter some weired behaviour while testing IOs with iovcnt > 1. I wrote a small test program that creates an iovec struct (let's say in size of 4) and each iov_base starts from page begining and 4k len: 0 iov_base 0xe25000 iov_len 4096 1 iov_base 0xe26000 iov_len 4096 2 iov_base 0xe27000 iov_len 4096 3 iov_base 0xe28000 iov_len 4096 I use readv/writev to send this iovec data. I saw that my driver get 4 different bio requests even thought this vector is aligned. This is surprising becasue in other OS I get only 1 bio. I have noticed the the physio function in sys/kern/kern_physio.c is praparing one bio for each iovec entry. is there a reason for this kind of implementation ? is there an option to send this array using 1 bio (some flag) ? We can improve performance if we send it in 1 bio instead of 4. My driver supports BIO_UNMAPPED. Thanks, Max Gurtovoy. Mellanox Technologies. From owner-freebsd-geom@freebsd.org Thu Dec 10 15:02:15 2015 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F10619D6796; Thu, 10 Dec 2015 15:02:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7376818F3; Thu, 10 Dec 2015 15:02:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBAF2AlF026079 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 10 Dec 2015 17:02:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBAF2AlF026079 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBAF2Awr026078; Thu, 10 Dec 2015 17:02:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 10 Dec 2015 17:02:10 +0200 From: Konstantin Belousov To: Max Gurtovoy Cc: freebsd-scsi@freebsd.org, Sagi Grimberg , Oren Duer , Hans Petter Selasky , freebsd-geom@freebsd.org Subject: Re: splitting iovecs to bios Message-ID: <20151210150210.GY82577@kib.kiev.ua> References: <56696E03.8050202@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56696E03.8050202@mellanox.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 15:02:16 -0000 On Thu, Dec 10, 2015 at 02:20:19PM +0200, Max Gurtovoy wrote: > Hi all, > > I'm developing an iSER (iSCSI extensions for RDMA) driver for FreeBSD > 11-Current and I encounter some weired behaviour while testing IOs with > iovcnt > 1. > I wrote a small test program that creates an iovec struct (let's say in > size of 4) and each iov_base starts from page begining and 4k len: > 0 iov_base 0xe25000 iov_len 4096 > 1 iov_base 0xe26000 iov_len 4096 > 2 iov_base 0xe27000 iov_len 4096 > 3 iov_base 0xe28000 iov_len 4096 > > I use readv/writev to send this iovec data. > I saw that my driver get 4 different bio requests even thought this > vector is aligned. This is surprising becasue in other OS I get only 1 bio. > I have noticed the the physio function in sys/kern/kern_physio.c is > praparing one bio for each iovec entry. > > is there a reason for this kind of implementation ? is there an option > to send this array using 1 bio (some flag) ? > We can improve performance if we send it in 1 bio instead of 4. > My driver supports BIO_UNMAPPED. There might be indeed a reason, it could be that some drivers expect blocking to be done by the userspace. The drivers could have some restrictions on transfer sizes and atomicity of transfer, which would be broken by the unconditional merge. I cannot give you an example of such driver, known block-aware drivers like sa(4) only require the bio size to be multiple of the basic block size. OTOH, I see no issue with adding a SI_PHYSIOMERGE flag and doing the merges for the driver in physio(), when unmapped request has consequtive iov elements ending and starting at the page boundary.