From owner-freebsd-scsi@freebsd.org Thu Dec 10 12:20:54 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79F3C9D72D5; Thu, 10 Dec 2015 12:20:54 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0075.outbound.protection.outlook.com [157.55.234.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89C6E13BA; Thu, 10 Dec 2015 12:20:52 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from AM3PR05CA0054.eurprd05.prod.outlook.com (10.162.114.22) by DB4PR05MB541.eurprd05.prod.outlook.com (10.242.155.142) with Microsoft SMTP Server (TLS) id 15.1.355.16; Thu, 10 Dec 2015 12:20:43 +0000 Received: from AM1FFO11FD035.protection.gbl (2a01:111:f400:7e00::111) by AM3PR05CA0054.outlook.office365.com (2a01:111:e400:52b7::22) with Microsoft SMTP Server (TLS) id 15.1.355.16 via Frontend Transport; Thu, 10 Dec 2015 12:20:43 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by AM1FFO11FD035.mail.protection.outlook.com (10.174.64.224) with Microsoft SMTP Server (TLS) id 15.1.337.8 via Frontend Transport; Thu, 10 Dec 2015 12:20:41 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Thu, 10 Dec 2015 14:20:37 +0200 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Thu, 10 Dec 2015 14:20:37 +0200 Received: from [10.222.32.225] (10.222.32.225) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 10 Dec 2015 14:20:25 +0200 To: , Sagi Grimberg , Oren Duer , Hans Petter Selasky , From: Max Gurtovoy Subject: splitting iovecs to bios Message-ID: <56696E03.8050202@mellanox.com> Date: Thu, 10 Dec 2015 14:20:19 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1255"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.222.32.225] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11FD035; 1:CoVvXxpIKf3OjH7y8IX/Eck1IkCjDV1v0/VmFb+JK2/Jc9xhnjUm8YqZ8zROvw4HM2JsZEhjR+ToVnZcKO6dOz0GZFWtw3ObloUTAQNZGGgg463c+qwlpnmM1fqtkBXTZOmd2CGiMKbai2J5wXovCw7N7mqelsDS3slTD42+DR+ed3GXxZ4cG5f/X1EWIgwQ4gJTFrrzAV/Aba4WB/cr3zuh22s/YvF/UbLFZjR5rs3Iyvhh9yBFQG7G8cDIn5TCZofZHJZffwFEjeI7Rw92cKeD36d1fDZ/bTlEJW2s3S0EqGPTuj9pTPhQM+h86qbkvdWfJgyV4eZmIYs/fX19D4jvZgsopmSq9VdVQz1AbAM= X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(164054003)(53754006)(199003)(189002)(230700001)(3846002)(1220700001)(80316001)(6116002)(586003)(1096002)(47776003)(50466002)(87266999)(36756003)(50986999)(11100500001)(64126003)(450100001)(189998001)(54356999)(107886002)(5004730100002)(6806005)(86362001)(5001770100001)(99136001)(229853001)(97736004)(106466001)(87936001)(33656002)(4001350100001)(5008740100001)(65956001)(65816999)(59896002)(65806001)(77096005)(83506001)(92566002)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB4PR05MB541; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:ErrorRetry; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 2:IdrhbVgZ1A9jB3sZDS/igSOoDhQMxePw4+U6wjng+Tp+SksIgADtWdyPbeZHSvrQubUomMyPg6XoKCdkqzD5FAI931d1kkI/AGfYmecupTW4Obyko9U/8P1IkI8i6h2i62GmEysC38hVpg6g3J2n5w==; 3:Kp7lOpjYDdQaZekFhllffd3CBDz9BduDnN9uJvVRW8dk4RNPj1/O6zwdv53I0WMXg7p2IhABuVPJS5PGc9Ty8yB11c6I7JVUncUkVZQwETZ8Tml1rMGMVZeGVDVOVgzM4tv+ZQ0XOwNG3FfjgmtG9AYRUAXrVcFet1p14GXU5tBh3cebm1h0yeZkp4KRj4+7rWX3fi3TslLCLTICD2av4Z1TLkKiwl9V9ldyCigV+hMe6Z7gmwq472lzNYz/TjKD8t8wcndxe/OSM8vGarA+Vw==; 25:ovoK2V69xQWKaZiEx5CArhRV4w4I2RFMCcK+/4IK+9PXYroQe2HVV9V+qHbC68p+LmEJ9pD4RUYNQ3/hiLGJ1Z0rPK7jvXxmKQNT1U4xKhfYHPgVR0bSGWf2XlbjVOvC5+ol3KXyO7cqwkAp70WdgXACif9c0fQMsDAyjhwMQPbsUUt4yJZcOnCdZGQr54zrRsZ2SsVEQ24Yet/bHYG4qEdjt0xFjEvYPdoKdXeyDyKBCGFoGyHAqaEdIKznEv3l13zG9GBfp6OyvuPBFcTJIA== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501001); SRVR:DB4PR05MB541; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 20:ygjIBYUI3PIwhnQ0Yip09fWybUufSBQZzfuZd4MxO1gJRcF/I+zz7HnCuc4aQVFitZCK9GGh1zNpGm2FbTAze7SnrbVafctXRXZywYJG+6Rm9I3UtfNpja7cxed5qvBuQbUDBcuC1SEySQMtiFbkniLQwctnqKMeNZcqdpgwNElF7vsAriJiJe9PfcGcjPfFHl7heJ6zukBJoa0T3gLnvt8x/N4A18ioHzzIm6fiuYr2E8TGynlLJvWW3lZWtQsHmRm2PMhxjNUYl+QZ+rxBYsLDcV/4W8wbzj+VLUv2tX6niNEblYB9lp6oOfr1Dd13y5dL75Csz6HthFw63FoCrPqdpjmFdQ87HOMjxUOC17oCNh4o3ycF1THhgEnUs+cqGaPoSKILvEzz2IVEVLGLOlP98B8vEjqNSKHT6B1AuO9qvq68z7/CLZ+0bsOyw3eyBQVGXMbQOg4VDLIS8eGZXodFJJtdw7xpHV9P8MELZiYk4Ma/OFiZUkvQQ3XtdSlK; 4:KTqgpnQeAppJWLGFNti/M6wP0zst3gQlmLYKiiSOOqVTafrOGzkvhZIFl+jM3YtpMnQQUWB8R6+HdRrXN2ZvNw2eoYDGl/CXfTklN4g2NQ4lj8wPj44UWAJNoataXu71v8g5NHyfTZaPsw1X6pPikGrxMrCsUklV9KMKuXn13Dg6xgHpOENjGgAE4ornu8XC1CRD3EALwQZjuhCkmHXRO2PYHfuMYWiVpgKpji/4pLY2FJnNmAyFstNuK1KNW8c2onvUl53cCJPlEuI6b2SvlRxZxNK0AXtOUWbKMede0QQFxunLJ1SwxYLCMq2lYKsflzRFPud0HkWBVfYmcho0btlGCQheJ6qwOw69eos57F4pDC4ZsCm0pXMjvNYhzNJG X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(8121501046)(5005006)(10201501046)(3002001); SRVR:DB4PR05MB541; BCL:0; PCL:0; RULEID:; SRVR:DB4PR05MB541; X-Forefront-PRVS: 078693968A X-Microsoft-Exchange-Diagnostics: =?windows-1255?Q?1; DB4PR05MB541; 23:SueQiPHTqb9ms/thhb4aApKaF5/7hm5gm2o/R+?= =?windows-1255?Q?S2nHfNZI7XppmkUJJ7szU5tPyvUwDtdGo2m6MZwrKaR+9CKn4ALxG/Bn?= =?windows-1255?Q?YQ1K/LDkAwSRxrMLbTweRwrWve9Y70mFqXDBOECQ0ge3sLftA7o0EPFZ?= =?windows-1255?Q?v4h++poRZ7JsKNGT+RMtp+JMVK/EnfO9xMcLmaT5ZKsof7sCCGgPeoi9?= =?windows-1255?Q?fFkFhlUZXSjUlhACffndzuN9IjxahzYT4aujG4t1N95dS6KPnrp90ivg?= =?windows-1255?Q?YNX2kC8pc5yEXt6YP/gQWytqboi7cYIrOdOtWGBrBIdHbRAH1H9gAYBX?= =?windows-1255?Q?kxZgRu9pTeADhmI5DoJ2Z03fFfUDOVjmXXNVkQMhSm5D2Nytcph9Ob3W?= =?windows-1255?Q?t2ppv2hhQ2jSN+3KgBWD93FyIvdHEizxCFa/IfuyXuPnGArZYrfjiIEN?= =?windows-1255?Q?EDay3BVKIoLWSSRqVzR4zce4ueigEDihCJyxJpiIs00104JjRS/AhIgV?= =?windows-1255?Q?T8Yvkrdv4FxGhfJcYXVlsOvQQwaAqlz95OfPJaIMahiCvlVG9jWBDy2u?= =?windows-1255?Q?bx3RyvAVfxWYzowViao/u5MS9+dMUPZVrKjTn9g+TPFAZegpTFrDvTx4?= =?windows-1255?Q?+XO/iwd3w7re4AaGGpnfYJUlIs9GLG+mTQ/rtRimBn/lHeRY9tWOr0N1?= =?windows-1255?Q?EH5Z0j7BbecuFdRbt8r/mX8BqQlSPv5e5afcUFXzyW8S0hF/3BlsJZVO?= =?windows-1255?Q?T6eGiqeWrKr/tTLzNEzmcx8LzmsIpcQvbuyfcdgm7Uy1R7vzxGsgWgSJ?= =?windows-1255?Q?lilBMWa6Zk7eS9uu5OV76L75iV2ETHc9sf/8thuXKZRScIaDiB69RJC9?= =?windows-1255?Q?Z0NJRonXKo14MQNS070fd4NK78EVsgfOra2NP9QFFZK9k3cXaopWvY+/?= =?windows-1255?Q?ZjVW/gaB6vhvko3MEf7BsbLHPXQyBT0diVZOr5KFAAwvcmJM2iE5GLGE?= =?windows-1255?Q?7AXSaqicZtt8GSp/VyUPbDFMEhKSGYJ5YyHx/n+wK+Dj6We1GCzR2DDC?= =?windows-1255?Q?9vpR9k4A8d9tahvn6RYQbemeY9JWmhAPpXqoSTprh0D+pGp43vOzf1Es?= =?windows-1255?Q?AzcEJOPRYKQocygDOXhrTiJR65Min/4DLzg8hB4SwLLmdVsxuWcB9NgP?= =?windows-1255?Q?pE4LJWO6iIiv0AG3rVIhkDSHXxJkHgkqa/iuARgpDK3qjFTNCaGpEGez?= =?windows-1255?Q?mLAu3VCZ3LAmk9VQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB541; 5:pNDsZVtdK1hjE0/1jY3ZLFjaMzmdOs9atPxpkOj9f1PtfYQWBhgeKqKo8/yzVMZeQ7clRGVP3H9NMKptM+qIvI2jBQlyRQgrORdikf+JSREuM8OD5YmdHTlDV9UaFK3GdFPajmWUr4v/R1/37a8uNA==; 24:Ji2sZTGcmR1Qaqu/e71mzOXJHQO4mf2LJUjGEZQ5lbjwxbSzcTrsD+TAobp/F6f5gGvfjSHyPSkCBXnJAL9KnkKr8urnPPMUbu86/PEkKHw= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Dec 2015 12:20:41.1852 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR05MB541 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 12:20:54 -0000 Hi all, I'm developing an iSER (iSCSI extensions for RDMA) driver for FreeBSD 11-Current and I encounter some weired behaviour while testing IOs with iovcnt > 1. I wrote a small test program that creates an iovec struct (let's say in size of 4) and each iov_base starts from page begining and 4k len: 0 iov_base 0xe25000 iov_len 4096 1 iov_base 0xe26000 iov_len 4096 2 iov_base 0xe27000 iov_len 4096 3 iov_base 0xe28000 iov_len 4096 I use readv/writev to send this iovec data. I saw that my driver get 4 different bio requests even thought this vector is aligned. This is surprising becasue in other OS I get only 1 bio. I have noticed the the physio function in sys/kern/kern_physio.c is praparing one bio for each iovec entry. is there a reason for this kind of implementation ? is there an option to send this array using 1 bio (some flag) ? We can improve performance if we send it in 1 bio instead of 4. My driver supports BIO_UNMAPPED. Thanks, Max Gurtovoy. Mellanox Technologies.