From owner-freebsd-scsi@freebsd.org Sun Aug 2 12:22:50 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D59559B0655 for ; Sun, 2 Aug 2015 12:22:50 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0084.outbound.protection.outlook.com [157.55.234.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F3431118D for ; Sun, 2 Aug 2015 12:22:49 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from AM3PR05MB337.eurprd05.prod.outlook.com (10.242.247.143) by AM3PR05MB547.eurprd05.prod.outlook.com (10.242.245.12) with Microsoft SMTP Server (TLS) id 15.1.225.19; Sun, 2 Aug 2015 12:22:41 +0000 Received: from AM3PR05CA010.eurprd05.prod.outlook.com (10.141.192.20) by AM3PR05MB337.eurprd05.prod.outlook.com (10.242.247.143) with Microsoft SMTP Server (TLS) id 15.1.225.19; Sun, 2 Aug 2015 12:22:38 +0000 Received: from DB3FFO11FD021.protection.gbl (2a01:111:f400:7e04::115) by AM3PR05CA010.outlook.office365.com (2a01:111:e400:882a::20) with Microsoft SMTP Server (TLS) id 15.1.225.19 via Frontend Transport; Sun, 2 Aug 2015 12:22:37 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD021.mail.protection.outlook.com (10.47.217.52) with Microsoft SMTP Server (TLS) id 15.1.243.9 via Frontend Transport; Sun, 2 Aug 2015 12:22:36 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Sun, 2 Aug 2015 15:22:35 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Sun, 2 Aug 2015 15:22:35 +0300 Received: from [10.223.0.78] (10.223.0.78) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Sun, 2 Aug 2015 15:21:41 +0300 Message-ID: <55BE0B55.8070807@mellanox.com> Date: Sun, 2 Aug 2015 15:21:41 +0300 From: Max Gurtovoy User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Subject: PIM_UNMAPPED question References: In-Reply-To: X-Forwarded-Message-Id: Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.223.0.78] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD021; 1:xBREHk8DRzxtDKYvJhNupCpGssqlKdm4NJsV6+nX7Gya1Ti57UgLRqGKBQnn4vqrFuPpvn5+bMd9IuJG+vLFbLiT6pmipBiwblRJYbNb3dWgW+J7OJi/xu3gM0WUgAnkbAG9Zt5olzBfxJz4YwFZ3ArT6sVV/54Y+qdNbGXUsxI3z0Q5U79hjgV9quzqZBg+nJM6uno0IDuoLwQoFgRi6NDpoEcZE9Bh9HVcZl2Xg/R4N9jarUZCF/OtDG9EshUOt8rXvsZivYPwPT+LZeGuIw6mXkKbZOrpVSjtX54de1M= X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(189002)(51874003)(199003)(15975445007)(2950100001)(23746002)(77096005)(80316001)(6806004)(106466001)(62966003)(59896002)(77156002)(36756003)(450100001)(19580395003)(189998001)(33656002)(110136002)(4001350100001)(83506001)(107886002)(2351001)(229853001)(46102003)(92566002)(47776003)(65956001)(65806001)(87936001)(87266999)(76176999)(64126003)(65816999)(54356999)(50986999)(86362001)(50466002)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM3PR05MB337; H:mtlcas13.mtl.com; FPR:; SPF:Pass; MLV:nov; MX:1; A:1; PTR:ErrorRetry; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; AM3PR05MB337; 2:KLVTjy8vmwNUYArHZCoxtj0gGVysM6j8g1NwclZKdbf9RyD/my+TtsXUfU9iVablZ0m5ZpVyYJ3YCDBe69wcFFzwEnDlg7Zunq0T2Vqt/1Z7+8P8hQqLy7PNxr+vdkVOXNth8M6HsAIUojfoYQRru0pqbQNBkathtJCGl4lbFX8=; 3:SLbot9Dsax6OkjXGYf7WxBdp1Q6bzipLXxKG0l7ILDvXyezeyX/YcA0AJ24AeSH2zmtqziGmYE/qklDlMATb59MTdP8keHG5RFr6qOvDTizfsXNaPORAimz5dJ4/Jeji2gC5Yxw+qjOJmkCq5V/AmGaqePMGZ9hH6wT/5lNmOEX38APt0OyxNxdzfwAaeZ1llutHgytQ6zJ4sJk6TIrMYtaQGkM9+lSMXPKr1p/aGB075BcORsR/nBV+ynHrUks0; 25:FI0OLl5cSjxN4N6/dQ6EwXRBhVQoao4QGl4uk/tPALY/C61fbV4FVs7joxwn3dz69XaSkgvW3+Zbt9kHmzmXRFFJcyOt7NfpsuWwhXXepwyP/6PKic1NyPNaPDHmb8HX+zQeluAiQTaDv91RlO0At+UBeatdZIshwtgHuq+dlt7Vi2Lg5r/DL0Ty42J25rOmtox0q7T9UNe3RBZVxINvVyrtlrsO0Y5E7np8lG6O3CWmeeWEIBOewa6+iX+XXl9hNJ/pljyn16OgMEfqJjILVg== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM3PR05MB337; UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM3PR05MB547; X-Microsoft-Exchange-Diagnostics: 1; AM3PR05MB337; 20:gHpdTryl57+E+/72REczv8SX+zzRq43rzuixts76gYBhiRnxrPPOUyH/RmbKJNKC8uk1PX2nnoybIACUBtpvpeFHTDpff2ejCb85O3IsOd1Y4uTiOGrbPN4PkAnffI/fDg9I5kFPqYb0YAm3d8gFylMZ27S6aN2AwKoIRuGucYuzCgMyRSQ2c3QMBooA68RC58eXKRSlqyXCgzrjCXXOs9raMTz9hYMfVR55KwHZ4DVSNjfaVGJTItAzsCZPG2e3J/zB1AZz8tVaeJycKxO8r3nEjvUph8AMxEWKXI70GU+68bz2kvjPC1oLprNY8jpGurkACxc1Nbwyc56DXrrysfEclg1uUnYHcpn9HkcIdk17uNTVQcSAkTFHqaPrQa4/RQMjmE7GwdmljAcaqdqF9ckrethKIQe9HlAV4sop5JB8Tzxtfy8Xn2icOH8sPY2HbsQ/sV8DVAfpb4OA1mj1zRpPoI0ZX05ZFaXOI8mT1fpSFHnmVdZoqdLAT1MmuXIo; 4:p3dlwpKgJpJB/D3Ru0uE2MkgEQJbx/fpiLbZReQQFCIH2oLf9ZR2lafHqp3JqKqn2hWOM5qSMRg37eA3teEeZxuz9zIUkKGMR4nr5ecIdHy1lFHZ/TmqdjMID0wT2VdIhWvT4B0L3ftsSiqjtDnxkyEi45KOkoSM3/69vCPbvx9dzQa03GB4tnYR5f53/BxwrTYgm6KGDXeesALfSM5k1FeNBSY+fODVYMjqjEOAm1nqClyO4g+iZPnVayDQAoMROGKYlqNKekmhtH96CJMs3CZJnqXyY/4gOeea3NW0FOo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:AM3PR05MB337; BCL:0; PCL:0; RULEID:; SRVR:AM3PR05MB337; X-Forefront-PRVS: 0656A4403B X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; AM3PR05MB337; 23:UCea1FJFHLBBkYgtie4mnr/tsouXtPgsbq44FH?= =?Windows-1252?Q?PczVVrfbltoDqC269Csd+qWVPIk9Kf79E7wJtvprxkcLD5RLLrrALBaa?= =?Windows-1252?Q?zCIsxr8CcU5H8Ak4gvyGT9Z8LmD/riZiTCTs1YRDx+ow3A2T1/bWrqVL?= =?Windows-1252?Q?w9IqKdG6xDx4l6mV0yRV4mI48TRsWol+E4Y29Gdwaks1bsvA0ods55W2?= =?Windows-1252?Q?Yw3aB/Asuc0rlDc3vivd0HY9edSb6t7nTnLD7nLQ9PJeDd8z3JMd589+?= =?Windows-1252?Q?twfD4UX7k/trGELICd/4jdC9H5/9Sjx9J+EpVKoo253SvDRVs23TJZmP?= =?Windows-1252?Q?WsTJkNPCAA2LEPRycSokbfVCWFtOiMXncfEkI2/GmsYEC6IH7FF0PMft?= =?Windows-1252?Q?PTEV5h+TGHqM1U2VjXXI9Y1LVyzFERnrmnfFVE0ozLck+iJ1ATquK5Fj?= =?Windows-1252?Q?0Uyv3Th5eIIY7TPiQeZH57QLPSZKbMUNg9fDXY5j0yRXlq2n+bdmq3Aq?= =?Windows-1252?Q?0rPugP94SKagCygLszTR0leEx0X6ZssGs+lmikHkVsYLx9tIlIR1ZVJ+?= =?Windows-1252?Q?JEtqLRhmbe8GAgHtVbB5c6B6L1q8/fZ1tFco4wROPAMvAsZ0JMr/Q+l5?= =?Windows-1252?Q?kMjrlm2SGGA2HCtU6ofPntkLkHQPITozyfDcHh4P04noLJ+LfeUxIm12?= =?Windows-1252?Q?XraH6lvv7v9mv0XHyaUBXIWwwng0PkKJ+lS4OqGMb5GfBJxCdVb/+Hgn?= =?Windows-1252?Q?JzxT8C/v87n2AXXHFvkpYH9DQ/CfvP7X0LltfsPuu4rk/urxm08zO+09?= =?Windows-1252?Q?DyE138EboP0lQU2SPcuTGmnRCBfvegNnN1WryIIMUuVmQFIolvVA6YIg?= =?Windows-1252?Q?Zmx7xrEomK8tQMBJ7VhgNSFSjaNOoM/IN9KDgEryM8hkL2fVe29emi4a?= =?Windows-1252?Q?69kTCoeCRbbmliR4TlhsQyyXjAJ3VZhKrlRJYOMHOI7RGQbCZzlTsH2o?= =?Windows-1252?Q?9+DJJMoGlxjK19KmOJPsLkdFme7r4KtTGbRrTwjplQDqqgE0vx8y6UyL?= =?Windows-1252?Q?E921p12zL4YTidlGhEmqcQK6RSzA97af5gfn+jS/RkeKN0VNOnRpVrJ2?= =?Windows-1252?Q?cMeYr2/JEXsYhfaZb9IPUTAUGS/wf3JOBnqo6FdMa8?= X-Microsoft-Exchange-Diagnostics: 1; AM3PR05MB337; 5:UDTDWL1atEjmkg7l1yX2mg5+HtDQr7H1gQkjd7xShWvFulI9/PwJ9YTA/dhP7k0D2InWIJgkeFmNcgZA9MHjdpp1OAthQUarFknNWnJswEpThG5dJijxFbZhubQQFNza/S1fEpxfgz18M3b7lI+ruw==; 24:9RPm4TyPhdzm8oGwP68oS9yGyJNt9iZp2O90P5E7y6LmSrbPiyqeuMz5a7RusL7fyGb4ogBpxMI3TNJuPcdW/09Zqn9LE/DoVfbCug1kp28=; 20:BOYRkZnvS7AJYtarTUSPU4yW55iiByGyMoey8LWYdNpGOaoQ2c+QeORhOyocisczF1u0w2LN2UeW2+rDMQ3QvQ== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2015 12:22:36.4346 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR05MB337 X-Microsoft-Exchange-Diagnostics: 1; AM3PR05MB547; 2:YgcHj3Btd1kGX3//CrK84mSO+nJrpMbDpHMyy+aBlNxUGve675XAOHakxdNavvySW9MZeW866rLYzd7JbrbtI288TMSvMn3Owt18yZBaWK4SHsB1nZnQV+7EKHUTM2yBmL5KRLEefCJx6+WOznU2jtnBfQbypLMNqQ12Le0bS8c=; 3:WFmfIG7a991UPhtHSk5/SDh9y2KhkUr6l00xG+wcOb/gWgmeuaRJ8E4JrPyKWk0NaYTXg/IdFkQeBMk1RVI3c2byuKgDxYHEmBuutP9JgAJmBgczk7q1FxlkeOSz3sIh2NsUL2P2OzK6FO1tWnCRiyqSbjTrczhtXv7EnajvN+hLjQ4bG5tuBbpipxU4FYiBKjHkzTold6KKvCczBUmPmoU62kU8vlMQq0backO0y1TpNTMUVhQlrwuwW7aHf/UX; 25:st1WgZvbrYvpTfZLmn0PEJ4vVXlQxtuJLuLngpTbCkghVQ4ItPgW52stCEBSTP7vKXqhyX3erHUwny4TY23hBypnfiExwukw7dQjsvvUCqoGFF02XvnuHztPc9K/HS8zIG77xQ6SyM4tcxKEgglMTt75nZRbPeTRhfMnAVfmdnbRCwxrP2GGEY/m4X2uWsEzw1vUiET2JgFl+Qv+YuCK7DpVtLW2iwcn0hiG9NOslLd1JAWWgoVM4ZmBYGc6srwveROzvsWEXlL0KkTyDP3zRQ==; 23:Lt6Kv1DiRrBt3tQPMkuhjPhtg/licVhNJWb0MkQyKA0Q0DARRyL3+u7E7XMzRoACVXoMtzWRVSwOnQXbuBfw3xnylLTlXBlam0NzuikT8ETJWux109HZuWfu/cTIqf90UhbnhFqIzjzNCF+HUOS2TElNF56HTKXHWL5B7yhePRyTH7A3vscGuFgIINltEWjrOtBVOBJQPLkaNfI8VGCmtwy9sfQgjcjvaci8Ge2xSJtHerxyyJMLSerFJ6MEpaTa X-OriginatorOrg: Mellanox.com X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2015 12:22:50 -0000 Hi, I am working on a new iscsi RDMA transport implementation. In order to avoid kernel buffer allocation in the IO path I added support for unmapped buffers in my driver (PIM_UNMAPPED hba_misc flag). This indeed helped performance by improving the IOPs By a factor of 4x vs. KVAs. However, I noticed that the bio_ma vector consists of physically contiguous addresses. I expected that user-space allocations would not be physically contiguous. Is it possible that I'm still not seeing the user IO buffers? I tested using dd/fio traffic generators and the phenomenon still persists. 128k Read output to show the phenomenon: kernel: dma_address 0x40608000 len 4096 kernel: dma_address 0x40609000 len 4096 kernel: dma_address 0x4060a000 len 4096 kernel: dma_address 0x4060b000 len 4096 kernel: dma_address 0x4060c000 len 4096 kernel: dma_address 0x4060d000 len 4096 kernel: dma_address 0x4060e000 len 4096 kernel: dma_address 0x4060f000 len 4096 kernel: dma_address 0x40610000 len 4096 kernel: dma_address 0x40611000 len 4096 kernel: dma_address 0x40612000 len 4096 kernel: dma_address 0x40613000 len 4096 kernel: dma_address 0x40614000 len 4096 kernel: dma_address 0x40615000 len 4096 kernel: dma_address 0x40616000 len 4096 kernel: dma_address 0x40617000 len 4096 kernel: dma_address 0x40618000 len 4096 kernel: dma_address 0x40619000 len 4096 kernel: dma_address 0x4061a000 len 4096 kernel: dma_address 0x4061b000 len 4096 kernel: dma_address 0x4061c000 len 4096 kernel: dma_address 0x4061d000 len 4096 kernel: dma_address 0x4061e000 len 4096 kernel: dma_address 0x4061f000 len 4096 kernel: dma_address 0x40620000 len 4096 kernel: dma_address 0x40621000 len 4096 kernel: dma_address 0x40622000 len 4096 kernel: dma_address 0x40623000 len 4096 kernel: dma_address 0x40624000 len 4096 kernel: dma_address 0x40625000 len 4096 kernel: dma_address 0x40626000 len 4096 kernel: dma_address 0x40627000 len 4096 Initiator machine info: - OS 11-current r284921 - 32G RAM - 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz CPUs The code is available at: https://github.com/sagigrimberg/iser-freebsd/tree/iser-rebase-11-current-r284921 Thanks in advanced, Max Gurtovoy.