From owner-freebsd-scsi@freebsd.org Tue Aug 16 00:46:12 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB4B6BBA607 for ; Tue, 16 Aug 2016 00:46:12 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0054.outbound.protection.outlook.com [104.47.2.54]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F01841F34 for ; Tue, 16 Aug 2016 00:46:11 +0000 (UTC) (envelope-from maxg@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=p+g82y3EvDMPqG78h9CSbSzt/5UXIEZyh/Gd+N7VsFg=; b=ehxoTHVs8SCxllFPYxdrLnjaQu3E4UiH0QTr4KpIEJwL2nUwRW3Fwr3qDF9I6GmpoNbI9UIcOoNlyqrNiKBnmWI9VfxNHzDIHXy1LAarR2WVUZ94Mn34rbhpj9yAT8K7XxtPiJaEIpbA6e5OeokTXcBI0UKauvdcg+CANn59V8E= Received: from HE1PR05CA0015.eurprd05.prod.outlook.com (10.162.181.25) by DB4PR05MB0847.eurprd05.prod.outlook.com (10.161.248.17) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.549.15; Mon, 15 Aug 2016 22:12:36 +0000 Received: from DB3FFO11FD030.protection.gbl (2a01:111:f400:7e04::151) by HE1PR05CA0015.outlook.office365.com (2a01:111:e400:51f9::25) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.557.21 via Frontend Transport; Mon, 15 Aug 2016 22:12:36 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD030.mail.protection.outlook.com (10.47.217.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.567.7 via Frontend Transport; Mon, 15 Aug 2016 22:12:36 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Tue, 16 Aug 2016 01:12:34 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4 via Frontend Transport; Tue, 16 Aug 2016 01:12:34 +0300 Received: from [172.16.0.237] (172.16.0.237) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.301.0; Tue, 16 Aug 2016 01:12:32 +0300 Subject: Re: Help to make iSER working To: Ben RUBSON , "freebsd-scsi@freebsd.org" References: <71E4886B-61C3-4091-A7FF-E5A4F59AEACE@gmail.com> <64d8bc44-b6a3-4c0a-6d13-f919bf26685a@mellanox.com> <41CCC244-A781-4087-A811-B33847EDEC01@gmail.com> <6a464f2c-c048-d29c-e602-c129b35a5d4a@mellanox.com> <621BCC32-4612-410F-B658-D1086A7A3A09@gmail.com> <8207D6A0-3647-48B4-A4E0-3F1FCF3A6723@gmail.com> CC: Oren Duer , Oded Shanoon , "Meny Yossefi" , Roy Shterman , "Hans Petter Selasky" From: Max Gurtovoy Message-ID: Date: Tue, 16 Aug 2016 01:12:32 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [172.16.0.237] X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:193.47.165.134; IPV:NLI; CTRY:IL; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(43784003)(199003)(377454003)(66654002)(189002)(24454002)(64126003)(81166006)(8676002)(81156014)(8936002)(2906002)(626004)(106466001)(31696002)(4326007)(7736002)(65806001)(305945005)(356003)(33646002)(7846002)(86362001)(11100500001)(2950100001)(47776003)(65956001)(23746002)(77096005)(107886002)(2501003)(31686004)(4001430100002)(189998001)(6116002)(230700001)(93886004)(36756003)(50466002)(3846002)(19580405001)(83506001)(19580395003)(5001770100001)(87936001)(4001350100001)(76176999)(586003)(50986999)(54356999)(92566002)(65826006)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB4PR05MB0847; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:mail13.mellanox.com; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD030; 1:lVC1ejm3uRPoJ9OtPKqadIQc0yN09pBFCwzqGrEAGnb+nHaAgISF600OMpxEFZXwpp9xJ21HU9eZQKE2+WjqsZ13y29fQgtVnABngiy8QBivw/7ZQe+DoIjQznGzD61BIk1TEPCLxIA30yiE8MEwKRGOo7RReIq2Blitz8xhl9UI0JwFjLnI+ACT0vXYhVApeJgjTXPifhJqHTzWmEp8fHYBrcn6jDuIlIFNZOabj+oc1r4waswy0qBZS6yZ6sy0LHL9XwDz4fDiy7V5NXwXGIerG/y8I/6b4w0T3cZ6pKvvzPlEhSelBO8x6PYUyPs32ue+axiwBneEV6ntOv6fIe0r2jZbX4ipza/xOkb31c5Ll4Xvz7nbPNbg2m9nzlhXTxey+cAVfpbTB7W3QN396gMXfa8tgyRZtm/gr2xkp6zUoe2Sayre6hjWIsFFw3Z1ffDywhozcBKCnzoJihBIUH3qbiY5fEiQM7Pm2Uh+85zSxbwVghdrQGxzuVqXAGxaXrPpEeYLc75F1QfuRkz2P0ou3Xj1s+8vlTT2mDL3zus= X-MS-Office365-Filtering-Correlation-Id: 0e79142b-8f2d-4ec7-54c8-08d3c5594383 X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB0847; 2:CIT5jQjs1wVMazCM20EuLL/KGreO1agxz/7X2Hu9JlU3WmX9BI3FlMtZka97lyObbjsCp/uQkHuRIC0Ybp9vWlHJMhi1+41EVIOo3ObRIYkUVfCmdyqhZIHvlU/6UYg5nhjL1eISvS/xaa9+MOgUsPkNQwqVZwc+420tcaluXm7XCRCHy1DSnzUzf7wsW1bg; 3:0+s9kE/PueZsJJ0WhYL5aF47k2AWD9Bt92CNqMdPVgyNJf43Tpa6i+EZy2TRaamnlxBGGWP4KP9fPOrkdcq2j9bKxWa0xOoDljkBG4qStgj0bZJanEdxdrIUUhAcRbLnObs1QCxl1rq1YCERUVccj64ZCAcvF/ID1VjmhY1Zf2jLlx+apib8HrUNAMFp5MgZeIK5His4XsGgvoQjxNh1t0fI6KEL9eltF8GNltqZaUJCq4Epz5O0QBkF5Ld1h4PxHXSRqT9Mmsb3rW/eJsnGOw==; 25:R1DgN4l7LbACD963Z3VGcmHPHpxCzgH3y4iW3l+cjngRMsWblBn9DhlJOQrpenCv95ctEDK1AsZO4GBhOmtFJ0QsBu+dmNdcD+ft26GH/rj2bb0zpgR24w3B7wnr8FfGQK8XEsvoGkLNRnEEl3HA4j2KEsI/o/567ujfa/BeHbflr4FkyyHl6sEXWJwr8q2MXsqT+JTleNuLiKXWEQc1wA2hlhEG63WGW+/qjhM/IBHdtkKxwuKhkSa/JpT84swS/Ws8cs222LTI30qKZS+8OwZZKLHq6uH+u8m1r6CB6rswOLW7t0APZ9nZ2ivMyiS//WgdG8bStd56HV9nF2SCAdDuAcY2bckSYCBnTSTJSJ7HVd/lOCxgDM+RYRVr5JBtFqT4RKUYCdWg6ULyYErNLP1UI4jxJuGN6Ms6ek1NBu4= X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:DB4PR05MB0847; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB0847; 31:jhOZ2CmAdNYCCOlZVXKEHlPXylR6AqZSxCcuSop6S+zgIFSk333DQBNcSf8t2jlw9vGpX4QZViJFNIs1ox19cRPjqEfLTlo+zvcaS3pG6JxVhb1OAQadXJ31JYHrlcg95g73WQdYFy4ZH6ipvTw/uuelyz64dYidS5n6oPEgnrIRJkqKHaGNUBK1meGnX049/lYEvVNB4SaU6LfSsCg2ryTWuROTDRwTuWFHmME/RRE=; 20:ElUK/LsrzOxrHtmj5LbiEuve8tvcdLSZEvM2swJhTG6mwjakV+fOMbwDMlPBdB2PgBADCA/+V9YagoRhGwv947yRkwjdlxsW8xgpnr1yVXblnCYSmYcrme1XdnipbuKA81CB+RvJwzOViY/lD7r1i0hyCFJL0aanWcVnDoU6XJSG/1Qy75ocHzxyfsN6WjVmwBl1oDFTxD3OBq7Aar1JEZDb8Rp8BADF4hTdtUjvnlSMA1Xv4ZiHeCOMcBUD3XhiFqPk25cbZ2sqyOIctMMMvg8F3nshF72vlkDSEy8+YmF+YUFsQvNC+zJuhnKg8ZgdEmqR1EzN65lP0v9dP3VUWV/faLqcHGBO+XAOvd6QB4dq7LvxJDgVSjcoTCsJnectccCOC4nzYaHqH9mA/C28FjIzN+JlyfM+4PjLVhKfnHIZhvKSjAqXzDh7nCtSXx0iR2oBGuiM//hp3K8fxXRHD8uVgOBYi5kNydOID7aGGg80in/O3wsBZbFqK/kl+KsH X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(13018025)(13016025)(5005006)(8121501046)(10201501046)(3002001)(6055026); SRVR:DB4PR05MB0847; BCL:0; PCL:0; RULEID:; SRVR:DB4PR05MB0847; X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB0847; 4:8iOb+gKahxUVXPiFVqr/6PgNUJHI4M9gVSaWBtesIjgSNQSp4hF7a2BQNmmBGYHzIo6Crk0uLma+meCK2Sz1yXg7+4N0MmQU52NwDouNMmsUXerp2hAhDhZpEQ396JFsu0ytR/dLDekxvHcaWlpB4XdJNQAQND3CfvKcKiz+m7hCBxE6CkhGF6sU8vj2NYusCn+oeje/R+K4u4j62IZcVB1t8VHY0rFu22riB57Eh20ebJDHB7bpbsuT996NJvZZQmPfSq/SMucUCWl9+iOA/30Jxa05e16M3xM392Y1fZQTZF6g4bzDSGrgSWLWDVaw07SAMnFvVNbuPEGmUSVH+Z/bQYN6ghlBtORKLOq9d94i+0g0JQgN016C9h6dbBsYF3kERv483+z+M5F6muaIEaEp5HeiWRcI5WuIF+no6R/7bb/NwcDafUh9q2WMt6o6 X-Forefront-PRVS: 0035B15214 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; DB4PR05MB0847; 23:l1j1fPBGBnIkmTCVuWAA52hedD5yVyBPqm4if?= =?Windows-1252?Q?FcHgoTs3AA2lOyGI4eIaDZJ4s/MHF8CGaw4+hgaAvVIMLv2cZjouNHES?= =?Windows-1252?Q?4/Z3iXuOMB16oBuXI+u8kDl2BMgCDgDjPJipaCAbWPsGqB/oM78slW/9?= =?Windows-1252?Q?mburc/ZixEiHvSf0CX4iqUcwVifhKlKFHp8CFM10AZxdXdqEammQTtBs?= =?Windows-1252?Q?zGqViNc5msaikFQ+fYK7GNMW8o6XfokbgHskS80HxHWKCudl2fq0PnKG?= =?Windows-1252?Q?hRZpVp4WPETnkGPPN7lmItz1NGjaXsvCWY2RqFaXnC5dPankT+kBD6i+?= =?Windows-1252?Q?G4xaDKfgzt6QoYHUstwkF1G7EDeDqlmR49ABa3sYM2qyMF6VBzfzEOZY?= =?Windows-1252?Q?fgm44DHKAZr7XrXDYn7r8cvmBO7xBfIJxjapsrRfl18H2rZLs9sYdfb9?= =?Windows-1252?Q?vm7KmtOyv0wwbpxcKH6IST9m7cj0gI5CST2YqdN3GeC8fMFA101arg5g?= =?Windows-1252?Q?CthLLa2xBzRkH66beyuKcYDVIj+hZb829BYYYx9MrYxZIwh33M8MSXhq?= =?Windows-1252?Q?Iu0TqOB1XpnT+DzVqUtj+v3+X0LRzYZuwuHvGHAiLyfxl4RjFn95GHpP?= =?Windows-1252?Q?I6IbrXmnkYvM8oGMAXRXDFpj+6F2L6btxPFi+G71pY435UkXUTfUi/HA?= =?Windows-1252?Q?QxtRFy7CA+izkxPxCB+e6QzJXF33ksyXJVlCPlgMqvyYU4EHVraThR5S?= =?Windows-1252?Q?7tNJUTBUtMPqPD1qSNp8JV4kGk3YWHm5WHG47SYOaUCnYekat7WGoddn?= =?Windows-1252?Q?ph6zJkP28hXImMn4tZz2nFoui7X01I1x2XFgYILmM2KS7XR7r/PP1CX9?= =?Windows-1252?Q?FNw9ANNM5DsiBOfxb6Mk1atI96CO/PNnhxToHIN3SmWREljuG7/UPnin?= =?Windows-1252?Q?wLslh7dZxsmlu8aXZr9xrmGk3NJMGegXlZ6wNptU3bsypW/3G8RuNFtA?= =?Windows-1252?Q?yYGjD9FYpkeK2h5tRrjvabW9Pmf+jEIx0fPPwr+4qp0BON9b6A5G1dvH?= =?Windows-1252?Q?77R5n00/SgKFCH2yBzhwLnzDvMMHNUnUtGDGa425CiQps7behVbwdI5R?= =?Windows-1252?Q?585v8lZZpFwYWrF1WdkuWixYbLl97ajvYKKFBLpvpssuj7WiKAv+cf6F?= =?Windows-1252?Q?0PAXHLU+BbHbpzUULwcdDJQBVg++Bp/P4RkANuARVwGj5cbxBGlFPkTb?= =?Windows-1252?Q?EzOAl8RSQGLHeHD1vQMyr92nm/zyFoq3HhPmOvYf+1VWXMvRKAEfluum?= =?Windows-1252?Q?Gssa+icC383o2vjd8kVtdbc1AuMTYwHkD1S3oXxNUPTO17DSgyjc9n/L?= =?Windows-1252?Q?Oy88w5Tkz97Aw5f1rnxZjy9VjEOiQdwzmnNCPQLPl4m3y8ZL450AdafQ?= =?Windows-1252?Q?hu2Y3C/6yNC/YBU5Mc9IlI+oMgF5R+MdcCqEBqBDxJ3I/r1APfV05FNr?= =?Windows-1252?Q?ybJO1Wp4ql6f+UsnFFRff135k8O?= X-Microsoft-Exchange-Diagnostics: 1; DB4PR05MB0847; 6:ljbUWKwKSyQGLfI09Ts6LBhKpUlPa7haOBoV29CgVCciJkgkaxNpRYHnzMg3UMBZvSuO/LndwS/2tTSoA2ubMdXSr63oMipon9Y01tTrgtUNaicjve5I0LeJhwnPmew1X/K2lF9Zk+kBkUKUONSg6Y6wRtmhJVcZdkIk2avurPcyMGp7aQ0NZ67H1jJSHMc4Z4eyPhybCZuVd5gp6GyZ/h1vsmtB1LKUtgeJpaxCXUjklT6FCGreqMeoxUJvWjBsizkUjaCmloNqRI6smwTtYkBKYmC/dYYP7D1zK5xOaDAQFrCGxAvLsysjv7Mjx0wqvmOWtoc347Q6vUiTq7VB3g==; 5:8rlBINfRxDFDM62C2eh5R8YdFpzaSGmrAwqktRf7TxLrHhljSzybJi/tILboJlobQf6bzhDE5TIsNnw1m+tz1/fZYqORP7gx4PArFBf8tOwwO2coR0li6R4sxo5wc6a8VV+WqQoFxxR/ADEVSzjLBw==; 24:xmHnFNZx71cljgp0jBCv597Cw+O//FnfKV/zrC5ND8uzbLmtkKE10lcDU2J1nVjn5ZYl+078mMgbd1qKEnYUbwfgtYysu+ydcjIjZcscgug=; 7:RJ1S8gs6dYI6expeoAINed8mWzZnKbViCezScqMW5Zj0z1/r7uXlMwQHUwe/gvPjBpj1bIKChq0ZsPQ96O2NZbBvZCvd6m1AKrXA3qBRBxhXu7PWmlCFRCvD3siqLv9Fdex4MwQFubkDVbnzTu2TGiOa8SVbcObZ7V0Ltt9EklQUbHU2Pd/kJz9OWpxqtEDvciJxo73Xd95KM4WZtlwcFHaa1rlsHFYlSZpyaU73dxSls3vMlKODUOizpI6fBogC SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2016 22:12:36.2941 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR05MB0847 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2016 00:46:12 -0000 Good to see that you got a nice progress there. You can try the fio tool and see even better performance. You can also use Mellanox community site to see how to reach wire spead and leaverage your numbers. The iser posts are mostly for linux users but you can find it helpfull for your needs to. Thanks, Max. On 8/15/2016 12:11 AM, Ben RUBSON wrote: > >> On 13 Aug 2016, at 15:12, Ben RUBSON wrote: >> >> Benchmarks will of course follow. > > (also posted @freebsd-fs) > > I then used Linux TGT as a test target. > > ### Target : local disk throughput, one thread : > # dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920 > 10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s > > ### Initiator : iscsi disk throughput, one thread : > # dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920 > 10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s > > ### Initiator : iSER disk throughput, one thread : > # dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920 > 10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s > > In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck. > > I also tested with a target RAM device : > > ### Initiator : iscsi RAM throughput, one thread : > # dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920 > 10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s > > ### Initiator : iSER RAM throughput, one thread : > # dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920 > 10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s > (just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s) > >> On 11 Aug 2016, at 01:13, Max Gurtovoy wrote: >> >> Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)). >> I'm not aware of other contributers plans. > > > Can't wait to see this then :) > > Many thanks again ! > > Best regards, > > Ben > >