From owner-freebsd-infiniband@FreeBSD.ORG Thu May 7 11:37:32 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B3676D2C for ; Thu, 7 May 2015 11:37:32 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5141417A1 for ; Thu, 7 May 2015 11:37:28 +0000 (UTC) Received: by wgic8 with SMTP id c8so13990613wgi.1 for ; Thu, 07 May 2015 04:37:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:content-type:content-transfer-encoding; bh=mNwVbIHcH35pSFmxORY22HZ+kOhA9EG3MhUZk0p2Njg=; b=iEXcGZsyZz/x8YxpzEFH9uKitJmnXl9hI5gFYwvhjboSE3rjW3Vm11k5/7r6jP24qt AX0OYj1TlHG7m9Hdt+h0qwZxzbEbeMXh6Pq+3Zl9lgm7mbo+JAoG4sxYsVH02fEua86B SRv7G+BK+LY0D2EZ9l7fZyQyllGh+zpSm0DFXVKqJRJiELTGRoUpP/q7EoxOca08YZvF Waw9uDfZxLLRB5MTud0ijcs+LrL5WaJ5m6Ev49Yr/KT/cu2h2d5SrLt/PcivGDlDeNp4 0ED3FkvBvClKs/xtqm9iSjsNi0uQdSW8BZlQ3QVVW/+wCkfSCmRKfESkjIDhazu0Uv9z wmWw== X-Gm-Message-State: ALoCoQnixeai541H5imoIojVc0MbiAJjTvMqSDE8UHDkPf0qYzC6T56MCcNZb3/sAF0PgMAnIBQ2 X-Received: by 10.180.230.199 with SMTP id ta7mr1082358wic.1.1430998226862; Thu, 07 May 2015 04:30:26 -0700 (PDT) Received: from [10.223.3.26] ([193.47.165.251]) by mx.google.com with ESMTPSA id u9sm2958909wjx.15.2015.05.07.04.30.24 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 04:30:25 -0700 (PDT) Message-ID: <554B4CDC.3090900@dev.mellanox.co.il> Date: Thu, 07 May 2015 14:30:36 +0300 From: Sagi Grimberg User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org, freebsd-infiniband@freebsd.org CC: Max Gurtovoy , Oren Duer , Hans Petter Selasky , Or Gerlitz , Oded Shanoon Subject: FreeBSD iSER initiator Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 11:37:32 -0000 Hello everyone, Mellanox Technologies is pleased to announce the availability of a new iSCSI Extensions for RDMA (iSER) initiator mode support for FreeBSD. What is it ? iSER is a computer network protocol that extends the iSCSI storage networking protocol to use Remote Direct Memory Access (RDMA). RDMA is provided by either RoCE (RDMA over converged Ethernet), iWARP or InfiniBand to enable efficient data movement using hardware offload capabilities. Why doing this ? RDMA infrastructure provides benefits such as Zero-Copy, CPU offload, Reliable transport, Fabric consolidation and many more. The iSER protocol eliminates some of the bottlenecks in the traditional iSCSI/TCP stack, provides low latency and high throughput and is well suited for latency aware workloads. What was added ? The iSER transport was added as another icl provider module, and it opens and RDMA channel to the target (icl_conn_connect). Other than that the addition of the iser module is pretty straight-forward. Some minor additions and changes were added to the iscsi layer for iser support: - Not relying on io_received when handling a scsi response since io_received is not maintained in iser (data is transferred via RDMA). - Added a PDU pointer to task_setup so iser can reference the scsiio in its private pdu context. - Added a -T flag to user-space to indicate using iser. - Added iSER relevant information to the login request. The code is available at https://github.com/sagigrimberg/iser-freebsd The code maturity level is considered as alpha. It has gone through limited testing on top of 11-current using Mellanox ConnectX-3 adapters (Login, Logout, I/O). We tested against Linux LIO and TGT targets and we have several known issues in error flows and session recovery areas (Documented in release_notes.txt). We also did some initial performance measurements and we can see that iser achieves x20 BW (~5.4 GB/s) and x2 IOPs (~100 KIOPs) compared to iscsi/tcp for a single LUN device. We would like to get some feedback on the code in general, and specifically on some of the framework changes that we did in the iscsi common layers (and user-space). Moreover, feel free to try out the code, report BUGs (to Max or me at this point) and even provide fixes. Cheers, Max Gurtovoy & Sagi Grimberg Mellanox Technologies From owner-freebsd-infiniband@FreeBSD.ORG Thu May 7 14:03:30 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 22491C1C for ; Thu, 7 May 2015 14:03:30 +0000 (UTC) Received: from mail.tds.net (smtp-fo.tds.net [64.8.70.105]) by mx1.freebsd.org (Postfix) with ESMTP id DB54B1A51 for ; Thu, 7 May 2015 14:03:29 +0000 (UTC) X_CMAE_Category: , , X-CNFS-Analysis: v=2.1 cv=TdqCtHgh c=1 sm=1 tr=0 a=ZUodfxLL0dFkpt6uoqiZUQ==:117 a=ZUodfxLL0dFkpt6uoqiZUQ==:17 a=K-v-2zaBAAAA:8 a=vC3ST6aIoYIA:10 a=N659UExz7-8A:10 a=5Bwo4ec7AAAA:8 a=NEAV23lmAAAA:8 a=6I5d2MoRAAAA:8 a=-BM0Rm_KmpwObZHXfjcA:9 a=pILNOxqGKmIA:10 X-CM-Score: 0 X-Scanned-by: Cloudmark Authority Engine X-Authed-Username: andiYWNvbkB0ZHMubmV0 Authentication-Results: smtp01.tds.cmh.synacor.com smtp.user=jwbacon@tds.net; auth=pass (LOGIN) Received: from [96.60.148.194] ([96.60.148.194:50413] helo=pro.local) by mail.tds.net (envelope-from ) (ecelerity 3.6.5.45644 r(Core:3.6.5.0)) with ESMTPSA (cipher=AES128-SHA) id 84/56-13657-9A07B455; Thu, 07 May 2015 10:03:22 -0400 Message-ID: <554B70A8.9050605@tds.net> Date: Thu, 07 May 2015 09:03:20 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Sagi Grimberg , freebsd-scsi@freebsd.org, freebsd-infiniband@freebsd.org CC: Max Gurtovoy , Or Gerlitz , Oren Duer Subject: Re: FreeBSD iSER initiator References: <554B4CDC.3090900@dev.mellanox.co.il> In-Reply-To: <554B4CDC.3090900@dev.mellanox.co.il> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 14:03:30 -0000 Many thanks to Mellanox for supporting FreeBSD! Having the capabilities of FreeBSD available to Infiniband HPC clusters, etc. will be a great advantage for the entire industry... On 5/7/15 6:30 AM, Sagi Grimberg wrote: > Hello everyone, > > Mellanox Technologies is pleased to announce the availability of a new > iSCSI Extensions for RDMA (iSER) initiator mode support for FreeBSD. > > What is it ? > > iSER is a computer network protocol that extends the iSCSI storage > networking protocol to use Remote Direct Memory Access (RDMA). RDMA is > provided by either RoCE (RDMA over converged Ethernet), iWARP or > InfiniBand to enable efficient data movement using hardware offload > capabilities. > > Why doing this ? > > RDMA infrastructure provides benefits such as Zero-Copy, CPU offload, > Reliable transport, Fabric consolidation and many more. The iSER > protocol eliminates some of the bottlenecks in the traditional > iSCSI/TCP stack, provides low latency and high throughput and is well > suited for latency aware workloads. > > What was added ? > > The iSER transport was added as another icl provider module, and it > opens and RDMA channel to the target (icl_conn_connect). Other than > that the addition of the iser module is pretty straight-forward. > > Some minor additions and changes were added to the iscsi layer for > iser support: > - Not relying on io_received when handling a scsi response since > io_received is not maintained in iser (data is transferred via RDMA). > - Added a PDU pointer to task_setup so iser can reference the scsiio in > its private pdu context. > - Added a -T flag to user-space to indicate using iser. > - Added iSER relevant information to the login request. > > > The code is available at https://github.com/sagigrimberg/iser-freebsd > > The code maturity level is considered as alpha. It has gone through > limited testing on top of 11-current using Mellanox ConnectX-3 adapters > (Login, Logout, I/O). We tested against Linux LIO and TGT targets and > we have several known issues in error flows and session recovery areas > (Documented in release_notes.txt). > > We also did some initial performance measurements and we can see that > iser achieves x20 BW (~5.4 GB/s) and x2 IOPs (~100 KIOPs) compared to > iscsi/tcp for a single LUN device. > > We would like to get some feedback on the code in general, and > specifically on some of the framework changes that we did in the iscsi > common layers (and user-space). Moreover, feel free to try out the > code, report BUGs (to Max or me at this point) and even provide fixes. > > Cheers, > > Max Gurtovoy & Sagi Grimberg > > Mellanox Technologies > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net If a problem can be solved, there's no need to worry. If it cannot be solved, then worrying will do no good. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~