From owner-freebsd-current@FreeBSD.ORG Mon Nov 3 20:08:54 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 14D99FD9; Mon, 3 Nov 2014 20:08:54 +0000 (UTC) Received: from mail.highsecure.ru (mail6.highsecure.ru [IPv6:2a01:4f8:190:43b5::99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 99F2C261; Mon, 3 Nov 2014 20:08:53 +0000 (UTC) Received: from [172.24.168.60] (global-2-11.nat.csx.cam.ac.uk [131.111.185.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by mail.highsecure.ru (Postfix) with ESMTPSA id 7D0FA3002A2; Mon, 3 Nov 2014 21:08:46 +0100 (CET) Message-ID: <5457E0CA.5010509@highsecure.ru> Date: Mon, 03 Nov 2014 20:08:42 +0000 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Freddie Cash , Allan Jude Subject: Re: ssh None cipher References: <5441E834.2000906@freebsd.org> <544246E8.1090001@ijs.si> <5442C040.4090200@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=highsecure.ru; s=dkim; t=1415045327; bh=j39li2jR3gk5XWBMrlDIkg4VBQlUmMQIb0Fd/7WggF4=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=YsTFOwkG3zdS3wqmC6kQFiiR4yYwYW2uH5imik+SR9nR5DWExjPQYgdTvPtNbplOHumE3vd+usr4pCtUopgTuB+oler3i0aJI0NogSP7a8MVbRqS+sb7SYEpQRhxDZiNg6lNB+fcFej0S+rouhLAzD0pP/RCUDcs4LkeHIWlLk0= Cc: FreeBSD-Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2014 20:08:54 -0000 On 03/11/14 17:59, Freddie Cash wrote: > On Sat, Oct 18, 2014 at 12:32 PM, Allan Jude wrote: > >> On 2014-10-18 13:21, Freddie Cash wrote: >>> On Oct 18, 2014 3:54 AM, "Mark Martinec" >>> wrote: >>>> >>>> If the purpose of having a none cipher is to have a fast >>>> file transfer, then one should be using sysutils/bbcp >>>> for that purposes. Uses ssd for authentication, and >>>> opens unencrypted channel(s) for the actual data transfer. >>>> It's also very fast, can use multiple TCP streams. >>> >>> That's an interesting alternative to rsync, scp, and ftp, but doesn't >> help >>> with zfs send/recv which is where the none cipher really shines. >>> >>> Without the none cipher, SSH becomes the bottleneck limiting transfers to >>> around 400 Mbps on a gigabit LAN. With the none cipher, the network >> becomes >>> the bottleneck limiting transfers to around 920 Mbps on the same gigabit >>> LAN. >>> >>> This is between two 8-core AMD Opteron 6200 systems using igb(4) NICs. >> >> Actually, looking into it, the bbcp command can support a pipe at each >> end instead of files, so you can actually do a zfs send | zfs receive >> via bbcp, and use multiple concurrent connections, to get around TCP >> window stuff when going transatlantic >> >> I am going to be trying it out shortly. >> >> Note: the other big improvement in newer ssh is the HPN stuff, that is >> switched on since 9.2 I think. > > > ​After much finagling and testing, I have managed to incorporate bbcp into > my ZFS send/recv script. And it works much better than regular, encrypted > SSH in my setup. > > Regular SSH transfers tended to top out around 400 Mbps, using 100% of 1 > CPU. Was not able to get the multi-threadded AES cipher working. > > SSH connections using the NONE cipher saturated the gigabit link with > minimal CPU usage. > > And a bbcp connection is currently running between 500-800 Mbps (depending > on the size of the snpashot), also with minimal CPU usage. > > NOTE: I expect this be running much better next week, as the receiving > pool is currently resilvering a drive, slowing everything down. > > > Got things working using the following bbcp command format: > > bbcp -N io "zfs send -I pool/fs@snap1 pool/fs@snap2" username@remotesys:"zfs > recv -d pool" > > > Have not played with any of the myriad tuning options for bbcp. Just > wanted to see if I could get it to work, and how an untuned connection > compared to an untuned SSH connection (with and without NONE cipher). So > far, I'm impressed. > > ​Thanks for the suggestion. It's another tool in the box. :)​ > I have also written an utility[1] to efficiently encrypt and authenticate data transferred over the network using the modern AES-GCM or chacha20-poly1305 ciphers with pre-shared keys (or passwords). It can use multiple cores at a time allowing up to 32 gigabits per second. [1] - https://github.com/vstakhov/hpenc