From owner-freebsd-geom@freebsd.org Wed Oct 12 18:54:49 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29588C0F311 for ; Wed, 12 Oct 2016 18:54:49 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 933C79C1 for ; Wed, 12 Oct 2016 18:54:47 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from [192.168.1.8] ([62.157.57.141]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LpObx-1bFqmu32RR-00fB8q for ; Wed, 12 Oct 2016 20:54:45 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> From: Michael Osipov <1983-01-06@gmx.net> Message-ID: <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> Date: Wed, 12 Oct 2016 20:54:45 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:9hYoP4Mx19YFRY/QqGG0DQLqtQmBWuTZQU3rWYcffngKJKNQMNL lU6RpaluNfyNXgsgax35lDHENr0IsQ289wjChfK6UZxJJ0foMVp1mwFbSxDROtc73A/9rQv qRSotRM5egSM+tnL6RPBoD9nFQ3JFahC9eixA8efrTjtLay8GVfyePYN8FOsGsTrtkHQKrM 9mYd0+4SUc2V5rWY4sHdw== X-UI-Out-Filterresults: notjunk:1;V01:K0:4QskYIo83Co=:mftlclqxBoz2NMiC0aPwQI D4AzJ0ju1s8blKKhzpsyRGgMoLXSqBKMFkxIy5V+/5+gVtcd3Sg3ZcdJc4yUS4A6E5PXNjoFb l+JVznfWItCjyDplYe6TQ6SDCmzkI2Rf50OEQoQu2ytOGYfqiUBMCG7GhOxTX+LsfitwF6ht7 JzYScdMPebgE7neuDippxYEGcKur4xBdEgJuPT0ZAuKfcboai0s3jt+VRIaBB4JP13QnGsTkc 9lRZW2lsIQCJZ7+gQE//kyVJk3ZRMwj52yh1sRRm/Xf5pGEPrhjO3Gx0qRXOMo59/PsdrDzBe uNykAVDgw6TQqddwz59C8jlEb/z8SZOo91sUdrqDoeA1Wxt0NjHFnZxOR12TKs/WQ9viK6dbM hR8apIk5Zu4peYFLNeOlFesdJPja/GBtWnytHs58aUad0TGh5vw1tj4GG46QPsYfdyRTYz9Yw FxdLyoNFXp/As1LahuXalqEl9EaI8POD5hb+em13CHSSbQ57hQCUU4koiq55++fGjMZ6KRz9m odrbb6GIl2BVyVmx1tzQvjv3LT+la2KaZzMO+8Na5VErL2DSf9gwIqLRBTMsx3SS2HMJRMawI GoNStwTj42B3222/OrpwuUhgwg3QrGyaTWZO+RnE5gtOV3+CpPMjzqcnyl7BK4GYKLzJ3uFBs EOQd20ILAN4fAmPBDVGBxB7eGdiJnOlcSE9FwG94E8oBQmJXFW/w5Ua0a4Ztkw7d6ues6eIO5 f0bkIL8CTu+iURDesZSvnM4hrnXVske9QFA9uGqgZUUq7pjdpyp/B7qUU3JXKpMcmUVAr30ax FXuyTl3 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 18:54:49 -0000 Am 2016-10-12 um 19:08 schrieb Jan Bramkamp: > > > On 12/10/2016 17:25, Michael Osipov wrote: >> Am 2016-10-06 um 20:34 schrieb Michael Osipov: >>> Hi folks, >>> >>> I am experiecing a performance breakdown when writing to some geom class >>> volume over network. I have tried SCP/SFTP, SMB, nc, ggated/ggatec from >>> Windows 10 and FreeBSD 10.3-RELEASE and I receive at most 500 kB/s. >>> Reading is superfast as well as writing to a regular, non-geom volume, >>> network is fully saturated. >>> >>> I am on: >>> # uname -a >>> FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 >>> 18:37:29 UTC 2016 >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 >>> >>> Any idea what the problem is? How can I investigate on the penality in >>> performance? >> >> Seriously no one able to help? > > Not with the description you've given. Probably, yes though I wasn't even sure where to start. > Please describe the network setup, how you access remote block devices, > what you mean by "geom volume". Simply put, I have this: $ uname -a FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:37:29 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 atapci0@pci0:2:13:0: class=0x018000 card=0x00051103 chip=0x00041103 rev=0x03 hdr=0x00 vendor = 'HighPoint Technologies, Inc.' device = 'HPT366/368/370/370A/372/372N' class = mass storage bfe0@pci0:2:5:0: class=0x020000 card=0x80a81043 chip=0x440114e4 rev=0x01 hdr=0x00 vendor = 'Broadcom Corporation' device = 'BCM4401 100Base-T' class = network subclass = ethernet # camcontrol devlist | grep WDC at scbus0 target 0 lun 0 (ada0,pass0) at scbus0 target 1 lun 0 (ada1,pass1) at scbus1 target 0 lun 0 (ada2,pass2) # graid3 label -rv data ada0 ada1 ada2 Metadata value stored on ada0. Metadata value stored on ada1. Metadata value stored on ada2. Done. # newfs -j /dev/raid3/data /dev/raid3/data: 381564.4MB (781443928 sectors) block size 32768, fragment size 4096 using 610 cylinder groups of 626.22MB, 20039 blks, 80256 inodes. with soft updates ... Using inode 4 in cg 0 for 33554432 byte journal newfs: soft updates journaling set # mount /dev/raid3/data /mnt After the filesystem has been mounted, I have copied huge files, hundreds of megabytes each, via SFTP and nc(1) to that /mnt from another machine: $ stat Image.bin File: 'Image.bin' Size: 654302880 Blocks: 638968 IO Block: 65536 regular file Device: 3041c3c6h/809616326d Inode: 562949953455390 Links: 1 Access: (0644/-rw-r--r--) Uid: (197609/ mosipov) Gid: (197121/ UNKNOWN) Access: 2016-01-23 09:30:50.807257500 +0100 Modify: 2010-10-10 12:35:43.609375000 +0200 Change: 2010-10-10 12:35:43.609375000 +0200 Birth: 2016-01-23 09:30:50.807257500 +0100 $ scp Image.bin 192.168.1.7:/mnt Password for mosipov@bsd1home: Image.bin 0% 5888KB 52.8KB/s 3:19:57 ETA Now let's copy to non-geom filesystem: $ scp Image.bin 192.168.1.7:/usr/home/mosipov Password for mosipov@bsd1home: Password for mosipov@bsd1home: Image.bin 100% 624MB 10.6MB/s 00:59 It it neither the network card nor the controller as you can see, non-geom is fast and local writes too: # dd if=/dev/zero bs=1M count=10240 | pv -s 10G -treab --progress > /mnt/file.bin 10240+0 records out 10737418240 bytes transferred in 216.084333 secs (49690869 bytes/sec) 10GiB 0:03:36 [47.4MiB/s] [47.4MiB/s] [==========================>] 100% As if there is a bottleneck between socket read and geom write to FS. Is that better? Best regards, Michael PS: iSCSI won't be an option because the machine will be access either from Windows via SMB or SFTP, alternatively via DNLA (HTTP) from phones and TV.