From owner-freebsd-geom@freebsd.org Sun Oct 9 05:07:00 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81E3CC07CCB for ; Sun, 9 Oct 2016 05:07:00 +0000 (UTC) (envelope-from terrence@darkness-reigns.com) Received: from mail.darkness-reigns.net (mail.darkness-reigns.net [149.210.220.190]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.darkness-reigns.net", Issuer "SSL.com DV CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 18576354 for ; Sun, 9 Oct 2016 05:06:58 +0000 (UTC) (envelope-from terrence@darkness-reigns.com) X-CGPDR-AntiVirus-Kaspersky: Scanned and found clean X-CGPDR-AntiVirus-McAfee: Scanned and found clean X-CGPDR-AntiVirus-Sophos: Scanned and found clean X-Abuse-Info: user was authenticated X-Abuse-Info: send abuse reports about this email to abuse@darkness-reigns.net X-ExtFilter: Niversoft's DomainKeys Helper DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; d=darkness-reigns.com; s=mail; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type: X-Mailer:Thread-Index:Content-Language; b=itweUQC1ktVrqsHWx/KrVZYE10CC3/zX4GwBiN2BhQqXm2jfXC0Z2Pjzdk1JcIXcdX 4v5A+qh12DDt35NvGQbA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=darkness-reigns.com; s=mail; t=1475989616; x=1476594416; q=dns/txt; h=From:To:Subject:Date:Message-ID:MIME-Version: Content-Type:X-Mailer:Thread-Index:Content-Language; bh=bf0oYjdS jMZM3tFVb62xenNo6XN6hx06fUib0cse3nU=; b=feg8cSdSVWPRixUMC5Q1ZvxP 98WZ/yYI4acpDioP7Y+s2iPuw/ZlqZEfnKCCx1wfGgX3S1Vg4PSYm4je3rikLw== Received: from internal.domain; Sun, 09 Oct 2016 07:06:54 +0200 From: "Terrence Koeman" To: Subject: fsck geom geli volume? Date: Sun, 9 Oct 2016 07:06:54 +0200 Message-ID: <467301d221ea$f3d6f320$db84d960$@darkness-reigns.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AdIh6dzR29tKIGMdSZmOUDh9ZEJ0Iw== Content-Language: en X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2016 05:07:00 -0000 I had to reboot a server that hung, and now I can't seem to fsck one of the geli volumes. I've found this thread: https://lists.freebsd.org/pipermail/freebsd-geom/2008-September/003042.html, but this doesn't work for me as given: [root@baphomet /usr]# geli attach -p -k /etc/geli/vtbd1.kf /dev/vtbd1 [root@baphomet /usr]# geli list [snip boot vol] [snip swap vol] Geom name: vtbd1.eli State: ACTIVE EncryptionAlgorithm: AES-CBC KeyLength: 256 Crypto: hardware Version: 7 UsedKey: 0 Flags: NONE KeysAllocated: 512 KeysTotal: 512 Providers: 1. Name: vtbd1.eli Mediasize: 2199023251456 (2.0T) Sectorsize: 4096 Mode: r0w0e0 Consumers: 1. Name: vtbd1 Mediasize: 2199023255552 (2.0T) Sectorsize: 512 Mode: r1w1e1 [root@baphomet /usr]# fsck -t ufs -pf /dev/vtbd1.eli /dev/vtbd1.eli: NO WRITE ACCESS /dev/vtbd1.eli: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. I did not attach with the -r option, but somehow fsck thinks it's read-only anyway. An fsck -y completes without finding errors: [root@baphomet /usr]# fsck -t ufs -y /dev/vtbd1.eli ** /dev/vtbd1.eli (NO WRITE) ** Last Mounted on /mnt/backup ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 10 files, 4255186 used, 515758487 free (23 frags, 64469808 blocks, 0.0% fragmentation) What am I missing? Thanks. -- Regards, Terrence Koeman, PhD/MTh/BPsy Darkness Reigns (Holding) B.V. Please quote relevant replies. From owner-freebsd-geom@freebsd.org Sun Oct 9 07:16:52 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32B7EC052B2 for ; Sun, 9 Oct 2016 07:16:52 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 02D68405 for ; Sun, 9 Oct 2016 07:16:51 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from localhost (unknown [24.6.107.161]) by mail.dawidek.net (Postfix) with ESMTPSA id 58C03D34; Sun, 9 Oct 2016 09:16:43 +0200 (CEST) Date: Sun, 9 Oct 2016 09:16:40 +0200 From: Pawel Jakub Dawidek To: Terrence Koeman Cc: freebsd-geom@freebsd.org Subject: Re: fsck geom geli volume? Message-ID: <20161009071639.GA26588@garage.freebsd.pl> References: <467301d221ea$f3d6f320$db84d960$@darkness-reigns.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="k1lZvvs/B4yU6o8G" Content-Disposition: inline In-Reply-To: <467301d221ea$f3d6f320$db84d960$@darkness-reigns.com> X-OS: FreeBSD 11.0-CURRENT amd64 User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2016 07:16:52 -0000 --k1lZvvs/B4yU6o8G Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 09, 2016 at 07:06:54AM +0200, Terrence Koeman wrote: > I had to reboot a server that hung, and now I can't seem to fsck one of t= he geli volumes. >=20 > I've found this thread: https://lists.freebsd.org/pipermail/freebsd-geom/= 2008-September/003042.html, but this doesn't work for me as given: >=20 >=20 > [root@baphomet /usr]# geli attach -p -k /etc/geli/vtbd1.kf /dev/vtbd1 > [root@baphomet /usr]# geli list > [snip boot vol] >=20 > [snip swap vol] >=20 > Geom name: vtbd1.eli > State: ACTIVE > EncryptionAlgorithm: AES-CBC > KeyLength: 256 > Crypto: hardware > Version: 7 > UsedKey: 0 > Flags: NONE > KeysAllocated: 512 > KeysTotal: 512 > Providers: > 1. Name: vtbd1.eli > Mediasize: 2199023251456 (2.0T) > Sectorsize: 4096 > Mode: r0w0e0 This looks good, ie. it is not open by anything yet. > Consumers: > 1. Name: vtbd1 > Mediasize: 2199023255552 (2.0T) > Sectorsize: 512 > Mode: r1w1e1 > [root@baphomet /usr]# fsck -t ufs -pf /dev/vtbd1.eli = =20 > /dev/vtbd1.eli: NO WRITE ACCESS > /dev/vtbd1.eli: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. Please start by running: # ktrace fsck -t ufs -pf /dev/vtbd1.eli and see where exactly it is failing and with what error (run kdump and look for attempts to open /dev/vtbd1.eli device). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://mobter.com --k1lZvvs/B4yU6o8G Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJX+e7XAAoJEJVLhSuxKFt1FUkP/2jJmQa0wyWjP4bNeZzph3WY Cvr4K+dVCqulSZ4uvtobiJvI8zk55wyPgLOfEQmbfW32gE7nmV8idAKVC0yHxKJL GBFJjocoOISa6l2BOsB4Kl4q0ogeU7RY/j5FJKscPehw4FMNf5oqbsaFwjXB59AI bL31nbI3gdrzalkLmhQdAzWcO4fPp9HHn46YTfzcvu+j4hBwvoOgeXiqfNT2AEvX +g9nsMTLXsuWMlXLyj36JrMUvgc8AXcZ9ItyBWAprkBMICq8VWk4Vosoh0oRZTRZ T0ll5DC7NdZpacDQJC/4N1YxCaWkV+5PVVA5r+b8cMPodVNp+ihLaiaqVKwxvq5R AbKdrC7BI1k/sjCb39S6XERWNbCOKkZMkkzY0+pYxiyebLNi9AmDjIYF8A83AQu+ IgLCCulkMzv39EggXXFSIn4MKWBC5z2+AMI9/ctPJxZkC8h2KNWHCBDQVVLeZlo6 aGadswVJeRa3N8h3M/B88A0EhxBpmeom43hAEyK7IYgA7nIC69pZNMhT1m802aHz ClV/yHho0CD++KN9/Gx5mBr2/oVCVtQkfqoMMMfMJsVx0A8+78CgzaxX32XSulGt XEfK/lXhGrlXUYGPs7zfBSDDuGENRL5LLKb5s6MhE6XNHr+jSb0lZiLEpYgU2EoX oj5QGVBrgD3ATYK4BnAA =fY3G -----END PGP SIGNATURE----- --k1lZvvs/B4yU6o8G-- From owner-freebsd-geom@freebsd.org Sun Oct 9 10:59:51 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27A26C07E9A for ; Sun, 9 Oct 2016 10:59:51 +0000 (UTC) (envelope-from terrence@darkness-reigns.com) Received: from mail.darkness-reigns.net (mail.darkness-reigns.net [149.210.220.190]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.darkness-reigns.net", Issuer "SSL.com DV CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A715A9F; Sun, 9 Oct 2016 10:59:49 +0000 (UTC) (envelope-from terrence@darkness-reigns.com) X-CGPDR-AntiVirus-Kaspersky: Scanned and found clean X-CGPDR-AntiVirus-McAfee: Scanned and found clean X-CGPDR-AntiVirus-Sophos: Scanned and found clean X-Abuse-Info: user was authenticated X-Abuse-Info: send abuse reports about this email to abuse@darkness-reigns.net X-ExtFilter: Niversoft's DomainKeys Helper DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; d=darkness-reigns.com; s=mail; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type:X-Mailer:Thread-Index:Content-Language; b=jzDKlMhr6mwucBZL3A/gQadOFyOH2xC5r+oBZzp/Od5W4w73kO4Y/baqON9kZ45570 XlVvuJjowKxsxnZITwsQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=darkness-reigns.com; s=mail; t=1476010786; x=1476615586; q=dns/txt; h=From:To:Cc:References:In-Reply-To:Subject:Date: Message-ID:MIME-Version:Content-Type:X-Mailer:Thread-Index: Content-Language; bh=O9s2aelnUmATVNDGop0cgy3gFeW22QAgYZW6tFJzADo =; b=TUtzSpOa+ocE9iTzgvChWD+o0zsvsrMqShG0erkemQm1MbsrhIhBihIavpp QoKElmVckxfHvF0AZJ27wEhG7bg== Received: from internal.domain; Sun, 09 Oct 2016 12:59:44 +0200 From: "Terrence Koeman" To: Cc: "'Pawel Jakub Dawidek'" References: <467301d221ea$f3d6f320$db84d960$@darkness-reigns.com> <20161009071639.GA26588@garage.freebsd.pl> In-Reply-To: <20161009071639.GA26588@garage.freebsd.pl> Subject: RE: fsck geom geli volume? Date: Sun, 9 Oct 2016 12:59:44 +0200 Message-ID: <46ad01d2221c$3e0aa780$ba1ff680$@darkness-reigns.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQBPvEzvB2aheWofDyAOE9UYSS5gIAIAGFEgo5S8pSA= Content-Language: en X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2016 10:59:51 -0000 [snip] > > Please start by running: > > # ktrace fsck -t ufs -pf /dev/vtbd1.eli > > and see where exactly it is failing and with what error (run kdump and > look for attempts to open /dev/vtbd1.eli device). Unfortunately, I don't have KTRACE compiled in on this machine. However, I found that fsck would work when I commented the attempt at mounting & rebooted. I investigated and I found that when I only run "geli attach" I can run fsck, but when I attach and then try to mount (which fails due to dirty fs) then fsck thinks the fs is read-only. Whether I then detach and attach again doesn't matter, fsck will think /dev/vtbd1.eli is read-only until reboot. -- Regards, Terrence Koeman, PhD/MTh/BPsy Darkness Reigns (Holding) B.V. Please quote relevant replies. From owner-freebsd-geom@freebsd.org Wed Oct 12 15:25:58 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF5F0C0FAE9 for ; Wed, 12 Oct 2016 15:25:58 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 24B1A276 for ; Wed, 12 Oct 2016 15:25:57 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from [192.168.1.8] ([62.157.57.141]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0M85r3-1azTMi2iPZ-00vdVQ for ; Wed, 12 Oct 2016 17:25:49 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> From: Michael Osipov <1983-01-06@gmx.net> Message-ID: <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> Date: Wed, 12 Oct 2016 17:25:43 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:pgdhObPneU7FBEUE8LBiMxzpsWRIz0AYof6Pfa9nL1wFdRVQC1e xYcBCX7Y6DeJrl8Y4pOaJi4f5x+t/blCvd5ttNeUQixWmLmcsuhru3NtpdIa7n4yQvFx5b6 D8vjAQiZK9ezkKYQhKTvFPW4l0FoFve+8uZzq5dLkFfzRRw3rIol2x1RC8AZ4fNRy8X070T ipP/j3TGGao3HQEYNEzTQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:jrn//joW5hA=:frNk+7PiGohjxu4MtIvVAa +pYo056VcwKBxnwVW8hTgxbBHqLm9wYpbUB3jBt6YgFiCz9ORQSQrNDF1OMFNz54RgiNtqJeC bUxM4za87TJlL6WGoqjNOKTBMG+CiVz1+lwjone3BZ1Uk/hGk/7AtOa4Hq+kkSNqLD29Fdxky vIht4fy3Oy03D/kGV7+QpR5XuqdH+PCvb0Tb3w0/CahaBJpNrcGtW+bEu/BqrTh3sMXb3y0IS 0apGmNMtiKciIM26sCmLywBvpJb0k3pEsHS91ceNYK2n7tUscCPS0EwBye13I4BOBdTbGoaND BRBUWY+WgwfTNowH+M4JiaVpgTaV8nvtUIz7APgNfw0NKZ9DrsPoQfMjDQMrFsyYvK14pOJKm X4+qwbk0unYLlxJ8BxXW73D+3hAUx9QnOkI+nA3ZJE2191RpiLdojQOWPjKcXoCTuKk/JfVwX 24roRoHtpVod13f+55Z1mU1gMSu12SYmuYvGuP4yn6vwiqnzNbCp7pgGj7AzFwm07gKovSUw9 YhA4PQ2EI9xm9aWJ/86huhhlG6moP6Uu7BM+8o0FBEn1+9dS9vRDGugvhn2a9OgHVW3HSiZLX sVIfW/ik3Hh5qWLw66yPRong6bb+2/Cbo66r1t+wSDvZr/YV/7B3n5rkN4Bp0FDv2+yeR66ni fVlBvI39m4iZ4bMIUMIstYWqXWF6O40k9rAnJvZma199IufHtaLBY03s9fVJG4O/Arx0eYmzc ROszhICAOSzW0fhYyEUOk15cYeMk7FOV0iqphP56GYaY8GnZnhfQx5ioZQ2ssBWRzxRgBGV0d 7di73+N X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 15:25:58 -0000 Am 2016-10-06 um 20:34 schrieb Michael Osipov: > Hi folks, > > I am experiecing a performance breakdown when writing to some geom class > volume over network. I have tried SCP/SFTP, SMB, nc, ggated/ggatec from > Windows 10 and FreeBSD 10.3-RELEASE and I receive at most 500 kB/s. > Reading is superfast as well as writing to a regular, non-geom volume, > network is fully saturated. > > I am on: > # uname -a > FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 > 18:37:29 UTC 2016 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 > > Any idea what the problem is? How can I investigate on the penality in > performance? Seriously no one able to help? Michael From owner-freebsd-geom@freebsd.org Wed Oct 12 17:08:27 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1025FC0FCFB for ; Wed, 12 Oct 2016 17:08:27 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4E2BBF5 for ; Wed, 12 Oct 2016 17:08:26 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from vader9.bultmann.eu (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 2432C5774 for ; Wed, 12 Oct 2016 19:08:24 +0200 (CEST) Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> From: Jan Bramkamp Message-ID: <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> Date: Wed, 12 Oct 2016 19:08:22 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 17:08:27 -0000 On 12/10/2016 17:25, Michael Osipov wrote: > Am 2016-10-06 um 20:34 schrieb Michael Osipov: >> Hi folks, >> >> I am experiecing a performance breakdown when writing to some geom class >> volume over network. I have tried SCP/SFTP, SMB, nc, ggated/ggatec from >> Windows 10 and FreeBSD 10.3-RELEASE and I receive at most 500 kB/s. >> Reading is superfast as well as writing to a regular, non-geom volume, >> network is fully saturated. >> >> I am on: >> # uname -a >> FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 >> 18:37:29 UTC 2016 >> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 >> >> Any idea what the problem is? How can I investigate on the penality in >> performance? > > Seriously no one able to help? Not with the description you've given. Please describe the network setup, how you access remote block devices, what you mean by "geom volume". The most common way to access remote storage at the block level supported by FreeBSD is iSCSI. FreeBSD 10.3 contains an iSCSI target and initiator in the base system. From owner-freebsd-geom@freebsd.org Wed Oct 12 18:54:49 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29588C0F311 for ; Wed, 12 Oct 2016 18:54:49 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 933C79C1 for ; Wed, 12 Oct 2016 18:54:47 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from [192.168.1.8] ([62.157.57.141]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LpObx-1bFqmu32RR-00fB8q for ; Wed, 12 Oct 2016 20:54:45 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> From: Michael Osipov <1983-01-06@gmx.net> Message-ID: <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> Date: Wed, 12 Oct 2016 20:54:45 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:9hYoP4Mx19YFRY/QqGG0DQLqtQmBWuTZQU3rWYcffngKJKNQMNL lU6RpaluNfyNXgsgax35lDHENr0IsQ289wjChfK6UZxJJ0foMVp1mwFbSxDROtc73A/9rQv qRSotRM5egSM+tnL6RPBoD9nFQ3JFahC9eixA8efrTjtLay8GVfyePYN8FOsGsTrtkHQKrM 9mYd0+4SUc2V5rWY4sHdw== X-UI-Out-Filterresults: notjunk:1;V01:K0:4QskYIo83Co=:mftlclqxBoz2NMiC0aPwQI D4AzJ0ju1s8blKKhzpsyRGgMoLXSqBKMFkxIy5V+/5+gVtcd3Sg3ZcdJc4yUS4A6E5PXNjoFb l+JVznfWItCjyDplYe6TQ6SDCmzkI2Rf50OEQoQu2ytOGYfqiUBMCG7GhOxTX+LsfitwF6ht7 JzYScdMPebgE7neuDippxYEGcKur4xBdEgJuPT0ZAuKfcboai0s3jt+VRIaBB4JP13QnGsTkc 9lRZW2lsIQCJZ7+gQE//kyVJk3ZRMwj52yh1sRRm/Xf5pGEPrhjO3Gx0qRXOMo59/PsdrDzBe uNykAVDgw6TQqddwz59C8jlEb/z8SZOo91sUdrqDoeA1Wxt0NjHFnZxOR12TKs/WQ9viK6dbM hR8apIk5Zu4peYFLNeOlFesdJPja/GBtWnytHs58aUad0TGh5vw1tj4GG46QPsYfdyRTYz9Yw FxdLyoNFXp/As1LahuXalqEl9EaI8POD5hb+em13CHSSbQ57hQCUU4koiq55++fGjMZ6KRz9m odrbb6GIl2BVyVmx1tzQvjv3LT+la2KaZzMO+8Na5VErL2DSf9gwIqLRBTMsx3SS2HMJRMawI GoNStwTj42B3222/OrpwuUhgwg3QrGyaTWZO+RnE5gtOV3+CpPMjzqcnyl7BK4GYKLzJ3uFBs EOQd20ILAN4fAmPBDVGBxB7eGdiJnOlcSE9FwG94E8oBQmJXFW/w5Ua0a4Ztkw7d6ues6eIO5 f0bkIL8CTu+iURDesZSvnM4hrnXVske9QFA9uGqgZUUq7pjdpyp/B7qUU3JXKpMcmUVAr30ax FXuyTl3 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 18:54:49 -0000 Am 2016-10-12 um 19:08 schrieb Jan Bramkamp: > > > On 12/10/2016 17:25, Michael Osipov wrote: >> Am 2016-10-06 um 20:34 schrieb Michael Osipov: >>> Hi folks, >>> >>> I am experiecing a performance breakdown when writing to some geom class >>> volume over network. I have tried SCP/SFTP, SMB, nc, ggated/ggatec from >>> Windows 10 and FreeBSD 10.3-RELEASE and I receive at most 500 kB/s. >>> Reading is superfast as well as writing to a regular, non-geom volume, >>> network is fully saturated. >>> >>> I am on: >>> # uname -a >>> FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 >>> 18:37:29 UTC 2016 >>> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 >>> >>> Any idea what the problem is? How can I investigate on the penality in >>> performance? >> >> Seriously no one able to help? > > Not with the description you've given. Probably, yes though I wasn't even sure where to start. > Please describe the network setup, how you access remote block devices, > what you mean by "geom volume". Simply put, I have this: $ uname -a FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:37:29 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386 atapci0@pci0:2:13:0: class=0x018000 card=0x00051103 chip=0x00041103 rev=0x03 hdr=0x00 vendor = 'HighPoint Technologies, Inc.' device = 'HPT366/368/370/370A/372/372N' class = mass storage bfe0@pci0:2:5:0: class=0x020000 card=0x80a81043 chip=0x440114e4 rev=0x01 hdr=0x00 vendor = 'Broadcom Corporation' device = 'BCM4401 100Base-T' class = network subclass = ethernet # camcontrol devlist | grep WDC at scbus0 target 0 lun 0 (ada0,pass0) at scbus0 target 1 lun 0 (ada1,pass1) at scbus1 target 0 lun 0 (ada2,pass2) # graid3 label -rv data ada0 ada1 ada2 Metadata value stored on ada0. Metadata value stored on ada1. Metadata value stored on ada2. Done. # newfs -j /dev/raid3/data /dev/raid3/data: 381564.4MB (781443928 sectors) block size 32768, fragment size 4096 using 610 cylinder groups of 626.22MB, 20039 blks, 80256 inodes. with soft updates ... Using inode 4 in cg 0 for 33554432 byte journal newfs: soft updates journaling set # mount /dev/raid3/data /mnt After the filesystem has been mounted, I have copied huge files, hundreds of megabytes each, via SFTP and nc(1) to that /mnt from another machine: $ stat Image.bin File: 'Image.bin' Size: 654302880 Blocks: 638968 IO Block: 65536 regular file Device: 3041c3c6h/809616326d Inode: 562949953455390 Links: 1 Access: (0644/-rw-r--r--) Uid: (197609/ mosipov) Gid: (197121/ UNKNOWN) Access: 2016-01-23 09:30:50.807257500 +0100 Modify: 2010-10-10 12:35:43.609375000 +0200 Change: 2010-10-10 12:35:43.609375000 +0200 Birth: 2016-01-23 09:30:50.807257500 +0100 $ scp Image.bin 192.168.1.7:/mnt Password for mosipov@bsd1home: Image.bin 0% 5888KB 52.8KB/s 3:19:57 ETA Now let's copy to non-geom filesystem: $ scp Image.bin 192.168.1.7:/usr/home/mosipov Password for mosipov@bsd1home: Password for mosipov@bsd1home: Image.bin 100% 624MB 10.6MB/s 00:59 It it neither the network card nor the controller as you can see, non-geom is fast and local writes too: # dd if=/dev/zero bs=1M count=10240 | pv -s 10G -treab --progress > /mnt/file.bin 10240+0 records out 10737418240 bytes transferred in 216.084333 secs (49690869 bytes/sec) 10GiB 0:03:36 [47.4MiB/s] [47.4MiB/s] [==========================>] 100% As if there is a bottleneck between socket read and geom write to FS. Is that better? Best regards, Michael PS: iSCSI won't be an option because the machine will be access either from Windows via SMB or SFTP, alternatively via DNLA (HTTP) from phones and TV. From owner-freebsd-geom@freebsd.org Thu Oct 13 02:11:25 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E91F2C0F03F for ; Thu, 13 Oct 2016 02:11:25 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B879DEC1 for ; Thu, 13 Oct 2016 02:11:25 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id u9D1fiFD020083 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 12 Oct 2016 18:41:44 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id u9D1fi4k020082; Wed, 12 Oct 2016 18:41:44 -0700 (PDT) (envelope-from jmg) Date: Wed, 12 Oct 2016 18:41:43 -0700 From: John-Mark Gurney To: Michael Osipov <1983-01-06@gmx.net> Cc: freebsd-geom@freebsd.org Subject: Re: Abysmally slow write to geom class volume over network Message-ID: <20161013014143.GA1669@funkthat.com> Mail-Followup-To: Michael Osipov <1983-01-06@gmx.net>, freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> X-Operating-System: FreeBSD 11.0-ALPHA2 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Wed, 12 Oct 2016 18:41:44 -0700 (PDT) X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2016 02:11:26 -0000 Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 +0200: > As if there is a bottleneck between socket read and geom write to FS. > > Is that better? Have you run gstat on the system to see if there is an IO bottle neck? Since you are using graid3, you want to look to see if it's %busy is ~100, while the underlying components are not. Also, if you do dd if=/dev/zero of=/mnt/somefile bs=1m count=1024, from the local system, are things still slow? P.S. This is basic debugging, have to identify which part of the system is misbehaving. P.P.S. Your "non-geom" test is still using geom, just not w/ the graid3 module. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-geom@freebsd.org Thu Oct 13 18:41:43 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6377FC10D1A for ; Thu, 13 Oct 2016 18:41:43 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B759813A2 for ; Thu, 13 Oct 2016 18:41:42 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from [192.168.1.8] ([62.157.47.35]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0M2ojS-1b26MF0uOM-00sbfe for ; Thu, 13 Oct 2016 20:41:33 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> From: Michael Osipov <1983-01-06@gmx.net> Message-ID: <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> Date: Thu, 13 Oct 2016 20:41:32 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161013014143.GA1669@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:F+hNP6Ra/8HSu0ON+9QARBqqCkvoKDBmv+zr21G2QtRBKV2iAID fTAMzVl1ZP6CAuWetLH88sH2MRxKwIR7ijmINZsFp7gXZVfYF/nGpEF/khwZRhKpx/kCENg y46K4h4EdiICFMeTKgvEkF+wTCUPk4cU4pZVsfXyoHqkLrL1xSva8SVIKyiBed4RTLaxmUu jqewe4gUmzbfqtSW1dwBQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:3Ij86d39uYs=:02eZ/w1sbTixB3eqo8T+xG +p8bYSEPSnxxPgoXqOwiyAxkqn8FRq1esZbtctNbXJc6pEyAvpv1aV99c1O0vQg9U4ltRVCoW JhyPnkeCbwKYWQ4jbmmFlwB9z3HBNW9m8hp6Y2Lm1psmaZ22hwO2IV/+cxSMa9D+r1S0tOJIs SHUVv6ae9hl8xc52uF/vLNNCxwE2C6r5nr1qyiFyIld4hxLcVvtxL//qiozsQ+RZIP3Q9M7Xr d2Tdj6HTtHwyqqwbJYlc8Y2Wsdde2i13J1DwLblac1p3whJu/hZhE1vZ3/3hzipsZ9hAr28At KL5QDHvwqk8bDmVcs8mTywkzk3vnQ/dltsQZH5vX8zpe8nJegxyx0zRQ+waGY0JgzM33ukpJv Ac46IqGMtDlWiou+QFkFyUwsrnYW0/eHmBBxAp/SWbw7pTyTze/+vbkxAh4RSmluVPlyZrRin 78jrEMWK0+oWCiolw3VrLltqtk+vqvSoLMKvStrwggbXv2w9pMzl0CKOwZgY0UJlThviumOpx d5C2/CWKBHD9qdA4tI1Z1ijzTjMPRh+pi02f8jOLnOo/BFUe/ZBdU2kI4xuPgtLqGEdzDnbLi 9+lfcr8GDvNuXLcQinFoaz8MxyQIK9wZZKtjD65U8ay4MDXFWU5E48RUaGbym0WbBDaRay0VS MT0dN7LgrIIhaUBCCDS4hWZUrbOBzju2kZJxSdEiuwt2Iu6X2k4Zr7M1Fxnsm4IANFTmGPrSA hUD93RrEVd8tCcEgBtFrKAlK0KL0q2tlEG3KKmi9XPxUUKwKOaflRFCYxFMfvsDXWB0dyBtcj OSkZ8Qy X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2016 18:41:43 -0000 Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: > Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 +0200: >> As if there is a bottleneck between socket read and geom write to FS. >> >> Is that better? > > Have you run gstat on the system to see if there is an IO bottle > neck? Since you are using graid3, you want to look to see if > it's %busy is ~100, while the underlying components are not. This is hardly impossible because as soon as I start some SFTP transfer, all of my SSH sessions free or receive connetion timeout/abort. Doing a SFTP from FreeBSD to FreeBSD gives me on both physical disks and RAID3 volume a busy of zero to one perfect. In other terms, the drives are bored. > Also, if you do dd if=/dev/zero of=/mnt/somefile bs=1m count=1024, > from the local system, are things still slow? I have done this in the first place, this is fast: # dd if=/dev/zero bs=1M count=10240 | pv -s 10G -treab --progress > /mnt/file.bin 10240+0 records out 10737418240 bytes transferred in 216.084333 secs (49690869 bytes/sec) 10GiB 0:03:36 [47.4MiB/s] [47.4MiB/s] [==========================>] 100% > P.S. This is basic debugging, have to identify which part of the > system is misbehaving. I am aware of that. Thank you. > P.P.S. Your "non-geom" test is still using geom, just not w/ the graid3 > module. Oh ok, I wasn't aware of that. Actually, I have tried all of them: concat, stripe, mirror, gvinum. Local operations are fast, though remote still crawls slowly. Michael From owner-freebsd-geom@freebsd.org Thu Oct 13 20:22:57 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF736C109F5 for ; Thu, 13 Oct 2016 20:22:57 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 9E8002D5 for ; Thu, 13 Oct 2016 20:22:57 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from localhost (unknown [24.6.107.161]) by mail.dawidek.net (Postfix) with ESMTPSA id ED2C6BAF; Thu, 13 Oct 2016 22:22:49 +0200 (CEST) Date: Thu, 13 Oct 2016 22:22:46 +0200 From: Pawel Jakub Dawidek To: Terrence Koeman Cc: freebsd-geom@freebsd.org Subject: Re: fsck geom geli volume? Message-ID: <20161013202246.GA39811@garage.freebsd.pl> References: <467301d221ea$f3d6f320$db84d960$@darkness-reigns.com> <20161009071639.GA26588@garage.freebsd.pl> <46ad01d2221c$3e0aa780$ba1ff680$@darkness-reigns.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="x+6KMIRAuhnl3hBn" Content-Disposition: inline In-Reply-To: <46ad01d2221c$3e0aa780$ba1ff680$@darkness-reigns.com> X-OS: FreeBSD 11.0-CURRENT amd64 User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2016 20:22:57 -0000 --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 09, 2016 at 12:59:44PM +0200, Terrence Koeman wrote: >=20 > [snip] > >=20 > > Please start by running: > >=20 > > # ktrace fsck -t ufs -pf /dev/vtbd1.eli > >=20 > > and see where exactly it is failing and with what error (run kdump and > > look for attempts to open /dev/vtbd1.eli device). >=20 > Unfortunately, I don't have KTRACE compiled in on this machine. However, = I found that fsck would work when I commented the attempt at mounting & reb= ooted. >=20 > I investigated and I found that when I only run "geli attach" I can run f= sck, but when I attach and then try to mount (which fails due to dirty fs) = then fsck thinks the fs is read-only. Whether I then detach and attach agai= n doesn't matter, fsck will think /dev/vtbd1.eli is read-only until reboot. I recall there was some hack that allowed to fsck device even if it was open already, but only once. Maybe this is somehow related. You could try setting kern.geom.debugflags to 4 and see where g_access(9) is failing. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://mobter.com --x+6KMIRAuhnl3hBn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJX/+0WAAoJEJVLhSuxKFt1SzEP/2D+rj7PxgPxJxO7b+74egXH Z9/s1GlFlnE2R/TS2cv31zjgPtl2qLiugzFy3LBr1xGUIngG2E66q1Xt2Zbi2aoH gvlAU696+hL+b4xfkZBQZ6HiHzh2ahDpgtIo2NX+A/k/a4GfSyw3DkSHrWF2DMsT m8PnGUuO1s3EbO/uOyV0knCzDXdg4cJRGiSNHsi5dn9g+vpxGdrPS16tDcRaUHfL TB6dvXgvBJ29/xhdRX18pUnT/OSYZv5KRECsVjx+KxkkTxhE5YscbQ1qZG0OtN1w 09fGY+XT8uwXEwXUHlv8uXTvihGq5z75KWUuo5/kf52dV7JGGV+R57G8ih6VZAdS GTLthdiTTBUvpglm2WtbQx7AAF55l/1AkEFzfzcpX0tgjotMwJVokGQPFdLVEjzO uARDJYRDqG31R7yzQCauKgfT9RR/02kRkqVjmdJgJ2GAIteov+AoUyEEV+2Z7ehG 11e8EJius6I9I0it9BGmOzf/EbZ3wQDLN+bxSyR7fobMdgA8jR0vVJiNGf99zuH5 858Mi+V02zxl4YzW6hrJrYkckuqoDzXH3jxyiTCkboSOjrEG1QEH5ejjYjgs0ki+ tCqCcaQIjqac7q8fvzyoiHHQ7efz1/uwa+i2Thnzh6kXGEnrejjI3PDcSwPyvZod +6WN2hyKAvDvWsiXmIiD =d3z6 -----END PGP SIGNATURE----- --x+6KMIRAuhnl3hBn-- From owner-freebsd-geom@freebsd.org Fri Oct 14 09:47:16 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3E4EC0F2F5 for ; Fri, 14 Oct 2016 09:47:16 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.18.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 770F115C for ; Fri, 14 Oct 2016 09:47:16 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.169.203] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (Exim 4.84) (envelope-from ) id 1buz1Y-0001Tj-Rs; Fri, 14 Oct 2016 11:43:32 +0200 Date: Fri, 14 Oct 2016 11:43:30 +0200 From: Fabian Keil To: Michael Osipov <1983-01-06@gmx.net> Cc: freebsd-geom@freebsd.org Subject: Re: Abysmally slow write to geom class volume over network Message-ID: <20161014114330.396fe534@fabiankeil.de> In-Reply-To: <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/cAhz0Jgv60TflM0qjGYUZ2u"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2016 09:47:16 -0000 --Sig_/cAhz0Jgv60TflM0qjGYUZ2u Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Michael Osipov <1983-01-06@gmx.net> wrote: > Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: > > Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 > > +0200: =20 > >> As if there is a bottleneck between socket read and geom write to FS. > >> > >> Is that better? =20 > > > > Have you run gstat on the system to see if there is an IO bottle > > neck? Since you are using graid3, you want to look to see if > > it's %busy is ~100, while the underlying components are not. =20 >=20 > This is hardly impossible because as soon as I start some SFTP transfer,= =20 > all of my SSH sessions free or receive connetion timeout/abort. Doing a= =20 > SFTP from FreeBSD to FreeBSD gives me on both physical disks and RAID3=20 > volume a busy of zero to one perfect. In other terms, the drives are > bored. Try checking the FAIL and SLEEP columns in the "vmstat -z" output. This could be related to: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209680#c2 Fabian --Sig_/cAhz0Jgv60TflM0qjGYUZ2u Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlgAqMMACgkQBYqIVf93VJ3DXQCeJglkl/Nfsdizp2qglBI1fo+i niYAnRm6Nyc01Mp+bqHgP9vmOOjjTmXy =Y7jL -----END PGP SIGNATURE----- --Sig_/cAhz0Jgv60TflM0qjGYUZ2u-- From owner-freebsd-geom@freebsd.org Fri Oct 14 16:12:55 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70FFAC10ACE for ; Fri, 14 Oct 2016 16:12:55 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D8B95132 for ; Fri, 14 Oct 2016 16:12:54 +0000 (UTC) (envelope-from 1983-01-06@gmx.net) Received: from [192.168.1.8] ([62.157.52.151]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LuKHz-1cwID62bgt-011i9e; Fri, 14 Oct 2016 18:12:51 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: Fabian Keil References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> <20161014114330.396fe534@fabiankeil.de> Cc: freebsd-geom@freebsd.org From: Michael Osipov <1983-01-06@gmx.net> Message-ID: <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> Date: Fri, 14 Oct 2016 18:12:49 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161014114330.396fe534@fabiankeil.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:XcSPIKUuXXLjQxsDfLLyiH81xpoaxV6tPBklRejca2Zii5FUTOZ hD8YhmCFtxoXIE+h7LyPQDtSrSYFqxy/Xdj/sOvZRBjFEaJ4eGBv4P9ZLwxvODRdCZzfjdN yAUL9j/1kQN24jnyzYatDacWOBoiFtYJni/YliXzKbkTEN8u3YzXD2/oA0LB/gMxVdmYmDf hhEfuQFkKoYnTd3vvCrqg== X-UI-Out-Filterresults: notjunk:1;V01:K0:8kk/7Bf3u+I=:cBf2KXV2i8XsBQuHP20kSS YV6y7hTQByu5NIXF/wc8RXI65jqCJXDy/QOzLZgdlJxRcPbW3APgYLV9e0/InRXSnZiC9UZKy 171km9xnFmm+Bya9TnPjO/kwD/YoIz6dN/T61n8RQ7cDADfYsOsecWCA/TnBADo95I1hH8iqD +dZtdwnvdGbVc2wqf/uOZ74+jPtaREO2k488bXHUreRfrDb8f223ryAwYB5TuHkDUtHV8R2QE 8zrO6qDqK/gF87Av7q+v0rvhM2HAcUhqg0WzVkPorBL/tA4KBH0mN48UA4BGuSrg1iFlODtSs j+DLzSlzyO9OOWQleGUcOQ51SL4rM++1llgutwTuqTuao2hkQaEyKJtYRdHMLI7USbaXLUSE9 fqyUCVUuAydhwGXdqLwkFiDAO0X2Q9asu5pfJp+YLMWCya1leNM4ZdmxNSpc0p0kwUeqqh0Th 0H6LX26bTFBPMdnl64zU9otwKtuyRvG3wizTFl+hdUJwVZvF6em4fTQ+U5hxkCTcUj+cVSIVv jfBUymLNL3JKVtNfhc4AX9Hxmu5xtmpRhvchVIem5guRchpj4+s4XfBs9Qa6uh8wx/hzAlRhL myHWVR9zvmJ+laQsshnhca7WlHqB8u0wTyXvZpcZPcmAPHkDB5lOoGubo9m7VN1mMz7eqfgGh NSNauZxYELIzVA5rwh/gxvtof3SSjWBEaX0adlQwxAkagYURpPkspPliuN6eb1a6GBU7t3Y18 o+Osnb+xuynj2/9BZVViwd3ATX0TzBWh98lDNia15RdSqRq6sv7entiS0kXkIkIGjBfTndas1 SdudXhw X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2016 16:12:55 -0000 Am 2016-10-14 um 11:43 schrieb Fabian Keil: > Michael Osipov <1983-01-06@gmx.net> wrote: > >> Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: >>> Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 >>> +0200: >>>> As if there is a bottleneck between socket read and geom write to FS. >>>> >>>> Is that better? >>> >>> Have you run gstat on the system to see if there is an IO bottle >>> neck? Since you are using graid3, you want to look to see if >>> it's %busy is ~100, while the underlying components are not. >> >> This is hardly impossible because as soon as I start some SFTP transfer, >> all of my SSH sessions free or receive connetion timeout/abort. Doing a >> SFTP from FreeBSD to FreeBSD gives me on both physical disks and RAID3 >> volume a busy of zero to one perfect. In other terms, the drives are >> bored. > > Try checking the FAIL and SLEEP columns in the "vmstat -z" output. I assume that you expect a rise on those numbers. I have made several runs. Rebooted the machine and then started SFTP transfer. After seconds my SSH sessions locked up. The transfer was aborted manually after 10 minutes which should have saturated the entire connection. After that, I reran vmstat -z, no or minimal rise in FAIL and SLEEP. Interesting to say that this happens if is is a UFS volume on gconcat/graid3/gvinum/gstripe configuration. Regular gpart with GPT has no performance penalty. Additionally, it is not limited to SSH but virtually everything with sockets: nc, ggate, smb. > This could be related to: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209680#c2 It pretty much sounds like it, though I do not use ipfw, pf or any NAT stuff. I will try your first patch and let you know. Do you want me to add my usecase to the issue? Michael From owner-freebsd-geom@freebsd.org Sat Oct 15 11:34:55 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49CF6C12F90 for ; Sat, 15 Oct 2016 11:34:55 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D459EA9 for ; Sat, 15 Oct 2016 11:34:54 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.156.164] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (Exim 4.84) (envelope-from ) id 1bvN3A-00018q-Vd for freebsd-geom@freebsd.org; Sat, 15 Oct 2016 13:22:49 +0200 Date: Sat, 15 Oct 2016 13:22:49 +0200 From: Fabian Keil To: freebsd-geom@freebsd.org Subject: Re: Abysmally slow write to geom class volume over network Message-ID: <20161015132249.607b374c@fabiankeil.de> In-Reply-To: <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> <20161014114330.396fe534@fabiankeil.de> <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> Reply-To: freebsd-geom@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/22vuQlpGpXF63./exq12l87"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Oct 2016 11:34:55 -0000 --Sig_/22vuQlpGpXF63./exq12l87 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Michael Osipov <1983-01-06@gmx.net> wrote: > Am 2016-10-14 um 11:43 schrieb Fabian Keil: > > Michael Osipov <1983-01-06@gmx.net> wrote: > > =20 > >> Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: =20 > >>> Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 > >>> +0200: =20 > >>>> As if there is a bottleneck between socket read and geom write to > >>>> FS. > >>>> > >>>> Is that better? =20 > >>> > >>> Have you run gstat on the system to see if there is an IO bottle > >>> neck? Since you are using graid3, you want to look to see if > >>> it's %busy is ~100, while the underlying components are not. =20 > >> > >> This is hardly impossible because as soon as I start some SFTP > >> transfer, all of my SSH sessions free or receive connetion > >> timeout/abort. Doing a SFTP from FreeBSD to FreeBSD gives me on both > >> physical disks and RAID3 volume a busy of zero to one perfect. In > >> other terms, the drives are bored. =20 > > > > Try checking the FAIL and SLEEP columns in the "vmstat -z" output. =20 >=20 > I assume that you expect a rise on those numbers. I have made several=20 > runs. Rebooted the machine and then started SFTP transfer. After seconds= =20 > my SSH sessions locked up. The transfer was aborted manually after 10=20 > minutes which should have saturated the entire connection. After that, I= =20 > reran vmstat -z, no or minimal rise in FAIL and SLEEP. IIRC the SLEEP column only showns currently sleeping requests, therefore you may want to run "vmstat -z" multiple times while the transfer is ongoing. Having said that, a custom DTrace script would probably be a better tool to diagnose the issue anyway. > Interesting to say that this happens if is is a UFS volume on=20 > gconcat/graid3/gvinum/gstripe configuration. Regular gpart with GPT has=20 > no performance penalty. Additionally, it is not limited to SSH but=20 > virtually everything with sockets: nc, ggate, smb. >=20 > > This could be related to: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209680#c2 =20 >=20 > It pretty much sounds like it, though I do not use ipfw, pf or any NAT=20 > stuff. I will try your first patch and let you know. >=20 > Do you want me to add my usecase to the issue? If the patch helps, that could be useful once a committer finds the time to look at the PR. Fabian --Sig_/22vuQlpGpXF63./exq12l87 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlgCEYkACgkQBYqIVf93VJ2onQCfeXtrcivSCDnFcysKRANjg8DK k3gAoIKMagZ3TK+DJdbvow21YmIqXqpe =fwh3 -----END PGP SIGNATURE----- --Sig_/22vuQlpGpXF63./exq12l87-- From owner-freebsd-geom@freebsd.org Sat Oct 15 12:02:05 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50276C12DE4 for ; Sat, 15 Oct 2016 12:02:05 +0000 (UTC) (envelope-from mosipov@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B59A9EBF for ; Sat, 15 Oct 2016 12:02:04 +0000 (UTC) (envelope-from mosipov@gmx.de) Received: from [192.168.1.8] ([62.157.52.151]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0McluX-1cDKBc40UT-00Hsxr for ; Sat, 15 Oct 2016 14:02:02 +0200 Subject: Re: Abysmally slow write to geom class volume over network To: freebsd-geom@freebsd.org References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> <20161014114330.396fe534@fabiankeil.de> <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> <20161015132249.607b374c@fabiankeil.de> From: Michael Osipov Message-ID: <90ca3d9c-6016-8a63-64c5-ce3f829756dd@gmx.de> Date: Sat, 15 Oct 2016 14:01:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161015132249.607b374c@fabiankeil.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:ktBN6280/2iCxUygzkxrG7QJvUKDuLPP5XeExw5Htm8wo7cL9Jw VpA5wn/pNOiQf4Ixq1D8x6TlAvMh5Pg3KlypBRhj9N71sRufvM3rbeJpiCztCJ/evmu+hbh lrxshEpUdmlMvuA2OMH5BVcSjaFq9okcEyGv4uPj+xrWuAAc12A44lWFPub3Afffd3TJ0mm i5GlRdMuWcUjbgFz7gcGg== X-UI-Out-Filterresults: notjunk:1;V01:K0:GCyEBmCcCqM=:yadeHeb6fXW89R2FXLtjdg QqaPAqRusLpEt0tGTnEWTPslNeNmQ1EkKpF4OieAwPARYOFEPirJSmwgw+OKvyy+t2ozwJp7j rq1D6Zd4kyQ5/Ffi76jvc/WOrpe1ANVrH8+AWiHdEvOTTOPgGJYfCyCKmJ2HEiNXjOKQy+b2h a1gLm3xyDWJRq/BXWImCvgz1nHj8nNof0uRZf6/06TtpvOxmKiHQlmW2nI0dGFEeNiQw28NoX n8tVoop/f1Xq5BWLIVqfhBlF+uf8Ju3UYRI7/UHfRVwY5WIbTR3AVVlAFzMmu7JukDpyoFkdP MA4+kD7htLTn5xryrOkK2l578Q7gJR4Fx/Msrt/8BPpsyKqp+/jV/rorE6vgw/LTgGM9BKQrT MVRvLKlE1ILHDkZsfBcgQ9bzxF6lEyLpHP7Y6pQamxB6f6q6gVuJtEBNgrkxuCcLO+knETGkP 5wWnJ8nAyDxCZxrXb6COYgCTBXpzw6BIZ/Pgle2BpksmifMO59akS/SFNWQGS6FlhYvME/lVH ZPG/lt8WTwokoJ+XCtqPM/qv/umTTx8CJQcODekgsTae3tzA7tVX2iyoGKwIVihgP4BD7qKrD 8Cd+T3xD606kgP5lgIvHdV+fEKaqWNtba3cezyD9Gc6kQBkzNYu59Ns2kSjTxbQAn0F7Irb0o 5ethoA4kG2Gsg2IAYodaGkNdzkB5sMN0FcOzzuNDqu0CLygjYNfvMQSTD/wD0MsUA6WtbIGdy 674NU03po4bqax9gTNakuNLMNxhLeVI5tMbAMNii7/YHakKvblP4rsexUmljy+pk7fFuMlu2b 7gMOmX1 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Oct 2016 12:02:05 -0000 Am 2016-10-15 um 13:22 schrieb Fabian Keil: > Michael Osipov <1983-01-06@gmx.net> wrote: > >> Am 2016-10-14 um 11:43 schrieb Fabian Keil: >>> Michael Osipov <1983-01-06@gmx.net> wrote: >>> >>>> Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: >>>>> Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 >>>>> +0200: >>>>>> As if there is a bottleneck between socket read and geom write to >>>>>> FS. >>>>>> >>>>>> Is that better? >>>>> >>>>> Have you run gstat on the system to see if there is an IO bottle >>>>> neck? Since you are using graid3, you want to look to see if >>>>> it's %busy is ~100, while the underlying components are not. >>>> >>>> This is hardly impossible because as soon as I start some SFTP >>>> transfer, all of my SSH sessions free or receive connetion >>>> timeout/abort. Doing a SFTP from FreeBSD to FreeBSD gives me on both >>>> physical disks and RAID3 volume a busy of zero to one perfect. In >>>> other terms, the drives are bored. >>> >>> Try checking the FAIL and SLEEP columns in the "vmstat -z" output. >> >> I assume that you expect a rise on those numbers. I have made several >> runs. Rebooted the machine and then started SFTP transfer. After seconds >> my SSH sessions locked up. The transfer was aborted manually after 10 >> minutes which should have saturated the entire connection. After that, I >> reran vmstat -z, no or minimal rise in FAIL and SLEEP. > > IIRC the SLEEP column only showns currently sleeping requests, > therefore you may want to run "vmstat -z" multiple times while > the transfer is ongoing. Having said that, a custom DTrace script > would probably be a better tool to diagnose the issue anyway. Ah ok, I need to switch physically to that machine because SSH is not possible. I don't mind to read the DTrace tutorial but what exactly should I trace? sshd? >> Interesting to say that this happens if is is a UFS volume on >> gconcat/graid3/gvinum/gstripe configuration. Regular gpart with GPT has >> no performance penalty. Additionally, it is not limited to SSH but >> virtually everything with sockets: nc, ggate, smb. >> >>> This could be related to: >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209680#c2 >> >> It pretty much sounds like it, though I do not use ipfw, pf or any NAT >> stuff. I will try your first patch and let you know. >> >> Do you want me to add my usecase to the issue? > > If the patch helps, that could be useful once a committer > finds the time to look at the PR. I will switch from 11.0-RELEASE to 11.0-STABLE via Subversion first, rebuild world and kernel and then apply your patches. Hand on. Michael