From owner-freebsd-geom@freebsd.org Sat Oct 15 11:34:55 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49CF6C12F90 for ; Sat, 15 Oct 2016 11:34:55 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D459EA9 for ; Sat, 15 Oct 2016 11:34:54 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.156.164] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (Exim 4.84) (envelope-from ) id 1bvN3A-00018q-Vd for freebsd-geom@freebsd.org; Sat, 15 Oct 2016 13:22:49 +0200 Date: Sat, 15 Oct 2016 13:22:49 +0200 From: Fabian Keil To: freebsd-geom@freebsd.org Subject: Re: Abysmally slow write to geom class volume over network Message-ID: <20161015132249.607b374c@fabiankeil.de> In-Reply-To: <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> References: <33da0f73-48f1-4727-fe76-41343dc4955b@gmx.net> <18314f27-849b-31df-d88d-af64e89c133f@gmx.net> <346c7da7-02b2-ba34-1463-f3f0a5a3cd9a@rlwinm.de> <15e9cb94-7ad8-e547-b06a-699ce2250624@gmx.net> <20161013014143.GA1669@funkthat.com> <636763f0-a732-18ba-262b-c3fc01f4342c@gmx.net> <20161014114330.396fe534@fabiankeil.de> <5a12f70d-3488-e799-b875-9b358ef7aff9@gmx.net> Reply-To: freebsd-geom@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/22vuQlpGpXF63./exq12l87"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Oct 2016 11:34:55 -0000 --Sig_/22vuQlpGpXF63./exq12l87 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Michael Osipov <1983-01-06@gmx.net> wrote: > Am 2016-10-14 um 11:43 schrieb Fabian Keil: > > Michael Osipov <1983-01-06@gmx.net> wrote: > > =20 > >> Am 2016-10-13 um 03:41 schrieb John-Mark Gurney: =20 > >>> Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54 > >>> +0200: =20 > >>>> As if there is a bottleneck between socket read and geom write to > >>>> FS. > >>>> > >>>> Is that better? =20 > >>> > >>> Have you run gstat on the system to see if there is an IO bottle > >>> neck? Since you are using graid3, you want to look to see if > >>> it's %busy is ~100, while the underlying components are not. =20 > >> > >> This is hardly impossible because as soon as I start some SFTP > >> transfer, all of my SSH sessions free or receive connetion > >> timeout/abort. Doing a SFTP from FreeBSD to FreeBSD gives me on both > >> physical disks and RAID3 volume a busy of zero to one perfect. In > >> other terms, the drives are bored. =20 > > > > Try checking the FAIL and SLEEP columns in the "vmstat -z" output. =20 >=20 > I assume that you expect a rise on those numbers. I have made several=20 > runs. Rebooted the machine and then started SFTP transfer. After seconds= =20 > my SSH sessions locked up. The transfer was aborted manually after 10=20 > minutes which should have saturated the entire connection. After that, I= =20 > reran vmstat -z, no or minimal rise in FAIL and SLEEP. IIRC the SLEEP column only showns currently sleeping requests, therefore you may want to run "vmstat -z" multiple times while the transfer is ongoing. Having said that, a custom DTrace script would probably be a better tool to diagnose the issue anyway. > Interesting to say that this happens if is is a UFS volume on=20 > gconcat/graid3/gvinum/gstripe configuration. Regular gpart with GPT has=20 > no performance penalty. Additionally, it is not limited to SSH but=20 > virtually everything with sockets: nc, ggate, smb. >=20 > > This could be related to: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209680#c2 =20 >=20 > It pretty much sounds like it, though I do not use ipfw, pf or any NAT=20 > stuff. I will try your first patch and let you know. >=20 > Do you want me to add my usecase to the issue? If the patch helps, that could be useful once a committer finds the time to look at the PR. Fabian --Sig_/22vuQlpGpXF63./exq12l87 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlgCEYkACgkQBYqIVf93VJ2onQCfeXtrcivSCDnFcysKRANjg8DK k3gAoIKMagZ3TK+DJdbvow21YmIqXqpe =fwh3 -----END PGP SIGNATURE----- --Sig_/22vuQlpGpXF63./exq12l87--