From owner-freebsd-hubs@FreeBSD.ORG Thu Jan 10 12:52:50 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4748E85C for ; Thu, 10 Jan 2013 12:52:50 +0000 (UTC) (envelope-from cejkar@fit.vutbr.cz) Received: from kazi.fit.vutbr.cz (kazi6.fit.vutbr.cz [IPv6:2001:67c:1220:808::93e5:80c]) by mx1.freebsd.org (Postfix) with ESMTP id B8087C3C for ; Thu, 10 Jan 2013 12:52:49 +0000 (UTC) Received: from kazi.fit.vutbr.cz (localhost [127.0.0.1]) by kazi.fit.vutbr.cz (8.14.6/8.14.6) with ESMTP id r0ACqjYs070252 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 10 Jan 2013 13:52:45 +0100 (CET) Received: (from cejkar@localhost) by kazi.fit.vutbr.cz (8.14.6/8.13.1/Submit) id r0ACqjBX070251 for freebsd-hubs@freebsd.org; Thu, 10 Jan 2013 13:52:45 +0100 (CET) (envelope-from cejkar@fit.vutbr.cz) X-Authentication-Warning: kazi.fit.vutbr.cz: cejkar set sender to cejkar@fit.vutbr.cz using -f Date: Thu, 10 Jan 2013 13:52:45 +0100 From: Cejka Rudolf To: freebsd-hubs@freebsd.org Subject: Planned ftp-master.eu.FreeBSD.org outages Message-ID: <20130110125245.GA69651@fit.vutbr.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.73 on 147.229.8.12 X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2013 12:52:50 -0000 Hello, I'm going to play with ftp-master.eu.FreeBSD.org upgrade from i386 to amd64 in-place, so please be patient with possible outages until the end of the week (until 2013/01/11 or at worst 2013/01/13). Thank you. -- Rudolf Cejka http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic From owner-freebsd-hubs@FreeBSD.ORG Fri Jan 11 15:33:47 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3644935A for ; Fri, 11 Jan 2013 15:33:47 +0000 (UTC) (envelope-from cejkar@fit.vutbr.cz) Received: from kazi.fit.vutbr.cz (kazi6.fit.vutbr.cz [IPv6:2001:67c:1220:808::93e5:80c]) by mx1.freebsd.org (Postfix) with ESMTP id C0BD6171 for ; Fri, 11 Jan 2013 15:33:46 +0000 (UTC) Received: from kazi.fit.vutbr.cz (localhost [127.0.0.1]) by kazi.fit.vutbr.cz (8.14.6/8.14.6) with ESMTP id r0BFXh37060043 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 11 Jan 2013 16:33:43 +0100 (CET) Received: (from cejkar@localhost) by kazi.fit.vutbr.cz (8.14.6/8.13.1/Submit) id r0BFXhRQ060038 for freebsd-hubs@freebsd.org; Fri, 11 Jan 2013 16:33:43 +0100 (CET) (envelope-from cejkar@fit.vutbr.cz) X-Authentication-Warning: kazi.fit.vutbr.cz: cejkar set sender to cejkar@fit.vutbr.cz using -f Date: Fri, 11 Jan 2013 16:33:43 +0100 From: Cejka Rudolf To: freebsd-hubs@freebsd.org Subject: Re: Planned ftp-master.eu.FreeBSD.org outages Message-ID: <20130111153343.GA51647@fit.vutbr.cz> References: <20130110125245.GA69651@fit.vutbr.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130110125245.GA69651@fit.vutbr.cz> User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.73 on 147.229.8.12 X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 15:33:47 -0000 Cejka Rudolf wrote (2013/01/10): > Hello, > I'm going to play with ftp-master.eu.FreeBSD.org upgrade from i386 > to amd64 in-place, so please be patient with possible outages until > the end of the week (until 2013/01/11 or at worst 2013/01/13). Done, upgrade has been successfully (I hope :o) done. Thank you for the patience. -- Rudolf Cejka http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 05:21:48 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 533E6949 for ; Sat, 12 Jan 2013 05:21:48 +0000 (UTC) (envelope-from john.marshall@riverwillow.com.au) Received: from mail4.riverwillow.net.au (mail4.riverwillow.net.au [202.125.45.59]) by mx1.freebsd.org (Postfix) with ESMTP id D7744400 for ; Sat, 12 Jan 2013 05:21:46 +0000 (UTC) Received: from [172.25.24.201] (riverw1.lnk.telstra.net [165.228.239.138]) (authenticated bits=0) by mail4.riverwillow.net.au (8.14.6/8.14.6) with ESMTP id r0C5LWpx074376 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT) for ; Sat, 12 Jan 2013 15:21:33 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=riverwillow.com.au; s=m4001; t=1357968094; bh=A3ZpdAZujE04pkHx42a14sH6syxWxdVcs0h2Ml5srKM=; h=Date:From:To:Subject; b=EknaS0aP8yOHyPk2bs8ZJB7TC0lPPf+pHgkQfLzlZbEENL+eagXCdZ8/HgjGrqxbm dovHYJiROBVnHD12OLjFdn8JEeDVemBHggj4XnaJ+g+Xe9fYKWAqkk+F+qS9yUbZ+N SzI2GQrd+uZWTjmWQciEYWnF/oqehvX1AnN7KKcY= Message-ID: <50F0F2CE.9050605@riverwillow.com.au> Date: Sat, 12 Jan 2013 16:21:18 +1100 From: John Marshall Organization: Riverwillow Pty Ltd User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-hubs@freebsd.org Subject: SLOW rsync from bit0.us-west X-Enigmail-Version: 1.4.6 OpenPGP: id=A29A84A2; url=http://pki.riverwillow.com.au/pgp/johnmarshall.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig1934F8F4CDBEB9CDE5AE8380" X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 05:21:48 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1934F8F4CDBEB9CDE5AE8380 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I have been updating the gnats collection on our mirror via rsync since the CVSup updates stopped but noticed this morning that the updates appeared to be stalled. I killed off jobs that appeared to be stuck downloading incremental file list (after about 20 minutes) but then thought to run rsync manually with -vvv. What I saw then was short bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with 15-20 seconds of nothing. tcpdump showed just a handful of packets every 15 seconds or so. The update eventually finished after more than 2 hours.= I ran the rsync again after about 10 minutes with only a single -v and included --stats in the command line. This time it took only 1 hour to update 9 files and produced the following output. ozsrv03# rsync -avz --stats bit0.us-west.freebsd.org::FreeBSD-bit/gnats/ /freebsd/current/gnats/gnats This is bit0.us-west.FreeBSD.org. receiving incremental file list bin/ bin/175214 docs/ docs/175224 gnats-adm/ gnats-adm/current gnats-adm/index gnats-adm/locks/ kern/ kern/151564 kern/152250 kern/153361 ports/175221 ports/175222 Number of files: 174424 Number of files transferred: 9 Total file size: 1738253321 bytes Total transferred file size: 33819340 bytes Literal data: 70300 bytes Matched data: 33749040 bytes File list size: 1967813 File list generation time: 0.075 seconds File list transfer time: 0.000 seconds Total bytes sent: 41303 Total bytes received: 1989084 sent 41303 bytes received 1989084 bytes 612.76 bytes/sec total size is 1738253321 speedup is 856.12 The network isn't any worse than usual. We have a 100Mbps connection and latency across the Pacific Ocean from Australia is normal (~170ms RTT). ozsrv03# ping -qc 3 bit0.us-west.FreeBSD.org. PING bit0.ysv.FreeBSD.org (8.8.178.111): 56 data bytes --- bit0.ysv.FreeBSD.org ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 168.805/168.922/169.023/0.090 ms Is it just me? --=20 John Marshall --------------enig1934F8F4CDBEB9CDE5AE8380 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlDw8toACgkQw/tAaKKahKI2vQCfflahqcu2k/Zw9/vRO/K699oT 4FYAn1ufXTkoIo1LQs/Gu9DVVatKNnpH =eMgk -----END PGP SIGNATURE----- --------------enig1934F8F4CDBEB9CDE5AE8380-- From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 05:30:23 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB7B8A3A for ; Sat, 12 Jan 2013 05:30:23 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from wilberforce.math.missouri.edu (wilberforce.math.missouri.edu [128.206.184.213]) by mx1.freebsd.org (Postfix) with ESMTP id 7160563D for ; Sat, 12 Jan 2013 05:30:22 +0000 (UTC) Received: from [127.0.0.1] (wilberforce.math.missouri.edu [128.206.184.213]) by wilberforce.math.missouri.edu (8.14.6/8.14.6) with ESMTP id r0C5ULFr072402 for ; Fri, 11 Jan 2013 23:30:21 -0600 (CST) (envelope-from stephen@missouri.edu) Message-ID: <50F0F4ED.4020203@missouri.edu> Date: Fri, 11 Jan 2013 23:30:21 -0600 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-hubs@freebsd.org Subject: Re: SLOW rsync from bit0.us-west References: <50F0F2CE.9050605@riverwillow.com.au> In-Reply-To: <50F0F2CE.9050605@riverwillow.com.au> X-Enigmail-Version: 1.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 05:30:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/11/2013 11:21 PM, John Marshall wrote: > I have been updating the gnats collection on our mirror via rsync > since the CVSup updates stopped but noticed this morning that the > updates appeared to be stalled. I killed off jobs that appeared to > be stuck downloading incremental file list (after about 20 minutes) > but then thought to run rsync manually with -vvv. What I saw then > was short bursts of messages like 'recv_file_name(bin/nnnn)' > interspersed with 15-20 seconds of nothing. tcpdump showed just a > handful of packets every 15 seconds or so. The update eventually > finished after more than 2 hours. > > I ran the rsync again after about 10 minutes with only a single -v > and included --stats in the command line. This time it took only 1 > hour to update 9 files and produced the following output. > > ozsrv03# rsync -avz --stats > bit0.us-west.freebsd.org::FreeBSD-bit/gnats/ > /freebsd/current/gnats/gnats This is bit0.us-west.FreeBSD.org. > > receiving incremental file list bin/ bin/175214 docs/ docs/175224 > gnats-adm/ gnats-adm/current gnats-adm/index gnats-adm/locks/ > kern/ kern/151564 kern/152250 kern/153361 ports/175221 > ports/175222 > > Number of files: 174424 Number of files transferred: 9 Total file > size: 1738253321 bytes Total transferred file size: 33819340 bytes > Literal data: 70300 bytes Matched data: 33749040 bytes File list > size: 1967813 File list generation time: 0.075 seconds File list > transfer time: 0.000 seconds Total bytes sent: 41303 Total bytes > received: 1989084 > > sent 41303 bytes received 1989084 bytes 612.76 bytes/sec total > size is 1738253321 speedup is 856.12 > > The network isn't any worse than usual. We have a 100Mbps > connection and latency across the Pacific Ocean from Australia is > normal (~170ms RTT). > > ozsrv03# ping -qc 3 bit0.us-west.FreeBSD.org. PING > bit0.ysv.FreeBSD.org (8.8.178.111): 56 data bytes > > --- bit0.ysv.FreeBSD.org ping statistics --- 3 packets transmitted, > 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev > = 168.805/168.922/169.023/0.090 ms > > Is it just me? > I was doing csup from cvsup-master this morning, and it was similarly extremely slow. I killed it, and then tried again several hours later, and it went at normal speed. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJQ8PToAAoJEC3xK9GaktgHotIH/2Vwr4vq5BNJ8fy1dmgAzGvi 2vSzvzYz8A2sl0yErQudEkdO9aBxps2D+I+fpI3Ob6mSlij839gq3cmGndtefNYA h8psA8TrJnh3W1VlaBEIC9NZkJooKSWq3A4NboMuYH2ReNcQeXZUZn7iL4WG8STo j+SyRHYo7SJ8DEeXtsSeTiOP6ltPcNU+fbIgpqFtGd4AA3Yd6fCWbdEpnp/nPf8a lRY+3eWPO3X0ujomRW8eO1dSlffz62eIC+NVzCPeqHkodFRHQ+Kp2AFFTvUBsUSj iaQ66MzA911Rl/bW0ZaAXkf+br2VjGVPR84urAr5hMCCNxKW79nHK+UEMlQDbTk= =PlQC -----END PGP SIGNATURE----- From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 10:55:46 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 07561D7B for ; Sat, 12 Jan 2013 10:55:46 +0000 (UTC) (envelope-from john.marshall@riverwillow.com.au) Received: from mail4.riverwillow.net.au (mail4.riverwillow.net.au [202.125.45.59]) by mx1.freebsd.org (Postfix) with ESMTP id 82C78FE9 for ; Sat, 12 Jan 2013 10:55:44 +0000 (UTC) Received: from [172.25.24.201] (riverw1.lnk.telstra.net [165.228.239.138]) (authenticated bits=0) by mail4.riverwillow.net.au (8.14.6/8.14.6) with ESMTP id r0CAtbEn050510 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT) for ; Sat, 12 Jan 2013 20:55:38 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=riverwillow.com.au; s=m4001; t=1357988138; bh=iYScHPfTWlZv7BvwCqYdS4W7xak3M3k2821B24+1VdE=; h=Date:From:To:Subject; b=se2kNZNVAJwJSj60E/AlpQsFbFim9N8G5xvXtrNNHU9PXQAsmZupMXARyZzgQSOPe bAiDwn5ThuyJlHPX6rge/VFHm44pEi/OqH+XRJWP7Xd3CyzfQEnRa1W7KhAVJk8GSo X4dBdzgCK43pmVr4NDIN2TT+kkTZ+TOPJgopmhN0= Message-ID: <50F1411C.7070800@riverwillow.com.au> Date: Sat, 12 Jan 2013 21:55:24 +1100 From: John Marshall Organization: Riverwillow Pty Ltd User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-hubs@freebsd.org Subject: cvsup-master choking? X-Enigmail-Version: 1.4.6 OpenPGP: id=A29A84A2; url=http://pki.riverwillow.com.au/pgp/johnmarshall.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig404BA39A91D6A287FA9EA196" X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 10:55:46 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig404BA39A91D6A287FA9EA196 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable cvsup.au has not been able to update from cvsup-master for the past 12 hours. Has there been an accidental config change or is there something bad happening over on that side of the planet? Append to archive/svn-src-projects Updating collection distrib/self Finished successfully CVSup update ends at 2013-01-12 07:19:10 CVSup update begins at 2013-01-12 08:18:00 Updating from cvsup-master.freebsd.org Connected to cvsup-master.freebsd.org Rejected by server: Access limit exceeded; try again later CVSup update ends at 2013-01-12 08:18:01 =2E =2E =2E CVSup update begins at 2013-01-12 20:18:00 Updating from cvsup-master.freebsd.org Connected to cvsup-master.freebsd.org Rejected by server: Access limit exceeded; try again later CVSup update ends at 2013-01-12 20:18:01 --=20 John Marshall --------------enig404BA39A91D6A287FA9EA196 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlDxQSgACgkQw/tAaKKahKJmegCguanj3Bkb37lpWbT3G7c7kq23 k9kAoIr2Dj0vThE3wcdFghiKKaih3A5x =mRXm -----END PGP SIGNATURE----- --------------enig404BA39A91D6A287FA9EA196-- From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 11:54:32 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1AC1D886 for ; Sat, 12 Jan 2013 11:54:32 +0000 (UTC) (envelope-from john.marshall@riverwillow.com.au) Received: from mail4.riverwillow.net.au (mail4.riverwillow.net.au [202.125.45.59]) by mx1.freebsd.org (Postfix) with ESMTP id B778C2A2 for ; Sat, 12 Jan 2013 11:54:30 +0000 (UTC) Received: from [172.25.24.201] (riverw1.lnk.telstra.net [165.228.239.138]) (authenticated bits=0) by mail4.riverwillow.net.au (8.14.6/8.14.6) with ESMTP id r0CBsKLp064197 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Sat, 12 Jan 2013 21:54:21 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=riverwillow.com.au; s=m4001; t=1357991661; bh=LrBiHRp6l0hc93ubg7VPBkJDmvVukyMIwHxk9QfoWm0=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=TRo2k2rk3Yqv3APy3yH/iH9IEari7KA/1XaWhyC369EqssLQM931KuciufRmylcy/ 0Dx0J6qr2nzgii58UAq/FTEwO70t9ybe8FbgAbw/xChWQuHiofKvqcchHXxIGOWLXy cqWOBmibyTls6xldSTHUKLainvq8masvRWUDcdUg= Message-ID: <50F14EE1.2070209@riverwillow.com.au> Date: Sat, 12 Jan 2013 22:54:09 +1100 From: John Marshall Organization: Riverwillow Pty Ltd User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Stephen Montgomery-Smith Subject: Re: SLOW rsync from bit0.us-west References: <50F0F2CE.9050605@riverwillow.com.au> <50F0F4ED.4020203@missouri.edu> In-Reply-To: <50F0F4ED.4020203@missouri.edu> X-Enigmail-Version: 1.4.6 OpenPGP: id=A29A84A2; url=http://pki.riverwillow.com.au/pgp/johnmarshall.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA632DDC0BF67A6DF4877D27E" Cc: freebsd-hubs@freebsd.org X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 11:54:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA632DDC0BF67A6DF4877D27E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable On 12/01/2013 16:30, Stephen Montgomery-Smith wrote: > On 01/11/2013 11:21 PM, John Marshall wrote: >> I have been updating the gnats collection on our mirror via rsync >> since the CVSup updates stopped but noticed this morning that the >> updates appeared to be stalled. I killed off jobs that appeared to >> be stuck downloading incremental file list (after about 20 minutes) >> but then thought to run rsync manually with -vvv. What I saw then >> was short bursts of messages like 'recv_file_name(bin/nnnn)' >> interspersed with 15-20 seconds of nothing. tcpdump showed just a >> handful of packets every 15 seconds or so. The update eventually >> finished after more than 2 hours. >> Is it just me? >=20 > I was doing csup from cvsup-master this morning, and it was similarly > extremely slow. I killed it, and then tried again several hours > later, and it went at normal speed. Thanks Stephen. Since it appears that both hosts are located in the same place, I took heart from your remarks and tried again, wondering if, perhaps, the gremlins had all gone to bed. No joy however: 50 minutes to transfer 30 files. ozsrv03# rsync -avz --stats bit0.us-west.freebsd.org::FreeBSD-bit/gnats/ /freebsd/current/gnats/gnats This is bit0.us-west.FreeBSD.org. receiving incremental file list =2E/ bin/ bin/175139 bin/175181 docs/ docs/175226 gnats-adm/ gnats-adm/current gnats-adm/index gnats-adm/locks/ kern/ kern/174749 kern/175179 kern/175220 kern/175227 misc/ ports/ ports/155991 ports/164600 ports/167374 ports/168010 ports/168464 ports/172372 ports/173620 ports/174318 ports/174385 ports/174955 ports/175167 ports/175168 ports/175187 ports/175203 ports/175206 ports/175210 ports/175216 ports/175217 ports/175228 ports/175229 ports/175230 Number of files: 174431 Number of files transferred: 30 Total file size: 1738385115 bytes Total transferred file size: 34275776 bytes Literal data: 171451 bytes Matched data: 34107102 bytes File list size: 1967889 File list generation time: 0.255 seconds File list transfer time: 0.000 seconds Total bytes sent: 45313 Total bytes received: 2025814 sent 45313 bytes received 2025814 bytes 677.17 bytes/sec total size is 1738385115 speedup is 839.34 --=20 John Marshall --------------enigA632DDC0BF67A6DF4877D27E Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlDxTusACgkQw/tAaKKahKLuFQCaAqHGDCUTe5BDbBSFMHrfFQCR 6dEAn375gP8evcAuCqh71KOD/EWEVG3Y =OnWS -----END PGP SIGNATURE----- --------------enigA632DDC0BF67A6DF4877D27E-- From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 15:37:32 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5F011459 for ; Sat, 12 Jan 2013 15:37:32 +0000 (UTC) (envelope-from simon@qxnitro.org) Received: from mail-la0-f41.google.com (mail-la0-f41.google.com [209.85.215.41]) by mx1.freebsd.org (Postfix) with ESMTP id D4DB5D32 for ; Sat, 12 Jan 2013 15:37:31 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id em20so2675421lab.0 for ; Sat, 12 Jan 2013 07:37:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qxnitro.org; s=google; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4g70KftbVGuz4en9mnziRjgY7Vwy6SUxNM7zGJPVAug=; b=pfSwNl/0NCj2+fSPPsfiW+4lACi2m9g1PlcEHkljJZTFmfR42xhTDSHDrdfwo0ezA6 O6s64vcOjt0quoPJnA5bt9V+khwFZe2hxwCKXkmhwqFxQZco/N69EUGS3mATqaRHu5ak fpenmHD6F6IAIxf+kkm8S13+Ozrr4MahLqS0E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=4g70KftbVGuz4en9mnziRjgY7Vwy6SUxNM7zGJPVAug=; b=MgRrWIawVE2iKZIpRkpPae+cOWD6rFozZDcyh3uiGevdimuoQecZMmEYDDgCWHwuJG xGweN0TicqYRp6+GiJ5Nzc5Xoxx5vs7DEfgoEywVwvAv3Y+7iNPB9bYhlT6Ix44LBL7Y ev7LDhuDKn8K4BDddNbZBCZdA5VoeJnISDl5MYhcerRMUlJLHLUVz8eqzhO0fwn3bCqf 0daejYHq1EXqxq5g5TQO/isisFd3D/OoYVcOJL7Nl0B/VsRbLNRyVsvivk7XHE0DUzCx PUukHApLFxfRxsG3Wv2WPkCParJTRc3kGl+KmZNYevzZyXuewV+9+R6vAFfmou64xIKq 8mNg== MIME-Version: 1.0 Received: by 10.112.29.229 with SMTP id n5mr33523279lbh.130.1358005050377; Sat, 12 Jan 2013 07:37:30 -0800 (PST) Sender: simon@qxnitro.org Received: by 10.112.19.41 with HTTP; Sat, 12 Jan 2013 07:37:30 -0800 (PST) X-Originating-IP: [89.100.233.26] In-Reply-To: <50F0F2CE.9050605@riverwillow.com.au> References: <50F0F2CE.9050605@riverwillow.com.au> Date: Sat, 12 Jan 2013 15:37:30 +0000 X-Google-Sender-Auth: qLeR8YiZdHqgvZT0HCnApieVJSU Message-ID: Subject: Re: SLOW rsync from bit0.us-west From: "Simon L. B. Nielsen" To: John Marshall Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQn1xIti9T/DI6cb7CP1pSI8owLMh5+WOZbPDUnp12I6VeL0Oe6Ssv/YVMIcfRNJ6+lf6sBE Cc: freebsd-hubs@freebsd.org X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 15:37:32 -0000 On 12 January 2013 05:21, John Marshall wrote: > I have been updating the gnats collection on our mirror via rsync since > the CVSup updates stopped but noticed this morning that the updates > appeared to be stalled. I killed off jobs that appeared to be stuck > downloading incremental file list (after about 20 minutes) but then > thought to run rsync manually with -vvv. What I saw then was short > bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with > 15-20 seconds of nothing. tcpdump showed just a handful of packets every > 15 seconds or so. The update eventually finished after more than 2 hours. The server which runs bit0.us-west (which is also svn0.us-west), is totally bogged down. CPU: 3.1% user, 0.0% nice, 96.0% system, 0.0% interrupt, 1.0% idle As it has just been updated to the latest stable/9 I'm a bit suspicious of that... Thanks for letting us know. I will start poking people. -- Simon L. B. Nielsen From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 19:58:21 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 020BCFFB for ; Sat, 12 Jan 2013 19:58:20 +0000 (UTC) (envelope-from simon@qxnitro.org) Received: from mail-la0-f49.google.com (mail-la0-f49.google.com [209.85.215.49]) by mx1.freebsd.org (Postfix) with ESMTP id 80D5678F for ; Sat, 12 Jan 2013 19:58:20 +0000 (UTC) Received: by mail-la0-f49.google.com with SMTP id fk20so2701402lab.22 for ; Sat, 12 Jan 2013 11:58:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qxnitro.org; s=google; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=yAPNMEquunOi5hs9yG2J56tG3mxsJBb9KmbbDUYB2R4=; b=lzt/moyKMDmN/ycgt0T1HDh2ZvHGGRF8beWeFnZeAVg6ZXmjN5JunoF/c1TOg9pvlF ViXk4xlbGPLY1MX3zaEIGVGAtFu8YeXlPgt9OiTxliLZWjZdvLXWmMulFMkqZz0CQ556 WnkcF/5KTet6ENQhCfp6/GZZLUmHdod47vBBA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=yAPNMEquunOi5hs9yG2J56tG3mxsJBb9KmbbDUYB2R4=; b=JS3K4W+y4Mxru88fhDKhphzq/FmV/QHlYOBM6cay1cZNlWrXxq0c+OzW0pVA4wwG4t 6xFxQweRYjlrwAO5WKhc2vPELOF+dy2NW0k0V8A75R/v8c+j3W+YiiyDGdAS0US3FoBl ldoXpR1Bt8khAovlLWX3nLMFLbosXaZURclDc18z9MY7yuWNuwhAZ85fuJ9JdHjd9yJt Qb9/bZEFeWUx7uScz+wxstXIezjyHjXNKHGnBdKwOmsAtQO5Svteugbt5LcW+2AX3lM9 gaDutcQJ5cCyaWl5PohIAiWQWy1oLLu7Pnn34EChToZ5BkS5e/IjQnBQDADo2q79thJi PhGw== MIME-Version: 1.0 Received: by 10.112.29.229 with SMTP id n5mr33767527lbh.130.1358020698537; Sat, 12 Jan 2013 11:58:18 -0800 (PST) Sender: simon@qxnitro.org Received: by 10.112.19.41 with HTTP; Sat, 12 Jan 2013 11:58:18 -0800 (PST) X-Originating-IP: [89.100.233.26] In-Reply-To: References: <50F0F2CE.9050605@riverwillow.com.au> Date: Sat, 12 Jan 2013 19:58:18 +0000 X-Google-Sender-Auth: 2ztWEUoEK3N-Y9HgmGvk6SE4YUI Message-ID: Subject: Re: SLOW rsync from bit0.us-west From: "Simon L. B. Nielsen" To: John Marshall Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQm33KNTwkG0JojbmfAks5DyylBFpY7DcoiWByK/VHgKTbDq98PwXR55asext6buckUg4zF/ Cc: freebsd-hubs@freebsd.org X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 19:58:21 -0000 On 12 January 2013 15:37, Simon L. B. Nielsen wrote: > On 12 January 2013 05:21, John Marshall > wrote: >> I have been updating the gnats collection on our mirror via rsync since >> the CVSup updates stopped but noticed this morning that the updates >> appeared to be stalled. I killed off jobs that appeared to be stuck >> downloading incremental file list (after about 20 minutes) but then >> thought to run rsync manually with -vvv. What I saw then was short >> bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with >> 15-20 seconds of nothing. tcpdump showed just a handful of packets every >> 15 seconds or so. The update eventually finished after more than 2 hours. > > The server which runs bit0.us-west (which is also svn0.us-west), is > totally bogged down. > > CPU: 3.1% user, 0.0% nice, 96.0% system, 0.0% interrupt, 1.0% idle > > As it has just been updated to the latest stable/9 I'm a bit > suspicious of that... > > Thanks for letting us know. I will start poking people. Please have another look and see if things aren't better now. It looks like we are stressing nullfs more than people had before - at least with way more vnode / files so the server was spending all of its time doing vnode lookups. -- Simon L. B. Nielsen From owner-freebsd-hubs@FreeBSD.ORG Sat Jan 12 21:13:50 2013 Return-Path: Delivered-To: freebsd-hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 81678E11; Sat, 12 Jan 2013 21:13:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D4C579AF; Sat, 12 Jan 2013 21:13:49 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0CLDdaw030998; Sat, 12 Jan 2013 23:13:39 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0CLDdaw030998 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0CLDd7K030997; Sat, 12 Jan 2013 23:13:39 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 12 Jan 2013 23:13:39 +0200 From: Konstantin Belousov To: "Simon L. B. Nielsen" Subject: Re: SLOW rsync from bit0.us-west Message-ID: <20130112211339.GM2561@kib.kiev.ua> References: <50F0F2CE.9050605@riverwillow.com.au> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Fm6XlJPzS/ViAqAU" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-hubs@freebsd.org, John Marshall X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 21:13:50 -0000 --Fm6XlJPzS/ViAqAU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 07:58:18PM +0000, Simon L. B. Nielsen wrote: > On 12 January 2013 15:37, Simon L. B. Nielsen wrote: > > On 12 January 2013 05:21, John Marshall > > wrote: > >> I have been updating the gnats collection on our mirror via rsync since > >> the CVSup updates stopped but noticed this morning that the updates > >> appeared to be stalled. I killed off jobs that appeared to be stuck > >> downloading incremental file list (after about 20 minutes) but then > >> thought to run rsync manually with -vvv. What I saw then was short > >> bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with > >> 15-20 seconds of nothing. tcpdump showed just a handful of packets eve= ry > >> 15 seconds or so. The update eventually finished after more than 2 hou= rs. > > > > The server which runs bit0.us-west (which is also svn0.us-west), is > > totally bogged down. > > > > CPU: 3.1% user, 0.0% nice, 96.0% system, 0.0% interrupt, 1.0% idle > > > > As it has just been updated to the latest stable/9 I'm a bit > > suspicious of that... > > > > Thanks for letting us know. I will start poking people. >=20 > Please have another look and see if things aren't better now. >=20 > It looks like we are stressing nullfs more than people had before - at > least with way more vnode / files so the server was spending all of > its time doing vnode lookups. What are your sysctl kerne.numvnodes and kern.maxvnodes ? How did you determined that lookups were the cause ? The deal with nullfs is that I MFCed a set of changes recently which allow to keep nullfs vnodes cached. As a consequence, the nullfs vnodes could be (usefully) locked shared not, which provides up to 100% increase in the speed of the metadata-intensive operations on nullfs, as measured by Peter Holm. The drawback of the change is that nullfs vnodes are cached, which effectively doubles the amount of vnodes the system operates on. I added the nocache nullfs mount option to return to the pre-cached behaviour, but this is not propagated to the stable/9 yet. Supposedly, you could use the option if increasing kern.maxvnodes is not possible. --Fm6XlJPzS/ViAqAU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8dIDAAoJEJDCuSvBvK1B+4kP/iNFjo8uREzRh2WyDcPzcc1c gWOHdqoZ8w/x/8DTSyFt0V3e1YNNASS7VnSA0rEhei20wEt49TL80WtgGMaeh4TD tpYJGcbMUdH0Zr/RommXUysN851AAQoj6ZKgEfnvvY7+sm8TUqO1aJa48dGf8Lgm NHXdjppY124CvzM6rflGwRZncJ7xttNsfOw3v/uyhNkZUwIcsVCHFkoGYCAUQL6S VZq51Ad6CrMotzXmHcOV2gUsauk4vzouOHX6SByiJI6S3nfMqRfajAkzSljfKJsB pFHgANO3mAdXiIaiT5A5rpbYqmpA6V7FSrlrgzdO46lIimY2Ou+NkPWs47P54Ea8 kqXPSir+BhVX2EfbnRMx4j5bTAH3uFv2Hn/o3Datt5CdyFvR0giGqgMOZZxRn43r P2obaQ9r5h+zTsUXzNIVTu3JYQaQR1skDib54rFsxPvn/BNOZyIM0Rmmv/w/pont S+i/GC+aoHvdq1iNjDEm2A4nyLKicKxBai1s6qMYc/n2KS46Scd1x+/DG+2w4Iw9 XnlAPfZJ2iYhdMBrxs8qh1ivodiMzXUrDJC8UQYF1KGq0h+xoMcAXbbUIH5FuMnh /9Ut4r1FMUWZpfdpvpznJigZA4TEPkwSn123RaqZIX0fp7lM54W7VfRWlBaW6mZS 5pll70eiLJk/i3KFfFXT =fwrS -----END PGP SIGNATURE----- --Fm6XlJPzS/ViAqAU--