From owner-freebsd-questions@freebsd.org Wed Apr 13 01:06:07 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 662FDB0CC47 for ; Wed, 13 Apr 2016 01:06:07 +0000 (UTC) (envelope-from benfell@parts-unknown.org) Received: from mail.parts-unknown.org (n4rky-1-pt.tunnel.tserv29.fmt1.ipv6.he.net [IPv6:2001:470:66:119::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E6BE1251 for ; Wed, 13 Apr 2016 01:06:07 +0000 (UTC) (envelope-from benfell@parts-unknown.org) Received: from [192.168.1.75] (unknown [50.250.218.173]) by mail.parts-unknown.org (Postfix) with ESMTPSA id 53A965979234 for ; Tue, 12 Apr 2016 18:06:06 -0700 (PDT) Subject: SOLVED, BUT UGLY, was Re: rsync update mixed environment To: freebsd-questions@freebsd.org References: <570C9DE3.6030601@parts-unknown.org> <570CA890.7090706@FreeBSD.org> From: David Benfell Message-ID: <570D9B74.80005@parts-unknown.org> Date: Tue, 12 Apr 2016 18:05:56 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.1 MIME-Version: 1.0 In-Reply-To: <570CA890.7090706@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="qJ0pb0IHfI3EAWGrCW5w3TU1Mf35EqMeF" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 01:06:07 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qJ0pb0IHfI3EAWGrCW5w3TU1Mf35EqMeF Content-Type: multipart/mixed; boundary="GT6aLDuu82TxIA2EHnGoWEI1OqWIQx8ev" From: David Benfell To: freebsd-questions@freebsd.org Message-ID: <570D9B74.80005@parts-unknown.org> Subject: SOLVED, BUT UGLY, was Re: rsync update mixed environment References: <570C9DE3.6030601@parts-unknown.org> <570CA890.7090706@FreeBSD.org> In-Reply-To: <570CA890.7090706@FreeBSD.org> --GT6aLDuu82TxIA2EHnGoWEI1OqWIQx8ev Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Thanks Matthew for your response. My specific responses to your questions and suggestions are below. For everybody else, you probably don't want to wade through all that; I'm top-posting the TL;DR version he= re. Basically, I couldn't get rsync to update from within a script. No idea why but it looks like some way, somehow the script was the problem. Realizing it would be completely unreasonable to expect people on this list to debug my fairly gnarly script, I gave up and am using a new script to find the local and (with ssh) remote modification times. In the script that wasn't working, I'm using this new script to obtain those times, compare them, and scp accordingly. This works like a charm. Yes, I'm wondering why I can go to all that trouble, get it working, and utterly fail to get rsync working. My script to obtain the modification time is at https://parts-unknown.org/stikked/view/2e00aa88 but be sure to read the comments because using stat turns out to be really, really, really ugly and weird, which means, among other things, you have to use it with /dev/null on STDIN (this seems to be true on both Linux and FreeBSD). Again, no idea why. I'm in debt to a couple stackoverflow posts here. On 04/12/2016 12:49 AM, Matthew Seaman wrote: > On 12/04/2016 08:04, David Benfell wrote: >> I'm used to rsync from Linux, where one can do rsync -auvz to update t= he >> files from one directory to another between networked systems. >> >> The z option gave me trouble in FreeBSD but I guess you can do zz >> instead. I guess I missed a memo on whatever is going on with this. > The FreeBSD port is using the zlib from the base system rather than wha= t > is bundled with rsync. This does have an effect on functionality, > particularly on interoperability with other versions of rsync. If > you're using rsync in a diverse environment, it might be an idea to > toggle that option setting in the rsync port and reinstall it. I found and disabled the option. It should now be using the bundled zlib. However, rsync from Linux to FreeBSD still fails. But testing further, the failure only occurs within a script (there are a number of files and directories I'm trying to keep in sync) in which I first try to sync from the FreeBSD system to the Linux system. This must be making a difference, but I'm not understanding why, have no idea what questions to ask about why it might be making a difference, and reversing the order of the operations makes no difference. More generally, it's even failing to create files on the (FreeBSD) destination that do not previously exist. An obvious but both rude and inconvenient solution to this might be to elide the u option or to go to something simpler like scp (from ssh) but we're talking lots and lots of gigabytes, sometimes to and from a notebook that might be operating remotely from a coffee shop. I found unison, but it's now fussy about versions so a mixed environment won't work at all. >> Either way, updates don't seem to happen reliably when I do this from = a >> Linux system to a FreeBSD system. The failure is silent; it just doesn= 't >> seem to recognize that a file in the source is newer than the version = in >> the destination. And then I wonder why an updated file didn't propagat= e. > Curious. Is there any problem going in the other direction? A test going the other direction succeeded. > FreeBSD to > Linux? What are the versions of rsync on either side? At first blush, the versions are the same. But there are some differences in capabilities and I just don't know enough to even guess whether they're significant: On Linux: rsync --version rsync version 3.1.2 protocol version 31 Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc On FreeBSD (both with and without bundled zlib): rsync --version rsync version 3.1.2 protocol version 31 Capabilities: 64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags > Does using > compression or not affect the reliability? (ie. try your rsync without= > any compression at all.) Yes, I have tried it without compression. That's actually the condition I initially encountered the problem under because my initial reaction to the error message about the z option was simply to elide it--these transfers are mostly occurring on a local network so speed is a bit less of a concern. > I'm thinking this could be a problem due to > the change in the way the compression option works with different > rsync(1) versions. Quoting the man page: > > This matching-data com- > pression comes at a cost of CPU, though, and can be disabled by > repeating the -z option, but only if both sides are at least > version 3.1.1. --=20 David Benfell, Ph.D. benfell@parts-unknown.org --GT6aLDuu82TxIA2EHnGoWEI1OqWIQx8ev-- --qJ0pb0IHfI3EAWGrCW5w3TU1Mf35EqMeF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJXDZt9AAoJEOoml8Ul4D7WwhYQAKEBsc3A24mxJhojzJFUAsBA R2GJ9b7BQbo5APU1bKBMcK5u2zPs8WzrXo2kmziwNvILCvfEOIX5OB6or52wmDtx wsKA1KnriAxK/awbE4kjou2Sd7TUzqdYeqJbvk0anxGPBDz9o9nsk8OmH6+e+gTY /Ic4aOxnBfefFeWYZJhUX/I8KXAjucfKImNjqAPHqThnM0aysG2wDmtrL2/3W4Dq ZbOFaiuLarEACkxq+NZnARXUXQjhGFZNK/e5G/G+8+ccBDUYlBHMYYtLWRi3d0/e fSvL6RmGuALkVwAzOW+VbWYFa88FqJ/fWv+KOLPqZB1pZugliHK+NDsgd0YG0Bo8 fEbOcUfX1/qaTE3nXAl0+NPLm7+lwbtE1rrXRgTihgvY38bGaj0sY5F8Ht+rMbne vYSW5ZT+hxs8BuCcfK3lwPdIU+mGHUI1k3PM2/z3Kg924wyclv2b/I0SFiSn/FXh XcjYMk+yQ224dxcivjTlXrA+FKxYp86SNp91JduJHjfnGzVQYtijjg3r/B3Ejt11 i6jE2HoP0PDBiQUxIX1HYUWBeHjkTQEz246Tipg4HS9tyz97445VqdiPWC5fJiJB fsUNqvq/rFYc2IvY4DZAC2rW9ROFxFN24XCVZD1nyNLA+S4IzPYmQsjangpAEkG/ Rd3Izulc92JwAvkR7gE5 =vT1Y -----END PGP SIGNATURE----- --qJ0pb0IHfI3EAWGrCW5w3TU1Mf35EqMeF--