From owner-freebsd-fs@FreeBSD.ORG Fri Apr 5 13:02:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1C9BE594 for ; Fri, 5 Apr 2013 13:02:14 +0000 (UTC) (envelope-from joar.jegleim@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) by mx1.freebsd.org (Postfix) with ESMTP id AB0BD5FB for ; Fri, 5 Apr 2013 13:02:13 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id c10so1763389wiw.2 for ; Fri, 05 Apr 2013 06:02:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=eqo+S71j2X/nCcHkciTRMzg+yLNKRi6pml4G7XCb1OQ=; b=J/QbZ0GRk6nhMweo7iCV3VIUBm51xYbIh/dP9n8CvOzIEOYHHfR0NLIw4wV5aytuAR sn3beT0X0G0Hu7YeWzgZsxAvSJX+SgH00MNqKGpgXFvihnzoqErkm1nO1/xc7BHXVy+J 0+gJL1sea+WH1vmVIatWgBPF4f9VtCFcirldL2O6uHNskFvR/iBEkgcAmCdNvaYSafIw XkUPlDXFl65zdRpc8+TKncY00OAB0/MA4AI0Ow7W2c5HvvaMdpwomUIUvDIJyjPI3Qp+ GT6abY4no2o/vVgEgwNKhYMnP4eycVY0l+V2dqADirdZUxjoBqISISp9ZgEUrlp+swCq +pTQ== MIME-Version: 1.0 X-Received: by 10.180.92.229 with SMTP id cp5mr4079652wib.20.1365166932680; Fri, 05 Apr 2013 06:02:12 -0700 (PDT) Received: by 10.216.34.9 with HTTP; Fri, 5 Apr 2013 06:02:12 -0700 (PDT) In-Reply-To: <515EB744.5000607@brockmann-consult.de> References: <8B0FFF01-B8CC-41C0-B0A2-58046EA4E998@my.gd> <515EB744.5000607@brockmann-consult.de> Date: Fri, 5 Apr 2013 15:02:12 +0200 Message-ID: Subject: Re: Regarding regular zfs From: Joar Jegleim To: Peter Maloney Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 13:02:14 -0000 You make some interesting points . I don't _think_ the script 'causes more than 1 zfs write at a time, and I'm sure 'nothing else' is doing that neither . But I'm gonna check that out because it does sound like a logical explanation. I'm wondering if the rsync from the receiving server (that is: the backup server is doing rsync from the zfs receive server) could 'cause the same problem, it's only reading though ... -- ---------------------- Joar Jegleim Homepage: http://cosmicb.no Linkedin: http://no.linkedin.com/in/joarjegleim fb: http://www.facebook.com/joar.jegleim AKA: CosmicB @Freenode ---------------------- On 5 April 2013 13:36, Peter Maloney wrote: > On 2013-04-05 13:07, Damien Fleuriot wrote: > > -I've implemented mbuffer for the zfs send / receive operations. With > mbuffer the sync went a lot faster, but still got the same symptoms > when the zfs receive is done, the hang / unresponsiveness returns for > 5-20 seconds > -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to > V28), same symptoms > -I've upgraded to 9.1-RELEASE, still same symptoms > > > So my question(s) to the list would be: > In my setup have I taken the use case for zfs send / receive too far > (?) as in, it's not meant for this kind of syncing and this often, so > there's actually nothing 'wrong'. > > > I do the same thing on an 8.3-STABLE system, with replication every 20 > minutes (compared to your 15 minutes), and it has worked flawlessly for > over a year. Before that point, it was hanging often, until I realized that > all hangs were from when there was more than 1 writing "zfs" command > running at the same time (snapshot, send, destroy, rename, etc. but not > list, get, etc.). So now *all my scripts have a common lock between them*(just a pid file like in /var/run; cured the hangs), and I don't run manual > zfs commands without stopping my cronjobs. If the hang was caused by a > destroy or smething during a send, I think it would usually unhang when the > send is done, do the destroy or whatever else was blocking, then be unhung > completely, smoothly working. In other cases, I think it would be > deadlocked. > > > NAME USED REFER USEDCHILD USEDDS USEDSNAP > AVAIL MOUNTPOINT > tank 38.5T 487G 37.4T 487G 635G > 9.54T /tank > tank/backup 7.55T 1.01T 5.08T 1.01T 1.46T > 9.54T /tank/backup > ... > > Sends are still quick with 38 T to send. The last replication run started > 2013-04-05 13:20:00 +0200 and finished 2013-04-05 13:22:18 +0200. I have > 234 snapshots at the moment (one per 20 min today + one daily for a few > months). > >