From owner-freebsd-stable@FreeBSD.ORG Tue Jul 5 17:03:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F7FF106566B for ; Tue, 5 Jul 2011 17:03:24 +0000 (UTC) (envelope-from cscotts@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id BD8C68FC08 for ; Tue, 5 Jul 2011 17:03:23 +0000 (UTC) Received: by qwc9 with SMTP id 9so4051732qwc.13 for ; Tue, 05 Jul 2011 10:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=v11ZcKbcNsR/TXw2y/6NyaHbZ079P/xM1klt4cXVxTo=; b=h8b2yuN7vsNzbzZ/mAQ2ZJiQvrCNVje4vmEGQBcVgVPQF11q62qdSmu3qKgOQBfVBI ctPpet9cljietUyjp3YUbRKM0hv8VyZv5HjNCLossIH6p0nfSZOo8jbgykCjet5Yo/sL uqYaVjodne84R/Ucaz4msulcA5uMCLrjnJzCo= Received: by 10.224.17.144 with SMTP id s16mr5393523qaa.233.1309885402891; Tue, 05 Jul 2011 10:03:22 -0700 (PDT) Received: from sahibkuran.cap-press.com (rrcs-24-172-62-94.midsouth.biz.rr.com [24.172.62.94]) by mx.google.com with ESMTPS id r33sm5679106qcs.6.2011.07.05.10.03.21 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 05 Jul 2011 10:03:21 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Scott Sipe In-Reply-To: <20110705170023.87183xz6ehxbeft3@webmail.in-berlin.de> Date: Tue, 5 Jul 2011 13:03:20 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <5BA0546B-DE70-4753-B445-551FD5336787@gmail.com> References: <20110701222232.GA33935@icarus.home.lan> <20110702045435.GA81502@DataIX.net> <54D65EC5-9A9B-4F96-BB45-1904F2147CBA@gmail.com> <20110704100845.94513n3znbabpthp@webmail.in-berlin.de> <20110705170023.87183xz6ehxbeft3@webmail.in-berlin.de> To: Peter Ross X-Mailer: Apple Mail (2.1084) Cc: freebsd-stable List , Jeremy Chadwick Subject: Re: scp: Write Failed: Cannot allocate memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 17:03:24 -0000 I'm running virtualbox 3.2.12_1 if that has anything to do with it. sysctl vfs.zfs.arc_max: 6200000000 While I'm trying to scp, kstat.zfs.misc.arcstats.size is hovering right = around that value, sometimes above, sometimes below (that's as it should = be, right?). I don't think that it dies when crossing over arc_max. I = can run the same scp 10 times and it might fail 1-3 times, with no = correlation to the arcstats.size being above/below arc_max that I can = see. Scott On Jul 5, 2011, at 3:00 AM, Peter Ross wrote: > Hi all, >=20 > just as an addition: an upgrade to last Friday's FreeBSD-Stable and to = VirtualBox 4.0.8 does not fix the problem. >=20 > I will experiment a bit more tomorrow after hours and grab some = statistics. >=20 > Regards > Peter >=20 > Quoting "Peter Ross" : >=20 >> Hi all, >>=20 >> I noticed a similar problem last week. It is also very similar to one = reported last year: >>=20 >> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.ht= ml >>=20 >> My server is a Dell T410 server with the same bge card (the same = pciconf -lvc output as described by Mahlon: >>=20 >> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058711.ht= ml >>=20 >> Yours, Scott, is a em(4).. >>=20 >> Another similarity: In all cases we are using VirtualBox. I just want = to mention it, in case it matters. I am still running VirtualBox 3.2. >>=20 >> Most of the time kstat.zfs.misc.arcstats.size was reaching = vfs.zfs.arc_max then, but I could catch one or two cases then the value = was still below. >>=20 >> I added vfs.zfs.prefetch_disable=3D1 to sysctl.conf but it does not = help. >>=20 >> BTW: It looks as ARC only gives back the memory when I destroy the = ZFS (a cloned snapshot containing virtual machines). Even if nothing = happens for hours the buffer isn't released.. >>=20 >> My machine was still running 8.2-PRERELEASE so I am upgrading. >>=20 >> I am happy to give information gathered on old/new kernel if it = helps. >>=20 >> Regards >> Peter >>=20 >> Quoting "Scott Sipe" : >>=20 >>>=20 >>> On Jul 2, 2011, at 12:54 AM, jhell wrote: >>>=20 >>>> On Fri, Jul 01, 2011 at 03:22:32PM -0700, Jeremy Chadwick wrote: >>>>> On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott Sipe wrote: >>>>>> I'm running 8.2-RELEASE and am having new problems with scp. When = scping >>>>>> files to a ZFS directory on the FreeBSD server -- most notably = large files >>>>>> -- the transfer frequently dies after just a few seconds. In my = last test, I >>>>>> tried to scp an 800mb file to the FreeBSD system and the transfer = died after >>>>>> 200mb. It completely copied the next 4 times I tried, and then = died again on >>>>>> the next attempt. >>>>>>=20 >>>>>> On the client side: >>>>>>=20 >>>>>> "Connection to home closed by remote host. >>>>>> lost connection" >>>>>>=20 >>>>>> In /var/log/auth.log: >>>>>>=20 >>>>>> Jul 1 14:54:42 freebsd sshd[18955]: fatal: Write failed: Cannot = allocate >>>>>> memory >>>>>>=20 >>>>>> I've never seen this before and have used scp before to transfer = large files >>>>>> without problems. This computer has been used in production for = months and >>>>>> has a current uptime of 36 days. I have not been able to notice = any problems >>>>>> copying files to the server via samba or netatalk, or any = problems in >>>>>> apache. >>>>>>=20 >>>>>> Uname: >>>>>>=20 >>>>>> FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19 = 01:02:54 EST >>>>>> 2011 root@xeon:/usr/obj/usr/src/sys/GENERIC amd64 >>>>>>=20 >>>>>> I've attached my dmesg and output of vmstat -z. >>>>>>=20 >>>>>> I have not restarted the sshd daemon or rebooted the computer. >>>>>>=20 >>>>>> Am glad to provide any other information or test anything else. >>>>>>=20 >>>>>> {snip vmstat -z and dmesg} >>>>>=20 >>>>> You didn't provide details about your networking setup (rc.conf, >>>>> ifconfig -a, etc.). netstat -m would be useful too. >>>>>=20 >>>>> Next, please see this thread circa September 2010, titled "Network >>>>> memory allocation failures": >>>>>=20 >>>>> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.ht= ml#58708 >>>>>=20 >>>>> The user in that thread is using rsync, which relies on scp by = default. >>>>> I believe this problem is similar, if not identical, to yours. >>>>>=20 >>>>=20 >>>> Please also provide your output of ( /usr/bin/limits -a ) for the = server >>>> end and the client. >>>>=20 >>>> I am not quite sure I agree with the need for ifconfig -a but some >>>> information about the networking driver your using for the = interface >>>> would be helpful, uptime of the boxes. And configuration of the = pool. >>>> e.g. ( zpool status -a ;zfs get all ) You should = probably >>>> prop this information up somewhere so you can reference by URL = whenever >>>> needed. >>>>=20 >>>> rsync(1) does not rely on scp(1) whatsoever but rsync(1) can be = made to >>>> use ssh(1) instead of rsh(1) and I believe that is what Jeremy is >>>> stating here but correct me if I am wrong. It does use ssh(1) by >>>> default. >>>>=20 >>>> Its a possiblity as well that if using tmpfs(5) or mdmfs(8) for = /tmp >>>> type filesystems that rsync(1) may be just filling up your temp ram = area >>>> and causing the connection abort which would be expected. ( df -h ) = would >>>> help here. >>>=20 >>> Hello, >>>=20 >>> I'm not using tmpfs/mdmfs at all. The clients yesterday were 3 = different OSX computers (over gigabit). The FreeBSD server has 12gb of = ram and no bce adapter. For what it's worth, the server is backed up = remotely every night with rsync (remote FreeBSD uses rsync to pull) to = an offsite (slow cable connection) FreeBSD computer, and I have not seen = any errors in the nightly rsync. >>>=20 >>> Sorry for the omission of networking info, here's the output of the = requested commands and some that popped up in the other thread: >>>=20 >>> http://www.cap-press.com/misc/ >>>=20 >>> In rc.conf: ifconfig_em1=3D"inet 10.1.1.1 netmask 255.255.0.0" >>>=20 >>> Scott >>>=20 >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>=20 >>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >=20 >=20