From owner-freebsd-net@freebsd.org Mon Jun 29 17:27:49 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1766D9908C3 for ; Mon, 29 Jun 2015 17:27:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6E831631 for ; Mon, 29 Jun 2015 17:27:48 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igblr2 with SMTP id lr2so57613057igb.0 for ; Mon, 29 Jun 2015 10:27:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=5fyqpj+zZWCXjAu6DzBj2/1mFwKdU2iRecBz0FPZqSM=; b=kXo2aljw5OS198Ut5ej9JMxjkfN82sKcLmMTWF2Z6ftN3mTu6MQvo+Bd9Wtte+HwQZ Xeiw76iWVBts/3Mc/tRvZNf7i2DhbrJb55vxkOL2Be2kM5opmy2UxinzVXf2pnqRpCcQ zxmQA9VX1iSB9rKOQ7DMoAT/eUZELIW03gtXfBUB5b8+XVGt/TrkNjc9sSw3nMEqkiJq 1c7i1lc/zzPGCvwWVW5ADeleRPkqUNwjmfqi3yNctELiOY5CKgBDt0bdhqZEcVXilocx Kg3ef89hC1wjZ3bkqmG/7S3AxxJlbJ1vVkiRYCFHI+SHsUjEF0UCe0RZMoUJQDrlWsjp q7Sw== MIME-Version: 1.0 X-Received: by 10.50.134.196 with SMTP id pm4mr16856132igb.6.1435598868182; Mon, 29 Jun 2015 10:27:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Mon, 29 Jun 2015 10:27:48 -0700 (PDT) In-Reply-To: <559109C3.7070900@field.hu> References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <557AB1BB.60502@field.hu> <557AD10D.5070205@field.hu> <557AD2FA.103@field.hu> <559109C3.7070900@field.hu> Date: Mon, 29 Jun 2015 10:27:48 -0700 X-Google-Sender-Auth: 6XmcJAzagmuij_mpHvPe6u_amco Message-ID: Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic From: Adrian Chadd To: Csaba Banhalmi Cc: Christopher Forgeron , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 17:27:49 -0000 hi, I asked for the output of vmstat -z and vmstat -m in a loop. :) -a On 29 June 2015 at 02:02, Csaba Banhalmi wrote: > Hi All, > > "vmstat 5" output when system freezes: > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us= sy > id > 0 0 0 8752M 126M 5663 0 0 0 4042 445 66 0 1219 7148 4870 = 3 > 2 95 > 0 0 0 8650M 145M 2167 0 0 0 3501 447 79 0 974 4042 3578 = 1 > 1 98 > 0 0 0 8374M 201M 3113 0 0 0 6790 441 5 0 1130 6670 3729 = 3 > 1 96 > 0 0 0 8252M 220M 2632 0 0 0 4014 435 4 0 726 11653 2401= 2 > 1 97 > 0 0 0 8188M 224M 1625 0 0 0 2189 434 5 0 713 6714 2376 = 1 > 1 98 > 0 0 0 7992M 233M 1504 0 0 0 2254 433 2 0 867 2890 2868 = 1 > 1 98 > 4 0 0 8032M 216M 2145 0 0 0 1995 435 18 0 526 3769 2048 = 1 > 1 98 > 0 0 0 8180M 195M 1949 0 0 0 1741 435 50 0 593 3441 2363 = 1 > 1 98 > 0 0 0 8186M 178M 2859 0 0 0 2525 436 6 0 499 3313 1733 = 2 > 1 97 > 1 0 0 8410M 146M 2521 0 0 0 1764 440 11 0 736 67271 2121= 4 > 2 94 > 0 0 0 8182M 205M 2910 0 0 0 6378 927 8 0 495 16043 1775= 1 > 1 98 > 1 1 0 7944M 210M 3009 0 0 0 3696 438 8 0 522 4247 1963 = 2 > 1 97 > 0 0 0 8091M 169M 7529 0 0 0 3601 436 105 0 1359 75290 4400= 9 > 3 88 > 0 0 0 8121M 141M 4607 0 0 0 3288 444 62 0 949 12169 3268= 5 > 1 94 > 0 0 0 8044M 201M 1782 0 0 0 4954 1795 9 0 446 3025 1927= 1 > 1 99 > 0 0 0 7916M 222M 1296 0 0 0 2671 438 5 0 525 2984 1920 = 1 > 1 98 > 1 0 0 7870M 230M 888 0 0 0 1677 432 8 0 473 6424 2126 = 1 > 1 99 > 0 0 0 7968M 228M 3375 0 0 0 2625 433 51 0 768 4100 2852 = 3 > 1 96 > 0 0 0 8238M 194M 7586 0 0 0 4758 436 88 0 1026 9631 3908 = 4 > 2 94 > 0 0 0 8293M 185M 3253 0 0 0 2362 437 52 0 747 4475 3105 = 2 > 1 97 > > I increased the vm.v_free_min, but did not help. It was a different froze= , > the system was unreacheable even through IPMI, needed a hard reset. > > Regards, > Csaba > > > > 2015.06.12. 20:17 keltez=C3=A9ssel, Adrian Chadd =C3=ADrta: >> >> On 12 June 2015 at 10:57, Christopher Forgeron >> wrote: >>> >>> I agree it shouldn't run out of memory. Here's what mine does under >>> network >>> load, or rsync load: >>> >>> 2 0 9 1822M 1834M 0 0 0 0 14 8 0 0 22750 724 >>> 136119 >>> 0 23 77 >>> >>> 0 0 9 1822M 1823M 0 0 0 0 0 8 0 0 44317 347 >>> 138151 >>> 0 16 84 >>> >>> 0 0 9 1822M 1761M 0 0 0 0 17 8 0 0 23818 820 921= 98 >>> 0 >>> 12 88 >>> >>> 0 0 9 1822M 1727M 0 0 0 0 14 8 0 0 40768 634 >>> 126688 >>> 0 17 83 >>> >>> 0 0 9 1822M 8192B 0 8 0 0 15 3 3 0 9236 305 5714= 9 >>> 0 >>> 33 67 >>> >>> >>> That's with a 5 second vmstat output. After the 8KiB, the system is >>> nearly >>> completely brain-dead and needs a hard power-off. >>> >>> >>> I've seen it go from 6 GiB free to 8KiB in 5 sec as well. Currently my >>> large >>> machines are set to 12 GiB free to keep them from crashing, from what I >>> presume is just network load due to lots of iSCSI / NFS traffic on my >>> 10GiB >>> network. >>> >>> >>> I haven't had time to type this up for the list yet, but I'm putting it >>> here >>> just to make sure people know it's real. >>> >> Hi, >> >> Then something is leaking or holding onto memory when it shouldn't be. >> >> Try doing vmstat -z and vmstat -m in a one second loop, post the data >> just before it falls over. >> >> >> -adrian > >