From owner-freebsd-net@freebsd.org Mon Jun 29 09:12:46 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA77198C7DF for ; Mon, 29 Jun 2015 09:12:46 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 793F31ABF; Mon, 29 Jun 2015 09:12:45 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (localhost [127.0.0.1]) by green.field.hu (Postfix) with ESMTP id 783E1250AC0; Mon, 29 Jun 2015 11:03:13 +0200 (CEST) X-Virus-Scanned: by Amavisd-new at field.hu Received: from green.field.hu ([127.0.0.1]) by green.field.hu (green.field.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Dq3zkq8AL1Kv; Mon, 29 Jun 2015 11:03:10 +0200 (CEST) Received: from [10.10.10.153] (unknown [188.227.229.50]) by green.field.hu (Postfix) with ESMTPA id 8C3A4250ACE; Mon, 29 Jun 2015 11:03:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=field.hu; s=mail; t=1435568590; bh=cRwSKxyfgBv5gvV/dpQonJvxA/tCHGscgeQ2UItthXE=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=OgVNvkCO0okU3+XGAJjXI/qjl7Vdo2ugU+AUV+j4OA39AW4bA+lXjSGI8uR2QhBBz CdqdY3ubeC8lBF+/U0aO9jxCut2Cn1j+Im67HdzGL6D/X5YmwRVn9aqsL4SwzlGTJ7 2QfFHU+bInQoNHx1QTYVPbVMKKzIitLgvUDDUM/E= Message-ID: <559109C3.7070900@field.hu> Date: Mon, 29 Jun 2015 11:02:59 +0200 From: Csaba Banhalmi User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Adrian Chadd , Christopher Forgeron CC: FreeBSD Net Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <557AB1BB.60502@field.hu> <557AD10D.5070205@field.hu> <557AD2FA.103@field.hu> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 09:12:46 -0000 Hi All, "vmstat 5" output when system freezes: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 0 0 8752M 126M 5663 0 0 0 4042 445 66 0 1219 7148 4870 3 2 95 0 0 0 8650M 145M 2167 0 0 0 3501 447 79 0 974 4042 3578 1 1 98 0 0 0 8374M 201M 3113 0 0 0 6790 441 5 0 1130 6670 3729 3 1 96 0 0 0 8252M 220M 2632 0 0 0 4014 435 4 0 726 11653 2401 2 1 97 0 0 0 8188M 224M 1625 0 0 0 2189 434 5 0 713 6714 2376 1 1 98 0 0 0 7992M 233M 1504 0 0 0 2254 433 2 0 867 2890 2868 1 1 98 4 0 0 8032M 216M 2145 0 0 0 1995 435 18 0 526 3769 2048 1 1 98 0 0 0 8180M 195M 1949 0 0 0 1741 435 50 0 593 3441 2363 1 1 98 0 0 0 8186M 178M 2859 0 0 0 2525 436 6 0 499 3313 1733 2 1 97 1 0 0 8410M 146M 2521 0 0 0 1764 440 11 0 736 67271 2121 4 2 94 0 0 0 8182M 205M 2910 0 0 0 6378 927 8 0 495 16043 1775 1 1 98 1 1 0 7944M 210M 3009 0 0 0 3696 438 8 0 522 4247 1963 2 1 97 0 0 0 8091M 169M 7529 0 0 0 3601 436 105 0 1359 75290 4400 9 3 88 0 0 0 8121M 141M 4607 0 0 0 3288 444 62 0 949 12169 3268 5 1 94 0 0 0 8044M 201M 1782 0 0 0 4954 1795 9 0 446 3025 1927 1 1 99 0 0 0 7916M 222M 1296 0 0 0 2671 438 5 0 525 2984 1920 1 1 98 1 0 0 7870M 230M 888 0 0 0 1677 432 8 0 473 6424 2126 1 1 99 0 0 0 7968M 228M 3375 0 0 0 2625 433 51 0 768 4100 2852 3 1 96 0 0 0 8238M 194M 7586 0 0 0 4758 436 88 0 1026 9631 3908 4 2 94 0 0 0 8293M 185M 3253 0 0 0 2362 437 52 0 747 4475 3105 2 1 97 I increased the vm.v_free_min, but did not help. It was a different froze, the system was unreacheable even through IPMI, needed a hard reset. Regards, Csaba 2015.06.12. 20:17 keltezéssel, Adrian Chadd írta: > On 12 June 2015 at 10:57, Christopher Forgeron wrote: >> I agree it shouldn't run out of memory. Here's what mine does under network >> load, or rsync load: >> >> 2 0 9 1822M 1834M 0 0 0 0 14 8 0 0 22750 724 136119 >> 0 23 77 >> >> 0 0 9 1822M 1823M 0 0 0 0 0 8 0 0 44317 347 138151 >> 0 16 84 >> >> 0 0 9 1822M 1761M 0 0 0 0 17 8 0 0 23818 820 92198 0 >> 12 88 >> >> 0 0 9 1822M 1727M 0 0 0 0 14 8 0 0 40768 634 126688 >> 0 17 83 >> >> 0 0 9 1822M 8192B 0 8 0 0 15 3 3 0 9236 305 57149 0 >> 33 67 >> >> >> That's with a 5 second vmstat output. After the 8KiB, the system is nearly >> completely brain-dead and needs a hard power-off. >> >> >> I've seen it go from 6 GiB free to 8KiB in 5 sec as well. Currently my large >> machines are set to 12 GiB free to keep them from crashing, from what I >> presume is just network load due to lots of iSCSI / NFS traffic on my 10GiB >> network. >> >> >> I haven't had time to type this up for the list yet, but I'm putting it here >> just to make sure people know it's real. >> > Hi, > > Then something is leaking or holding onto memory when it shouldn't be. > > Try doing vmstat -z and vmstat -m in a one second loop, post the data > just before it falls over. > > > -adrian