Date: Mon, 29 Jun 2015 10:27:48 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Csaba Banhalmi <bimmer@field.hu> Cc: Christopher Forgeron <csforgeron@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic Message-ID: <CAJ-VmokfH3XBXeDgH1cjeCzyjcTY6ve2zWLz9q2XZG5dW%2B8u2w@mail.gmail.com> In-Reply-To: <559109C3.7070900@field.hu> References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <CAB2_NwA-D7bH47=Qkf9QLF3=mZOQBVo81bUsQzQr02W9U4vHMA@mail.gmail.com> <557AB1BB.60502@field.hu> <CAB2_NwA9i-wMXGH2%2BcP9SWxDMNomFRjoVP25hsGWaTDGjBxFTw@mail.gmail.com> <557AD10D.5070205@field.hu> <CAB2_NwAeD43tSwWO3LGuniRMNZ3TVupOuLWj3aUm228jLT2y1A@mail.gmail.com> <557AD2FA.103@field.hu> <CAB2_NwCgEvmMxqmAotO1USsipXOSaGkwK3Uu%2BiVbKd9_bn%2BLWg@mail.gmail.com> <CAJ-VmokXk69V_YURWOjLOQmKrW%2B2-YAiFQnhLOA7SKO6ipw_KQ@mail.gmail.com> <CAB2_NwANWRB2SJY0rO7n%2B_8RK61dyGJ5FCphH_ewQG-E7eOAUg@mail.gmail.com> <CAJ-Vmo=hSE%2Bk1q_JrX9wKOshSRa_WJ78hbL54ZaXMH03PrNFdg@mail.gmail.com> <559109C3.7070900@field.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
hi, I asked for the output of vmstat -z and vmstat -m in a loop. :) -a On 29 June 2015 at 02:02, Csaba Banhalmi <bimmer@field.hu> wrote: > Hi All, > > "vmstat 5" output when system freezes: > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us= sy > id > 0 0 0 8752M 126M 5663 0 0 0 4042 445 66 0 1219 7148 4870 = 3 > 2 95 > 0 0 0 8650M 145M 2167 0 0 0 3501 447 79 0 974 4042 3578 = 1 > 1 98 > 0 0 0 8374M 201M 3113 0 0 0 6790 441 5 0 1130 6670 3729 = 3 > 1 96 > 0 0 0 8252M 220M 2632 0 0 0 4014 435 4 0 726 11653 2401= 2 > 1 97 > 0 0 0 8188M 224M 1625 0 0 0 2189 434 5 0 713 6714 2376 = 1 > 1 98 > 0 0 0 7992M 233M 1504 0 0 0 2254 433 2 0 867 2890 2868 = 1 > 1 98 > 4 0 0 8032M 216M 2145 0 0 0 1995 435 18 0 526 3769 2048 = 1 > 1 98 > 0 0 0 8180M 195M 1949 0 0 0 1741 435 50 0 593 3441 2363 = 1 > 1 98 > 0 0 0 8186M 178M 2859 0 0 0 2525 436 6 0 499 3313 1733 = 2 > 1 97 > 1 0 0 8410M 146M 2521 0 0 0 1764 440 11 0 736 67271 2121= 4 > 2 94 > 0 0 0 8182M 205M 2910 0 0 0 6378 927 8 0 495 16043 1775= 1 > 1 98 > 1 1 0 7944M 210M 3009 0 0 0 3696 438 8 0 522 4247 1963 = 2 > 1 97 > 0 0 0 8091M 169M 7529 0 0 0 3601 436 105 0 1359 75290 4400= 9 > 3 88 > 0 0 0 8121M 141M 4607 0 0 0 3288 444 62 0 949 12169 3268= 5 > 1 94 > 0 0 0 8044M 201M 1782 0 0 0 4954 1795 9 0 446 3025 1927= 1 > 1 99 > 0 0 0 7916M 222M 1296 0 0 0 2671 438 5 0 525 2984 1920 = 1 > 1 98 > 1 0 0 7870M 230M 888 0 0 0 1677 432 8 0 473 6424 2126 = 1 > 1 99 > 0 0 0 7968M 228M 3375 0 0 0 2625 433 51 0 768 4100 2852 = 3 > 1 96 > 0 0 0 8238M 194M 7586 0 0 0 4758 436 88 0 1026 9631 3908 = 4 > 2 94 > 0 0 0 8293M 185M 3253 0 0 0 2362 437 52 0 747 4475 3105 = 2 > 1 97 > > I increased the vm.v_free_min, but did not help. It was a different froze= , > the system was unreacheable even through IPMI, needed a hard reset. > > Regards, > Csaba > > > > 2015.06.12. 20:17 keltez=C3=A9ssel, Adrian Chadd =C3=ADrta: >> >> On 12 June 2015 at 10:57, Christopher Forgeron <csforgeron@gmail.com> >> wrote: >>> >>> I agree it shouldn't run out of memory. Here's what mine does under >>> network >>> load, or rsync load: >>> >>> 2 0 9 1822M 1834M 0 0 0 0 14 8 0 0 22750 724 >>> 136119 >>> 0 23 77 >>> >>> 0 0 9 1822M 1823M 0 0 0 0 0 8 0 0 44317 347 >>> 138151 >>> 0 16 84 >>> >>> 0 0 9 1822M 1761M 0 0 0 0 17 8 0 0 23818 820 921= 98 >>> 0 >>> 12 88 >>> >>> 0 0 9 1822M 1727M 0 0 0 0 14 8 0 0 40768 634 >>> 126688 >>> 0 17 83 >>> >>> 0 0 9 1822M 8192B 0 8 0 0 15 3 3 0 9236 305 5714= 9 >>> 0 >>> 33 67 >>> >>> >>> That's with a 5 second vmstat output. After the 8KiB, the system is >>> nearly >>> completely brain-dead and needs a hard power-off. >>> >>> >>> I've seen it go from 6 GiB free to 8KiB in 5 sec as well. Currently my >>> large >>> machines are set to 12 GiB free to keep them from crashing, from what I >>> presume is just network load due to lots of iSCSI / NFS traffic on my >>> 10GiB >>> network. >>> >>> >>> I haven't had time to type this up for the list yet, but I'm putting it >>> here >>> just to make sure people know it's real. >>> >> Hi, >> >> Then something is leaking or holding onto memory when it shouldn't be. >> >> Try doing vmstat -z and vmstat -m in a one second loop, post the data >> just before it falls over. >> >> >> -adrian > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokfH3XBXeDgH1cjeCzyjcTY6ve2zWLz9q2XZG5dW%2B8u2w>