From owner-freebsd-net@FreeBSD.ORG Fri Jun 12 12:39:32 2015 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D73869B4 for ; Fri, 12 Jun 2015 12:39:32 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EFB951952 for ; Fri, 12 Jun 2015 12:39:31 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (localhost [127.0.0.1]) by green.field.hu (Postfix) with ESMTP id 90F13250A78; Fri, 12 Jun 2015 14:39:29 +0200 (CEST) X-Virus-Scanned: by Amavisd-new at field.hu Received: from green.field.hu ([127.0.0.1]) by green.field.hu (green.field.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NX8DRwHmQ6sS; Fri, 12 Jun 2015 14:39:25 +0200 (CEST) Received: from [10.10.10.153] (unknown [188.227.229.50]) by green.field.hu (Postfix) with ESMTPA id 34070250A7B; Fri, 12 Jun 2015 14:39:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=field.hu; s=mail; t=1434112765; bh=qocKw4h97ktnQLuPHhXioFpyc2e3Iai9RP9sd6N5mlk=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=Qjc59lLpQ7oJUU7EcHJIW+wM1/46N8TazVllJ4OxKfI8z/9k8CJuaoN2twBOXomTV KSWapSFveoCbbtGT0zQweUsiuhBLlmy0ydp9YAlEQpnVIylADysrT8ujBq0OGGraU4 5Bc1PKYgC7MNliMMS/3UCUuS/2Qusk31P1jpWjmA= Message-ID: <557AD2FA.103@field.hu> Date: Fri, 12 Jun 2015 14:39:22 +0200 From: Cs User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Christopher Forgeron CC: FreeBSD Net Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <557AB1BB.60502@field.hu> <557AD10D.5070205@field.hu> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 12:39:32 -0000 but why is that machine runs fine except the network if it's memory related? swap didn't increased before the network outage. 2015.06.12. 14:37 keltezéssel, Christopher Forgeron írta: > rsycn burns memory - I'd say you have a good chance you're running out > of mem before it's replenished. > > For vmstat 5 - Don't run it on console. Connect via a second box with > ssh, and run it there - That way it's the last thing on the ssh > terminal screen when the box dies, and you'll have your proof. > > On Fri, Jun 12, 2015 at 9:31 AM, Cs > wrote: > > machine has been restarted before I could check the "vmstat 5" > output. Yep, it's rsync. Anyway I disabled the backup transfer > it'll solve, but I can't really accept this for solution. > > > 2015.06.12. 14 :29 keltezéssel, Christopher > Forgeron írta: > > Well, even at low speed it could drop due to memory from what > I've seen. > > What was the last line from vmstat 5 before it locked up? > > I find that the em driver isn't crap, but there is a deeper > problem inside > of FreeBSD that is being exposed now - For me it's due to > faster network > connections. > > Are you using rsync to move the files? > > On Fri, Jun 12, 2015 at 7:17 AM, Cs > wrote: > > it seems it's not memory related. Server just died a few > minutes ago > during transporting the backup (400GB) around 800Mbps speed.. > will disable remote backup, it's a shame that em driver is > such a crap. > > > 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta: > > You know what helped me: > > 'vmstat 5' > > Leave that running. If the last thing on the console > after a crash/hang is > vmstat showing 8k of memory left, then you're in the > same problem-park as > me. > > My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in less > than 5 seconds, and > then crashing/panicking/hanging. > > There's others with this issues if you search for it; > a sysctl > to vm.v_free_min to double or triple that value may > help, but first let us > know if that's what is bonking your sever. > > > > On Sun, Jun 7, 2015 at 11:03 AM, Cs > wrote: > > ok, just lowered it to 1500 but please also note > that it was on 1500 for > > 2 > years > > 2015.06.07. 14 :57 > keltezéssel, Rick Macklem írta: > > Since disabling TSO didn't help, you could try > dropping to 1500mtu > > on both interfaces. Some people run into > problems when 9K jumbo clusters > fragment the kernel address space used to > allocate mbufs. > > Good luck with it, rick > > ----- Original Message ----- > > Hi All, > > It worked fine for two weeks but I had a > network outage 2 days ago > then > today. Tried to disable rxcsum and txcsum > after the first one, didn't > help. Don't know what else to do it's a > shame that I can't use this > card > with fbsd i REALLY don't want to install > linux instead but my > production > servers outages are not welcomed by the > customers.. > > 2015.05.26. 10 :36 > keltezéssel, Cs írta: > > Thanks Mark, good idea. I found this > thread which is exactly the > > same > problem as mine: > > > https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/ > > Will see if it helps in a couple weeks. > > Regards, > Csaba > > 2015.05.26. 10 > :30 keltezéssel, > Mark Schouten írta: > > Oh, didn't see your lowest remark. > Then, the next thing that comes > > past here a few times per week is > 'Try disabling TSO'. > > > Met vriendelijke groeten, > > -- > Kerio Operator in de Cloud? > https://www.kerioindecloud.nl/ > Mark Schouten | Tuxis Internet > Engineering > KvK: 61527076 | http://www.tuxis.nl/ > T: 0318 200208 | info@tuxis.nl > > > > > Van: Cs > > Aan: Mark Schouten > > > Cc: > > > Verzonden: 25-5-2015 11:12 > Onderwerp: Re: FreeBSD > 10.1-REL - network unaccessible after > high > traffic > > It was on 1500 for ~3 years :) > Regards, > Csaba > On May 25, 2015, 10:30, > at 10:30, Mark Schouten > > > wrote: > > Try lowering your mtu to 1500, > that worked miracles for me.. > > -- > Mark Schouten > Tuxis Internet Engineering > mark@tuxis.nl > / 0318 > 200208 > > On 25 May 2015, at 09:36, > "Cs" > wrote: > > Hi all, > I have two FreeBSd > 10.1-RELEASE servers > connected to each > other. > They > > were connected via cross > link, but they are > connected to a cisco > > switch > now (the problem was the same > with cross link too). When > transferring > huge files (50-500GB backup > files) via Gigabit (it is > important!) > the > network randomly dies. The > backup runs every day/week and > sometimes the > connection is ok for months > sometimes it happens twice a week. > When the > network dies I can log in to > the server via IPMI and use the > console > everything is OK, but can't > send anything out on the network. > ifconfig > em0 down/up doesn't help nor > netif restart. The problem never > occured > when I used 100Mbit connection > between them, but it was 3com NIC > (xl), > gigabit adapter is Intel > (em0). When I limit the > transfer rate > (rsync > bandwith limit or ipfw pipe) > the problem is much more rare. > > I tried to set these > tuning parameters on both > servers with > > different > > buffer size but nothing > helped: > > # cat /etc/sysctl.conf > > security.bsd.see_other_uids=0 > net.inet.tcp.recvspace=512000 > net.route.netisr_maxqlen=2048 > kern.ipc.nmbclusters=1310720 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > kern.ipc.soacceptqueue=32768 > # cat /boot/loader.conf > geom_mirror_load="YES" # > RAID1 disk driver (see > gmirror(8)) > ipfw_load="YES" > net.inet.ip.fw.default_to_accept=1 > kern.maxusers=4096 > accf_data_load="YES" > The duplex settings > are identical on both servers. > Server A: > em1: > flags=8843 > metric 0 > mtu > > 9000 > > > options=4219b > > > ether > 00:25:90:24:52:66 > > inet x.x.x.x > netmask 0xfffffe00 > broadcast x.x.x.x > nd6 > options=29 > media: Ethernet > autoselect (1000baseT > ) > status: active > Server B: > em0: > flags=8843 > metric 0 > mtu > > 9000 > > > options=4219b > > > ether > 00:30:48:dd:fe:3e > > inet x.x.x.x > netmask 0xfffffe00 > broadcast x.x.x.x > nd6 > options=29 > media: Ethernet > autoselect (1000baseT > ) > status: active > Today I tried to set > mtu to 9000 but in tcpdump > I see that > during > scp > > it is still 1500: > > x.x.x.x.222 > > x.x.x.x.37612: Flags [.], > cksum 0xb6ee > > (incorrect -> > > 0xda6f), seq 35749, ack > 113701596, win 7986, > options [nop,nop,TS > > val > 3103966325 > ecr > 853712893], length 0 > > 09:27:33.912354 IP (tos 0x8, > ttl 64, id 1028, offset 0, flags > > [DF], > > proto TCP (6), length 1500) > > 09:27:33.912358 IP (tos 0x8, > ttl 64, id 1029, offset 0, flags > > [DF], > > proto TCP (6), length 1500) > > Any ideas? Thanks guys! > > _______________________________________________ > freebsd-net@freebsd.org > > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any > mail to > > > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > > freebsd-net@freebsd.org > > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > > _______________________________________________ > > freebsd-net@freebsd.org > > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > >