From owner-freebsd-net@FreeBSD.ORG Fri Jun 12 12:31:22 2015 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A7AA72F for ; Fri, 12 Jun 2015 12:31:22 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BA5651831 for ; Fri, 12 Jun 2015 12:31:20 +0000 (UTC) (envelope-from bimmer@field.hu) Received: from green.field.hu (localhost [127.0.0.1]) by green.field.hu (Postfix) with ESMTP id 9BC6C250A7B for ; Fri, 12 Jun 2015 14:31:17 +0200 (CEST) X-Virus-Scanned: by Amavisd-new at field.hu Received: from green.field.hu ([127.0.0.1]) by green.field.hu (green.field.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4JuM3FQAEWw0 for ; Fri, 12 Jun 2015 14:31:12 +0200 (CEST) Received: from [10.10.10.153] (unknown [188.227.229.50]) by green.field.hu (Postfix) with ESMTPA id C19ED250A64 for ; Fri, 12 Jun 2015 14:31:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=field.hu; s=mail; t=1434112272; bh=2SlvQq4Ctv+byzVAHW1OTYIRy9YnYIGHe31dlEl5EY0=; h=Date:From:To:Subject:References:In-Reply-To; b=Wy4NGFpqVH7cNi5hsvnaTz59EUZFKe9Ibb2Uuq1x7NkmergbgRhCYqb4S/70TgSCj e3gketEu0zF8Ig11EnpaPYEC4CRTwL7P7rHGpI3ybEvqAIfSKYbyKuzwDWOJcwglIO MC2dLxAKZ77SnSf1RTXedT7hACl3WKy0m5WZO5yg= Message-ID: <557AD10D.5070205@field.hu> Date: Fri, 12 Jun 2015 14:31:09 +0200 From: Cs User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <557AB1BB.60502@field.hu> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 12:31:22 -0000 machine has been restarted before I could check the "vmstat 5" output. Yep, it's rsync. Anyway I disabled the backup transfer it'll solve, but I can't really accept this for solution. 2015.06.12. 14:29 keltezéssel, Christopher Forgeron írta: > Well, even at low speed it could drop due to memory from what I've seen. > > What was the last line from vmstat 5 before it locked up? > > I find that the em driver isn't crap, but there is a deeper problem inside > of FreeBSD that is being exposed now - For me it's due to faster network > connections. > > Are you using rsync to move the files? > > On Fri, Jun 12, 2015 at 7:17 AM, Cs wrote: > >> it seems it's not memory related. Server just died a few minutes ago >> during transporting the backup (400GB) around 800Mbps speed.. >> will disable remote backup, it's a shame that em driver is such a crap. >> >> >> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta: >> >>> You know what helped me: >>> >>> 'vmstat 5' >>> >>> Leave that running. If the last thing on the console after a crash/hang is >>> vmstat showing 8k of memory left, then you're in the same problem-park as >>> me. >>> >>> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in less than 5 seconds, and >>> then crashing/panicking/hanging. >>> >>> There's others with this issues if you search for it; a sysctl >>> to vm.v_free_min to double or triple that value may help, but first let us >>> know if that's what is bonking your sever. >>> >>> >>> >>> On Sun, Jun 7, 2015 at 11:03 AM, Cs wrote: >>> >>> ok, just lowered it to 1500 but please also note that it was on 1500 for >>>> 2 >>>> years >>>> >>>> 2015.06.07. 14:57 keltezéssel, Rick Macklem írta: >>>> >>>> Since disabling TSO didn't help, you could try dropping to 1500mtu >>>>> on both interfaces. Some people run into problems when 9K jumbo clusters >>>>> fragment the kernel address space used to allocate mbufs. >>>>> >>>>> Good luck with it, rick >>>>> >>>>> ----- Original Message ----- >>>>> >>>>> Hi All, >>>>>> It worked fine for two weeks but I had a network outage 2 days ago >>>>>> then >>>>>> today. Tried to disable rxcsum and txcsum after the first one, didn't >>>>>> help. Don't know what else to do it's a shame that I can't use this >>>>>> card >>>>>> with fbsd i REALLY don't want to install linux instead but my >>>>>> production >>>>>> servers outages are not welcomed by the customers.. >>>>>> >>>>>> 2015.05.26. 10:36 keltezéssel, Cs írta: >>>>>> >>>>>> Thanks Mark, good idea. I found this thread which is exactly the >>>>>>> same >>>>>>> problem as mine: >>>>>>> >>>>>>> >>>>>>> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/ >>>>>>> >>>>>>> Will see if it helps in a couple weeks. >>>>>>> >>>>>>> Regards, >>>>>>> Csaba >>>>>>> >>>>>>> 2015.05.26. 10:30 keltezéssel, Mark Schouten írta: >>>>>>> >>>>>>> Oh, didn't see your lowest remark. Then, the next thing that comes >>>>>>>> past here a few times per week is 'Try disabling TSO'. >>>>>>>> >>>>>>>> >>>>>>>> Met vriendelijke groeten, >>>>>>>> >>>>>>>> -- >>>>>>>> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ >>>>>>>> Mark Schouten | Tuxis Internet Engineering >>>>>>>> KvK: 61527076 | http://www.tuxis.nl/ >>>>>>>> T: 0318 200208 | info@tuxis.nl >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Van: Cs >>>>>>>> Aan: Mark Schouten >>>>>>>> Cc: >>>>>>>> Verzonden: 25-5-2015 11:12 >>>>>>>> Onderwerp: Re: FreeBSD 10.1-REL - network unaccessible after >>>>>>>> high >>>>>>>> traffic >>>>>>>> >>>>>>>> It was on 1500 for ~3 years :) >>>>>>>> Regards, >>>>>>>> Csaba >>>>>>>> On May 25, 2015, 10:30, at 10:30, Mark Schouten >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Try lowering your mtu to 1500, that worked miracles for me.. >>>>>>>>> -- >>>>>>>>> Mark Schouten >>>>>>>>> Tuxis Internet Engineering >>>>>>>>> mark@tuxis.nl / 0318 200208 >>>>>>>>> >>>>>>>>> On 25 May 2015, at 09:36, "Cs" wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> I have two FreeBSd 10.1-RELEASE servers connected to each >>>>>>>>>> other. >>>>>>>>>> They >>>>>>>>>> >>>>>>>>>> were connected via cross link, but they are connected to a cisco >>>>>>>>> switch >>>>>>>>> now (the problem was the same with cross link too). When >>>>>>>>> transferring >>>>>>>>> huge files (50-500GB backup files) via Gigabit (it is important!) >>>>>>>>> the >>>>>>>>> network randomly dies. The backup runs every day/week and >>>>>>>>> sometimes the >>>>>>>>> connection is ok for months sometimes it happens twice a week. >>>>>>>>> When the >>>>>>>>> network dies I can log in to the server via IPMI and use the >>>>>>>>> console >>>>>>>>> everything is OK, but can't send anything out on the network. >>>>>>>>> ifconfig >>>>>>>>> em0 down/up doesn't help nor netif restart. The problem never >>>>>>>>> occured >>>>>>>>> when I used 100Mbit connection between them, but it was 3com NIC >>>>>>>>> (xl), >>>>>>>>> gigabit adapter is Intel (em0). When I limit the transfer rate >>>>>>>>> (rsync >>>>>>>>> bandwith limit or ipfw pipe) the problem is much more rare. >>>>>>>>> >>>>>>>>> I tried to set these tuning parameters on both servers with >>>>>>>>>> different >>>>>>>>>> >>>>>>>>>> buffer size but nothing helped: >>>>>>>>> # cat /etc/sysctl.conf >>>>>>>>>> security.bsd.see_other_uids=0 >>>>>>>>>> net.inet.tcp.recvspace=512000 >>>>>>>>>> net.route.netisr_maxqlen=2048 >>>>>>>>>> kern.ipc.nmbclusters=1310720 >>>>>>>>>> net.inet.tcp.sendbuf_max=16777216 >>>>>>>>>> net.inet.tcp.recvbuf_max=16777216 >>>>>>>>>> kern.ipc.soacceptqueue=32768 >>>>>>>>>> # cat /boot/loader.conf >>>>>>>>>> geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8)) >>>>>>>>>> ipfw_load="YES" >>>>>>>>>> net.inet.ip.fw.default_to_accept=1 >>>>>>>>>> kern.maxusers=4096 >>>>>>>>>> accf_data_load="YES" >>>>>>>>>> The duplex settings are identical on both servers. >>>>>>>>>> Server A: >>>>>>>>>> em1: flags=8843 metric 0 >>>>>>>>>> mtu >>>>>>>>>> >>>>>>>>>> 9000 >>>>>>>>> >>>>>>>>> options=4219b >>>>>>>>> >>>>>>>>> >>>>>>>>> ether 00:25:90:24:52:66 >>>>>>>>> >>>>>>>>>> inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x >>>>>>>>>> nd6 options=29 >>>>>>>>>> media: Ethernet autoselect (1000baseT ) >>>>>>>>>> status: active >>>>>>>>>> Server B: >>>>>>>>>> em0: flags=8843 metric 0 >>>>>>>>>> mtu >>>>>>>>>> >>>>>>>>>> 9000 >>>>>>>>> >>>>>>>>> options=4219b >>>>>>>>> >>>>>>>>> >>>>>>>>> ether 00:30:48:dd:fe:3e >>>>>>>>> >>>>>>>>>> inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x >>>>>>>>>> nd6 options=29 >>>>>>>>>> media: Ethernet autoselect (1000baseT ) >>>>>>>>>> status: active >>>>>>>>>> Today I tried to set mtu to 9000 but in tcpdump I see that >>>>>>>>>> during >>>>>>>>>> scp >>>>>>>>>> >>>>>>>>>> it is still 1500: >>>>>>>>> x.x.x.x.222 > x.x.x.x.37612: Flags [.], cksum 0xb6ee >>>>>>>>>> (incorrect -> >>>>>>>>>> >>>>>>>>>> 0xda6f), seq 35749, ack 113701596, win 7986, options [nop,nop,TS >>>>>>>>> val >>>>>>>>> 3103966325 ecr 853712893], length 0 >>>>>>>>> >>>>>>>>> 09:27:33.912354 IP (tos 0x8, ttl 64, id 1028, offset 0, flags >>>>>>>>>> [DF], >>>>>>>>>> >>>>>>>>>> proto TCP (6), length 1500) >>>>>>>>> 09:27:33.912358 IP (tos 0x8, ttl 64, id 1029, offset 0, flags >>>>>>>>>> [DF], >>>>>>>>>> >>>>>>>>>> proto TCP (6), length 1500) >>>>>>>>> Any ideas? Thanks guys! >>>>>>>>>> _______________________________________________ >>>>>>>>>> freebsd-net@freebsd.org mailing list >>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>>>>>> To unsubscribe, send any mail to >>>>>>>>>> >>>>>>>>>> "freebsd-net-unsubscribe@freebsd.org" >>>>>>>>> _______________________________________________ >>>>>>>> freebsd-net@freebsd.org mailing list >>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>>>> To unsubscribe, send any mail to >>>>>>>> "freebsd-net-unsubscribe@freebsd.org" >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>> freebsd-net@freebsd.org mailing list >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>>> To unsubscribe, send any mail to >>>>>>> "freebsd-net-unsubscribe@freebsd.org" >>>>>>> >>>>>>> _______________________________________________ >>>>>> freebsd-net@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>> To unsubscribe, send any mail to >>>>>> "freebsd-net-unsubscribe@freebsd.org" >>>>>> >>>>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>>> >>>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"