From owner-freebsd-net@FreeBSD.ORG Sun May 10 00:16:58 2015 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D496CDE7 for <freebsd-net@freebsd.org>; Sun, 10 May 2015 00:16:58 +0000 (UTC) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8EE5E1513 for <freebsd-net@freebsd.org>; Sun, 10 May 2015 00:16:58 +0000 (UTC) Received: by qgfi89 with SMTP id i89so52854495qgf.1 for <freebsd-net@freebsd.org>; Sat, 09 May 2015 17:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=HX4JM86I7F5u5pDQM0sqO2Wtml1jpCRMnnXnFNrcQRQ=; b=jf1Hjh3GJgi0qmJFcKu+jifHM99LsI3yrDe28YEp73oL2ITP+1+67VgpnDbRNZG2w9 6EpOWsBsCJS9tcH/y4Lv+/1TcmdPAiPF2wlaWBSCAFC0sMCC5iu/AwP70wjSI28UD8cM NyCp7wFi6aDPE7XAwJdc9PKAhsCPH6KIboUTv3nBxGwBOBLrZZBNrpwSdceynyMPL1uk GxIvctFb4TfQ1cBKi5ZQSIwNbwRurZsSygSLTHc2C+J5a1i9JVws7JBz8Xmm2gxIzvvV eauE8NFDl5CXIROpfa1f5IRsC8sqiofNGhFjN5eDJ9kIihhiz36Qtj2tkL3W4D3UOcyg VSXg== MIME-Version: 1.0 X-Received: by 10.140.105.198 with SMTP id c64mr5576124qgf.61.1431217017622; Sat, 09 May 2015 17:16:57 -0700 (PDT) Received: by 10.96.110.229 with HTTP; Sat, 9 May 2015 17:16:57 -0700 (PDT) In-Reply-To: <554A2D3D.3060408@tuxis.nl> References: <137094161.27589033.1430255162390.JavaMail.root@uoguelph.ca> <5540889A.5030904@tuxis.nl> <21824.58754.452182.195043@hergotha.csail.mit.edu> <554A2D3D.3060408@tuxis.nl> Date: Sat, 9 May 2015 21:16:57 -0300 Message-ID: <CAB2_NwAR11OTM5N+S4A4om9Bfat+GbdBHMNJZ_Zg7EpmaJ5cKQ@mail.gmail.com> Subject: Re: Frequent hickups on the networking layer From: Christopher Forgeron <csforgeron@gmail.com> To: Mark Schouten <mark@tuxis.nl> Cc: FreeBSD Net <freebsd-net@freebsd.org> Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sun, 10 May 2015 00:16:58 -0000 Mark, did switching to a MTU of 1500 ever help? I'm currently reliving a problem with this - I'm down to a MTU of 4000, but I still see jumbo pages being allocated - I believe it's my iSCSI setup (using 4k block size, which means the packet is bigger than 4k), but I'm not sure where it's all coming from yet. I'm on 10.1 RELEASE fyi. I'm going to patch my network devs to not use MJUM9BYTES and see if that has an effect. For me, the problem all started again once I really started putting storage load on the FreeBSD machines. At times, I'm seeing 7 Gbits on the 10 Gbit adapters. Oh, and there are gremlins in the new ctld / iscsi as well. I'll get into that later, but if a heavily loaded iscsi target goes down, when it reboots, the reconnect storm from all the iscsi machines kernel panics the FreeBSD iscsi target host. My machine looped through three boot-start-panic loops before I caught it and put it into single-user mode. Starting ctld manually seems to make everything okay. On Wed, May 6, 2015 at 12:03 PM, Mark Schouten <mark@tuxis.nl> wrote: > Hi, > > On 04/29/2015 04:06 PM, Garrett Wollman wrote: > > If you're using one of the drivers that has this problem, then yes, >> keeping your layer-2 MTU/MRU below 4096 will probably cause it to use >> 4k (page-sized) clusters instead, which are perfectly safe. >> >> As a side note, at least on the hardware I have to support, Infiniband >> is limited to 4k MTU -- so I have one "jumbo" network with 4k frames >> (that's bridged to IB) and one with 9k frames (that everything else >> uses). >> > > So I was thinking, a customer of mine runs mostly the same setup, and has > no issues at all. The only difference, MTU of 1500 vs MTU of 9000. > > I also created a graph in munin, graphing the number of mbuf_jumbo > requests and failures. I find that when lots of writes occur to the > iscsi-layer, the number of failed requests grow, and so so the number of > errors on the ethernet interface. See attached images. My customer is also > not suffering from crashing ctld-daemons, which crashes every other minute > in my setup. > > So tonight I'm going to switch to an MTU of 1500, I'll let you know if > that helped. > > > Regards, > > Mark Schouten > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >