From owner-freebsd-net@FreeBSD.ORG Mon Dec 19 02:28:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 658E11065672 for ; Mon, 19 Dec 2011 02:28:20 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 258528FC15 for ; Mon, 19 Dec 2011 02:28:20 +0000 (UTC) Received: by iadj38 with SMTP id j38so5504641iad.13 for ; Sun, 18 Dec 2011 18:28:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=4C+qV2q7jPqWHrtqUM4IpRso1rqXsELEsrMg8AaZMws=; b=xMZzO8bTjKkWP8V7ju5fwLowCYpu2t6w+IbUzWm9OcZ1a+HThRJUoSpguD4T7TRuxS f2p7G0sWMosMGd4TCrD6++DLxG5eDcLvHHE549HX7u9RuoDiIHgEeXkhjoMKXnwZCQg0 1GyKjgw8E0gavQzZk3btCRK4vcF7omhP1iCAQ= Received: by 10.50.42.166 with SMTP id p6mr24274865igl.17.1324261699677; Sun, 18 Dec 2011 18:28:19 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id j3sm62863052ibj.1.2011.12.18.18.28.16 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 18 Dec 2011 18:28:18 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 18 Dec 2011 18:26:24 -0800 From: YongHyeon PYUN Date: Sun, 18 Dec 2011 18:26:24 -0800 To: Sami Halabi Message-ID: <20111219022624.GB1704@michelle.cdnetworks.com> References: <20111205022730.GE1729@michelle.cdnetworks.com> <20111207005958.GE5825@michelle.cdnetworks.com> <20111207235810.GB9211@michelle.cdnetworks.com> <20111208215256.GB13605@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Arnaud Lacombe Subject: Re: Problem on re0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 02:28:20 -0000 On Sat, Dec 17, 2011 at 06:49:48PM +0200, Sami Halabi wrote: > Hi, > > I've moved to a new server with bge card driver, similar configuration: > /etc/sysctl.conf > net.inet.flowtable.enable=0 > net.inet.ip.fastforwarding=1 > kern.ipc.somaxconn=8192 > kern.ipc.shmmax=2147483648 > kern.ipc.maxsockets=204800 > kern.ipc.maxsockbuf=2097152 > hw.intr_storm_threshold=9000 > kern.maxfiles=256000 > kern.maxfilesperproc=230400 > net.inet.ip.dummynet.pipe_slot_limit=1000 > #net.inet.ip.dummynet.io_fast=1 > net.link.ether.ipfw=1 > kern.ipc.nmbclusters=409600 > net.graph.recvspace=40960 > net.graph.maxdgram=40960 > > Kernel > ----------- > device lagg > options IPFIREWALL > options IPFIREWALL_FORWARD > options IPFIREWALL_VERBOSE > options DUMMYNET > options HZ=1000 > options TCP_SIGNATURE > device crypto # core crypto support > device cryptodev # /dev/crypto for access to h/w > options IPSEC > options DEVICE_POLLING > device if_bridge > > /boot/loader.conf > net.graph.maxalloc=128000 > net.graph.maxdata=128000 > net.graph.threads=4 > > and i have similar problem.. i check each minute for ping and i see > sometimes loss of 50% or 100% (ping -c 2 -t 4 otherside.ip.com), > here are the sysctl dev.bge.1.stats: > dev.bge.1.stats.FramesDroppedDueToFilters: 0 > dev.bge.1.stats.DmaWriteQueueFull: 2291350 > dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 > dev.bge.1.stats.NoMoreRxBDs: 0 > dev.bge.1.stats.InputDiscards: 0 > dev.bge.1.stats.InputErrors: 0 > dev.bge.1.stats.RecvThresholdHit: 124120300 > dev.bge.1.stats.DmaReadQueueFull: 13470948 > dev.bge.1.stats.DmaReadHighPriQueueFull: 962 > dev.bge.1.stats.SendDataCompQueueFull: 0 > dev.bge.1.stats.RingSetSendProdIndex: 349289174 > dev.bge.1.stats.RingStatusUpdate: 260304688 > dev.bge.1.stats.Interrupts: 260304688 > dev.bge.1.stats.AvoidedInterrupts: 0 > dev.bge.1.stats.SendThresholdHit: 0 > dev.bge.1.stats.rx.ifHCInOctets: 2165908673 > dev.bge.1.stats.rx.Fragments: 0 > dev.bge.1.stats.rx.UnicastPkts: 237503495 > dev.bge.1.stats.rx.MulticastPkts: 0 > dev.bge.1.stats.rx.FCSErrors: 0 > dev.bge.1.stats.rx.AlignmentErrors: 0 > dev.bge.1.stats.rx.xonPauseFramesReceived: 0 > dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 > dev.bge.1.stats.rx.ControlFramesReceived: 0 > dev.bge.1.stats.rx.xoffStateEntered: 0 > dev.bge.1.stats.rx.FramesTooLong: 0 > dev.bge.1.stats.rx.Jabbers: 0 > dev.bge.1.stats.rx.UndersizePkts: 0 > dev.bge.1.stats.rx.inRangeLengthError: 0 > dev.bge.1.stats.rx.outRangeLengthError: 0 > dev.bge.1.stats.tx.ifHCOutOctets: 1956421618 > dev.bge.1.stats.tx.Collisions: 0 > dev.bge.1.stats.tx.XonSent: 0 > dev.bge.1.stats.tx.XoffSent: 0 > dev.bge.1.stats.tx.flowControlDone: 0 > dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 > dev.bge.1.stats.tx.SingleCollisionFrames: 0 > dev.bge.1.stats.tx.MultipleCollisionFrames: 0 > dev.bge.1.stats.tx.DeferredTransmissions: 0 > dev.bge.1.stats.tx.ExcessiveCollisions: 0 > dev.bge.1.stats.tx.LateCollisions: 0 > dev.bge.1.stats.tx.UnicastPkts: 347260508 > dev.bge.1.stats.tx.MulticastPkts: 0 > dev.bge.1.stats.tx.BroadcastPkts: 30306 > dev.bge.1.stats.tx.CarrierSenseErrors: 0 > dev.bge.1.stats.tx.Discards: 0 > dev.bge.1.stats.tx.Errors: 0 > > this driver gives more statics... > > please help me, this causes us serious problems with customers. > Given that you see the same issue with bge(4) it looks like the root cause is not in ethernet driver. I also see no evidence of dropped frames from bge(4) hardware MAC statistics. It seems you have really complex network configuration(ipfw(4), lagg(4), dummynet(4), bridge(4) and ipsec(4) etc). Finding out simplest network setup that shows the issue would be required here.