From owner-freebsd-net@FreeBSD.ORG Mon Dec 19 06:54:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC3A5106566B for ; Mon, 19 Dec 2011 06:54:08 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id AE2A08FC08 for ; Mon, 19 Dec 2011 06:54:08 +0000 (UTC) Received: by iadj38 with SMTP id j38so5983130iad.13 for ; Sun, 18 Dec 2011 22:54:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/D7mG3ns2AEydbmw+BKKWCgvhKY1PtlnBasRV7A31pA=; b=XKGduIF0gijb7dK2N0vThdm+5VpnmQb9Jjx0pS5XW3UpORBHKeIT5//CZXf5xYpoUF btNTywV3O33uVIXUW7mf2JOFzHuwH67Zt1H2ypaIQQ7jE8tYzRZlymXVV4MisD9lHwRw THFEPEDrVRSohJFp1QS9MxsHUpk3GHg+1XfQI= MIME-Version: 1.0 Received: by 10.50.219.226 with SMTP id pr2mr15562975igc.77.1324277648102; Sun, 18 Dec 2011 22:54:08 -0800 (PST) Received: by 10.231.41.206 with HTTP; Sun, 18 Dec 2011 22:54:07 -0800 (PST) In-Reply-To: <20111219022624.GB1704@michelle.cdnetworks.com> References: <20111205022730.GE1729@michelle.cdnetworks.com> <20111207005958.GE5825@michelle.cdnetworks.com> <20111207235810.GB9211@michelle.cdnetworks.com> <20111208215256.GB13605@michelle.cdnetworks.com> <20111219022624.GB1704@michelle.cdnetworks.com> Date: Mon, 19 Dec 2011 08:54:07 +0200 Message-ID: From: Sami Halabi To: pyunyh@gmail.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Arnaud Lacombe Subject: Re: Problem on re0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 06:54:09 -0000 Hi, >It seems you have really complex network configuration(ipfw(4), >lagg(4), dummynet(4), bridge(4) and ipsec(4) etc). Finding out >simplest network setup that shows the issue would be required here. hmm.. i actually don't use lagg nor bridge nor ipsec, i compile them for any future use that might be. however i use IPFW, the MPD port limit connected users via pptp/l2tp with ipfw rules via netgraph (i must admit i don't see them when I use ipfw show) and probably dummynet. Archetecture is simple: bge0 -- connected to border router, bge1 connected to cables company where customers connect we are routing 172.16.0.0/12 to bge1, default gateway is our peer in the other side of bge0. installed port as pptp/l2tp server is MPD. please help me its really making us troubles... Sami On Mon, Dec 19, 2011 at 4:26 AM, YongHyeon PYUN wrote: > On Sat, Dec 17, 2011 at 06:49:48PM +0200, Sami Halabi wrote: > > Hi, > > > > I've moved to a new server with bge card driver, similar configuration: > > /etc/sysctl.conf > > net.inet.flowtable.enable=0 > > net.inet.ip.fastforwarding=1 > > kern.ipc.somaxconn=8192 > > kern.ipc.shmmax=2147483648 > > kern.ipc.maxsockets=204800 > > kern.ipc.maxsockbuf=2097152 > > hw.intr_storm_threshold=9000 > > kern.maxfiles=256000 > > kern.maxfilesperproc=230400 > > net.inet.ip.dummynet.pipe_slot_limit=1000 > > #net.inet.ip.dummynet.io_fast=1 > > net.link.ether.ipfw=1 > > kern.ipc.nmbclusters=409600 > > net.graph.recvspace=40960 > > net.graph.maxdgram=40960 > > > > Kernel > > ----------- > > device lagg > > options IPFIREWALL > > options IPFIREWALL_FORWARD > > options IPFIREWALL_VERBOSE > > options DUMMYNET > > options HZ=1000 > > options TCP_SIGNATURE > > device crypto # core crypto support > > device cryptodev # /dev/crypto for access to h/w > > options IPSEC > > options DEVICE_POLLING > > device if_bridge > > > > /boot/loader.conf > > net.graph.maxalloc=128000 > > net.graph.maxdata=128000 > > net.graph.threads=4 > > > > and i have similar problem.. i check each minute for ping and i see > > sometimes loss of 50% or 100% (ping -c 2 -t 4 otherside.ip.com), > > here are the sysctl dev.bge.1.stats: > > dev.bge.1.stats.FramesDroppedDueToFilters: 0 > > dev.bge.1.stats.DmaWriteQueueFull: 2291350 > > dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 > > dev.bge.1.stats.NoMoreRxBDs: 0 > > dev.bge.1.stats.InputDiscards: 0 > > dev.bge.1.stats.InputErrors: 0 > > dev.bge.1.stats.RecvThresholdHit: 124120300 > > dev.bge.1.stats.DmaReadQueueFull: 13470948 > > dev.bge.1.stats.DmaReadHighPriQueueFull: 962 > > dev.bge.1.stats.SendDataCompQueueFull: 0 > > dev.bge.1.stats.RingSetSendProdIndex: 349289174 > > dev.bge.1.stats.RingStatusUpdate: 260304688 > > dev.bge.1.stats.Interrupts: 260304688 > > dev.bge.1.stats.AvoidedInterrupts: 0 > > dev.bge.1.stats.SendThresholdHit: 0 > > dev.bge.1.stats.rx.ifHCInOctets: 2165908673 > > dev.bge.1.stats.rx.Fragments: 0 > > dev.bge.1.stats.rx.UnicastPkts: 237503495 > > dev.bge.1.stats.rx.MulticastPkts: 0 > > dev.bge.1.stats.rx.FCSErrors: 0 > > dev.bge.1.stats.rx.AlignmentErrors: 0 > > dev.bge.1.stats.rx.xonPauseFramesReceived: 0 > > dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 > > dev.bge.1.stats.rx.ControlFramesReceived: 0 > > dev.bge.1.stats.rx.xoffStateEntered: 0 > > dev.bge.1.stats.rx.FramesTooLong: 0 > > dev.bge.1.stats.rx.Jabbers: 0 > > dev.bge.1.stats.rx.UndersizePkts: 0 > > dev.bge.1.stats.rx.inRangeLengthError: 0 > > dev.bge.1.stats.rx.outRangeLengthError: 0 > > dev.bge.1.stats.tx.ifHCOutOctets: 1956421618 > > dev.bge.1.stats.tx.Collisions: 0 > > dev.bge.1.stats.tx.XonSent: 0 > > dev.bge.1.stats.tx.XoffSent: 0 > > dev.bge.1.stats.tx.flowControlDone: 0 > > dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 > > dev.bge.1.stats.tx.SingleCollisionFrames: 0 > > dev.bge.1.stats.tx.MultipleCollisionFrames: 0 > > dev.bge.1.stats.tx.DeferredTransmissions: 0 > > dev.bge.1.stats.tx.ExcessiveCollisions: 0 > > dev.bge.1.stats.tx.LateCollisions: 0 > > dev.bge.1.stats.tx.UnicastPkts: 347260508 > > dev.bge.1.stats.tx.MulticastPkts: 0 > > dev.bge.1.stats.tx.BroadcastPkts: 30306 > > dev.bge.1.stats.tx.CarrierSenseErrors: 0 > > dev.bge.1.stats.tx.Discards: 0 > > dev.bge.1.stats.tx.Errors: 0 > > > > this driver gives more statics... > > > > please help me, this causes us serious problems with customers. > > > > Given that you see the same issue with bge(4) it looks like the > root cause is not in ethernet driver. I also see no evidence of > dropped frames from bge(4) hardware MAC statistics. > It seems you have really complex network configuration(ipfw(4), > lagg(4), dummynet(4), bridge(4) and ipsec(4) etc). Finding out > simplest network setup that shows the issue would be required here. > -- Sami Halabi Information Systems Engineer NMS Projects Expert