From owner-freebsd-net@FreeBSD.ORG Sat Dec 17 16:54:35 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CA171065676 for ; Sat, 17 Dec 2011 16:54:35 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id D0E028FC17 for ; Sat, 17 Dec 2011 16:54:34 +0000 (UTC) Received: by iadj38 with SMTP id j38so3053003iad.13 for ; Sat, 17 Dec 2011 08:54:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2ne6G8hnc91AcujYpBq5dIziNz/Ih9vzVyhQGMt9Vm4=; b=jOpChoR7oiwzb8xUpJHkhAq1VqJXa3z9tc/l4aszeqEtx05hINUeUnqCmpNwlbuoLf PlRDWKfWtwzC0wisjZPPK8Ze7oJQmKnA0qRQebMEGWdacy5IDnD6T8WoBtxV41S1tBNV 9kLsc0keRJSKfzwUvNbFbYo5eOrMAp76pzu/U= MIME-Version: 1.0 Received: by 10.50.42.166 with SMTP id p6mr16116288igl.17.1324140874495; Sat, 17 Dec 2011 08:54:34 -0800 (PST) Received: by 10.231.41.206 with HTTP; Sat, 17 Dec 2011 08:54:34 -0800 (PST) In-Reply-To: References: <20111205022730.GE1729@michelle.cdnetworks.com> <20111207005958.GE5825@michelle.cdnetworks.com> <20111207235810.GB9211@michelle.cdnetworks.com> <20111208215256.GB13605@michelle.cdnetworks.com> Date: Sat, 17 Dec 2011 18:54:34 +0200 Message-ID: From: Sami Halabi To: Arnaud Lacombe Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pyunyh@gmail.com, freebsd-net@freebsd.org Subject: Re: Problem on re0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 16:54:35 -0000 updated sysctl: dev.bge.1.stats.FramesDroppedDueToFilters: 0 dev.bge.1.stats.DmaWriteQueueFull: 2291364 dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 dev.bge.1.stats.NoMoreRxBDs: 0 dev.bge.1.stats.InputDiscards: 0 dev.bge.1.stats.InputErrors: 0 dev.bge.1.stats.RecvThresholdHit: 124125195 dev.bge.1.stats.DmaReadQueueFull: 13471017 dev.bge.1.stats.DmaReadHighPriQueueFull: 962 dev.bge.1.stats.SendDataCompQueueFull: 0 dev.bge.1.stats.RingSetSendProdIndex: 349298815 dev.bge.1.stats.RingStatusUpdate: 260314059 dev.bge.1.stats.Interrupts: 260314059 dev.bge.1.stats.AvoidedInterrupts: 0 dev.bge.1.stats.SendThresholdHit: 0 dev.bge.1.stats.rx.ifHCInOctets: 2168102685 dev.bge.1.stats.rx.Fragments: 0 dev.bge.1.stats.rx.UnicastPkts: 237511300 dev.bge.1.stats.rx.MulticastPkts: 0 dev.bge.1.stats.rx.FCSErrors: 0 dev.bge.1.stats.rx.AlignmentErrors: 0 dev.bge.1.stats.rx.xonPauseFramesReceived: 0 dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 dev.bge.1.stats.rx.ControlFramesReceived: 0 dev.bge.1.stats.rx.xoffStateEntered: 0 dev.bge.1.stats.rx.FramesTooLong: 0 dev.bge.1.stats.rx.Jabbers: 0 dev.bge.1.stats.rx.UndersizePkts: 0 dev.bge.1.stats.rx.inRangeLengthError: 0 dev.bge.1.stats.rx.outRangeLengthError: 0 dev.bge.1.stats.tx.ifHCOutOctets: 1965505340 dev.bge.1.stats.tx.Collisions: 0 dev.bge.1.stats.tx.XonSent: 0 dev.bge.1.stats.tx.XoffSent: 0 dev.bge.1.stats.tx.flowControlDone: 0 dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 dev.bge.1.stats.tx.SingleCollisionFrames: 0 dev.bge.1.stats.tx.MultipleCollisionFrames: 0 dev.bge.1.stats.tx.DeferredTransmissions: 0 dev.bge.1.stats.tx.ExcessiveCollisions: 0 dev.bge.1.stats.tx.LateCollisions: 0 dev.bge.1.stats.tx.UnicastPkts: 347270086 dev.bge.1.stats.tx.MulticastPkts: 0 dev.bge.1.stats.tx.BroadcastPkts: 30311 dev.bge.1.stats.tx.CarrierSenseErrors: 0 dev.bge.1.stats.tx.Discards: 0 dev.bge.1.stats.tx.Errors: 0 Sami On Sat, Dec 17, 2011 at 6:53 PM, Sami Halabi wrote: > Hi, > > here is # pciconf -lv | grep -A4 bge\[0-9]\@ > bge0@pci0:2:0:0: class=0x020000 card=0x03781014 chip=0x165a14e4 > rev=0x00 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'Broadcom NetXtreme BCM5722 Gigabit (94309)' > class = network > subclass = ethernet > -- > bge1@pci0:3:1:0: class=0x020000 card=0x026f1014 chip=0x16c714e4 > rev=0x10 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'BCM5703A3 NetXtreme Gigabit Ethernet' > class = network > subclass = ethernet > > and dmesg: > bge1: > mem 0xe8400000-0xe840ffff irq 21 at device 1.0 on pci3 > bge1: CHIP ID 0x00001100; ASIC REV 0x01; CHIP REV 0x11; PCI > miibus1: on bge1 > brgphy1: PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bge1: Ethernet address: 00:1a:64:21:86:29 > bge1: [ITHREAD] > > > Thanks > > On Sat, Dec 17, 2011 at 6:49 PM, Sami Halabi wrote: > >> Hi, >> >> I've moved to a new server with bge card driver, similar configuration: >> /etc/sysctl.conf >> net.inet.flowtable.enable=0 >> net.inet.ip.fastforwarding=1 >> kern.ipc.somaxconn=8192 >> kern.ipc.shmmax=2147483648 >> kern.ipc.maxsockets=204800 >> kern.ipc.maxsockbuf=2097152 >> hw.intr_storm_threshold=9000 >> kern.maxfiles=256000 >> kern.maxfilesperproc=230400 >> net.inet.ip.dummynet.pipe_slot_limit=1000 >> #net.inet.ip.dummynet.io_fast=1 >> net.link.ether.ipfw=1 >> kern.ipc.nmbclusters=409600 >> net.graph.recvspace=40960 >> net.graph.maxdgram=40960 >> >> Kernel >> ----------- >> device lagg >> options IPFIREWALL >> options IPFIREWALL_FORWARD >> options IPFIREWALL_VERBOSE >> options DUMMYNET >> options HZ=1000 >> options TCP_SIGNATURE >> device crypto # core crypto support >> device cryptodev # /dev/crypto for access to h/w >> options IPSEC >> options DEVICE_POLLING >> device if_bridge >> >> /boot/loader.conf >> net.graph.maxalloc=128000 >> net.graph.maxdata=128000 >> net.graph.threads=4 >> >> and i have similar problem.. i check each minute for ping and i see >> sometimes loss of 50% or 100% (ping -c 2 -t 4 otherside.ip.com), >> here are the sysctl dev.bge.1.stats: >> dev.bge.1.stats.FramesDroppedDueToFilters: 0 >> dev.bge.1.stats.DmaWriteQueueFull: 2291350 >> dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 >> dev.bge.1.stats.NoMoreRxBDs: 0 >> dev.bge.1.stats.InputDiscards: 0 >> dev.bge.1.stats.InputErrors: 0 >> dev.bge.1.stats.RecvThresholdHit: 124120300 >> dev.bge.1.stats.DmaReadQueueFull: 13470948 >> dev.bge.1.stats.DmaReadHighPriQueueFull: 962 >> dev.bge.1.stats.SendDataCompQueueFull: 0 >> dev.bge.1.stats.RingSetSendProdIndex: 349289174 >> dev.bge.1.stats.RingStatusUpdate: 260304688 >> dev.bge.1.stats.Interrupts: 260304688 >> dev.bge.1.stats.AvoidedInterrupts: 0 >> dev.bge.1.stats.SendThresholdHit: 0 >> dev.bge.1.stats.rx.ifHCInOctets: 2165908673 >> dev.bge.1.stats.rx.Fragments: 0 >> dev.bge.1.stats.rx.UnicastPkts: 237503495 >> dev.bge.1.stats.rx.MulticastPkts: 0 >> dev.bge.1.stats.rx.FCSErrors: 0 >> dev.bge.1.stats.rx.AlignmentErrors: 0 >> dev.bge.1.stats.rx.xonPauseFramesReceived: 0 >> dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 >> dev.bge.1.stats.rx.ControlFramesReceived: 0 >> dev.bge.1.stats.rx.xoffStateEntered: 0 >> dev.bge.1.stats.rx.FramesTooLong: 0 >> dev.bge.1.stats.rx.Jabbers: 0 >> dev.bge.1.stats.rx.UndersizePkts: 0 >> dev.bge.1.stats.rx.inRangeLengthError: 0 >> dev.bge.1.stats.rx.outRangeLengthError: 0 >> dev.bge.1.stats.tx.ifHCOutOctets: 1956421618 >> dev.bge.1.stats.tx.Collisions: 0 >> dev.bge.1.stats.tx.XonSent: 0 >> dev.bge.1.stats.tx.XoffSent: 0 >> dev.bge.1.stats.tx.flowControlDone: 0 >> dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 >> dev.bge.1.stats.tx.SingleCollisionFrames: 0 >> dev.bge.1.stats.tx.MultipleCollisionFrames: 0 >> dev.bge.1.stats.tx.DeferredTransmissions: 0 >> dev.bge.1.stats.tx.ExcessiveCollisions: 0 >> dev.bge.1.stats.tx.LateCollisions: 0 >> dev.bge.1.stats.tx.UnicastPkts: 347260508 >> dev.bge.1.stats.tx.MulticastPkts: 0 >> dev.bge.1.stats.tx.BroadcastPkts: 30306 >> dev.bge.1.stats.tx.CarrierSenseErrors: 0 >> dev.bge.1.stats.tx.Discards: 0 >> dev.bge.1.stats.tx.Errors: 0 >> >> this driver gives more statics... >> >> please help me, this causes us serious problems with customers. >> >> Thanks in advance, >> Sami >> >> On Fri, Dec 9, 2011 at 8:51 AM, Arnaud Lacombe wrote: >> >>> Hi, >>> >>> On Thu, Dec 8, 2011 at 4:52 PM, YongHyeon PYUN wrote: >>> > On Thu, Dec 08, 2011 at 11:56:51AM +0200, Sami Halabi wrote: >>> >> hi, >>> >> i can see what it does. >>> >> how do i compile the driver and load it instead of the running driver >>> >> without restarting? its a production server. >>> > >>> > There is no such way. You have to rebuild/install kernel and >>> > reboot. >>> > >>> How so ? AFAIK, you should be able to build the driver as a module, >>> tell the bootloader to pre-load the module so that the built-in driver >>> ends up being ignored. I did that some time ago with em(4). From >>> memory, the steps should be: >>> >>> # From the source root >>> % cd sys/modules/re >>> % make >>> % scp if_re.ko :/boot/kernel >>> >>> then on the target, add in `/boot/loader.conf': >>> >>> if_re_load="YES" >>> >>> - Arnaud >>> >> >> >> >> -- >> Sami Halabi >> Information Systems Engineer >> NMS Projects Expert >> >> > > > -- > Sami Halabi > Information Systems Engineer > NMS Projects Expert > > -- Sami Halabi Information Systems Engineer NMS Projects Expert