Date: Tue, 25 Oct 2011 22:28:23 +0300 From: Bojidara Marinchovska <quintessence@bobi.gateit.net> To: freebsd-net@freebsd.org Subject: bce huge amount of input errors Message-ID: <4EA70DD7.9040207@bobi.gateit.net>
next in thread | raw e-mail | index | archive | help
Hello,
I'm running FreeBSD 8.2-STABLE #1: Thu May 19 15:05:33 EEST 2011
CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz (2400.10-MHz
K8-class CPU)
real memory = 25769803776 (24576 MB)
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci2
miibus0: <MII bus> on bce0
brgphy0: <BCM5709C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Ethernet address: xx:xx:xx:xx:xx:xx
bce0: [ITHREAD]
bce0@pci0:2:0:0: class=0x020000 card=0x7055103c chip=0x163914e4
rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II Gigabit Ethernet (BCM5709)'
class = network
subclass = ethernet
When bandwidth hits about 250Mb/s I get ~70 input errors/s, which causes
packet loss.
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
Settings:
loader.conf:
kern.hz=200
kern.maxusers=1024
sysctl.conf:
kern.ipc.maxsockets=204800
kern.ipc.nmbclusters=131072
kern.ipc.somaxconn=16384
net.inet.tcp.maxtcptw=40960
About 4 million input errors on bce0.
Today I changed kern.hz 200 -> 2000 and for ~7 hour uptime I get ~2
million input errors on bce0.
vmstat:
1.) -i (fast forwarding is disabled)
interrupt total rate
irq1: atkbd0 6 0
irq17: atapci0 35 0
irq22: uhci2 uhci4 42643 1
cpu0: timer 48658417 1999
irq256: ciss0 11143030 457
irq257: bce0 198633062 8163
irq258: bce1 16803717 690
cpu1: timer 48649953 1999
cpu11: timer 48649228 1999
cpu3: timer 48649973 1999
cpu10: timer 48649217 1999
cpu6: timer 48649994 1999
cpu15: timer 48649262 1999
cpu14: timer 48649263 1999
cpu2: timer 48649866 1999
cpu12: timer 48649206 1999
cpu4: timer 48649976 1999
cpu8: timer 48649077 1999
cpu7: timer 48649994 1999
cpu9: timer 48649240 1999
cpu5: timer 48649994 1999
cpu13: timer 48649207 1999
Total 1005024360 41304
2.) -z (server does not hit any limit)
ITEM SIZE LIMIT USED FREE REQUESTS
FAILURES
UMA Kegs: 208, 0, 102, 0,
102, 0
UMA Zones: 704, 0, 102, 3,
102, 0
UMA Slabs: 568, 0, 16658, 2,
24197, 0
UMA RCntSlabs: 568, 0, 4830, 0,
4830, 0
UMA Hash: 256, 0, 0, 15,
3, 0
16 Bucket: 152, 0, 192, 8,
192, 0
32 Bucket: 280, 0, 341, 9,
341, 0
64 Bucket: 536, 0, 481, 2,
493, 65
128 Bucket: 1048, 0, 2829, 0, 3226,
118368
VM OBJECT: 216, 0, 467472, 33576,
8255068, 0
MAP: 232, 0, 7, 25,
7, 0
KMAP ENTRY: 120, 827700, 47, 728,
25413, 0
MAP ENTRY: 120, 0, 665651, 3050,
3248202, 0
DP fakepg: 120, 0, 0, 0,
0, 0
SG fakepg: 120, 0, 0, 0,
0, 0
mt_zone: 2056, 0, 256, 7,
256, 0
16: 16, 0, 2837, 2539,
36713029, 0
32: 32, 0, 3229, 2225,
847494, 0
64: 64, 0, 12458, 3670,
769223924, 0
128: 128, 0, 14520, 2358,
7645834, 0
256: 256, 0, 8250, 3330,
2945807, 0
512: 512, 0, 8115, 1069,
7803084, 0
1024: 1024, 0, 75, 917,
124714, 0
2048: 2048, 0, 3961, 1033,
48456, 0
4096: 4096, 0, 4000, 819,
7480557, 0
Files: 80, 0, 11131, 1694,
77493053, 0
TURNSTILE: 136, 0, 4327, 273,
4327, 0
umtx pi: 96, 0, 0, 0,
0, 0
MAC labels: 40, 0, 0, 0,
0, 0
PROC: 1136, 0, 3660, 402,
17921, 0
THREAD: 1120, 0, 4140, 186,
4140, 0
SLEEPQUEUE: 80, 0, 4327, 545,
4327, 0
VMSPACE: 392, 0, 3639, 1101,
17899, 0
cpuset: 72, 0, 2, 98,
2, 0
audit_record: 952, 0, 0, 0,
0, 0
mbuf_packet: 256, 0, 1716, 4684,
475185964, 0
mbuf: 256, 0, 2560, 4480,
786007357, 0
mbuf_cluster: 2048, 131072, 6400, 770,
6400, 0
mbuf_jumbo_page: 4096, 33280, 126, 1119,
7766976, 0
mbuf_jumbo_9k: 9216, 16640, 0, 0,
0, 0
mbuf_jumbo_16k: 16384, 8320, 0, 0,
0, 0
mbuf_ext_refcnt: 4, 0, 2419, 3461,
152993298, 0
g_bio: 232, 0, 0, 3632,
44700603, 0
ttyinq: 160, 0, 135, 273,
600, 0
ttyoutq: 256, 0, 72, 138,
320, 0
ata_request: 320, 0, 0, 48,
20, 0
ata_composite: 336, 0, 0, 0,
0, 0
VNODE: 472, 0, 392140, 33452,
7823148, 0
VNODEPOLL: 112, 0, 0, 0,
0, 0
S VFS Cache: 108, 0, 404747, 33328,
7828800, 0
L VFS Cache: 328, 0, 450, 594,
4331, 0
NAMEI: 1024, 0, 0, 1024,
327010570, 0
DIRHASH: 1024, 0, 1856, 2872,
17086719, 0
NFSMOUNT: 632, 0, 0, 0,
0, 0
NFSNODE: 688, 0, 0, 0,
0, 0
pipe: 728, 0, 3, 407,
11069, 0
ksiginfo: 112, 0, 4048, 803,
5396, 0
itimer: 344, 0, 0, 0,
0, 0
KNOTE: 128, 0, 3622, 1221,
76994, 0
socket: 680, 204804, 40034, 5380,
18020394, 0
unpcb: 240, 204800, 15, 385,
759, 0
ipq: 56, 4158, 0, 0,
0, 0
udp_inpcb: 336, 204809, 2, 471,
19993, 0
udpcb: 16, 204960, 2, 2182,
19993, 0
tcp_inpcb: 336, 204809, 55675, 7223,
17999640, 0
tcpcb: 880, 204800, 18626, 4718,
17999640, 0
tcptw: 72, 41000, 37049, 3951,
11173499, 409
syncache: 144, 65546, 224, 1934,
17303968, 0
hostcache: 136, 65548, 22382, 3490,
89062, 0
tcpreass: 40, 8232, 162, 2274,
1817401, 0
sackhole: 32, 0, 106, 3328,
39148671, 0
sctp_ep: 1272, 66561, 0, 0,
0, 0
sctp_asoc: 2240, 40000, 0, 0,
0, 0
sctp_laddr: 48, 80064, 0, 144,
2, 0
sctp_raddr: 600, 80004, 0, 0,
0, 0
sctp_chunk: 136, 400008, 0, 0,
0, 0
sctp_readq: 104, 400032, 0, 0,
0, 0
sctp_stream_msg_out: 96, 400026, 0, 0,
0, 0
sctp_asconf: 40, 400008, 0, 0,
0, 0
sctp_asconf_ack: 48, 400032, 0, 0,
0, 0
ripcb: 336, 204809, 0, 0,
0, 0
rtentry: 200, 0, 6, 51,
6, 0
pfsrctrpl: 152, 10000, 0, 0,
0, 0
pfrulepl: 912, 0, 20, 8,
20, 0
pfstatepl: 392, 10000, 764, 2416,
224673, 0
pfaltqpl: 240, 0, 0, 0,
0, 0
pfpooladdrpl: 88, 0, 8, 76,
8, 0
pfrktable: 1296, 1002, 2, 4,
3, 0
pfrkentry: 216, 100008, 1, 35,
1, 0
pfrkentry2: 216, 0, 0, 0,
0, 0
pffrent: 32, 5050, 0, 202,
8, 0
pffrag: 80, 0, 0, 90,
4, 0
pffrcache: 80, 10035, 0, 0,
0, 0
pffrcent: 24, 50022, 0, 0,
0, 0
pfstatescrub: 40, 0, 0, 0,
0, 0
pfiaddrpl: 120, 0, 2, 60,
2, 0
pfospfen: 112, 0, 696, 30,
696, 0
pfosfp: 40, 0, 407, 97,
407, 0
selfd: 56, 0, 6834, 3120,
157089498, 0
SWAPMETA: 288, 116519, 0, 0,
0, 0
Mountpoints: 752, 0, 6, 9,
6, 0
FFS inode: 168, 0, 392105, 33617,
7823018, 0
FFS1 dinode: 128, 0, 0, 0,
0, 0
FFS2 dinode: 256, 0, 392105, 33685,
7822979, 0
netstat
1) -m (a lot of free resources)
5055/8385/13440 mbufs in use (current/cache/total)
1907/5263/7170/131072 mbuf clusters in use (current/cache/total/max)
1907/4493 mbuf+clusters out of packet secondary zone in use (current/cache)
117/1128/1245/33280 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/16640 9k jumbo clusters in use (current/cache/total/max)
0/0/0/8320 16k jumbo clusters in use (current/cache/total/max)
5563K/17134K/22697K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
8664222 requests for I/O initiated by sendfile
0 calls to protocol drain routines
2.) -ni ( a lot of input errors )
Name Mtu Network Address Ipkts Ierrs Idrop
Opkts Oerrs Coll
bce0 1500 <Link#1> {hidden} 401171627 2001104 0 254047107
0 0
bce0 1500 {hidden} {hidden} 405636437 - - 454597924 - -
bce1 1500 <Link#2> {hidden} 17418213 209 0 14402350 0 0
bce1 1500 {hidden} {hidden} 17235746 - -
14211395 - -
bce2* 1500 <Link#3> {hidden} 0 0 0 0 0 0
bce3* 1500 <Link#4> {hidden} 0 0 0 0 0 0
usbus 0 <Link#5> 0 0 0
0 0 0
usbus 0 <Link#6> 0 0 0
0 0 0
usbus 0 <Link#7> 0 0 0
0 0 0
usbus 0 <Link#8> 0 0 0
0 0 0
usbus 0 <Link#9> 0 0 0
0 0 0
usbus 0 <Link#10> 0 0 0
0 0 0
lo0 16384 <Link#11> 7075150 0 0
7075326 0 0
lo0 16384 127.0.0.0/8 127.0.0.1 0 - -
7075400 - -
pflog 33152 <Link#12> 0 0 0
36748 0 0
sysctl:
hw.bce.msi_enable: 1
hw.bce.tso_enable: 1
dev.bce.0.%desc: HP NC382i DP Multifunction Gigabit Server Adapter (C0)
dev.bce.0.%driver: bce
dev.bce.0.%location: slot=0 function=0
dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x103c
subdevice=0x7055 class=0x020000
dev.bce.0.%parent: pci2
dev.bce.0.l2fhdr_error_count: 0
dev.bce.0.mbuf_alloc_failed_count: 0
dev.bce.0.mbuf_frag_count: 0
dev.bce.0.dma_map_addr_rx_failed_count: 0
dev.bce.0.dma_map_addr_tx_failed_count: 52
dev.bce.0.unexpected_attention_count: 0
dev.bce.0.stat_IfHcInOctets: 77735343678
dev.bce.0.stat_IfHCInBadOctets: 1453703
dev.bce.0.stat_IfHCOutOctets: 596056312299
dev.bce.0.stat_IfHCOutBadOctets: 0
dev.bce.0.stat_IfHCInUcastPkts: 404441471
dev.bce.0.stat_IfHCInMulticastPkts: 134
dev.bce.0.stat_IfHCInBroadcastPkts: 3076
dev.bce.0.stat_IfHCOutUcastPkts: 524112124
dev.bce.0.stat_IfHCOutMulticastPkts: 0
dev.bce.0.stat_IfHCOutBroadcastPkts: 56
dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.0.stat_Dot3StatsFCSErrors: 0
dev.bce.0.stat_Dot3StatsAlignmentErrors: 0
dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.0.stat_Dot3StatsLateCollisions: 0
dev.bce.0.stat_EtherStatsCollisions: 0
dev.bce.0.stat_EtherStatsFragments: 0
dev.bce.0.stat_EtherStatsJabbers: 0
dev.bce.0.stat_EtherStatsUndersizePkts: 0
dev.bce.0.stat_EtherStatsOversizePkts: 0
dev.bce.0.stat_EtherStatsPktsRx64Octets: 221079621
dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 116359437
dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 463526
dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 1310927
dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 58127062
dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 7104108
dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.0.stat_EtherStatsPktsTx64Octets: 38475943
dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 31773632
dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 32482350
dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 25713275
dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 20790569
dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 374876411
dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.0.stat_XonPauseFramesReceived: 0
dev.bce.0.stat_XoffPauseFramesReceived: 0
dev.bce.0.stat_OutXonSent: 0
dev.bce.0.stat_OutXoffSent: 0
dev.bce.0.stat_FlowControlDone: 0
dev.bce.0.stat_MacControlFramesReceived: 0
dev.bce.0.stat_XoffStateEntered: 0
dev.bce.0.stat_IfInFramesL2FilterDiscards: 19159
dev.bce.0.stat_IfInRuleCheckerDiscards: 0
dev.bce.0.stat_IfInFTQDiscards: 0
dev.bce.0.stat_IfInMBUFDiscards: 0
dev.bce.0.stat_IfInRuleCheckerP4Hit: 3210
dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.0.stat_CatchupInFTQDiscards: 0
dev.bce.0.stat_CatchupInMBUFDiscards: 0
dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.0.com_no_buffers: 2019766
RX/TX pages - 2 (default by driver), until now I didn't try to set a
higher value (I saw in -current this option is available for changing
via sysctl)
load averages: 4.84, 4.81, 4.86
CPU: 5.4% user, 0.0% nice, 10.9% system, 4.4% interrupt, 79.3% idle
Mem: 3179M Active, 11G Inact, 4253M Wired, 56K Cache, 2465M Buf, 5097M Free
Swap: 24G Total, 24G Free
I found in similar thread this patch as suggested:
http://people.freebsd.org/~yongari/bce/bce.20100305.diff, but I didn't
try it yet, because noone confirmed if this is a working solution.
Anyone tested on -stable RX/TX pages = 8 ?
Did anyone test suggested patch from Yongari and can confirm it is a
working solution ?
Thanks
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EA70DD7.9040207>