Date: Mon, 30 Oct 2006 10:34:06 +0100 (CET) From: Hartmut Brandt <hartmut.brandt@dlr.de> To: Thomas <freebsdlists@bsdunix.ch> Cc: freebsd-net@freebsd.org Subject: Re: paket loss on freebsd router if (b)snmpd is running##SPAM Message-ID: <20061030102241.V60872@knop-beagle.kn.op.dlr.de> In-Reply-To: <45424017.2030905@bsdunix.ch> References: <4541AB5C.2060009@bsdunix.ch> <20061027104553.D27619@knop-beagle.kn.op.dlr.de> <454226C7.10505@bsdunix.ch> <20061027173957.S27619@knop-beagle.kn.op.dlr.de> <45424017.2030905@bsdunix.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Thomas, it seems that 5.4 has the old routing table code. This code used a TAILQ to hold all the routes. This turned out to be a problem for large routing tables so I replaced it with a red-black tree. This happened between 6.0 and 6.1 - 6.0 has still the old code, 6.1 the new one. The old code consumes a huge amount of memory (I think around 600 byte per route) and a huge amount of time, because it needs to loop through the ordered tailq for each routing update. It is espacially bad for the initial full read of the routing table and the full read that happens periodically (when you do an SNMP operation on the routing table), because it builds and ordered list from an ordered input set. Because the daemon opens the routing socket to get routing updates, it will do some work even when no SNMP requests come in. I suppose this is what is happening in your case. As a first step I would recommend to update the bsnmp daemon to at least RELENG_6_1. Updating contrib/bsnmp, lib/libbsnmp and usr.sbin/bsnmp and rebuilding/installing first the library and then the daemon should do the trick. 6.1 should also have the 64-bit counters in the interface statistics if I interpret the CVS log correctly. In any case it is still a question why a user space program doing work causes the interfaces to loose packets. I don't know enough about the POLLING stuff though to make a qualified guess. harti On Fri, 27 Oct 2006, Thomas wrote: T>Hello Harti T> T>Harti Brandt schrieb: T>> On Fri, 27 Oct 2006, Thomas wrote: T>> T>> T>Hello Harti T>> T> T>> T>Harti Brandt schrieb: T>> T>> On Fri, 27 Oct 2006, Thomas wrote: T>> T>> T>> T>> T>Hello T>> T>> T> T>> T>> T>I use several 5.4 and 6.1 FBSD machines as router (with quagga). The T>> T>> T>average traffic is 300mbit/s (em interfaces with polling enabled). It T>> T>> T>works more or less. T>> T>> T> T>> T>> T>Problem: T>> T>> T>If bsnmpd is running and I'm doing a snmpwalk from a remote machine the T>> T>> T>router has some significant packet loss. We are talking about 3-15% of T>> T>> T>packet loss. It's a simple snmpwalk (ipRouteTable and the interfaces stats). T>> T>> T>> T>> Does this happen also if you walk only the interface tables? What size is T>> T>> your routing table? T>> T> T>> T>The routing table has approx. 200k entries. I've to correct my T>> T>statement. I don't walk over the RoutingTable. T>> T>> Oh. That would be a nice test how good the RB-tree implementation of the T>> routing table is :-) T>> T>> T>It's only an interface table walk (IF-MIB). bsnmpd consumes approx. T>> T>6-10% cpu. I already have significant packet loss if bsnmpd is started. T>> T> T>> T>bsnmpd not running: T>> T> T>> T>ping www.switch.ch T>> T> T>> T>--- oreius.switch.ch ping statistics --- T>> T>44 packets transmitted, 44 packets received, 0% packet loss T>> T>round-trip min/avg/max/stddev = 6.825/7.425/8.007/0.252 ms T>> T> T>> T>zero packet loss. T>> T> T>> T> T>> T>netstat -w 1 shows no error. T>> T> T>> T> input (Total) output T>> T> packets errs bytes packets errs bytes colls T>> T> 39877 0 34330807 38614 0 34087518 0 T>> T> 45522 0 38730142 44124 0 38424881 0 T>> T> 44671 0 38783909 43455 0 38604698 0 T>> T> 43140 0 36939542 42059 0 36691583 0 T>> T> 43428 0 36858817 42213 0 36700954 0 T>> T> 43748 0 37994949 42687 0 37780190 0 T>> T> 42698 0 36451927 41409 0 36270936 0 T>> T> 38695 0 32184588 37119 0 31791417 0 T>> T> T>> T> T>> T> T>> T> T>> T>bsnmpd started: T>> T> T>> T>ping www.switch.ch T>> T>--- oreius.switch.ch ping statistics --- T>> T>57 packets transmitted, 53 packets received, 7% packet loss T>> T>round-trip min/avg/max/stddev = 6.715/12.638/95.183/19.041 ms T>> T> T>> T>7% packet loss. T>> T> T>> T> T>> T>netstat shows some errors too. T>> T>netstat -w 1 T>> T> input (Total) output T>> T> packets errs bytes packets errs bytes colls T>> T> 33597 0 27508009 32433 0 27344720 0 T>> T> 38204 0 32117848 36950 0 31852251 0 T>> T> 39171 0 33870984 37999 0 33696346 0 T>> T> 32160 0 26825650 31065 0 26591079 0 T>> T> 26915 726 20902878 25771 0 20791170 0 T>> T> 35167 0 29173014 33945 0 28904867 0 T>> T> 33282 370 27478393 32140 0 27297079 0 T>> T> 29738 0 23859586 28464 0 23524984 0 T>> T> 38456 598 31559055 36881 0 31313396 0 T>> T> 34738 0 28088584 33247 0 27743827 0 T>> T> 38193 648 30854513 36657 0 30556948 0 T>> T> T>> T> T>> T>A snmpwalk makes it a bit worse but not much. I already have packet loss T>> T>if bsnmpd is just started without any walks. T>> T>> You mean, you have loss if it is just ideling around? The only thing T>> bsnmpd does in this situation is polling the interface statistics, which T>> is needed because the kernel has only 32-bit counters. Could you please T>> check what the polling interval is? You find this in the BEGEMOT-MIB2-MIB. T>> Something like T>> T> T>Yes, bsnmpd is just ideling. T> T>> snmpwalk ... <host> begemotMib2 T> T>On this freebsd 5.4 machine I have bsnmpd-1.12: T>share/snmp/mibs/BEGEMOT-MIB.txt T>share/snmp/mibs/BEGEMOT-NTP-MIB.txt T>share/snmp/mibs/BEGEMOT-SNMPD.txt T> T>but no BEGEMOT-MIB2-MIB. T> T> T>snmpwalk -v 2c -m /usr/share/snmp/mibs/BEGEMOT-MIB.txt -c public T>localhost BEGEMOT T>BEGEMOT-MIB::begemot.1.1.1.1.0 = INTEGER: 2048 T>BEGEMOT-MIB::begemot.1.1.1.2.0 = INTEGER: 2048 T>BEGEMOT-MIB::begemot.1.1.1.3.0 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.1.4.0 = IpAddress: 0.0.0.0 T>BEGEMOT-MIB::begemot.1.1.1.5.0 = Gauge32: 3 T>BEGEMOT-MIB::begemot.1.1.2.1.3.192.168.1.10.162 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.2.1.4.192.168.1.10.162 = STRING: "SNTrap" T>BEGEMOT-MIB::begemot.1.1.2.1.5.192.168.1.10.162 = INTEGER: 2 T>BEGEMOT-MIB::begemot.1.1.4.1.3.127.0.0.1.161 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.4.1.3.192.168.1.100.161 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.6.1.2.5.109.105.98.73.73 = STRING: T>"/usr/local/lib/snmp_mibII.so" T>BEGEMOT-MIB::begemot.1.1.6.1.3.5.109.105.98.73.73 = STRING: "This module T>implements the interface and ip groups." T>BEGEMOT-MIB::begemot.1.1.7.1.0 = Counter32: 0 T>BEGEMOT-MIB::begemot.1.1.7.2.0 = Counter32: 0 T>BEGEMOT-MIB::begemot.1.1.7.3.0 = Counter32: 0 T>BEGEMOT-MIB::begemot.1.1.7.4.0 = Counter32: 0 T>BEGEMOT-MIB::begemot.1.1.8.1.0 = INTEGER: 2 T>BEGEMOT-MIB::begemot.1.1.8.2.0 = Gauge32: 0 T>BEGEMOT-MIB::begemot.1.1.8.3.0 = INTEGER: 7 T>BEGEMOT-MIB::begemot.1.1.9.1.2.19.47.118.97.114.47.114.117.110.47.115.110.109.112.100.46.115.111.99.107 T>= INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.9.1.3.19.47.118.97.114.47.114.117.110.47.115.110.109.112.100.46.115.111.99.107 T>= INTEGER: 4 T>BEGEMOT-MIB::begemot.1.1.10.1.1.2.3.117.100.112 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.10.1.1.2.5.108.115.111.99.107 = INTEGER: 1 T>BEGEMOT-MIB::begemot.1.1.10.1.1.3.3.117.100.112 = OID: T>BEGEMOT-MIB::begemot.1.1.10.2 T>BEGEMOT-MIB::begemot.1.1.10.1.1.3.5.108.115.111.99.107 = OID: T>BEGEMOT-MIB::begemot.1.1.10.3 T>BEGEMOT-MIB::begemot.3.1.1.1.0 = Counter64: 1000000000 T>BEGEMOT-MIB::begemot.3.1.1.2.0 = Timeticks: (2000) 0:00:20.00 T>BEGEMOT-MIB::begemot.3.1.1.3.0 = Timeticks: (0) 0:00:00.00 T> T>> given that /usr/share/snmp/mibs is in your MIB path. T> T>snmpd.conf: T> T># Configuration T>%snmpd T>begemotSnmpdDebugDumpPdus = 2 T>begemotSnmpdDebugSyslogPri = 7 T> T>begemotSnmpdCommunityString.0.1 = $(read) T>begemotSnmpdCommunityString.0.2 = $(write) T>begemotSnmpdCommunityDisable = 1 T> T># open standard SNMP ports T>begemotSnmpdPortStatus.[$(host)].161 = 1 T>begemotSnmpdPortStatus.127.0.0.1.161 = 1 T> T># open a unix domain socket T>begemotSnmpdLocalPortStatus."/var/run/snmpd.sock" = 1 T>begemotSnmpdLocalPortType."/var/run/snmpd.sock" = 4 T> T># send traps to the traphost T>begemotTrapSinkStatus.[$(traphost)].$(trapport) = 4 T>begemotTrapSinkVersion.[$(traphost)].$(trapport) = 2 T>begemotTrapSinkComm.[$(traphost)].$(trapport) = $(trap) T> T>sysContact = $(contact) T>sysLocation = $(location) T>sysObjectId = 1.3.6.1.4.1.12325.1.1.2.1.$(system) T> T>snmpEnableAuthenTraps = 2 T> T>begemotSnmpdModulePath."mibII" = "/usr/local/lib/snmp_mibII.so" T> T>Regards, T>Thomas T> T> T>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061030102241.V60872>