Date: Mon, 24 Feb 2003 20:02:05 +0100 From: Paul Schenkeveld <fb-isp@psconsult.nl> To: Wolfpaw - Dale Corse <admin-lists@wolfpaw.net> Cc: freebsd-bugs@FreeBSD.ORG, freebsd-isp@FreeBSD.ORG Subject: Re: FW: Netstat byte counters, and IPFW (1 or 2) counters don't match Message-ID: <20030224200205.A44721@psconsult.nl> In-Reply-To: <AJENJFOLCLAHHIIGCCHNKEMCFIAA.admin-lists@wolfpaw.net>; from admin-lists@wolfpaw.net on Mon, Feb 24, 2003 at 11:00:13AM -0700 References: <20030224181805.A43509@psconsult.nl> <AJENJFOLCLAHHIIGCCHNKEMCFIAA.admin-lists@wolfpaw.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi dale, First a little request, could you please set your mailer/editor to produce lines no longer than 72 characters? Follow-ups become very unreadable with long lines. On Mon, Feb 24, 2003 at 11:00:13AM -0700, Wolfpaw - Dale Corse wrote: > Hi Paul, > > Thanks for the reply :) I did have a few things to add; > > >> I can't seem to find a reason for this.. but if I graph the values of fxp0 using netstat -bin, like this: > >> > >> Name Mtu Network Address Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll > >> fxp0 1500 <Link#1> 00:02:b3:bc:12:7c 3417242 0 232992484 3997590 0 764014344 0 > >> > >> and IPFW-2 rules like this (I also tried ipfw 1, and ipf): > >> > >> 00100 388827 18466235 count ip from any to any in via fxp0 > >> 00101 454678 80243192 count ip from any to any out via fxp0 > >> > >> both with the same version of MRTG, the netstat version matches what my provider tells me I am using, but the > >> one using ipfw values is always 50 - 200 kbits/sec off. Does anyone know why? I am stumped :( I have tried > > > According to the numbers above, ipfw only counts 7.9% of all input > > bytes and 10.5% of all output bytes counted by netstat. > > Could it be that your (32-bit) counters have wrapped around? > > The ipfw counters are reset every 5 minutes - the netstat ones aren't. I was not aware > of a way to zero those without a reboot (which would be bad - the link in question does > about 3 - 5 mb/s constantly, people get upset when it goes down :( They seem to be out > as I mentioned above contstantly I seemed to remember a -z option to netstat but I just checked the man page to learn that -z can zero all kinds of netstat counters but not these. Sorry. Opportunity to get ones name engraved in *BSD sources? :-) > > I can think of there reasons for /small/ differences: > > > > - Netstat counts all bytes in the ethernet frame, ipfw probably only sees > > the IP packet minus ethernet header. > > Doesn't the new IPFW-2 layer2 count everything, even non-ip traffic? If not, is there > a way I can block non-ip traffic from leaving my network .. I see no reason at all that > it should be leaving it. I run no applications that should require any of the below protocols. Need to know more about your topology to see where they could come from but if you check with tcpdump at least you know what to look for and perhaps even what generates them. To block non-ip frames, look atr your WS-C6509 docs. I've configured a couple of those some three years ago and saw that they have nice layer-2 filters besides the access-lists you get with the optional layer-3 module. The 6509s I configures run Cisco Cat-OS, not IOS but it seems reasonable that the IOS versions also do layer-2 access-lists. > > - Netstat sees all protocols, ipfw only IP. Other notable protocols that > > can offset your netstat counters: > > * CDP (Cisco Discovery Protocol, sent out on every Cisco port unless > > explicitely disabled in the config of the router/switch) > > Hmm - I totally forgot about CDP - thanks much :) But with your traffic, CDP will not be a big issue. > > * IPv6 > > * IPX (are there any Novell servers or PCs with IPX enables in the > > subnet/broadcast domain). > > * Netbeui (any PC in the same subnet/broadcast domain with netbeui > > enabled). > > Again - any ways to forbid these from crossing the gateway to the world? I only > run IPv4 and IP based applications (they are all that is allowed), we have nothing > that should be using IPX, or (gasp) anything using netbeui enabled, but a customer > might without realizing it. > > > It could well be a combination of several of these factors. Make > > sure both start counting at the same time (or zero both counters > > simultaneously), look with 'tcpdump -ni <interface> not ip' to see all > > protocols other than IP. > > I'll try tcpdump, thanks much :) > > > After evicting all PCs sending out IPX or Netbeui, kindly asking your > > switch to not send CDP and taming all connected nodes to not use IPv6 > > should bring you a bit closer but the simple calculation: > > > > ipfw_no_of_bytes + ipfw_no_of_packets * ether_header_size > > > > will still not give you thenumber of bytes as IP datagrams may be > > fragmented and reassembled between the ethernet driver and the IP > > layer. > > Doesn't ipfw2's layer2 stuff handle this? Ipfw2 will see reassembled packets I think. So, if for example a FTP data connection sends a 4096 byte TCP frame, layer-2 will break this up in several fragments depending on the MTU of every link in the path between server and client. Assume the 4K packet is sent over a ethernet link first, you'll see three ethernet frames, 2 of 1500 octets and 1 a bit shorter. If the next link is a gif tunnel with a MTU of 1280, you'll get 5 packets because the two 1500 octet frames get fragmented again. So your next link is a modem link with PPP and a MTU of 296? You'll end up with more than a dozen frames. Each frame has it's own ethernet header when it reaches your FreeBSD box over ethernet. During reassembly all these headers are thrown away. So if I'm right and IPFW2 sees the reassembled packet you can see a big difference in octet counts between the link layer (netstat) and IPFW2 which only sees 4096 bytes.. > >> using both ipfw 1, and ipfw 2 (counting only layer 3, and only layer 2, and both types (which was way off) of > >> packets).. and I have been unable to solve the problem :( The other odd thing being, it seems to be traffic that > >> is maybe not IP based.. because the graph on the uplink of the Cisco switch behind the firewall in question does > >> not see the traffic ethier.. but the Cisco 6509 in front of it sees it.. :/ > >> > >> Please let me know if you have any ideas :) > >> > >> Thanks in advance :) > >> Dale. > > >Hope this helps you. > > It has been quite helpful - thanks much :) Sorry about all the questions :( No problems about the questions. Hopefully my answers help you further. BTW. I've only used ipfw1 and ipfilter so I'm right about ipfw2 and reassembly but it should be documented somewhere or otherwise a simple experiment should reveal the truth. Regards, Paul Schenkeveld, Consultant PSconsult ICT Services BV To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030224200205.A44721>