Date: Thu, 19 Oct 2006 23:42:06 -0700 From: "Ted Mittelstaedt" <tedm@toybox.placo.com> To: "Moses Leslie" <marmoset@malformed.org> Cc: freebsd-questions@freebsd.org Subject: Re: increasing transmit speeds in WAN setting? Message-ID: <001601c6f412$dd30a820$3c01a8c0@coolf89ea26645> References: <20061018222030.S11323@fincher.users.accretive-networks.net> <001801c6f349$1198b320$3c01a8c0@coolf89ea26645> <20061019011206.N11323@fincher.users.accretive-networks.net>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message ----- From: "Moses Leslie" <marmoset@malformed.org> To: "Ted Mittelstaedt" <tedm@toybox.placo.com> Cc: <freebsd-questions@freebsd.org> Sent: Thursday, October 19, 2006 1:33 AM Subject: Re: increasing transmit speeds in WAN setting? > Hi Ted, > > While I don't totally discount that possibility, I really don't think > that's the case. I told you that you wouldn't believe me. > We have over 500 servers, most of them running FreeBSD, > and we've seen this happen in multiple cases on different hardware. Except all of them gigabit cards, right? So much for "different hardware" > When > it's linux, exact same hardware, exact same cables, this doesn't happen. > > It's an intel card gbit card, using the em driver. They're uplinked to > Cisco 2948-getx switches, which are uplinked to 65xx's, which then go to > 12xxx borders. There aren't any collision errors on the port at all: > > 24 totalCollisionCount = 0 > 25 lateCollisionCount = 0 > 26 singleCollisionFrames = 0 > 27 multipleCollisionFrames = 0 > 28 excessiveCollisionFrames = 0 > > and no real errors to speak of, period. > > The port is auto, since it needs to be to get gbit. All of the non-gbit > servers we have are forced 100/full, all cisco switches, all intel 100/pro > (fxp) drivers, they all show this same problem. > Well right there you are doing things wrong. You should always set ethernet cards to auto. The only time you ever force 100/full or force anything, speed/duplex, is when your plugged into a hub that does NOT autoswitch. There's very few of them around that are 100base T, but there are some, and there's a lot more 10baseT stuff that wasn't autoswitching. Any halfway decent 100baseT hub will support nway autonegotiation and when you hard-code post speeds you will cause drops and speed loss. But, please don't take my word for it since you seem to like disbelieving me, just try it out yourself. Go to your fxp servers, login to your switches, set the switch port to the server to autonegotiation, on the server remove all the media options in /etc/rc.conf, shut down the server (you must power it down for the ports to switch into autonegotiation) and bring it up and you will see both sides negotiate to 100base T full, and a lot of your problems in throughput will disappear. Both the switch and the servermust be set to autonegotiation. If they don't autonegotiate to 100baseTfull, then you have a cable problem, simple as that. I've been doing ethernet since the late 80's and doing it professionally for a decade, and I've seen and use more different types of ethernet in my life than you will ever see in the rest of your career. The idea that your supposed to override the autonegotiation and hard code stuff originated from network admins who plugged early 100baseT stuff together then couldn't figure out why it didn't autonegotiate to 100baseT full. What they didn't realze is that the cabling they were using - CAT-3 mostly, or CAT-5 that had been incorrectly terminated with the wrong connectors, or wrong plugs, or wrong wiring pattern, or bad crimps because they were using stranded plugs on solid core wire, or some other such thing, what the real culprit, and the autonegotiation chips were in fact detecting the problem and trying to protect the network. Unfortunately, 90% of network admins out there don't know the first thing about layer-1, they assume the wiring contractors handle all that. The wiring contractors by contrast are mostly minimum-wage goobers who's heads are filled with a lot of rediculous nonsense about how Ethernet really works. > If the server is a 4.9 server, I can get ~400KB/s. If it's 6.1, ~300KB/s. > Linux 2.6, ~650KB/s, which is about what I'd expect given the latency and > the default settings. All on the same hardware, same switches, same > cables. > The Linux device drivers are simply different than the FreeBSD drivers. I don't know how much more I can tell you this over and over. The em driver has got some problems, granted. But, this has absolutely nothing to do with the FreeBSD version or the TCP/IP stack. Until you do what I told you to do and properly setup and test under fxp0, I am just not going to waste my time on this anymore. I will leave you with a printout of a test run on a new mailserver I'm building up right now, in fact, using an fxp card, to prove it's a not a stack problem. You can choose to believe it or you can choose to continue wasting your time chasing ghosts in the TP stack when the problem is the driver: $ whoami tedm $ $ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ls-lR.gz ls-lR.gz 100% of 18 MB 1057 kBps 00m00s $ $ ping ftp.freebsd.org PING ftp.freebsd.org (62.243.72.50): 56 data bytes 36 bytes from ge2-16.1000M.d5.opa.tdk.net (195.41.33.70): Communication prohibited by filter Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 82f2 0 0000 33 01 6e38 65.75.206.14 62.243.72.50 ^C --- ftp.freebsd.org ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss $ ping 195.41.33.70 PING 195.41.33.70 (195.41.33.70): 56 data bytes 64 bytes from 195.41.33.70: icmp_seq=0 ttl=242 time=171.189 ms 64 bytes from 195.41.33.70: icmp_seq=1 ttl=242 time=171.470 ms 64 bytes from 195.41.33.70: icmp_seq=2 ttl=242 time=171.185 ms 64 bytes from 195.41.33.70: icmp_seq=3 ttl=242 time=171.179 ms ^C --- 195.41.33.70 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max/stddev = 171.179/171.256/171.470/0.124 ms $ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ls-lR.gz ls-lR.gz 100% of 18 MB 1040 kBps 00m00s $ date Thu Oct 19 23:35:13 PDT 2006 $ $ uname -a FreeBSD mail.seasurf.net 6.1-RELEASE FreeBSD 6.1-RELEASE #0: Wed Sep 20 19:38:01 PDT 2006 tedm@mail.seasurf.net:/usr/src/sys/i386/compile/SEAMAIL i386 $ $ ifconfig -a fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=8<VLAN_MTU> inet6 fe80::250:8bff:fee0:4f03%fxp0 prefixlen 64 scopeid 0x1 inet 65.75.206.14 netmask 0xffffffe0 broadcast 65.75.206.31 ether 00:50:8b:e0:4f:03 media: Ethernet autoselect (100baseTX <full-duplex>) status: active Ted
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?001601c6f412$dd30a820$3c01a8c0>