Date: Fri, 24 Jun 2005 18:12:06 -0400 From: Greg Rowe <greg@qwest.net> To: freebsd-net@freebsd.org Subject: Looking For Ideas or Suggestions Message-ID: <200506241812.07076.greg@qwest.net>
next in thread | raw e-mail | index | archive | help
Greetings, I've been chasing a network interface "freeze" problem on and off for some time now and it's driving me nuts ! The problem occurs on two identical mail servers that sit behind a firewall. Both systems have two ethernet interfaces and when I'm having this problem the external interface will "freeze" once or twice an hour for between 10-15 seconds. The systems continue to run during these freezes and it doesn't effect the traffic on the 2nd interface. The problem is also intermittent in that it will effect one system for several weeks and then just go away. Today it's effecting both systems. The systems are Sun Fire V60X dual 3.06GHZ Xeon processor systems with integrated Intel PRO/1000 (em0) ethernet ports, 2GB of memory. We have a number of these systems and these are the only two experiencing the problem. They are running 4.11 STABLE although they were originally installed with 4.10 STABLE and upgraded to see if this fixed the problem. The one system currently has an Intel EtherExpress Pro/100B card installed as the primary interface to see if the em0 was my problem, but I still have freezes using fxp0. Both systems are very lightly loaded and running Sendmail and anti-spam packages. The systems hang off Catalyst switches that have been checked and rechecked. No errors or config issues. Duplex, speed and mediaopt are all set in rc.conf and aren't autodetected. Cables and ports have all been swapped. No errors in netstat or any logs. Sysctl " log_in_vain's" aren't showing me any clues. The interface just freezes and then starts again with no messages. Tests using pings from system to the other out each interface prove that the emo/fxp0 freezes with packet loss while pings to the em1 interface have no problems. Now, here's where it gets stranger. By accident I found one way to guarantee that a freeze won't occur. If I log into the system via the fxp0/em0 interface and start a ping against the IP of that interface. As long as the ping is running (I've tried days) and outputing the ping stats every second, the interface is freeze free ! I liken it to keeping the interface "warm" !! Doing the same ping with a -q for some reason doesn't stop the freezes. It needs the ping output to keep "warm". Pinging the em0 address from another system or while logged in through the other interface also won't stop the freeze. The freeze isn't login window related, although it may sound that way. The interface just stops working for no apparent reason and then starts again after 10 or 15 seconds. I've gone the network sniffer route and really can't see anything out of the ordinary happening when the freeze occurs. Most ports are blocked by the firewall and the systems also have ipfw enabled (taken out of the kernel on one to see if maybe that was causing the problem). I'm running out of ideas short of replacing bigger hardware than an ethernet card. The problem is I don't know what to replace. I've been building and running FreeBSD systems for many years and this one has me and everyone else stumped. I'm looking for any suggestions as to what I could enable or tweak that may give me some info as to why the interfaces are intermittently freezing. I'm willing to try just about anything right now. Thanks.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200506241812.07076.greg>