Date: Sat, 29 Sep 2001 23:14:44 -0600 From: "Kenneth D. Merry" <ken@kdm.org> To: Terry Lambert <tlambert2@mindspring.com> Cc: Jonathan Lemon <jlemon@flugsvamp.com>, the_srinivas@hotmail.com, hackers@FreeBSD.ORG Subject: Re: TCP&IP cksum offload on FreeBSD 4.2 Message-ID: <20010929231444.B52304@panzer.kdm.org> In-Reply-To: <3BB69F42.5C1ED705@mindspring.com>; from tlambert2@mindspring.com on Sat, Sep 29, 2001 at 09:27:46PM -0700 References: <200109270157.f8R1vZ546863@prism.flugsvamp.com> <3BB42E50.B32F5E95@mindspring.com> <20010928074347.A36960@panzer.kdm.org> <3BB61A6B.A97E955A@mindspring.com> <20010929153932.A50715@panzer.kdm.org> <3BB69F42.5C1ED705@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Sep 29, 2001 at 21:27:46 -0700, Terry Lambert wrote: > "Kenneth D. Merry" wrote: > > > Unfortunately, it can not correctly interoperate with a > > > number of cards in jumbogram mode, so unless you know the > > > card on the other end and manually configure it (it can't > > > negotiate properly), you can't really use jumbograms. Or > > > you could rewrite the firmware (of course). > > > > Huh? Care to explain? Why wouldn't it interoperate with jumbo frames? > > Why are jumbo frames something that need be negotiated by the card? > > They're not negotiated by the card, but rather by the TCP stack. > > Because at the time the Tigon II was released, the jumbogram > wire format had not solidified. Therefore cards built during > that time used different wire data for the jumbogram framing. Can you give more details? Did someone decide on a different ethertype than 0x8870 or something? That's really the only thing that's different between a standard ethernet frame and a jumbo frame. (other than the size) > > > I have never seen the board with the stock FreeBSD driver > > > get better than about half wire rate with normal size > > > packets and window depth. > > > > Yep, it doesn't handle large numbers of small packets very well. You can > > easily get wire rate with jumbo frames, though. > > Again, that's not useful, since it requires hand negotiation; > you can't automatically set the MTU up as a result of doing a > driver level negotiation with the other end of the link (unless > the other end is also a Tigon II). Negotiation, at least for packet size, is generally done at a higher level than the transport layer (ethernet in this case). Your routing hardware should be able to send back ICMP need fragment responses if an IP packet has the don't frag bit set, or it should be able to fragment the packet if it doesn't have that bit set. > > > On the other hand, the Tigon III > > > is capable of 960 megabits -- about the wire rate limit -- > > > with normal size packets, if you implement software interrupt > > > coelescing (which doesn't help, unless you crank the load up > > > to close to wire speed and/or do more of the stack processing > > > at interrupt time). > > > > > > The Tigon II also has the unfortunate need to download its > > > firmware, and a FreeBSD bug -- the lack of a distinct state > > > for firmware download -- means that the firmware get sent > > > to the card each time the thing is config'ed up, or an IP > > > alias is added to the thing. Plus the damn things are more > > > expensive than the Tigon III based cards. > > > > That's partially because Bill wanted to have a process context > > when he was downloading the firmware, and didn't want to do a > > kernel thread to do it. > > Bill and I discussed this at length, in person. The problem is > that FreeBSD resets the ethernet cards, for fear that it is the > user's intent to recover a hung card tis way. > > The firmware download issue is because there is not a seperate > entry point for firmware download, combined with the initial > gratuitous ARP request being placed on the transmit queue for > the card before it is really up, and the only other alternative > place to hook the firmware download meant that you would have > to hang the ifconfig command -- nad therefore the /etc/rc, and > therefore the boot process -- waiting for th download to complete. > > The upshot of this is that adding a virtual IP address to the > interface is assumed to be a very infrequent event. > > This assumption is incorrect for a large number of IP takeover > protocols (e.g. VRRP), whose timing granularity is small enough > that the firmware download exceeds the interval by a factor of > 10 or more. By the time the alias address assignment (and the > concommitant firmware download) are completed, the takeover > window has been greatly exceeded. > > For all three of the takeover protocols for which there is > published documentation available, this results in the address > being configured on, the firmware downloaded, the takeover > window being missed, losing the contention race for the takeover, > and the address being configured off -- and the firmware being > downloaded yet again. > > Another "amusing" consequence is that, if there is traffic at > the time of the event, the driver goes into a perpetual > "watchdog timeout: resetting" loop, which can oly be resolved > by a system reboot. That certainly sounds like a problem. > > So it gets done when the card is ifconfiged up. I wrote some code for my > > former employer to wait up to 10 seconds for link to show up when the board > > is configured. The alternative is a 'sleep 5' (for fiber boards) or 'sleep > > 8' (for copper boards) after it is ifconfiged up. > > The kernel code is trivial, but it's not in there by default, > and wouldn't need to be there, if there were a firmware download > phase for the card which could be triggered to occur asynchronously > (e.g. so that the driver did not hand user space in the attach). True. > The "sleep" workaround doesn't really work, unless you're willing > to hack up your rc files to deal correctly with a mix of cards; > even so, then it adds an additional ~10 seconds per ti interface > to the boot process, and to any reconfiguration process that can > occur later (and later reconfiguration processes risk going into > the "watchdog timeout: resetting" infinite loop, if there are any > proceses with network links active at the time -- something which > is almost inevitable). Yep. > > Having downloadable firmware is actually a huge advantage. You can do > > things with the Tigon II that just aren't possible with any other chip, > > either because they don't have downloadable firmware, or because the vendor > > won't release firmware source. > > I agree; I'm well aware of a number of people who have done > things to the firmware. The problem is the FreeBSD ethernet > driver architecture doesn't provide for an entry point hook for > asynchronous firmware download. I spent quite a lot of time working on Tigon II boards, in fact. > > This is a problem with the Broadcom Tigon III boards, and to some extent > > with the Tigon II. Basically it looks like the firmware for the Tigon II > > is very hard to get now that 3Com has control of it, and I don't think > > Broadcom will release the Tigon III firmware. (Assuming it is a > > firmware-based chip.) > > The Tigon II firmware is available from their web site. Do you have a URL for that? All I can find is a download page for developer information on mostly obsolete products, not the 3c985B or the Alteon ACEnics that they bought from Alteon/Nortel. > You can license the Tigon III firmware, but I agree, that it > is definitely a step backward, compared to general availability. It is a shame. > > > To put a final nail in the coffin, the card can't offload > > > TCP checksum with VLANs enabled (it breaks). > > > > Probably something that could be fixed in the firmware. Too bad > > the vendor isn't really supporting it very much now. > > The card has been end of lifed. > > > > All in all, the Tigon II is a bad board. > > > > That's your opinion, but I disagree. > > I can more than double my connections per second, and triple > or quadruple my overall throughput by switching from a Tigon II > to a Tigon III, with no other changes. The Tigon III is the > first board to be able to do Gigabit wire rate with normal sized > packets. I'm sure the Tigon III is a better chip -- it's almost inevitable that as time goes on better things will come out. The main drawback there is that you can't get firmware without an NDA, and that hinders development. Ken -- Kenneth Merry ken@kdm.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010929231444.B52304>