Date: Fri, 8 Jan 1999 15:55:37 -0500 From: Chuck Cranor <chuck@research.att.com> To: freebsd-atm@FreeBSD.ORG Cc: Poul-Henning Kamp <phk@FreeBSD.ORG>, Tim Salo <salo@networkcs.com>, "John D. DeHart" <jdd@arl.wustl.edu>, Zubin Dittia <zubin@ccrc.wustl.edu>, guru@arl.wustl.edu Subject: Re: ATM in FreeBSD, status requested! Message-ID: <19990108155537.A1179348@research.att.com>
next in thread | raw e-mail | index | archive | help
greetings-
i've received several copies of messages from this list regarding
the status of my BSDATM code WRT FreeBSD over the holidays. i have
not had time to reply until now due to the holidays and the context
switch overhead of starting a new job (I'm now at AT&T Labs-Research
and am still trying to get my world back in order).
from reading the archive of this list it seems pretty clear to me
that there are some misconceptions about the goals of the BSDATM
project that has lead to some unfair and inaccurate criticism of it.
this is partly my fault for not providing better documentation for
what i was trying to do with BSDATM. over the past couple of days
i've spent some time looking at the HARP code in -current and
re-reading the HARP web page to get a better handle on the goals of
HARP. i'd like to now clarify the goals of the BSDATM project and how
it fits in with respect to HARP.
Goals of the BSDATM project:
at the time i started BSDATM there were no freely available ATM
device drivers for BSD, the only thing out there was Werner
Almesberger's stuff for Linux. this was a problem for us (wustl.edu)
because we were using BSD as a research platform, but we were unable
to do anything with ATM under BSD. i decided to address this problem
by creating BSDATM. my goals in creating BSDATM were:
[1] to write a device driver for the Efficient Networks
midway-based ATM cards that:
- is robust and not likely to crash your system.
- is portable across BSD operating systems and ENI card types (e.g.
both SBus and PCI).
- is well commented and easy to debug (even via e-mail).
- provides reasonable performance with minimal data copying.
- supports the midway's bandwidth settings (QoS)
[2] to write an upper-level ATM networking framework that has as
small a footprint on the BSD kernel and operating system as
possible while still meeting our research needs. our needs include:
- being able to transmit and receive plain AAL0 cells and AAL5
frames between our host systems and our ATM-based multimedia
devices. for example, we have a relatively simple video
source/sink box called an "MMX" that communicates with our
BSD-based video server using AAL0 netnatm sockets. the AAL0
connection used for transmitting data from the video server to
the MMX requires the use of the ENI's bandwidth control because
the MMX box has limited buffering and no way to throttle the
sender. the AAL0 connection used for recording real-time video
data from an MMX puts a heavy interrupt load on the receiving
system (interrupt per cell) and is a good stress test for the
ATM device driver. another example is Jon Turner's next
generation of ATM switch which is being distributed as part of
the "Gigabit Network Technology Distribution" sponsored by the
NSF and wustl. the new switch's control processor will be
a PC running BSD. the PC communicates with the switch by
sending and receiving control data over a netnatm socket.
- being able to transmit and receive IP packets over PVCs without
clashing with VCs currently in use by netnatm sockets (and
vice-versa).
- being able to take advantage of the hardware demux feature of the
ATM host interface (netnatm does this).
again, note that the goals of BSDATM are to produce a robust
lower-level device layer capable of supporting all of midway's
features combined with a lean (minimal footprint) upper-level ATM
layer. thus, supporting SVCs, complex ATM signaling protocols,
etc... was never a goal of BSDATM and is not included in BSDATM's
higher-level ATM networking framework. i chose to do it this way
because i was working on BSDATM alone in my spare time (when not
working on my doctoral research) and i did not have the time to do
the standards research necessary to adequately support those
upper-layer features in BSDATM. instead, i focused my efforts on
the lower levels of ATM support with the idea that future work in
the upper layers could build on my lower-level work and either
replace or build on my lean upper-layer ATM framework.
so despite the assertions made on this list that BSDATM is a
poorly written and poorly supported piece of work i believe that
BSDATM has been quite successful in reaching the goals set for it.
i have had great success in the area of getting other people to
use and/or build on it (both in the research area and in the ISP
business). for example, in addition to the ATM projects at
wustl.edu, Kenjiro Cho has done the ALTQ work and helped maintain
the FreeBSD version of the driver. I worked with Anne Hutton and
the folks at ISI/CARIN vie e-mail to add support for the Adaptec
version of the midway card to the driver and get the bandwidth
limiting stuff going for them. [The Adaptec version of the card
has a better DMA interface than the older ENI version.] The
driver has been ported to all the free BSD systems and also to
other operating systems such as Mach. The number of pending bug
reports relating to the handling the midway device is zero ---
both Kenjiro and I have worked hard to eliminate the final kinks
from the driver and we have not had any problems with it for many
months. In addition to my midway work, I have been working with
some folks who want to add support for ENI's Lanai-based 25Mbps
ATM card to FreeBSD, NetBSD, and OpenBSD. for that project it
appears that we will need to eventually get PPP over ATM working.
Goals of HARP:
from looking at the HARP web page (http://www.msci.magic.net/harp/) it
should be pretty clear to everyone that the HARP folks' major focus is
on supporting the standard protocols at the upper layers of the ATM
framework and the bulk of their work lies in that area rather than in
lower-level driver support (where BSDATM's focus lies). i believe
that in some sense the HARP work and the BSDATM work complement each
other rather than conflict. looking at the code size footprint of the
the projects confirms this:
HARP: total lines of code = ~80000 (see
http://www.msci.magic.net/harp/usenix/ppframe.htm, the
"lessons learned I" slide)
the HARP midway driver is only ~6% of those ~80000 lines of code
BSDATM: total lines of code = ~5700
the midway driver is about 70% of BSDATM
I spent some time looking at the HARP version of the midway driver and
comparing it to my driver. some points of interest include:
- the HARP driver uses structures to access midway device registers,
the BSDATM midway driver uses the more portable bus_space
macros with offsets to access the card. i believe FreeBSD is
moving in the bus_space/bus_dma direction (according to
conversations i've had with justin gibbs).
- the HARP driver does not support Adaptec versions of the card.
- the HARP driver does not support sbus versions of the card and may
not handle byte ordering conflicts properly.
- the HARP driver does not test the DMA engine at boot time to verify
proper operation.
- the HARP driver does not dynamically determine what DMA burst
sizes are valid on a specific system and does not handle the
strict DMA alignment required on some systems (e.g. the
"alburst" restriction common on sparcs). Instead the HARP
driver hardwares the max DMA burst to 8 words and doesn't make
use of 16 word DMA bursts.
- the HARP driver doesn't support setting bandwidth limits on VCs and
thus only uses the channel 0 transmit buffer (BSDATM supports
multiple transmission channels).
- the HARP driver doesn't appear to check for bogus AAL5 pdu length
values in the AAL5 frame.
- the HARP driver requires the allocation of an mbuf even if the
received frame is in error (e.g. CRC error). the "receiver
to lock" if the mbuf allocation fails.
- the HARP driver clears the "VCI_IN_SERVICE" bit without advancing
its servread pointer in error conditions. strictly speaking,
i don't believe this should be allowed.
- the HARP driver may not handle DMA descriptor wraparound properly
(Kenjiro detected that the empty and full conditions for the
DMA descriptors are the same and thus you've got to be
careful.) The transmit buffer in HARP is handled properly.
- the HARP driver doesn't have a VC drain state to drain off DMA and
I/O operations when a VC is being closed and thus may
prematurely free resources (often happens with our multimedia
ATM devices).
- the HARP driver handles unaligned data by doing an in-place bcopy
(KM_COPY)... it appears to be re-writing a cluster mbuf's
data area (M_EXT). i believe that is illegal (especially
when the data area is referenced by more than one mbuf header).
- BSDATM driver doesn't have code to read the seeprom on the ENI card
(the upper layer ATM code doesn't require it)
- HARP dynamically allocates receive buffers on the card, but the
allocations are of a fixed size. BSDATM allocates fixed sized
buffers on the card at boot time (doesn't have a memory allocator).
- HARP has code to talk to the SUNI chip (PHY chip) to read the
counters and clear SUNI interrupts, BSDATM doesn't.
- both BSDATM and HARP use a header in front of the data to pass data
around (in BSDATM it is the atm_pseudohdr structure, in HARP
it is un-named).
- BSDATM caches allocated mbufs if an "out of DMA descriptor" error
condition occurs so that they don't have to be reallocated
later.
- BSDATM has a DDB-based debugging interface that is useful for
debugging other people's problems via e-mail.
that's a partial list of differences i came up with after spending a
day or two looking at code. my impression is that there is a
significant amount of duplicate work to be done at the driver level to
get HARP's midway support up to speed with BSDATM. likewise, there is
a significant amount of duplicate work to be done to get BSDATM's
upper layer ATM support caught up with HARP.
based on that, i believe that it may be worthwhile to look at the
driver/upper-layer interface differences between the two systems and
seeing if the two interfaces can be merged so that both HARP and
BSDATM can share lower level drivers. if that can be done, then it
may be possible to get "the best of both worlds"? it isn't fully
clear to me how ALTQ fits into this picture.
chuck
--
Chuck Cranor <chuck@research.att.com>
Senior Technical Staff Member, AT&T Labs-Research
Room B180, 180 Park Ave, Florham Park NJ, 07932
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-atm" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19990108155537.A1179348>