Date: Sun, 16 Feb 2003 23:16:45 -0800 From: "Sam Leffler" <sam@errno.com> To: "Justin T. Gibbs" <gibbs@scsiguy.com>, "Perforce Change Reviews" <perforce@freebsd.org> Subject: Re: PERFORCE change 24844 for review Message-ID: <311a01c2d654$87c7cb50$52557f42@errno.com> References: <200302082030.h18KUuR9063864@repoman.freebsd.org> <47870000.1045456722@aslan.scsiguy.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > http://perforce.freebsd.org/chv.cgi?CH=24844 > > > > Change 24844 by sam@sam_ebb on 2003/02/08 12:30:08 > > > > bus_dma'd em driver (works on x86, not on sparc) > > replace rx logic with jumbo buffers (needs more work) > > There are lots of things that are a bit wierd about how this > was bus dma'ed... > > o The TX and RX rings are allocated separately instead of as a > single chunk. While on the x86, this doesn't matter > since each chunk is exactly a PAGE_SIZE, on Alpha, this will > likely result in 4K of wasted space allocated for each ring > due to its larger page size. I don't know that any implementation > will track these kinds of allocations at sub-page granularity, > but if tag's size were not hardcoded, at least the implementaiton > would have a chance to use the space. > This was a straight transcription from the previous code. I don't recall how much memory is involved but I intentionally allocated them separately to minimize risk of contigmalloc failure due to fragmentation. > o The TX/RX rings are not synced when updated prior to informing > the adapter of a change in the queues. I believe that NetBSD has > the concept of requesting that stuff be mapped in "consistent memory", > but the allocator need not honor that. If the consistent mapping succeeds, > sync operations cost either nothing or are significantly cheaper. > Yes, I mentioned to some other folks that I hadn't sync'd the descriptor/ring. Ideally the sync should be on a per-descriptor basis and I wasn't sure if it would be better to break things up to avoid sync'ing the entire ring. Interestingly there's been discussion recently on the netbsd mailing lists about devices that assume cache-coherent bus implementations (e.g. i82557) and also about adding support for non-cached memory. The x86 is a poor platform to develop on for this sort of stuff; it's time for me to get a sparc test machine. > o bus dmamaps have a lifetime of a single packet. While loading a map > is usually cheap, allocating a map may be very costly. The implementation > may use the allocation of a map as a hint to increase its mapping space, > bounce buffer pool, or to perform some other action that will decrease > the chance that a load will block should all maps currently allocated > against the tag be used at once. The map may be a fairly large object > depending on the needs of the implementation. I would suggest changing > the maps to have the same lifetime as the TX descriptors. > Ok. I believe this follows techniques taken from netbsd which may not be appropriate. I'll look at it. Since there haven't been any equivalent drivers to crib from for freebsd most times I've referred to netbsd or openbsd drivers. > o The driver allocates a duplicate "S/G list" for each tx descriptor. > If the driver were converted to a "continuation style" and had the > "bottom half" of packet processing performed in the callback, the > mappings could be placed directly into the TX descriptors. One of my > original complaints to thorpej about his bus dma implmentation was > this forced "extra copy" of the mapping information in a format that > neither the drivers nor most implementations (Not in IOMMU or SGMAP > format) could directly pass on to hardware. On devices that can support > large S/G lists, this can make dmamaps very costly (at 16bytes/element > and many I/O devices supporting a list of 256 or more elements, each > map could require a page or more!). The callback allows the > implementation to reuse this space as soon as the callback returns. > Having drivers make a copy that they cannot directly use negates > this benefit. > For network drivers the size of the s/g list is typically very small. I'll have to look, but moving the logic to the callback routine may require shuffling state around. I know it has consequences for the crypto drivers (ubsec, hifn). Thanks for the comments. Sam To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe p4-projects" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?311a01c2d654$87c7cb50$52557f42>