Date: Mon, 11 Aug 1997 10:19:59 -0700 (MST) From: Terry Lambert <terry@lambert.org> To: ccsanady@friley01.res.iastate.edu (Chris Csanady) Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: DISCUSS: interface for raw network driver.. Message-ID: <199708111719.KAA15515@phaeton.artisoft.com> In-Reply-To: <199708110552.AAA01853@friley01.res.iastate.edu> from "Chris Csanady" at Aug 11, 97 00:52:24 am
next in thread | previous in thread | raw e-mail | index | archive | help
> I am writing a pseudo device that basically sits on top of a network driver, > does memory management, and exports the buffers to user-space. [ ... ] > A program will use it by opening the special file, and mmapping the > whole chunk of device memory. It will then create an endpoint, > specifying the foreign address, port number, queue length > restrictions, etc. Now it can do IO by specificifying offset, size, > and VCI. (Currently, I am using a series of ioctls for this. They > include allocation, freeing, sending, etc.. of endpoints and buffers) Basically, the inverse of mmap'ing a file, and then sending an address in the mmap'ed region to avoid the copy overhead. I think the main difference I see is that this would allow the buffers to be non-file buffers, so for generated rather than raw file contents, you save a copy, but for files, you are still one user/kernel space copy over the minimum possible. > The obvious drawback is that anyone using this interface will be > able to trash eachothers buffers, but this seems a reasonable > compromise. With only one page in use, there will be no TLB thrashing, > and no overhead for vm mappings. Ideally, it will be as close as > possible to providing a user level network driver without comprimising > overall system integrity. Our initial use will be providing high > performance, low latency communication for our cluster. [ ... ] > Initially, this arhitecture will be used for gigabit and fast ethernet, > although if there are any glaring problems which would prevent use on > other network architectures, I would like to know. Even with ethernet > however, it will allow use of non-standard frame sizes on hardware > which supports it, and will be a huge win. > > Thoughts? What about establishing an anonymous memory mapping? The mmap'ing has long been used to save the copy overhead, by causing the uiomove() to take the transfer fault in kernel space (the user space reference only exists to get a DT entry of some kind). If you mapped a file into a process (specifically, grabbed some anonymous pages from /dev/zero), it seems to me that it would provide the same facilities, without the danger that someone else could stomp on your memory region which was pinned in the KVA space. Obviously, if the mmap() stuff was fixed, the kernel mapping would go away, so this might be a consideration. On the other hand, the per-process descriptor table must be present for your process to be running anyway, and it's unlikely to be LRU'ed out -- especially if, on I/O request, you pin the pages and have the completion unpin the pages? I think the remaining issue is the number of DT entries which need to be rewritten on context switch. With your method, you add one, but this is an artifact of the 4M page size -- you can get the same effect by using a small region, and/or by making mmap() use 4M pages as well (a good optimization in any case). Is there anything I've missed? I'm not sure it's necessary to go to the driver level an push the mapping up from the kernel instead of down from user space... ? Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199708111719.KAA15515>