Date: Thu, 01 Aug 2002 13:04:14 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: Alan Cox <alc@cs.rice.edu>, Peter Wemm <peter@wemm.org>, John Baldwin <jhb@FreeBSD.org>, freebsd-smp@FreeBSD.org Subject: Re: INTR_MPSAFE network drivers Message-ID: <3D49943E.65FF5AF0@mindspring.com> References: <XFMail.20020729175342.jhb@FreeBSD.org> <20020730000345.E0D9F2A7D6@canning.wemm.org> <15686.48913.150407.307190@grasshopper.cs.duke.edu> <20020730171226.GA26599@cs.rice.edu> <15688.22002.772933.316104@grasshopper.cs.duke.edu> <20020801021235.GD9934@cs.rice.edu> <15689.8042.488037.968605@grasshopper.cs.duke.edu> <20020801174855.GE9934@cs.rice.edu> <15689.31447.889898.252092@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Andrew Gallatin wrote: > Unless I'm mistaken, that means that non-legacy network drivers will > not be able to move out from under Giant until there is a way to > allocate mbufs which is is certain to never call into the vm system > for anything (eg, make M_DONTWAIT really mean M_DONTWAIT). Hmm.. we > could have a mclalloc kproc (like -stable does for MCLBYTES > > PAGE_SIZE), which would be awakened by the mbuf code in the case that > it could not fufill an interrupt time allocation which required > calling malloc. > > I suppose another way to go would be if drivers are prepared to drop > their driver locks and get Giant before calling into m_get*. I think > most drivers can easily deal with m*get returning NULL, but that it > would complicate the locking quite a bit to have to drop driver locks > in the middle of the receive path, get Giant, call m_get*, drop Giant, > then get driver locks back. ALL drivers must deal with m_get* returning NULL. It's possible to run out of mbufs entirely, when one runs out of memory, and removing mbufs from the interrupt receive rings is only permitted when there are empty mbufs available to replace the ones you intend to remove (and shove up the stack). Therefore, just returning NULL when the per CPU pool is empty is a valid approach to solving the problem. The only real question is how to refill the per CPU pool in a way that is (effectively) asynchronous, without adding a pool lock. This should be easy to accomplish with a freelist as a linked list, where the update of the list head pointer can be done in an atomic way. It is really very tempting to preallocate kernel mappings for all memory like this that you care about being able to obtain any non-blocking allocations out of; this was the case with the mbuf allocator before it was rewritten, and was the case will all the zalloci zone allocators. I think that the allocation requirements for statistics keeping requiring obtaining a memory manager lock were in fact a mistake; I would rather have statistical statistics (meaning a snapshot, not guaranteed to match reality exactly), and no locks, than the locks that are there now. This was discussed as "future work" when the current code was committed, and I raised this spectre at that time, but it was never committed. One way to face that problem would be to preallocate mappings, but *not* explicitly assign them to a zone. This also resolves the static tuning issues for mbufs and mbuf clusters, which, today, require a precommit of a chunk of KVA space whse size is already known. By creating a mapping pool of KVA space to begin with, *all* memory has the preexisting mappings, which then lets you allocate without creating mappings, and without allocating the mappings to a particular pool (though a rough two tier map for system vs. per-CPU page-pools would be useful). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D49943E.65FF5AF0>