Date: Tue, 02 Jan 2001 12:16:09 -0800 (PST) From: John Baldwin <jhb@FreeBSD.org> To: Matthew Jacob <mjacob@feral.com> Cc: alpha@FreeBSD.org, Doug Rabson <dfr@nlsystems.com> Subject: RE: Interrupt threads Message-ID: <XFMail.010102121609.jhb@FreeBSD.org> In-Reply-To: <Pine.LNX.4.21.0101021150361.14503-100000@zeppo.feral.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 02-Jan-01 Matthew Jacob wrote: > On Tue, 2 Jan 2001, John Baldwin wrote: > >> >> On 28-Dec-00 Doug Rabson wrote: >> > I started trying to work on changing the interrupt system so that the >> > ithread runs with i/o interrupts disabled, which would allow us to remove >> > the ugly code which tries to disable the interrupt sources. >> > >> > Unfortunately I immediately came up against a brick wall - the very first >> > interrupt interrupted a proc which owned Giant and the ithread also wanted >> > Giant. It tried to block, which switched back to the Giant owner with >> > interrupts enabled, causing the interrupt to fire again. >> > >> > I'm not sure what the right approach to solving this is. Possibly we could >> > 'lend' the mutex holder the IPL of the blocking thread in a similar way to >> > priority propagation. Another idea is to link thread IPL directly to its >> > priority so that when the ithread lends its priority, the mutex owner will >> > automagically run with raised IPL. >> >> I detailed what I think is a way of using IPL to do this in a pair of >> e-mails >> to Drew that Matt was copied on. I'll go dig them up and forward them to >> here. >> However, tying IPL to a process or a mutex is a mistake I think. You can't >> tie >> it to Giant because Giant is going away eventually. You can't tie it to a >> process because the first context switch that doesn't switch into the >> ithread >> will cause the interrupt to fire again. Instead, you basically have to futz >> with the trapframe in the interrupt handler so it doesn't lower IPL upon >> returning from teh interrupt, and don't lower the IPL again until the >> interrupt >> handler completes and the ithread finishes. Interacting properly with ast() >> on >> this makes it a bit more tricky. Let me go dig up my e-mails and forward >> them >> to the list. > > Please do. I agree this is tricky. > > I'll reiterate what I said a while back (insofar as I recall) and add: > > 1. It should be a bug to have the scheduler switch to a process that returns > to user space while leaving ithreads blocked. Hell, it should be a bug for > any > process that enters kernel context and acquires an SMP lock to not release it > before returning to user space. This is already true... > 2. Giant is like any other mutex in this context- whoever holds Giant should > be resumed if an ithread hits Giant. As soon as they drop Giant there should > be an immediate cpu_switch which had better resume the [ one of the ] > ithread[s] that had been blocked. Since ithreads are a higher priority than any other process (they are in their own separate run queue for goodness sake) this is already true.. > 3. You should not have to do anything with a trapframe until you actually are > returning from an interrupt. You can't assume that a 'return from interrupt' > can return with IPL set to non-zero. This is such a POLA violation that I > wouldn't expect it to work much of anywhere. Basically, you should take an > interrupt and immediately have the ithread for it run (unless fast- and until > we can get this right, only allow one non-fast interrupt at a time for > alpha). If this ithread hits a lock and can't get it, the holder of that lock > (whether Giant, Fred, Foo, Bar) gets resumed until it drops that lock, > whereupon the ithread should be resumed. This can repeat until the ithread is > done, whereupon a real return from interrupt can occur. For nested interrupts (clock interrupt during a normal interrupt for example) the IPL can _already_ be non-zero in the trapframe. However, the lock contention you describe already holds true because ithreads have such a high priority. > Unfortunately, I think we've begun to experience what Solaris had to put up > with in this respect, but so it goes. I would rather have a lot more in the > above model but this strikes me as something which very closely adheres to > what was the advertised SMP model we got from BSDi- and seems to have some > commented out code about this. The light weight ithreads will make performance slightly better, but they won't solve this problem. When an ithread blocks on a mutex, you still have the problem of switching to another process and running it while still needing to disable the interrupt. That we won't ever get away from. > 4. What defeated me immediately upon looking at Doug's stuff is all of the > thread pri seems to be a software construct, so has no real hardware > linkage. Additionally, I did not take the time to try and find an easy way to > find and reschdule the proc that holds a lock if I'm trying to acquire it > from > an ithread and am blocking. Seems to be a bit of work there. Yes, you can't tie the IPL to a thread. The IPL is tied to two events: 1) An interrupt trigger raises the IPL 2) Handling the interrupt lowers the IPL back to where it was. Since 2) is now asynchronous with respect to 1) sort of, we have to be a bit more tricky with when we lower the IPL rather than letting the machine do it for us. > -matt -- John Baldwin <jhb@FreeBSD.org> -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-alpha" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?XFMail.010102121609.jhb>