Date: Sun, 26 Jan 2014 23:19:44 -0800 From: Juli Mallett <jmallett@FreeBSD.org> To: "Jayachandran C." <c.jayachandran@gmail.com> Cc: "freebsd-mips@freebsd.org" <freebsd-mips@freebsd.org> Subject: Re: More trapframe panics Message-ID: <CACVs6=8ie2x16c4bafG4zaJS4UV%2BXT1ooJkm%2BdwT5%2BdSS%2B-p0Q@mail.gmail.com> In-Reply-To: <CA%2B7sy7ATMTjdWD5%2BNYCJPb=AWmeJoUwbgRdf=%2Bb4uYsJO0UUfw@mail.gmail.com> References: <52E42A1B.3040907@rewt.org.uk> <CACVs6=--Qy_8poWdHdCXYKqkO22=dvHhW8=Uma8kLR%2BhCoZDxw@mail.gmail.com> <CA%2B7sy7ATMTjdWD5%2BNYCJPb=AWmeJoUwbgRdf=%2Bb4uYsJO0UUfw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jan 26, 2014 at 11:06 PM, Jayachandran C. <c.jayachandran@gmail.com>wrote: > Hi Juli, > > On Sun, Jan 26, 2014 at 3:36 AM, Juli Mallett <jmallett@freebsd.org> > wrote: > > > > This happens easily on 64-bit MIPS because due to slightly-crummy design > > on our part there's proportionally less room on the stack than on a > 32-bit > > system. Several people have nebulous plans to address the problem of the > > stack being too small, but I don't know of anyone intending concrete > action > > going forward. > > I had not seen the issue so far. > Honestly, I haven't seen it much either, certainly not as much as many people seem to report on the lists, and I built lots of packages, etc. But clearly my workload was different or something has changed in the kernel, but it's definitely a real problem. > When I had done the pmap changes ealier, I had thought of adding the > option to use a higher order page for kernel stack and using the KSEG0 > address as the stack pointer, instead of using the wired entry as we > do now. Is there any reason this will not work. I don't have a problem with physically-contiguous kernel stacks and use of KSEG0/XKPHYS. It's better in some ways, and I'd worry about a system on which finding 4 physically-contiguous pages was impossible. We can't swap out kstacks then, but that's probably okay. My major concern is that then a kernel stack overflow is probably a lot harder to detect, and an attacker probably has other interesting kernel data structures nearby. I'm sure there's other drawbacks to consider, but it'd simplify a lot of bad code, and it seems worthwhile as an intermediate step anyway, even if it does mean MIPS can't support guard pages for the kernel stack. To recapitulate other constraints for anyone who might be reading with interest, and on the off chance something useful shakes out: You really need the stack to be entirely within wired pages or you can take a fault which is unserviceable because you'll end up endlessly faulting for the kernel stack pointer if you hit the slow fault path and not just TLB refill, which is unfortunately not reliable on at least some CPUs. You could write very careful code to load the stack pointer from PCPU and bounds check sp and proceed very carefully, but I'm not sure we really want or need to go there. That said, it only matters for faults taken within the kernel; anything coming from userland including interrupts and syscalls will load sp and start at the top of the stack anyway, so you could at least avoid adding the overhead for the most performance-critical exception handlers. I know that on some CPUs we care about, having to use the extra wired TLB entry is a real pain. There are things we could do to lessen this pain, such as getting rid of the wired PCPU TLB entry. There's two possibilities I'd suggest for that, and only one of them relies on kstacks not being shared between CPUs: (1) Move to keeping the PCPU pointer in "gp". (2) Always store a PCPU pointer at the top of the kernel stack, adding a little indirection to load PCPU, but not a lot. It's nice to have the PCPU wired because you do want to be able to access it quickly from things like exception handlers, which gets messier with the "gp" approach, since then you have to load "gp" in some annoying manner when coming from userland. Just some thoughts. I'd be delighted to see any forward-moving approach that's actually taken :) Thanks, Juli.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACVs6=8ie2x16c4bafG4zaJS4UV%2BXT1ooJkm%2BdwT5%2BdSS%2B-p0Q>