Date: Mon, 30 Mar 2015 22:59:16 -0400 From: Yue Chen <ycyc321@gmail.com> To: Mateusz Guzik <mjguzik@gmail.com>, Oliver Pinter <oliver.pinter@hardenedbsd.org>, Yue Chen <ycyc321@gmail.com>, Benjamin Kaduk <bjk@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, HardenedBSD Core <core@hardenedbsd.org>, PaX Team <pageexec@freemail.hu> Subject: Re: How to traverse kernel threads? Message-ID: <CAKtBrB5WB0eCQqwwUXCKM8RVGaqeiatyKisyQZist4Zq4tJPKQ@mail.gmail.com> In-Reply-To: <20150330173358.GB9095@dft-labs.eu> References: <CAKtBrB4h13ZFJ=V0fvkDeTG-L6=e5Uz9%2BHfYc8vY523Y3X6N0A@mail.gmail.com> <20150321220246.GE14650@dft-labs.eu> <CAKtBrB5KNqt6UJ1R_BQpPfTvQZdUzGvZZtT7Uz5qd4VrrfgEdw@mail.gmail.com> <20150321232622.GF14650@dft-labs.eu> <alpine.GSO.1.10.1503221644440.22210@multics.mit.edu> <CAPQ4ffuszSi%2B_SopJdCkoFr4OoY9=BZVbO6oo_s0sKrn8Rgjrw@mail.gmail.com> <CAKtBrB6ZF2FVExmDd%2Bt8yFpN0H7xHwaieWgvryR535Vc2cNBjw@mail.gmail.com> <20150327194920.GB18158@dft-labs.eu> <CAKtBrB6qeNTyemG-ws7X_OKcB2t-6muxJrOZ1Eyr0CRjjso5mQ@mail.gmail.com> <CAPQ4fft%2BmNJObMSw8SFvsweZsB-UKNUV2yghg4jB%2BdvSg9ho2w@mail.gmail.com> <20150330173358.GB9095@dft-labs.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
> This would be a performance hit. The cost is to traverse the .text area and some other data in memory. It's linear and only happens after a given time interval, like several minutes, and does not put overhead on the normal execution. So the performance overhead won't be high. > The only somewhat workable solution I see would require leaving function entry points at constant addresses and only insert jumps to new locations. Yes, we do this. For the PIC direct jump/call/"instructions including %RIP", we just need to update them. > Also runtime relocation of everything definitely has a lot of unknown unknowns, so all in all I would say this is a non-starter. FreeBSD kernel itself only has 1500+ indirect jumps. There are two types: 1. jump table. Like switch-case. 2. Tail call. We have not found other types like manual assembly code, or context restoration like that in longjmp in libc. Maybe there are some other but we have not found. > Leave a lot of data at constant addresses, defeating the point to some extent. > One could consider a different approach where kernel data is randomly shuffled around with some granularity and > relevant symbol relocated prior to booting. Why do you think the data address exposure could be a serious problem? It is non-executable and the code is randomized. Although for data itself, it is really hard to protect from tampering and may cause other problems. Why the randomized data leads to kernel panic? I think if the data goes somewhere else, the instruction addressing would follow, or page fault may happen. If randomizing code, the exploit attempt may hit an illegal instruction and the kernel panics. > Is there an operating system which got this to work or what? Not yet. On Mon, Mar 30, 2015 at 1:33 PM, Mateusz Guzik <mjguzik@gmail.com> wrote: > On Fri, Mar 27, 2015 at 11:20:39PM +0100, Oliver Pinter wrote: > > On Fri, Mar 27, 2015 at 9:55 PM, Yue Chen <ycyc321@gmail.com> wrote: > > >> Except it seems there routines are supposed to be only used when > > >> execution is 'frozen' (e.g. when escaped to the debugger). > > > > > > It means probably we can run the code in ``smp_rendezvous()'' function, > > > right? > > > > > >> Still nobody knows what you are trying to do. > > > > > > We are trying to enhance FreeBSD's security by randomizing kernel code > basic > > > blocks periodically at runtime, to mitigate the attacks like > return-oriented > > > programming (ROP). It is basically a stronger form of ASLR. > > > After each randomization procedure, the function return addresses > saved in > > > the stack are the ``old'' addresses before randomization, so we need to > > > update them to the new addresses. > > > That's why we need to get all the stack ranges to find those addresses. > > > > > > Also, in kernel, we believe that not only the return addresses in > stacks > > > need to be updated, there may exist other ``old'' saved instruction > (PC) > > > addresses in memory. Like in exception handling (maybe, do not know), > > > debugging-purpose code and restartable atomic sequences (RAS) > > > implementation. That's why I asked how to traverse all the kernel > pages and > > > get their virtual addresses here: > > > > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html > > > > > > Now we found that it seems needed to traverse the ``pv_entry'' > structure for > > > x86_64 MMU. > > > > > > Another problem is that we do not know if FreeBSD has any form of > special > > > encodings or offset form for 64-bit instruction addresses (e.g., saved > %RIP) > > > on X86_64, instead of hard-coded addresses. For example, using a 32-bit > > > offset instead of the 64-bit full address; and doing what glibc does > for the > > > setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the saved > > > register values). > > > > > > Any suggestion or correction are highly appreciated. > > > > > > Best, > > > Yue > > > > (Added HardenedBSD core and PaXTeam to CC.) > > > > Until you can not fixed all of the infoleaks from kernel (try sysctl > > -a | grep 0x or similar command) the KASLR and other kernel address > > space randomization techniques are easily bypass-able... > > > > I do not believe this can be implemented reliably without some serious > tinkering around the kernel. Surely I'm not a live patching expert (not > any other kind of expert), but hear my out. > > It seems proposed approach is to move the kernel around and then updated > all relevant pointers in all pages. > > I do not believe this can be done reliably without corrupting data, > unless a major surgery is performed on the kernel. > > Consider: > struct meh { > func_t *m_func; > size_t m_len; > data *m_buf; > }; > > Here we have 'm_func' which needs to be updated, but we don't know if > that's the only pointer so we have to scan the entire struct. But m_buf > can have data which looks like kernel pointers (i.e. matches some kernel > funcs) and how will you know not to modify it? What if this is some sort > of a hack and in fact you *should* modify it? > > CTF or some other solutions like that don't help if you just traverse > all allocated buffers since you have no idea what the object you found > really is. > > So, assuming this has to be done in runtime, the only somewhat workable > solution I see would require leaving function entry points at contant > addresses and only insert jumps to new locations. > > This would be a performance hit and would leave a lot of data at > constant addresses, defeating the point to some extent. > > Also runtime relocation of everything definitely has a lot of unknown > unknowns, so all in all I would say this is a non-starter. > > One could consider a different approach where kernel data is randomly > shuffled around with some granularity and relevant symbol relocated > prior to booting. > > This should provide unique enough layout, which paired with big > likelyhood of a kernel panic on first bad exploit attempt may be an > acceptable solution. > > But then the kernel may still have enough info leaks for this to not > matter, so I would not be so eager to implement it. > > That said, what prompted this entire effort? Is there an operating > system which got this to work or what? > > > > > > > > > > > > > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik <mjguzik@gmail.com> > wrote: > > >> > > >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > > >> > When using the following code on kernel module loading: > > >> > > > >> > > ------------------------------------------------------------------------------------------ > > >> > struct thread *td = kdb_thr_first(); > > >> > td = kdb_thr_next(td); > > >> > > > >> > > ------------------------------------------------------------------------------------------ > > >> > The kernel panics. > > >> > > > >> > > >> Panics how? > > >> > > >> Also you can easily see these functions don't lock anything, so it > would > > >> be assumed you took appropriate locks. > > >> > > >> Except it seems there routines are supposed to be only used when > > >> execution is 'frozen' (e.g. when escaped to the debugger). > > >> > > >> > > > >> > And when printing all threads in proc0 (all kernel threads?): > > >> > > > >> > > ------------------------------------------------------------------------------------------ > > >> > struct proc *p = pfind(0); > > >> > FOREACH_THREAD_IN_PROC(p, td) { > > >> > uprintf("td: %x\n", td); > > >> > } > > >> > > > >> > > >> proc0 is an exported symbol, no need to pfind. > > >> > > >> > td = curthread; > > >> > uprintf("cur td: %x\n", td); > > >> > > > >> > > ------------------------------------------------------------------------------------------ > > >> > The ``curthread'' (from this kernel module running the above code) > is > > >> > not > > >> > in the 0 process group. > > >> > > > >> > > >> There is no 'curthread from kernel module'. > > >> > > >> My guess is you do this work from module initializator, and in that > case > > >> curthread is the thread which loads the module, and such a thread is > > >> definitely not linked into proc0. > > >> > > >> Still nobody knows what you are trying to do. > > >> > > >> -- > > >> Mateusz Guzik <mjguzik gmail.com> > > > > > > > > -- > Mateusz Guzik <mjguzik gmail.com> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKtBrB5WB0eCQqwwUXCKM8RVGaqeiatyKisyQZist4Zq4tJPKQ>