From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 4 22:09:57 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95E06BEF; Sat, 4 Apr 2015 22:09:57 +0000 (UTC) Received: from mail-pd0-x234.google.com (mail-pd0-x234.google.com [IPv6:2607:f8b0:400e:c02::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40E3F1A2; Sat, 4 Apr 2015 22:09:57 +0000 (UTC) Received: by pdbni2 with SMTP id ni2so1252888pdb.1; Sat, 04 Apr 2015 15:09:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=gLk2lt+UCmj2jB8DDtRaRY+aZIAqvKo+LoI0co80Ep0=; b=zROoRM/jeOULYo7INbmNxyskai/Au44DIhVqpqN6f5MNJpyhG+La3wv4loAWy5RIlu 1gNIqKjQFnhImRqD8mvnSS+lSXEMsxiRQDrMdEacVWYw2y63A2yM3pHklIjzxKN8lits Zo2a+XgkvpYenfIQkK+ZymH92UXajtDSoWkWnxKP95zTXcT3xtRK7zX49tDXB5HhoGoZ EQGiAJ6UynO+6wL+T89xIhpaEf8mjz3UsjTsoq4sqOv1r6J+lz+hkUvcr2fSSnTHPMqB XB0S39iCeHsAeoq/GFbXhu4k6X1VcszQ5XGqycbXwfhWsW5awDNNjXG3JvSmNUBd8qs8 h5Dg== X-Received: by 10.66.160.197 with SMTP id xm5mr15181566pab.51.1428185396598; Sat, 04 Apr 2015 15:09:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.67.2.42 with HTTP; Sat, 4 Apr 2015 15:09:26 -0700 (PDT) In-Reply-To: <20150402214720.GE549@dft-labs.eu> References: <20150321232622.GF14650@dft-labs.eu> <20150327194920.GB18158@dft-labs.eu> <20150330173358.GB9095@dft-labs.eu> <20150402214720.GE549@dft-labs.eu> From: Yue Chen Date: Sat, 4 Apr 2015 18:09:26 -0400 Message-ID: Subject: Re: How to traverse kernel threads? To: Mateusz Guzik , Yue Chen , Oliver Pinter , Benjamin Kaduk , "freebsd-hackers@freebsd.org" , HardenedBSD Core , PaX Team Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2015 22:09:57 -0000 Thank you so much for the suggestions and kind help. On Thu, Apr 2, 2015 at 5:47 PM, Mateusz Guzik wrote: > On Mon, Mar 30, 2015 at 10:59:16PM -0400, Yue Chen wrote: > > > This would be a performance hit. > > > > The cost is to traverse the .text area and some other data in memory. > It's > > linear and only happens after a given time interval, like several > minutes, > > and does not put overhead on the normal execution. So the performance > > overhead won't be high. > > > > This was a remark about a runtime cost of executing kernel code with > with an extra jump in every function. > > > > The only somewhat workable solution I see would require leaving > function > > entry points at constant addresses and only insert jumps to new > locations. > > > > Yes, we do this. For the PIC direct jump/call/"instructions including > > %RIP", we just need to update them. > > > > I have trouble parsing what you wrote here. > > What I wrote here suggests taking existing foo_func, placing a copy at a > new location and putting a jump to that address as the very first > instruction of foo_func. (and zeroing/whatever the rest of old > foo_func). Quite terrible. > > Also quite irrelevant to how you arrive at given func. > > Also see below. > > > > Also runtime relocation of everything definitely has a lot of unknown > unknowns, > > so all in all I would say this is a non-starter. > > > > FreeBSD kernel itself only has 1500+ indirect jumps. There are two types: > > 1. jump table. Like switch-case. 2. Tail call. We have not found other > > types like manual assembly code, or context restoration like that in > > longjmp in libc. Maybe there are some other but we have not found. > > > > Unknown unknowns have the problem of not being known. You have to be > quite skilled in the area to come up with a working solution and I > highly doubt either of us is such a person. > > > > Leave a lot of data at constant addresses, defeating the point to some > > extent. > > > One could consider a different approach where kernel data is randomly > shuffled > > around with some granularity and > > > relevant symbol relocated prior to booting. > > > > Why do you think the data address exposure could be a serious problem? It > > is non-executable and the code is randomized. Although for data itself, > it > > is really hard to protect from tampering and may cause other problems. > > > > I see I wrote 'data', where I should have written '.text' or 'code', > although from the context it should have been clear this refers to > function entry points left behind. > > if you can get away with one function call, you don't need any infoleaks > - you know the address of the function. That's one of the problems with > leaving func entry points at known addresses. > > > Why the randomized data leads to kernel panic? I think if the data goes > > somewhere else, the instruction addressing would follow, or page fault > may > > happen. If randomizing code, the exploit attempt may hit an illegal > > instruction and the kernel panics. > > > > What? > > Anyway I still don't understand how are you trying to achieve > randomisation in the kernel. > > Previous mail suggests you just want to move funcs around and update all > callsites and structs which store addresses to them. > > As outlined with 'struct meh' example (which can be found below) I do > not believe this can work on real-world systems without a major surgery > simply because there is no way to tell what looking like a pointer > should be changed and what should not. I'm happy to be proved wrong. > > Kernel already leaks tons of addresses to userspace in documented ways, > and I'm sure there are tons of situations where it does so as a result > of a bug. > > I believe an alternative implementation I proposed which randomizes > *all* kernel pages prior to booting it would be doable, but given > aforementioned leaks not worth it. > > (well, striclty speaking you would want to move functions around with > respect to each other, but keep the kernel physically contiguous so > that you can still use superpages) > > Your previous mail in this thread suggests you are just starting any > kind of work on kernel-level, which makes this very hard task even > harder. > > Exploitation mitigation is a complicated subject on its own, I recommend > reading what people were doing it have to say. Here is a piece about > KASLR: https://forums.grsecurity.net/viewtopic.php?f=7&t=3367 > > In conclusion I strongly recommend dropping this project for the time > being and focusing on something easier. > > > > return-oriented > > > > > programming (ROP). It is basically a stronger form of ASLR. > > > > > After each randomization procedure, the function return addresses > > > saved in > > > > > the stack are the ``old'' addresses before randomization, so we > need to > > > > > update them to the new addresses. > > > > > That's why we need to get all the stack ranges to find those > addresses. > > > > > > > > > > Also, in kernel, we believe that not only the return addresses in > > > stacks > > > > > need to be updated, there may exist other ``old'' saved instruction > > > (PC) > > > > > addresses in memory. Like in exception handling (maybe, do not > know), > > > > > debugging-purpose code and restartable atomic sequences (RAS) > > > > > implementation. That's why I asked how to traverse all the kernel > > > pages and > > > > > get their virtual addresses here: > > > > > > > > > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html > > > > > > > > > > Now we found that it seems needed to traverse the ``pv_entry'' > > > structure for > > > > > x86_64 MMU. > > > > > > > > > > Another problem is that we do not know if FreeBSD has any form of > > > special > > > > > encodings or offset form for 64-bit instruction addresses (e.g., > saved > > > %RIP) > > > > > on X86_64, instead of hard-coded addresses. For example, using a > 32-bit > > > > > offset instead of the 64-bit full address; and doing what glibc > does > > > for the > > > > > setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the > saved > > > > > register values). > > > > > > > > > > Any suggestion or correction are highly appreciated. > > > > > > > > > > Best, > > > > > Yue > > > > > > > > (Added HardenedBSD core and PaXTeam to CC.) > > > > > > > > Until you can not fixed all of the infoleaks from kernel (try sysctl > > > > -a | grep 0x or similar command) the KASLR and other kernel address > > > > space randomization techniques are easily bypass-able... > > > > > > > > > > I do not believe this can be implemented reliably without some serious > > > tinkering around the kernel. Surely I'm not a live patching expert (not > > > any other kind of expert), but hear my out. > > > > > > It seems proposed approach is to move the kernel around and then > updated > > > all relevant pointers in all pages. > > > > > > I do not believe this can be done reliably without corrupting data, > > > unless a major surgery is performed on the kernel. > > > > > > Consider: > > > struct meh { > > > func_t *m_func; > > > size_t m_len; > > > data *m_buf; > > > }; > > > > > > Here we have 'm_func' which needs to be updated, but we don't know if > > > that's the only pointer so we have to scan the entire struct. But m_buf > > > can have data which looks like kernel pointers (i.e. matches some > kernel > > > funcs) and how will you know not to modify it? What if this is some > sort > > > of a hack and in fact you *should* modify it? > > > > > > CTF or some other solutions like that don't help if you just traverse > > > all allocated buffers since you have no idea what the object you found > > > really is. > > > > > > So, assuming this has to be done in runtime, the only somewhat workable > > > solution I see would require leaving function entry points at contant > > > addresses and only insert jumps to new locations. > > > > > > This would be a performance hit and would leave a lot of data at > > > constant addresses, defeating the point to some extent. > > > > > > Also runtime relocation of everything definitely has a lot of unknown > > > unknowns, so all in all I would say this is a non-starter. > > > > > > One could consider a different approach where kernel data is randomly > > > shuffled around with some granularity and relevant symbol relocated > > > prior to booting. > > > > > > This should provide unique enough layout, which paired with big > > > likelyhood of a kernel panic on first bad exploit attempt may be an > > > acceptable solution. > > > > > > But then the kernel may still have enough info leaks for this to not > > > matter, so I would not be so eager to implement it. > > > > > > That said, what prompted this entire effort? Is there an operating > > > system which got this to work or what? > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik > > > wrote: > > > > >> > > > > >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > > > > >> > When using the following code on kernel module loading: > > > > >> > > > > > >> > > > > > ------------------------------------------------------------------------------------------ > > > > >> > struct thread *td = kdb_thr_first(); > > > > >> > td = kdb_thr_next(td); > > > > >> > > > > > >> > > > > > ------------------------------------------------------------------------------------------ > > > > >> > The kernel panics. > > > > >> > > > > > >> > > > > >> Panics how? > > > > >> > > > > >> Also you can easily see these functions don't lock anything, so it > > > would > > > > >> be assumed you took appropriate locks. > > > > >> > > > > >> Except it seems there routines are supposed to be only used when > > > > >> execution is 'frozen' (e.g. when escaped to the debugger). > > > > >> > > > > >> > > > > > >> > And when printing all threads in proc0 (all kernel threads?): > > > > >> > > > > > >> > > > > > ------------------------------------------------------------------------------------------ > > > > >> > struct proc *p = pfind(0); > > > > >> > FOREACH_THREAD_IN_PROC(p, td) { > > > > >> > uprintf("td: %x\n", td); > > > > >> > } > > > > >> > > > > > >> > > > > >> proc0 is an exported symbol, no need to pfind. > > > > >> > > > > >> > td = curthread; > > > > >> > uprintf("cur td: %x\n", td); > > > > >> > > > > > >> > > > > > ------------------------------------------------------------------------------------------ > > > > >> > The ``curthread'' (from this kernel module running the above > code) > > > is > > > > >> > not > > > > >> > in the 0 process group. > > > > >> > > > > > >> > > > > >> There is no 'curthread from kernel module'. > > > > >> > > > > >> My guess is you do this work from module initializator, and in > that > > > case > > > > >> curthread is the thread which loads the module, and such a thread > is > > > > >> definitely not linked into proc0. > > > > >> > > > > >> Still nobody knows what you are trying to do. > > > > >> > > > > >> -- > > > > >> Mateusz Guzik > > > > > > > > > > > > > > > > -- > > > Mateusz Guzik > > > > > -- > Mateusz Guzik >