From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 30 17:34:03 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 952A4319; Mon, 30 Mar 2015 17:34:03 +0000 (UTC) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17999766; Mon, 30 Mar 2015 17:34:03 +0000 (UTC) Received: by wibgn9 with SMTP id gn9so140056940wib.1; Mon, 30 Mar 2015 10:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=6Ccns7CxnvzFZsbJBcxnHu1K/Qmyg+8RABcFYq+sm1g=; b=siKQTFbzmW3wwCWh2pSXDyRS+z7GTMNQ3i4dx5ojjEGgVyjuEEtUM+HfmT2iaAmJqX 4xzGAzYCRZLcPqJ2QNqbaZWDTB5d1gx3ilng1ZQJKU6fvoSvp4kTfz7Hr0f7D3K974sf I7saAq6ReYuvExZqBnFuO8ZqNT3RaMPEDgLwtXxXXRBKdYfENQJOUTsOCPUt3trVhI1S vKGKN9o3bEijZEsCE8DqmGjH0OLA6DF1sMfyNQyex3mJCxNPzQM2WIv0ZQ8NffYNno9H jiWp4V985NES6r6qY77pH3kSNVJGW3xXjm/jv2Ff9OF4getKSvA8Sv1hwi7FPyfRaM7+ P9ug== X-Received: by 10.194.185.68 with SMTP id fa4mr64303925wjc.111.1427736841623; Mon, 30 Mar 2015 10:34:01 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id uo6sm16650017wjc.49.2015.03.30.10.34.00 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Mon, 30 Mar 2015 10:34:00 -0700 (PDT) Date: Mon, 30 Mar 2015 19:33:58 +0200 From: Mateusz Guzik To: Oliver Pinter Subject: Re: How to traverse kernel threads? Message-ID: <20150330173358.GB9095@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Oliver Pinter , Yue Chen , Benjamin Kaduk , "freebsd-hackers@freebsd.org" , HardenedBSD Core , PaX Team References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> <20150327194920.GB18158@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: PaX Team , Benjamin Kaduk , HardenedBSD Core , Yue Chen , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2015 17:34:03 -0000 On Fri, Mar 27, 2015 at 11:20:39PM +0100, Oliver Pinter wrote: > On Fri, Mar 27, 2015 at 9:55 PM, Yue Chen wrote: > >> Except it seems there routines are supposed to be only used when > >> execution is 'frozen' (e.g. when escaped to the debugger). > > > > It means probably we can run the code in ``smp_rendezvous()'' function, > > right? > > > >> Still nobody knows what you are trying to do. > > > > We are trying to enhance FreeBSD's security by randomizing kernel code basic > > blocks periodically at runtime, to mitigate the attacks like return-oriented > > programming (ROP). It is basically a stronger form of ASLR. > > After each randomization procedure, the function return addresses saved in > > the stack are the ``old'' addresses before randomization, so we need to > > update them to the new addresses. > > That's why we need to get all the stack ranges to find those addresses. > > > > Also, in kernel, we believe that not only the return addresses in stacks > > need to be updated, there may exist other ``old'' saved instruction (PC) > > addresses in memory. Like in exception handling (maybe, do not know), > > debugging-purpose code and restartable atomic sequences (RAS) > > implementation. That's why I asked how to traverse all the kernel pages and > > get their virtual addresses here: > > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html > > > > Now we found that it seems needed to traverse the ``pv_entry'' structure for > > x86_64 MMU. > > > > Another problem is that we do not know if FreeBSD has any form of special > > encodings or offset form for 64-bit instruction addresses (e.g., saved %RIP) > > on X86_64, instead of hard-coded addresses. For example, using a 32-bit > > offset instead of the 64-bit full address; and doing what glibc does for the > > setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the saved > > register values). > > > > Any suggestion or correction are highly appreciated. > > > > Best, > > Yue > > (Added HardenedBSD core and PaXTeam to CC.) > > Until you can not fixed all of the infoleaks from kernel (try sysctl > -a | grep 0x or similar command) the KASLR and other kernel address > space randomization techniques are easily bypass-able... > I do not believe this can be implemented reliably without some serious tinkering around the kernel. Surely I'm not a live patching expert (not any other kind of expert), but hear my out. It seems proposed approach is to move the kernel around and then updated all relevant pointers in all pages. I do not believe this can be done reliably without corrupting data, unless a major surgery is performed on the kernel. Consider: struct meh { func_t *m_func; size_t m_len; data *m_buf; }; Here we have 'm_func' which needs to be updated, but we don't know if that's the only pointer so we have to scan the entire struct. But m_buf can have data which looks like kernel pointers (i.e. matches some kernel funcs) and how will you know not to modify it? What if this is some sort of a hack and in fact you *should* modify it? CTF or some other solutions like that don't help if you just traverse all allocated buffers since you have no idea what the object you found really is. So, assuming this has to be done in runtime, the only somewhat workable solution I see would require leaving function entry points at contant addresses and only insert jumps to new locations. This would be a performance hit and would leave a lot of data at constant addresses, defeating the point to some extent. Also runtime relocation of everything definitely has a lot of unknown unknowns, so all in all I would say this is a non-starter. One could consider a different approach where kernel data is randomly shuffled around with some granularity and relevant symbol relocated prior to booting. This should provide unique enough layout, which paired with big likelyhood of a kernel panic on first bad exploit attempt may be an acceptable solution. But then the kernel may still have enough info leaks for this to not matter, so I would not be so eager to implement it. That said, what prompted this entire effort? Is there an operating system which got this to work or what? > > > > > > > > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik wrote: > >> > >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > >> > When using the following code on kernel module loading: > >> > > >> > ------------------------------------------------------------------------------------------ > >> > struct thread *td = kdb_thr_first(); > >> > td = kdb_thr_next(td); > >> > > >> > ------------------------------------------------------------------------------------------ > >> > The kernel panics. > >> > > >> > >> Panics how? > >> > >> Also you can easily see these functions don't lock anything, so it would > >> be assumed you took appropriate locks. > >> > >> Except it seems there routines are supposed to be only used when > >> execution is 'frozen' (e.g. when escaped to the debugger). > >> > >> > > >> > And when printing all threads in proc0 (all kernel threads?): > >> > > >> > ------------------------------------------------------------------------------------------ > >> > struct proc *p = pfind(0); > >> > FOREACH_THREAD_IN_PROC(p, td) { > >> > uprintf("td: %x\n", td); > >> > } > >> > > >> > >> proc0 is an exported symbol, no need to pfind. > >> > >> > td = curthread; > >> > uprintf("cur td: %x\n", td); > >> > > >> > ------------------------------------------------------------------------------------------ > >> > The ``curthread'' (from this kernel module running the above code) is > >> > not > >> > in the 0 process group. > >> > > >> > >> There is no 'curthread from kernel module'. > >> > >> My guess is you do this work from module initializator, and in that case > >> curthread is the thread which loads the module, and such a thread is > >> definitely not linked into proc0. > >> > >> Still nobody knows what you are trying to do. > >> > >> -- > >> Mateusz Guzik > > > > -- Mateusz Guzik