Date: Tue, 17 Nov 2015 00:04:39 +0100 From: Marius Strobl <marius@alchemy.franken.de> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-arch@freebsd.org Subject: Re: Supporting cross-debugging vmcores in libkvm (Testing needed) Message-ID: <20151116230439.GA77914@alchemy.franken.de> In-Reply-To: <5385051.zAN7Yc63R0@ralph.baldwin.cx> References: <3121152.ujdxFEovO3@ralph.baldwin.cx> <111428878.dys4vNKReT@ralph.baldwin.cx> <20151112234146.GA76156@alchemy.franken.de> <5385051.zAN7Yc63R0@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Nov 13, 2015 at 11:50:37AM -0800, John Baldwin wrote: > On Friday, November 13, 2015 12:41:46 AM Marius Strobl wrote: > > On Thu, Nov 12, 2015 at 02:36:42PM -0800, John Baldwin wrote: > > > On Monday, August 31, 2015 02:21:19 PM John Baldwin wrote: > > > > On Wednesday, August 12, 2015 10:50:20 AM John Baldwin wrote: > > > > > On Tuesday, August 04, 2015 10:56:09 AM John Baldwin wrote: > > > > > > Many debuggers (recent gdb and lldb) support cross-architecture debugging > > > > > > just fine. My current WIP port of kgdb to gdb7 supports cross-debugging for > > > > > > remote targets already, but I wanted it to also support cross-debugging for > > > > > > vmcores. > > > > > > > > > > > > The existing libkvm/kgdb code in the tree has some limited support for > > > > > > cross-debugging. It requires building a custom libkvm (e.g. libkvm-i386.a) > > > > > > and custom kgdb for each target platform. However, gdb (and lldb) both > > > > > > support multiple targets in a single binary, so I'd like to have a single > > > > > > kgdb binary that can cross-debug anything. > > > > > > > > > > > > I started hacking on libkvm last weekend and have a prototype that I've used > > > > > > (along with some patches to my kgdb port) to debug an amd64 vmcore on an > > > > > > i386 machine and vice versa. > > > > > > > > > > > > ... > > > > > > > > > > > > What I'm mostly after is comments on the API, etc. Once that is settled I > > > > > > will move forward on converting and/or stubbing the other backends (the > > > > > > stub route would be to only support other backends on native systems for > > > > > > now). > > > > > > > > > > I guess this is closer to a nuclear power plant than a bikeshed judging by the > > > > > feedback. I have ported the rest of the MD backends and verified that the > > > > > updated libkvm passes a universe build (including various static assertions > > > > > for the duplicated constants in other backends). What I have not done is any > > > > > runtime testing and I would like to ask for help with that now. In particular > > > > > I need someone to test that kgdb and/or ps works against a native core dump > > > > > on all platforms other than amd64 and i386. Note that some of the trickiness > > > > > is that the backends now have to make runtime decisions for things that were > > > > > previously compile-time decisions. The biggest one affected by this is the > > > > > MIPS backend as that backend handles three ABIs (mipso32, mipsn32, and mipsn64). > > > > > I believe I have the handling for that correct (mips[on]32 use 32-bit KSEGs > > > > > where as mipsn64 uses the extended segments and compat32 KSEGS, and mipso32 > > > > > uses 32-bit PTEs and mipsn32/n64 both use 64-bit PTEs) (plus both endians > > > > > for both in theory). The ARM backend also handles both endians (in theory). > > > > > > > > > > Another wrinkle is that sparc64 uses its own dump format instead of writing > > > > > out an ELF file. I had to convert the header structures to use fixed-width > > > > > types to be cross-friendly. It would be good to ensure that a new libkvm > > > > > can read a vmcore from an old kernel and vice versa to make sure my conversion > > > > > is correct (I added an explicit padding field that I believe was implicit > > > > > before). > > > > > > > > > > The code is currently available for review in phabric at > > > > > https://reviews.freebsd.org/D3341 > > > > > > > > > > To test, you can run 'arc patch D3341' in a clean tree to apply the patch. > > > > > > > > I've just rebased this to port aarch64's minidump support. I just need people > > > > willing and able to test on non-x86. Testing with the in-tree kgdb using an > > > > updated libkvm would be sufficient. > > > > > > After a lot of crickets, I have updated the manpages for the new API. I will > > > commit this "soon". If you want kgdb to keep working on your non-x86 > > > platform, this is your chance to test this before it hits the tree. > > > > > > > What exact test procedure do you suggest for full coverage of an > > architecture? > > Just ensuring that kgdb and things like ps -M <core> -N <kernel> still work. With the patch from D3341 applied, kgdb(1) still seems to work fine on sparc64. However, `ps -M <core> -N <kernel>` doesn't; it just prints the header and then exists after a short pause. Using the same core and kernel with ps(1) on a machine with userland built without your patch, ps(1) just segfaults after a short period of time. I can't tell whether that's a regression or not as I've never used ps(1) on a core before and you also have added padding to struct sparc64_dump_hdr, which might be responsible for triggering the segfault. On the other hand, an old kgb(1) seemingly works fine with the new core. FYI, I needed the follow patch on top of D3341 (based on the amd64 counterpart): --- lib/libkvm/kvm_minidump_aarch64.c 2015-11-16 23:41:58.075242000 +0100 +++ lib/libkvm/kvm_minidump_aarch64.c 2015-11-16 13:25:26.411577000 +0100 @@ -122,7 +122,7 @@ return (-1); } if (pread(kd->pmfd, bitmap, vmst->hdr.bitmapsize, off) != - vmst->hdr.bitmapsize) { + (ssize_t)vmst->hdr.bitmapsize) { _kvm_err(kd, kd->program, "cannot read %d bytes for page bitmap", vmst->hdr.bitmapsize); @@ -215,7 +215,7 @@ } invalid: - _kvm_err(kd, 0, "invalid address (0x%lx)", va); + _kvm_err(kd, 0, "invalid address (0x%jx)", (uintmax_t)va); return (0); } Also, parallel builds failed with something not finding libelf but building with a single jobs succeeded. I don't know whether D3341 introduces that or if it's a bug in head (the latter probably is unlikely but I didn't investigate). > Btw, Mark Linimon tried to generate a crashdump for me on his sparc64 running > HEAD recently so I could test the updated kgdb but it failed to generate a > dump. Ah, that reminds me of something; fixed in r290957. > I was hoping that the thread on sparc64 with the guy from qemu would > result in working qemu as that would let me do the testing I need for this > and kgdb locally. > Yeah, I also thought that after all that time, OpenBIOS and QEMU would have progressed some more. Well, some bugs are fixed now but they're also still not quite there, yet. Marius
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151116230439.GA77914>