From owner-freebsd-fs@freebsd.org Sun Aug 27 16:34:43 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 447A9DF30B8 for ; Sun, 27 Aug 2017 16:34:43 +0000 (UTC) (envelope-from aijazbaig1@gmail.com) Received: from mail-it0-x243.google.com (mail-it0-x243.google.com [IPv6:2607:f8b0:4001:c0b::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 002207680A; Sun, 27 Aug 2017 16:34:42 +0000 (UTC) (envelope-from aijazbaig1@gmail.com) Received: by mail-it0-x243.google.com with SMTP id c196so2866112itc.3; Sun, 27 Aug 2017 09:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6vFvUyiu32+jKkwImaNIOQ7n4ghKvDnu8bWT9q9oUKo=; b=ZYq4HysG7nWJZUOnh1KU7OmBYi91AvMD/PY03fl42BE8WZgPa83/HRq1thsRqI+vWu tmHUj4qa7BlyFMOrnM6NIPfmoWZ3dyM+rth5yb++8gYZTEzobYzrk4O6wiTtTZZrF205 CMjaCVmIElKPzZJmm/fdFpyAbsx1KLkOKXuNqDj17QUizAiXaKvJd8m5MLdPeSnlpmN3 wJBsaSj5YZXeivrFmxyaBAt0jmUHxinCRxkUTEnwYVG3DE7x4fHQm7KHIkbCUHcsqkbL eZOwy1E+ox29vdppccMtA/zAa4/D0S7bDsYzpmXwHIKqHIsRg2ITGSLBm4flGAwQV7KW /S5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6vFvUyiu32+jKkwImaNIOQ7n4ghKvDnu8bWT9q9oUKo=; b=c76vpfnIoU7pZBU2kntx0nbEvs03hu+qorBI5G9smklUw4C4FRf0NkFcYLs0wsa8UP KVahlZaknrrwqgXeQWUrjFH3Xw5P8A+y2W/0Fv4UNv7LaSh5Vw7rCSK+h6QIX6ScgUcT 50ngcNs561g/cQsy+/HAAjL6W+kHm9nTuCGr/Z8B4od3uf2+llHmsjKec7yRKnzTi77p Cb+huJjf/eU003FG58JB3kd2P2PqI477LTbjM/mVAm+nk4nrapSYQfdQOg2Cvi6KOuNm 1xq1VE4j9emqKF359HT4kRVsefWarypjri/8oWAAyQsqkg0dz1hK4sBpxdJDFd+EKNZ5 5M4g== X-Gm-Message-State: AHYfb5iGX/vwLzaTtYadaCnYD0gupUsRn51Bjs+BeCnurpqqpV2JfS6Y vzBmpka5Pj8JQJMLaEeP9BKsYz3OJQ== X-Received: by 10.36.149.197 with SMTP id m188mr681337itd.112.1503851682164; Sun, 27 Aug 2017 09:34:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.22.66 with HTTP; Sun, 27 Aug 2017 09:34:41 -0700 (PDT) In-Reply-To: <201708250251.v7P2pwcl029687@chez.mckusick.com> References: <201708250251.v7P2pwcl029687@chez.mckusick.com> From: Aijaz Baig Date: Sun, 27 Aug 2017 22:04:41 +0530 Message-ID: Subject: Re: Tips on remote debugging for filesystem code To: Kirk McKusick Cc: jhb@freebsd.org, Julian Elischer , freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Aug 2017 16:34:43 -0000 Thanks for the heads up Kirk. Unfortunately I just ran into a roadblock with Bhyve since I am running FreeBSD on core2duo which "officially" doesn't support Bhyve yet. Is there some other way I can get this to work? I have installed FreeBSD bare metal now if it helps. On Fri, Aug 25, 2017 at 8:21 AM, Kirk McKusick wrote: > > From: Aijaz Baig > > Date: Fri, 25 Aug 2017 06:33:25 +0530 > > Subject: Re: Tips on remote debugging for filesystem code > > To: Kirk McKusick > > Cc: Julian Elischer , freebsd-fs@freebsd.org > > X-ASK-Info: Message Queued (2017/08/24 18:09:30) > > X-ASK-Info: Confirmed by User (2017/08/24 18:40:26) > > > > On Fri, Aug 25, 2017 at 12:00 AM, Kirk McKusick > wrote: > > > >>> To: Aijaz Baig , freebsd-fs@freebsd.org > >>> Subject: Re: Tips on remote debugging for filesystem code > >>> From: Julian Elischer > >>> Date: Thu, 24 Aug 2017 16:24:50 +0800 > >>> > >>> On 24/8/17 1:40 pm, Aijaz Baig wrote: > >>> > >>>> ... > >>>> How do you guys normally do it? Keen to hear tips and best practices > >>> > >>> I have had more success recently using BHype to make a Virtual FreeBSD > >>> machine and connecting to it using the built-in gdb interface. > >>> > >>> Firstly it is easier than a serial interface and secondly you don't > >>> need two machines. > >> > >> Is there documentation that describes how to do this? I have been using > >> a patched up set of scripts provided by John Baldwin a couple of years > >> ago to use kgdb with a bhyve VM. But if there is an existing way to do > >> it now, I would rather switch to it. > >> > >> Kirk McKusick > > > > Yes please I would love to hear about this as well. Nonetheless if > > any one of you Kirk or Julian could help me get started a bit with > > what how you guys use it (and those scripts that Kirk talks about), > > it would be really nice. > > Below I enclose an email from John Baldwin that got me going on > debugging under bhyve. I am copying John as he likely has progressed > considerably in the last two years, so can likely provide some more > up to date tips. Though as far as I can see, he never added his > updates to the bhyve/vmrun.sh script. So I am not sure how Julian > has set up his bhyve debugging which is what lead to my initial question. > > > By the way is this bhyve the only way to go for fs code debugging? > > > > Best Regards, > > Aijaz Baig > > You could use multiple machines or run under other hypervisors, but > bhyve lets you do everything on one machine and has the benefit of > easily integrating with kgdb. > > Another option is to read the several chapters in my book that describes > how the VFS layer works ;-) > > Kirk McKusick > > =-=-= > > From: John Baldwin > Date: Tue, 08 Sep 2015 16:53:26 -0700 > To: Kirk McKusick > Subject: Re: kgdb ported to devel/gdb > > On Tuesday, September 08, 2015 04:05:06 PM Kirk McKusick wrote: > >> From: John Baldwin > >> To: freebsd-current@freebsd.org > >> Subject: kgdb ported to devel/gdb > >> Date: Mon, 31 Aug 2015 14:32:04 -0700 > >> > >> Over the past several months I have ported kgdb to the version of > >> gdb in ports. I have a pending patch to the gdb port to add fork > >> following, but once that is done (and possibly after updating to > >> 7.10) I will try to add my existing work as a KGDB option on the > >> port. Until such time, you can try the newer kgdb by checking out > >> my branch from git. > >> > >> Here's my cheat sheet on how to build the newer kgdb. Note that > >> if you build a world with my cross-libkvm patches you should get a > >> kgdb that can debug i386 cores on amd64 and vice versa. > >> > >> All of the targets that the native devel/gdb support have their > >> backends ported (so x86, sparc64, powerpc and powerpc64). I have > >> not yet ported arm or mips since those don't work for userland yet > >> in upstream gdb. I have only compiled non-x86 backends. Testing > >> of the new kgdb on sparc64 and powerpc would be appreciated. > >> > >> Steps: > >> > >> % git clone https://github.com/bsdjhb/gdb.git > >> % git checkout freebsd-7.9.1-kgdb > >> % fetch http://www.freebsd.org/~jhb/gdb/build > >> % pkg install devel/gdb > >> > >> # Having gdb installed will mean you get the python bindings in the > right > >> # place. > >> > >> % pkg install gmake > >> > >> # I think this is the only build tool you need? > >> > >> % ./build > >> % cd obj > >> > >> # Replace 'obj' with 'obj.' for all but amd64 > >> > >> % gmake > >> > >> # ... wait > >> > >> You will now have a binary at 'obj/gdb/kgdb'. I just run it from my obj > >> tree currently when testing. Once it becomes part of the port it will > get > >> installed as /usr/local/bin/kgdb791 or some such. > > > > This is very helpful, thanks for doing it. > > > > I recently dumped my (very ancient) i386 debugging machine and started > > debugging in a bhyve virtual machine. Much quicker to boot and test. > > But, I lost the ability to remote debug over a serial line. Is there > > some way to remote attach to my virtual machine so I can once again > > set breakpoints, single step it, etc? > > Yes, I do this with my VMs (and this is how I tested the i386 <--> amd64 > cross debugging). The way I do this is to attach a nmdm device to COM2 > (uart1) in my VMs and then run gdb over that. First, I use a patched > vmrun.sh > to always create a /dev/nmdm2{A,B} pair of devices attached to COM2 > on each VM. The 'B' device is used by bhyve, and I use the 'A' device on > the > host with 'target remote' in kgdb. My change to vmrun.sh is this: > > --- /usr/share/examples/bhyve/vmrun.sh 2015-08-15 16:31:57.891092000 > -0700 > +++ bhyve/vmrun.sh 2015-09-08 16:46:05.000000000 -0700 > @@ -91,9 +91,13 @@ > loader_opt="" > bhyverun_opt="-H -A -P" > pass_total=0 > +com2="" > > -while getopts ac:C:d:e:g:hH:iI:m:p:t: c ; do > +while getopts 2:ac:C:d:e:g:hH:iI:m:p:t: c ; do > case $c in > + 2) > + com2=${OPTARG} > + ;; > a) > bhyverun_opt="${bhyverun_opt} -a" > ;; > @@ -269,7 +273,13 @@ > i=$(($i + 1)) > done > > + if [ -n "${com2}" ]; then > + devargs="$devargs -l com2,${com2}" > + elif kldstat -qm nmdm; then > + devargs="$devargs -l com2,/dev/nmdm${vmname}2B" > + fi > + > ${FBSDRUN} -c ${cpus} -m ${memsize} ${bhyverun_opt} \ > -g ${gdbport} \ > -s 0:0,hostbridge \ > -s 1:0,lpc \ > > Note that if you have nmdm kldloaded, this version will place a nmdm > device on COM2 automatically. > > Next, I enable gdb on COM2 by adding 'hint.uart.1.flags="0x80"' to > /boot/device.hints inside the VM. > > To test it, I boot the VM and then run 'sysctl debug.kdb.enter=1' on the > console (or over an ssh session). On the host I run my new kgdb (since > it can do i386 cross debugging via remote out of the box). Note that > you will need the kernel.debug on the host. I do this by keeping my > work tree on my host machine and exporting it via NFS into the guest. > I then build the kernel the "old" way using config and make directly so > that the kernel.debug is at 'sys/i386/compile//kernel.debug'. You > can then run the new kgdb as 'kgdb /path/to/kernel.debug'. In the VM > console, run 'gdb' at the db> prompt to switch over to GDB. You can then > use 'target remote /dev/nmdm2A' to connect to it. For example, > if your VM is named vm0, you would use /dev/nmdmvm02A. > > -- > John Baldwin > -- Best Regards, Aijaz Baig