From owner-freebsd-hackers@freebsd.org Sun May 21 17:00:32 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9445D77850 for ; Sun, 21 May 2017 17:00:32 +0000 (UTC) (envelope-from aijazbaig1@gmail.com) Received: from mail-vk0-x229.google.com (mail-vk0-x229.google.com [IPv6:2607:f8b0:400c:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8486A1D20 for ; Sun, 21 May 2017 17:00:32 +0000 (UTC) (envelope-from aijazbaig1@gmail.com) Received: by mail-vk0-x229.google.com with SMTP id y190so31138827vkc.1 for ; Sun, 21 May 2017 10:00:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=/bCBS9cfgh/wAGQaOknkSo5MQ7jfoVAkVkpaWJdT1Eo=; b=dI9eOvvG6TlITqFSwOt2/LGxIrGP67/cpi7iSS3glmAUmNho8NKvzR1EOytaF+tO02 5uXolZDsNuDvQJM3k7Bzmr1A/6zTUbijZyYHy3tQQzhNyTCk57aQnHfHkzr+FBUPS+4d QDrE/dbIzKzBs2Q8Sd+9TMBJ9bLt7kdQD4OgYAtvUukRSBI5dhWguA/RvFxv4pcKYowD PieUI3is+IXX3MmA5M0ZJXsgUOyjPPseQS11+izqQfsPMNHkOLQ4CHw1vQVuxgmqzweJ 40m61bv8LFrp914b/8kVtw0U5YoZDy/4zgQmUytT1yL1wHKiZcxUAWRw8vp2nifVMah4 ln/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=/bCBS9cfgh/wAGQaOknkSo5MQ7jfoVAkVkpaWJdT1Eo=; b=B8VYLm9HiExOh1wGUQUe2T1OdYmZTNNF/BqGSepIsnJngUIFRxyB6z73Ofv8UgsJJs n+b/9rcXkblsSWLTFGSjSuzGFQvx4KrHRc31igReLsLFVsgyx8GHOFhTEGPhXC6cOGDD 4vGnuWdxaAw5ojvAxUHl+C7bjS8IbESM4/m/Reu95uDYRej+3yhF9/F5Zqz9d5zipCJ6 ANIsx+IA9ZOd19CtO9A7Ag2O0XJ1js34H8SRx5JI/qnGSDUY7dudevY9jKsfUXhOxxWH AreLHlITBkgA1hcPfEKcQnRhsGhHPMfdhm8AbaEyP5HbOhysMq7tO2SRBwF4dv7Lxaak ChNw== X-Gm-Message-State: AODbwcADj8DUSHL80x4mKi+wVLnShhr+uHPoFkLbeYIN+mugb09yjt6P 7pYVrrLk7qA5jTN8jtGzHJYQGSWCGu1K X-Received: by 10.31.165.79 with SMTP id o76mr6671321vke.91.1495386030524; Sun, 21 May 2017 10:00:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.204.195 with HTTP; Sun, 21 May 2017 10:00:29 -0700 (PDT) From: Aijaz Baig Date: Sun, 21 May 2017 22:30:29 +0530 Message-ID: Subject: Debugging nullfs kernel module - cannot access memory at address To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 May 2017 17:00:32 -0000 I am trying to debug the nullfs kernel module so to that end, I do the following: On the target machine: kldstat gives Id Refs Address Size Name 1 10 0xffffffff80200000 17e10c8 kernel 2 1 0xffffffff819e2000 4cf0 vmxnet.ko 3 1 0xffffffff819e7000 16e0 echo.ko 4 1 0xffffffff81c11000 23dc vmmemctl.ko 5 1 0xffffffff81c14000 641b nullfs.ko nm /boot/kernel/nullfs.ko | grep mount 00000000000018f0 t null_getwritemount 0000000000000540 t nullfs_mount 0000000000000930 t nullfs_unmount U vfs_mountedfrom U vop_getwritemount_desc On the local machine (which connects to the target via a named pipe acting as a serial console (I am using virtual machines): (kgdb) tr0 kdb_sysctl_enter (oidp=, arg1=, arg2=0xfffffe004e7cc7f0, req=) at /usr/src/sys/kern/subr_kdb.c:446 446 kdb_why = KDB_WHY_UNSET; Current language: auto; currently minimal (kgdb) getsyms During symbol reading, Incomplete CFI data; unspecified registers at 0xffffffff8099497a. Id Refs Address Size Name 1 10 0x80200000 17e10c8 kernel 2 1 0x819e2000 4cf0 vmxnet.ko 3 1 0x819e7000 16e0 echo.ko 4 1 0x81c11000 23dc vmmemctl.ko 5 1 0x81c14000 641b nullfs.ko Select the list above with the mouse, paste into the screen and then press ^D. Yes, this is annoying. 5 1 0x81c14000 641b nullfs.ko add symbol table from file "/usr/obj/usr/src/sys/AIJAZ-DEBUG/modules/usr/src/sys/modules/nullfs/nullfs.ko.debug" at .text_addr = 0x81c14000 .data_addr = 0x81c14000 .bss_addr = 0x81c14000 (kgdb) add-kld nullfs.ko add symbol table from file "/boot/kernel/nullfs.ko.symbols" at .text_addr = 0xffffffff81c14000 set_sysinit_set_addr = 0xffffffff81c15c90 set_sysuninit_set_addr = 0xffffffff81c15cb0 .rodata.str1.1_addr = 0xffffffff81c15cc8 set_modmetadata_set_addr = 0xffffffff81c15e48 set_sysctl_set_addr = 0xffffffff81c15e58 .data_addr = 0xffffffff81c15e60 .bss_addr = 0xffffffff81c16360 (y or n) y Reading symbols from /boot/kernel/nullfs.ko.symbols... location expression too complex...done. (kgdb) b nullfs_mount Cannot access memory at address 0x81c14540 As one can see from the output of 'nm' and 'kldstat' above, the addresses are indeed proper. I even tried setting a "hardware breakpoint" at the above address (kgdb) hbreak *0x81c14540 Hardware assisted breakpoint 1 at 0x81c14540: file /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vfsops.c, line 74. (kgdb) c Continuing. Warning: Cannot insert breakpoint 1. Error accessing memory address 0x81c14540: Input/output error. On searching for this error on Linux, it appears that this is taken care of by turning off CONFIG_DEBUG_RODATA as part of the kernel config (which as per this: http://elinux.org/Overwrite_detection_for_kernel_text_and_read-only_data link appears to be some sort of a protection mechanism which detects when the text section of the kernel is being overwritten for some reason). This helps with the setting of software breakpoints which would otherwise be not set. Is there something similar for FreeBSD systems as well?? Keen to hear -- Best Regards, Aijaz Baig