From owner-freebsd-current@FreeBSD.ORG Tue Jul 26 18:16:02 2005 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 84FBF16A41F; Tue, 26 Jul 2005 18:16:02 +0000 (GMT) (envelope-from cracauer@schlepper.zs64.net) Received: from schlepper.zs64.net (schlepper.zs64.net [212.12.50.230]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BD0C43D45; Tue, 26 Jul 2005 18:16:01 +0000 (GMT) (envelope-from cracauer@schlepper.zs64.net) Received: from schlepper.zs64.net (schlepper [212.12.50.230]) by schlepper.zs64.net (8.13.1/8.12.9) with ESMTP id j6QIG0v1011938; Tue, 26 Jul 2005 20:16:00 +0200 (CEST) (envelope-from cracauer@schlepper.zs64.net) Received: (from cracauer@localhost) by schlepper.zs64.net (8.13.1/8.12.9/Submit) id j6QIG0kG011937; Tue, 26 Jul 2005 14:16:00 -0400 (EDT) (envelope-from cracauer) Date: Tue, 26 Jul 2005 14:16:00 -0400 From: Martin Cracauer To: freebsd-current@freebsd.org Message-ID: <20050726141600.A10751@cons.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Cc: jkoshy@freebsd.org Subject: PMC (performance counters) problems - kernel panic in 6-beta1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2005 18:16:02 -0000 [I don't think there is a mailing list for pmc, let me know if there's something better than -current] I tried out the pmc performance counters in 6-beta1 and 7-current. In FreeBSD 6-beta1, source of yesterday, on a Pentium-M I get a kernel panic. In FreeBSD 7-current, source of yesterday, on AMD64 (single processor) I always get "0" for all counters no matter what I do. %% First, the zero counters on 7-current on a one-processor Athlon 64: $ sudo kldload hwpmc $ pmccontrol -l #CPU 0: #N NAME CLASS STATE ROW-DISP 0 TSC TSC ENABLED FREE 1 K8-0 K8 ENABLED FREE 2 K8-1 K8 ENABLED FREE 3 K8-2 K8 ENABLED FREE 4 K8-3 K8 ENABLED FREE # pid 7100 is a gzip on some random data $ pmcstat -p k8-dc-l1-dtlb-miss-and-l2-dtlb-hit -p k8-dc-l1-dtlb-miss-and-l2-dtlb-miss -w1 -t 7100 # p/k8-dc-l1-dtlb-miss-and-l2-dtlb-hit p/k8-dc-l1-dtlb-miss-and-l2-dtlb-miss # p/k8-dc-l1-dtlb-miss-and-l2-dtlb-hit p/k8-dc-l1-dtlb-miss-and-l2-dtlb-miss 0 0 0 0 0 0 [etc] I tried all values from `pmccontrol -L` (in a loop) and all of them are always zero (some fail with e.g. "pmcstat: ERROR: Cannot allocate process-mode pmc with specification "k8-nb-memory-controller-page-access-event": Invalid argument"). I also tried running programs directly from the commandline. Did I forget to initialize anything? Machine dmesg, kernel conf, pciconf etc is on: http://www.cons.org/cracauer/machines/wings/ %% Now the 6-beta 1 panic: $ sudo kldload hwpmc $ pmcstat -w 1 -p p6-l2-lines-in ls ==> panic, page fault while in kernel mode Machine dmesg, kernel conf, pciconf etc is on: http://www.cons.org/cracauer/machines/grisu/ Here is the backtrace. I loaded the symbols from the hwpmc module and it looks like kgdb picked them up. Obviously, poor p6_get_config() gets a NULL pointer as the ppm pointer-to-pointer argument which it tries to follow. This is passed from pmc_process_exit(), deliberately, via /* * Pick up the pmc pointer from hardware * state similar to the CSW_OUT code. */ pm = NULL; (void) (*md->pmd_get_config)(cpu, ri, &pm); It seems to me that none of the *_get_config functions are prepared to take a NULL pointer, so I wonder what that quoted code is trying a achieve here. (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc056a6f5 in boot (howto=260) at ../../../kern/kern_shutdown.c:397 #2 0xc056aa6d in panic (fmt=0xc0764506 "from debugger") at ../../../kern/kern_shutdown.c:553 #3 0xc0456e42 in db_panic (addr=-1027275291, have_addr=0, count=-1, modif=0xf7cc0a44 "") at ../../../ddb/db_command.c:435 #4 0xc0456db2 in db_command (last_cmdp=0xc07d1544, cmd_table=0x0, aux_cmd_tablep=0xc0796c78, aux_cmd_tablep_end=0xc0796c7c) at ../../../ddb/db_command.c:349 #5 0xc0456ec5 in db_command_loop () at ../../../ddb/db_command.c:455 #6 0xc0459075 in db_trap (type=12, code=0) at ../../../ddb/db_main.c:221 #7 0xc0589117 in kdb_trap (type=0, code=0, tf=0xf7cc0bd4) at ../../../kern/subr_kdb.c:473 #8 0xc073bd1b in trap_fatal (frame=0xf7cc0bd4, eva=0) at ../../../i386/i386/trap.c:831 #9 0xc073b334 in trap (frame= {tf_fs = -137625592, tf_es = 40, tf_ds = -1028653016, tf_edi = 0, tf_esi = 3, tf_ebp = -137622508, tf_isp = -137622528, tf_ebx = 0, tf_edx = -1026180480, tf_ecx = -137622464, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1027275291, tf_cs = 32, tf_eflags = 66182, tf_esp = -137622448, tf_ss = -1027290778}) at ../../../i386/i386/trap.c:271 #10 0xc072820a in calltrap () at ../../../i386/i386/exception.s:137 #11 0xf7cc0008 in ?? () ---Type to continue, or q to quit--- #12 0x00000028 in ?? () #13 0xc2b00028 in ?? () #14 0x00000000 in ?? () #15 0x00000003 in ?? () #16 0xf7cc0c14 in ?? () #17 0xf7cc0c00 in ?? () #18 0x00000000 in ?? () #19 0xc2d5ba80 in ?? () #20 0xf7cc0c40 in ?? () #21 0x00000000 in ?? () #22 0x0000000c in ?? () #23 0x00000000 in ?? () #24 0xc2c505e5 in p6_get_config (cpu=0, ri=0, ppm=0x0) at /mnt/part2/usr/src/sys/modules/hwpmc/../../dev/hwpmc/hwpmc_ppro.c:487 #25 0xc2c4c966 in pmc_process_exit (arg=0x0, p=0xc2e9a830) at /mnt/part2/usr/src/sys/modules/hwpmc/../../dev/hwpmc/hwpmc_mod.c:3709 #26 0xc054b7ec in exit1 (td=0xc2e9c900, rv=0) at ../../../kern/kern_exit.c:199 #27 0xc054b3dd in sys_exit (td=0x0, uap=0x0) at ../../../kern/kern_exit.c:94 #28 0xc073c140 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 25, tf_esi = -1077942068, tf_ebp = -1077943752, tf_isp = -137622172, tf_ebx = 1074955492, tf_edx = 0, tf_ecx = 0, tf_eax = 1, tf_trapno = 12, tf_err = 2, tf_eip = 1074373979, tf_cs = 51, tf_eflags = 646, tf_esp = -1077943764, tf_ss = 59}) ---Type to continue, or q to quit--- at ../../../i386/i386/trap.c:985 #29 0xc072825f in Xint0x80_syscall () at ../../../i386/i386/exception.s:198 #30 0x0000003b in ?? () #31 0x0000003b in ?? () #32 0x0000003b in ?? () #33 0x00000019 in ?? () #34 0xbfbfe8cc in ?? () #35 0xbfbfe238 in ?? () #36 0xf7cc0d64 in ?? () #37 0x401284e4 in ?? () #38 0x00000000 in ?? () #39 0x00000000 in ?? () #40 0x00000001 in ?? () #41 0x0000000c in ?? () #42 0x00000002 in ?? () #43 0x4009a55b in ?? () #44 0x00000033 in ?? () #45 0x00000286 in ?? () #46 0xbfbfe22c in ?? () #47 0x0000003b in ?? () #48 0xd0d0d0d0 in ?? () #49 0xd0d0d0d0 in ?? () #50 0xd0d0d0d0 in ?? () ---Type to continue, or q to quit--- #51 0xd0d0d0d0 in ?? () #52 0x3c8a2000 in ?? () #53 0x00000000 in ?? () #54 0xc054f410 in fork_exit () at ../../../kern/kern_fork.c:804