Date: Thu, 22 Dec 2016 19:49:11 -0800 From: John Baldwin <jhb@freebsd.org> To: src-committers@freebsd.org Cc: svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r310445 - in head/sys/mips: include mips Message-ID: <1808755.8aTbUPpvbL@ralph.baldwin.cx> In-Reply-To: <201612230327.uBN3RBC1099442@repo.freebsd.org> References: <201612230327.uBN3RBC1099442@repo.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday, December 23, 2016 03:27:11 AM John Baldwin wrote: > Author: jhb > Date: Fri Dec 23 03:27:11 2016 > New Revision: 310445 > URL: https://svnweb.freebsd.org/changeset/base/310445 > > Log: > Teach DDB how to unwind across a kernel stack overflow. > > Kernel stack overflows in MIPS call panic() directly from an assembly > handler after storing the interrupted context's registers in a > trapframe. Rather than inferring the location of ra, sp, and pc from > the instruction stream, recognize the pc of a kernel stack overflow > and pull the registers from the trapframe. > > Sponsored by: DARPA / AFRL We should possibly use this same logic for other subroutines that use trapframes. Trying to run a program that core dumps over NFS in my qemu mips64 machine triggers the stack overflow on a kernel built with gcc5.3 from ports. The DDB stack trace doesn't get very far: panic: kernel stack overflow - trapframe at 0xffffffff80699eb0 KDB: stack backtrace: db_trace_self+1c (?,?,?,?) ra ffffffff801620d8 sp ffffffff80699ac0 sz 16 ffffffff801620ac+2c (?,?,?,?) ra ffffffff80357788 sp ffffffff80699ad0 sz 800 ffffffff80357744+44 (?,?,?,?) ra ffffffff80304ff4 sp ffffffff80699df0 sz 16 vpanic+f4 (?,?,?,?) ra ffffffff80305d84 sp ffffffff80699e00 sz 48 panic+30 (?,ffffffff80699eb0,ffffffffffffffec,ffffffff802c901c) ra ffffffff805388bc sp ffffffff80699e30 sz 96 MipsTLBInvalidException+360 (?,?,?,?) ra 0 sp ffffffff80699e90 sz 0 --- Kernel Stack Overflow --- ffffffff802c9018+4 (?,?,?,?) ra ffffffff805544e4 sp c00000002a5b5fa0 sz 96 ffffffff80554400+e4 (?,?,?,?) ra ffffffff802c90f4 sp c00000002a5b6000 sz 48 ffffffff802c9018+dc (?,?,?,?) ra ffffffff805563dc sp c00000002a5b6030 sz 96 cpu_intr+248 (?,?,?,?) ra ffffffff80538110 sp c00000002a5b6090 sz 80 MipsKernIntr+188 (?,?,?,?) ra 0 sp c00000002a5b60e0 sz 368 pid 607 KDB: enter: panic (In particular, we probably should be using the trapframe logic employed here for MipsKernIntr.) I've patched kgdb to add a MIPS kernel target and taught it how to handle trapframes, etc. (and it can cross-debug, so I used an amd64 host to examine the dump) and it gives a more useful stack trace. Unfortunately, increasing the kstack size on MIPS as a workaround for this issue seems to be very hard as the code in cpu_switch() hard codes exactly two TLB entries to cover the kernel stack and PCB to avoid TLB faults during a context switch. I think if you create a kthread with a non-default kstack size on MIPS it's probably a ticking time bomb in that you might get a TLB miss when accessing the pcb. savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 171 SAVE_U_PCB_CONTEXT(ra, PCB_REG_PC, a0) (kgdb) where #0 savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 #1 0xffffffff80304dfc in doadump (textdump=textdump@entry=0) at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:297 #2 0xffffffff8015d868 in db_dump (dummy=<optimized out>, dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>) at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:546 #3 0xffffffff8015e474 in db_command ( last_cmdp=last_cmdp@entry=0xffffffff8069e778 <db_last_command>, cmd_table=<optimized out>, cmd_table@entry=0xffffffff8069e730 <db_cmd_table>, dopager=dopager@entry=1) at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:453 #4 0xffffffff8015eab8 in db_command_loop () at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:506 #5 0xffffffff80162040 in db_trap (type=<optimized out>, code=<optimized out>) at /usr/home/john/work/git/freebsd/sys/ddb/db_main.c:248 #6 0xffffffff803583e8 in kdb_trap (type=type@entry=9, code=code@entry=0, tf=tf@entry=0xffffffff80699ca0 <pcpu_space+7328>) at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:654 #7 0xffffffff8054ba10 in trap (trapframe=0xffffffff80699ca0 <pcpu_space+7328>) at /usr/home/john/work/git/freebsd/sys/mips/mips/trap.c:828 #8 <signal handler called> #9 kdb_enter (why=0xffffffff805b6138 "panic", msg=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:444 #10 0xffffffff8030503c in vpanic (fmt=<optimized out>, ap=ap@entry=0xffffffff80699e58 <pcpu_space+7768>) at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:752 ---Type <return> to continue, or q <return> to quit--- #11 0xffffffff80305d84 in panic (fmt=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:690 #12 <signal handler called> #13 0xffffffff802c901c in intr_event_handle (ie=0x9800000000a42400, frame=frame@entry=0x0) at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1397 #14 0xffffffff805544e4 in gt_pci_intr (v=0x9800000000ab0800) at /usr/home/john/work/git/freebsd/sys/mips/malta/gt_pci.c:236 #15 0xffffffff802c90f4 in intr_event_handle (ie=0x9800000000a43900, frame=frame@entry=0xc00000002a5b6100) at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1436 #16 0xffffffff805563dc in cpu_intr (tf=0xc00000002a5b6100) at /usr/home/john/work/git/freebsd/sys/mips/mips/intr_machdep.c:264 #17 <signal handler called> #18 le_pci_wrcsr (sc=0x9800000000ae9400, port=<optimized out>, val=<optimized out>) at /usr/home/john/work/git/freebsd/sys/dev/le/if_le_pci.c:187 #19 0xffffffff801893e4 in am79900_start_locked (sc=0x9800000000ae9400) at /usr/home/john/work/git/freebsd/sys/dev/le/am79900.c:594 #20 0xffffffff8018b1c0 in lance_start (ifp=<optimized out>) at /usr/home/john/work/git/freebsd/sys/dev/le/lance.c:243 #21 0xffffffff804137e4 in if_start (ifp=ifp@entry=0x9800000000ae4800) at /usr/home/john/work/git/freebsd/sys/net/if.c:3630 #22 0xffffffff80413998 in if_transmit (ifp=0x9800000000ae4800, m=<optimized out>) at /usr/home/john/work/git/freebsd/sys/net/if.c:3642 #23 0xffffffff804174fc in ether_output_frame ( ---Type <return> to continue, or q <return> to quit--- ifp=ifp@entry=0x9800000000ae4800, m=0x980000000162d100) at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:457 #24 0xffffffff80417b94 in ether_output (ifp=0x9800000000ae4800, m=0x980000000162d100, dst=0x98000000032878f0, ro=<optimized out>) at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:429 #25 0xffffffff8045cf28 in ip_output (m=m@entry=0x980000000162d100, opt=<optimized out>, ro=0x98000000032878d0, flags=<optimized out>, imo=<optimized out>, imo@entry=0x0, inp=<optimized out>) at /usr/home/john/work/git/freebsd/sys/netinet/ip_output.c:664 #26 0xffffffff80470348 in tcp_output (tp=0x9800000001f4f000) at /usr/home/john/work/git/freebsd/sys/netinet/tcp_output.c:1432 #27 0xffffffff80480d1c in tcp_usr_send (so=0x98000000032a8a20, flags=<optimized out>, m=0x9800000001621a00, nam=0x0, control=<optimized out>, td=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/netinet/tcp_usrreq.c:956 #28 0xffffffff803a6ef8 in sosend_generic (so=0x98000000032a8a20, addr=<optimized out>, uio=0x0, top=0x9800000001621a00, control=<optimized out>, flags=<optimized out>, td=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1359 #29 0xffffffff803a7060 in sosend (so=<optimized out>, addr=addr@entry=0x0, uio=uio@entry=0x0, top=top@entry=0x9800000001621a00, control=control@entry=0x0, flags=flags@entry=0, td=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1403 #30 0xffffffff8049220c in clnt_vc_call (cl=<optimized out>, ext=0xc00000002a5b69e8, proc=<optimized out>, args=0x980000000162d200, resultsp=0xc00000002a5b6b40, utimeout=...) ---Type <return> to continue, or q <return> to quit--- at /usr/home/john/work/git/freebsd/sys/rpc/clnt_vc.c:398 #31 0xffffffff804908a4 in clnt_reconnect_call (cl=0x9800000001630400, ext=0xc00000002a5b69e8, proc=<optimized out>, args=0x980000000162d200, resultsp=0xc00000002a5b6b40, utimeout=...) at /usr/home/john/work/git/freebsd/sys/rpc/clnt_rc.c:271 #32 0xffffffff801dd360 in newnfs_request (nd=0xc00000002a5b6b40, nmp=0x980000000163b800, clp=clp@entry=0x0, nrp=0x980000000163b928, vp=0x98000000044aa000, td=td@entry=0x9800000003275a60, cred=cred@entry=0x9800000001670000, prog=prog@entry=100003, vers=3, retsum=retsum@entry=0x0, toplevel=toplevel@entry=1, xidp=xidp@entry=0x0, sep=sep@entry=0x0) at /usr/home/john/work/git/freebsd/sys/fs/nfs/nfs_commonkrpc.c:746 #33 0xffffffff8022c738 in nfscl_request (nd=nd@entry=0xc00000002a5b6b40, vp=vp@entry=0x98000000044aa000, p=p@entry=0x9800000003275a60, cred=cred@entry=0x9800000001670000, stuff=stuff@entry=0x0) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clport.c:951 #34 0xffffffff802116a8 in nfsrpc_commit (vp=0x98000000044aa000, offset=offset@entry=65536, cnt=cnt@entry=65536, cred=cred@entry=0x9800000001670000, p=p@entry=0x9800000003275a60, nap=nap@entry=0xc00000002a5b6cc0, attrflagp=attrflagp@entry=0xc00000002a5b6d88, stuff=stuff@entry=0x0) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clrpcops.c:3640 #35 0xffffffff80220e74 in ncl_commit (vp=vp@entry=0x98000000044aa000, offset=offset@entry=65536, cnt=<optimized out>, cred=cred@entry=0x9800000001670000, td=td@entry=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2553 ---Type <return> to continue, or q <return> to quit--- #36 0xffffffff8022130c in ncl_flush (vp=0x98000000044aa000, waitfor=<optimized out>, cred=cred@entry=0x0, td=0x9800000003275a60, commit=commit@entry=1, called_from_renewthread=called_from_renewthread@entry=0) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2774 #37 0xffffffff8022220c in nfs_fsync (ap=<optimized out>) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2619 #38 0xffffffff80559508 in VOP_FSYNC_APV (vop=<optimized out>, a=a@entry=0xc00000002a5b6fa0) at vnode_if.c:1331 #39 0xffffffff803bcbd4 in VOP_FSYNC (td=<optimized out>, waitfor=<optimized out>, vp=<optimized out>) at vnode_if.h:549 #40 bufsync (bo=<optimized out>, waitfor=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/vfs_bio.c:4605 #41 0xffffffff803e6088 in bufobj_invalbuf (bo=bo@entry=0x98000000044aa0d0, flags=<optimized out>, slpflag=<optimized out>, slptimeo=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1633 #42 0xffffffff803e63a8 in vinvalbuf (vp=vp@entry=0x98000000044aa000, flags=flags@entry=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1711 #43 0xffffffff8022df5c in ncl_vinvalbuf (vp=vp@entry=0x98000000044aa000, flags=flags@entry=1, td=td@entry=0x9800000003275a60, intrflg=<optimized out>, intrflg@entry=1) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1387 #44 0xffffffff8021d5ec in nfs_setattr (ap=0xc00000002a5b7298) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:937 #45 0xffffffff80558564 in VOP_SETATTR_APV (vop=<optimized out>, ---Type <return> to continue, or q <return> to quit--- a=a@entry=0xc00000002a5b7298) at vnode_if.c:799 #46 0xffffffff80230eac in VOP_SETATTR (cred=0x9800000001670000, vap=0xc00000002a5b71f0, vp=0x98000000044aa000) at vnode_if.h:335 #47 ncl_write (ap=<optimized out>) at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1280 #48 0xffffffff80558b70 in VOP_WRITE_APV (vop=<optimized out>, a=a@entry=0xc00000002a5b74f0) at vnode_if.c:1000 #49 0xffffffff803f9708 in VOP_WRITE (cred=<optimized out>, ioflag=<optimized out>, uio=0xc00000002a5b7690, vp=<optimized out>) at vnode_if.h:413 #50 vn_io_fault_doio (args=args@entry=0xc00000002a5b76d8, uio=uio@entry=0xc00000002a5b7690, td=td@entry=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:952 #51 0xffffffff803f9894 in vn_io_fault1 (vp=vp@entry=0x98000000044aa000, uio=uio@entry=0xc00000002a5b7690, args=args@entry=0xc00000002a5b76d8, td=td@entry=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:1060 #52 0xffffffff803fca30 in vn_rdwr (rw=rw@entry=UIO_WRITE, vp=vp@entry=0x98000000044aa000, base=base@entry=0x160085000, len=len@entry=16384, offset=offset@entry=131072, segflg=segflg@entry=UIO_USERSPACE, ioflg=ioflg@entry=16641, active_cred=active_cred@entry=0x9800000001670000, file_cred=file_cred@entry=0x0, aresid=aresid@entry=0xc00000002a5b77a0, td=td@entry=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:576 #53 0xffffffff803fcc68 in vn_rdwr_inchunks (rw=rw@entry=UIO_WRITE, ---Type <return> to continue, or q <return> to quit--- vp=0x98000000044aa000, base=0x160085000, len=16384, offset=131072, offset@entry=98304, segflg=segflg@entry=UIO_USERSPACE, ioflg=ioflg@entry=16641, active_cred=0x9800000001670000, file_cred=0x0, aresid=aresid@entry=0x0, td=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:641 #54 0xffffffff802954f4 in core_write (seg=UIO_USERSPACE, offset=98304, len=<optimized out>, base=<optimized out>, p=0xc00000002a5b7860) at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1229 #55 core_output (tmpbuf=0x0, p=0xc00000002a5b7860, offset=98304, len=<optimized out>, base=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1243 #56 elf64_coredump (td=<optimized out>, vp=0x98000000044aa000, limit=9223372036854775807, flags=<optimized out>) at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1384 #57 0xffffffff8030805c in coredump (td=td@entry=0x9800000003275a60) at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3477 #58 0xffffffff8030b818 in sigexit (td=td@entry=0x9800000003275a60, sig=sig@entry=7) at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3072 #59 0xffffffff8030c19c in postsig (sig=sig@entry=7) at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:2985 #60 0xffffffff8036f13c in ast (framep=0xc00000002a5b7d30) at /usr/home/john/work/git/freebsd/sys/kern/subr_trap.c:314 #61 <signal handler called> #62 0x0000000120000ff4 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1808755.8aTbUPpvbL>