Date: Fri, 23 Dec 2016 12:59:37 -0800 From: Adrian Chadd <adrian.chadd@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: "src-committers@freebsd.org" <src-committers@freebsd.org>, "svn-src-all@freebsd.org" <svn-src-all@freebsd.org>, "svn-src-head@freebsd.org" <svn-src-head@freebsd.org> Subject: Re: svn commit: r310445 - in head/sys/mips: include mips Message-ID: <CAJ-Vmo=OmDPGsDCMXqxfnkhkifhQ%2BqiigijwDmfdBABUT1A=Vw@mail.gmail.com> In-Reply-To: <1808755.8aTbUPpvbL@ralph.baldwin.cx> References: <201612230327.uBN3RBC1099442@repo.freebsd.org> <1808755.8aTbUPpvbL@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
That's a ... deeep stack. :( How big is each stackframe? -adrian On 22 December 2016 at 19:49, John Baldwin <jhb@freebsd.org> wrote: > On Friday, December 23, 2016 03:27:11 AM John Baldwin wrote: >> Author: jhb >> Date: Fri Dec 23 03:27:11 2016 >> New Revision: 310445 >> URL: https://svnweb.freebsd.org/changeset/base/310445 >> >> Log: >> Teach DDB how to unwind across a kernel stack overflow. >> >> Kernel stack overflows in MIPS call panic() directly from an assembly >> handler after storing the interrupted context's registers in a >> trapframe. Rather than inferring the location of ra, sp, and pc from >> the instruction stream, recognize the pc of a kernel stack overflow >> and pull the registers from the trapframe. >> >> Sponsored by: DARPA / AFRL > > We should possibly use this same logic for other subroutines that use > trapframes. Trying to run a program that core dumps over NFS in my > qemu mips64 machine triggers the stack overflow on a kernel built with > gcc5.3 from ports. The DDB stack trace doesn't get very far: > > panic: kernel stack overflow - trapframe at 0xffffffff80699eb0 > KDB: stack backtrace: > db_trace_self+1c (?,?,?,?) ra ffffffff801620d8 sp ffffffff80699ac0 sz 16 > ffffffff801620ac+2c (?,?,?,?) ra ffffffff80357788 sp ffffffff80699ad0 sz 800 > ffffffff80357744+44 (?,?,?,?) ra ffffffff80304ff4 sp ffffffff80699df0 sz 16 > vpanic+f4 (?,?,?,?) ra ffffffff80305d84 sp ffffffff80699e00 sz 48 > panic+30 (?,ffffffff80699eb0,ffffffffffffffec,ffffffff802c901c) ra ffffffff805388bc sp ffffffff80699e30 sz 96 > MipsTLBInvalidException+360 (?,?,?,?) ra 0 sp ffffffff80699e90 sz 0 > --- Kernel Stack Overflow --- > ffffffff802c9018+4 (?,?,?,?) ra ffffffff805544e4 sp c00000002a5b5fa0 sz 96 > ffffffff80554400+e4 (?,?,?,?) ra ffffffff802c90f4 sp c00000002a5b6000 sz 48 > ffffffff802c9018+dc (?,?,?,?) ra ffffffff805563dc sp c00000002a5b6030 sz 96 > cpu_intr+248 (?,?,?,?) ra ffffffff80538110 sp c00000002a5b6090 sz 80 > MipsKernIntr+188 (?,?,?,?) ra 0 sp c00000002a5b60e0 sz 368 > pid 607 > KDB: enter: panic > > (In particular, we probably should be using the trapframe logic employed > here for MipsKernIntr.) > > I've patched kgdb to add a MIPS kernel target and taught it how to handle > trapframes, etc. (and it can cross-debug, so I used an amd64 host to examine > the dump) and it gives a more useful stack trace. Unfortunately, increasing > the kstack size on MIPS as a workaround for this issue seems to be very > hard as the code in cpu_switch() hard codes exactly two TLB entries to cover > the kernel stack and PCB to avoid TLB faults during a context switch. I think > if you create a kthread with a non-default kstack size on MIPS it's probably > a ticking time bomb in that you might get a TLB miss when accessing the pcb. > > savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 > 171 SAVE_U_PCB_CONTEXT(ra, PCB_REG_PC, a0) > (kgdb) where > #0 savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 > #1 0xffffffff80304dfc in doadump (textdump=textdump@entry=0) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:297 > #2 0xffffffff8015d868 in db_dump (dummy=<optimized out>, > dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:546 > #3 0xffffffff8015e474 in db_command ( > last_cmdp=last_cmdp@entry=0xffffffff8069e778 <db_last_command>, > cmd_table=<optimized out>, > cmd_table@entry=0xffffffff8069e730 <db_cmd_table>, dopager=dopager@entry=1) > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:453 > #4 0xffffffff8015eab8 in db_command_loop () > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:506 > #5 0xffffffff80162040 in db_trap (type=<optimized out>, code=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/ddb/db_main.c:248 > #6 0xffffffff803583e8 in kdb_trap (type=type@entry=9, code=code@entry=0, > tf=tf@entry=0xffffffff80699ca0 <pcpu_space+7328>) > at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:654 > #7 0xffffffff8054ba10 in trap (trapframe=0xffffffff80699ca0 <pcpu_space+7328>) > at /usr/home/john/work/git/freebsd/sys/mips/mips/trap.c:828 > #8 <signal handler called> > #9 kdb_enter (why=0xffffffff805b6138 "panic", msg=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:444 > #10 0xffffffff8030503c in vpanic (fmt=<optimized out>, > ap=ap@entry=0xffffffff80699e58 <pcpu_space+7768>) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:752 > ---Type <return> to continue, or q <return> to quit--- > #11 0xffffffff80305d84 in panic (fmt=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:690 > #12 <signal handler called> > #13 0xffffffff802c901c in intr_event_handle (ie=0x9800000000a42400, > frame=frame@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1397 > #14 0xffffffff805544e4 in gt_pci_intr (v=0x9800000000ab0800) > at /usr/home/john/work/git/freebsd/sys/mips/malta/gt_pci.c:236 > #15 0xffffffff802c90f4 in intr_event_handle (ie=0x9800000000a43900, > frame=frame@entry=0xc00000002a5b6100) > at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1436 > #16 0xffffffff805563dc in cpu_intr (tf=0xc00000002a5b6100) > at /usr/home/john/work/git/freebsd/sys/mips/mips/intr_machdep.c:264 > #17 <signal handler called> > #18 le_pci_wrcsr (sc=0x9800000000ae9400, port=<optimized out>, > val=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/dev/le/if_le_pci.c:187 > #19 0xffffffff801893e4 in am79900_start_locked (sc=0x9800000000ae9400) > at /usr/home/john/work/git/freebsd/sys/dev/le/am79900.c:594 > #20 0xffffffff8018b1c0 in lance_start (ifp=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/dev/le/lance.c:243 > #21 0xffffffff804137e4 in if_start (ifp=ifp@entry=0x9800000000ae4800) > at /usr/home/john/work/git/freebsd/sys/net/if.c:3630 > #22 0xffffffff80413998 in if_transmit (ifp=0x9800000000ae4800, > m=<optimized out>) at /usr/home/john/work/git/freebsd/sys/net/if.c:3642 > #23 0xffffffff804174fc in ether_output_frame ( > ---Type <return> to continue, or q <return> to quit--- > ifp=ifp@entry=0x9800000000ae4800, m=0x980000000162d100) > at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:457 > #24 0xffffffff80417b94 in ether_output (ifp=0x9800000000ae4800, > m=0x980000000162d100, dst=0x98000000032878f0, ro=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:429 > #25 0xffffffff8045cf28 in ip_output (m=m@entry=0x980000000162d100, > opt=<optimized out>, ro=0x98000000032878d0, flags=<optimized out>, > imo=<optimized out>, imo@entry=0x0, inp=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/netinet/ip_output.c:664 > #26 0xffffffff80470348 in tcp_output (tp=0x9800000001f4f000) > at /usr/home/john/work/git/freebsd/sys/netinet/tcp_output.c:1432 > #27 0xffffffff80480d1c in tcp_usr_send (so=0x98000000032a8a20, > flags=<optimized out>, m=0x9800000001621a00, nam=0x0, > control=<optimized out>, td=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/netinet/tcp_usrreq.c:956 > #28 0xffffffff803a6ef8 in sosend_generic (so=0x98000000032a8a20, > addr=<optimized out>, uio=0x0, top=0x9800000001621a00, > control=<optimized out>, flags=<optimized out>, td=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1359 > #29 0xffffffff803a7060 in sosend (so=<optimized out>, addr=addr@entry=0x0, > uio=uio@entry=0x0, top=top@entry=0x9800000001621a00, > control=control@entry=0x0, flags=flags@entry=0, td=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1403 > #30 0xffffffff8049220c in clnt_vc_call (cl=<optimized out>, > ext=0xc00000002a5b69e8, proc=<optimized out>, args=0x980000000162d200, > resultsp=0xc00000002a5b6b40, utimeout=...) > ---Type <return> to continue, or q <return> to quit--- > at /usr/home/john/work/git/freebsd/sys/rpc/clnt_vc.c:398 > #31 0xffffffff804908a4 in clnt_reconnect_call (cl=0x9800000001630400, > ext=0xc00000002a5b69e8, proc=<optimized out>, args=0x980000000162d200, > resultsp=0xc00000002a5b6b40, utimeout=...) > at /usr/home/john/work/git/freebsd/sys/rpc/clnt_rc.c:271 > #32 0xffffffff801dd360 in newnfs_request (nd=0xc00000002a5b6b40, > nmp=0x980000000163b800, clp=clp@entry=0x0, nrp=0x980000000163b928, > vp=0x98000000044aa000, td=td@entry=0x9800000003275a60, > cred=cred@entry=0x9800000001670000, prog=prog@entry=100003, vers=3, > retsum=retsum@entry=0x0, toplevel=toplevel@entry=1, xidp=xidp@entry=0x0, > sep=sep@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfs/nfs_commonkrpc.c:746 > #33 0xffffffff8022c738 in nfscl_request (nd=nd@entry=0xc00000002a5b6b40, > vp=vp@entry=0x98000000044aa000, p=p@entry=0x9800000003275a60, > cred=cred@entry=0x9800000001670000, stuff=stuff@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clport.c:951 > #34 0xffffffff802116a8 in nfsrpc_commit (vp=0x98000000044aa000, > offset=offset@entry=65536, cnt=cnt@entry=65536, > cred=cred@entry=0x9800000001670000, p=p@entry=0x9800000003275a60, > nap=nap@entry=0xc00000002a5b6cc0, > attrflagp=attrflagp@entry=0xc00000002a5b6d88, stuff=stuff@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clrpcops.c:3640 > #35 0xffffffff80220e74 in ncl_commit (vp=vp@entry=0x98000000044aa000, > offset=offset@entry=65536, cnt=<optimized out>, > cred=cred@entry=0x9800000001670000, td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2553 > ---Type <return> to continue, or q <return> to quit--- > #36 0xffffffff8022130c in ncl_flush (vp=0x98000000044aa000, > waitfor=<optimized out>, cred=cred@entry=0x0, td=0x9800000003275a60, > commit=commit@entry=1, > called_from_renewthread=called_from_renewthread@entry=0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2774 > #37 0xffffffff8022220c in nfs_fsync (ap=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2619 > #38 0xffffffff80559508 in VOP_FSYNC_APV (vop=<optimized out>, > a=a@entry=0xc00000002a5b6fa0) at vnode_if.c:1331 > #39 0xffffffff803bcbd4 in VOP_FSYNC (td=<optimized out>, > waitfor=<optimized out>, vp=<optimized out>) at vnode_if.h:549 > #40 bufsync (bo=<optimized out>, waitfor=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_bio.c:4605 > #41 0xffffffff803e6088 in bufobj_invalbuf (bo=bo@entry=0x98000000044aa0d0, > flags=<optimized out>, slpflag=<optimized out>, slptimeo=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1633 > #42 0xffffffff803e63a8 in vinvalbuf (vp=vp@entry=0x98000000044aa000, > flags=flags@entry=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1711 > #43 0xffffffff8022df5c in ncl_vinvalbuf (vp=vp@entry=0x98000000044aa000, > flags=flags@entry=1, td=td@entry=0x9800000003275a60, > intrflg=<optimized out>, intrflg@entry=1) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1387 > #44 0xffffffff8021d5ec in nfs_setattr (ap=0xc00000002a5b7298) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:937 > #45 0xffffffff80558564 in VOP_SETATTR_APV (vop=<optimized out>, > ---Type <return> to continue, or q <return> to quit--- > a=a@entry=0xc00000002a5b7298) at vnode_if.c:799 > #46 0xffffffff80230eac in VOP_SETATTR (cred=0x9800000001670000, > vap=0xc00000002a5b71f0, vp=0x98000000044aa000) at vnode_if.h:335 > #47 ncl_write (ap=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1280 > #48 0xffffffff80558b70 in VOP_WRITE_APV (vop=<optimized out>, > a=a@entry=0xc00000002a5b74f0) at vnode_if.c:1000 > #49 0xffffffff803f9708 in VOP_WRITE (cred=<optimized out>, > ioflag=<optimized out>, uio=0xc00000002a5b7690, vp=<optimized out>) > at vnode_if.h:413 > #50 vn_io_fault_doio (args=args@entry=0xc00000002a5b76d8, > uio=uio@entry=0xc00000002a5b7690, td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:952 > #51 0xffffffff803f9894 in vn_io_fault1 (vp=vp@entry=0x98000000044aa000, > uio=uio@entry=0xc00000002a5b7690, args=args@entry=0xc00000002a5b76d8, > td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:1060 > #52 0xffffffff803fca30 in vn_rdwr (rw=rw@entry=UIO_WRITE, > vp=vp@entry=0x98000000044aa000, base=base@entry=0x160085000, > len=len@entry=16384, offset=offset@entry=131072, > segflg=segflg@entry=UIO_USERSPACE, ioflg=ioflg@entry=16641, > active_cred=active_cred@entry=0x9800000001670000, > file_cred=file_cred@entry=0x0, aresid=aresid@entry=0xc00000002a5b77a0, > td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:576 > #53 0xffffffff803fcc68 in vn_rdwr_inchunks (rw=rw@entry=UIO_WRITE, > ---Type <return> to continue, or q <return> to quit--- > vp=0x98000000044aa000, base=0x160085000, len=16384, offset=131072, > offset@entry=98304, segflg=segflg@entry=UIO_USERSPACE, > ioflg=ioflg@entry=16641, active_cred=0x9800000001670000, file_cred=0x0, > aresid=aresid@entry=0x0, td=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:641 > #54 0xffffffff802954f4 in core_write (seg=UIO_USERSPACE, offset=98304, > len=<optimized out>, base=<optimized out>, p=0xc00000002a5b7860) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1229 > #55 core_output (tmpbuf=0x0, p=0xc00000002a5b7860, offset=98304, > len=<optimized out>, base=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1243 > #56 elf64_coredump (td=<optimized out>, vp=0x98000000044aa000, > limit=9223372036854775807, flags=<optimized out>) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1384 > #57 0xffffffff8030805c in coredump (td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3477 > #58 0xffffffff8030b818 in sigexit (td=td@entry=0x9800000003275a60, > sig=sig@entry=7) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3072 > #59 0xffffffff8030c19c in postsig (sig=sig@entry=7) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:2985 > #60 0xffffffff8036f13c in ast (framep=0xc00000002a5b7d30) > at /usr/home/john/work/git/freebsd/sys/kern/subr_trap.c:314 > #61 <signal handler called> > #62 0x0000000120000ff4 in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > -- > John Baldwin >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=OmDPGsDCMXqxfnkhkifhQ%2BqiigijwDmfdBABUT1A=Vw>