From owner-svn-src-all@freebsd.org Fri Dec 23 20:59:40 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06E7BC8EBFA; Fri, 23 Dec 2016 20:59:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9AFB3F2; Fri, 23 Dec 2016 20:59:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id k184so25605705wme.1; Fri, 23 Dec 2016 12:59:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=WwgRQobpuBvJ/QYpkFf6u3FJcLvRHfO09fWmkjspHvc=; b=Hb+Z1uiwN8ZVWPMAcwWhYsAl7Uap/QZqZQzK1grj90XJmJ6oZPzc7TIStfZqxdquSr CZv98J0Tt7Mq6skfi0twJGHiOE6A9PiTLCqb2VtR9Q4XVqzoVCVZ2gFdLc71fOM7mxLh 9CfORhnAepw/mlyWmtMbn8zwn9gzBwSN34iXcd+PWenrTBGscPzJA3n9A0AWOSCY5w8o jd843vnhaWPXPHCwV9w5QG8SUKUr1Uo1nk7gr/5GgLy220z4yUT6rfyvrNdqjsp0ImLL tAIo5j0wdziXy6GbshA/lYDN5i6dBPWENglEG904txnVfnQbb+pyvqQoGK2Bo9cAtxa/ Gxmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=WwgRQobpuBvJ/QYpkFf6u3FJcLvRHfO09fWmkjspHvc=; b=AvpIkKrcwRqfrgKt6pLQtQ/eBamLEfcbMM7aRulQlav0xQEUDEgW8/6nNPg4jsPmka oq5/LFwmPQP6GscZuHR13p/0s6kLaX77tpXc4nPdS2+BGgs0/aHFksegXRHN+jRnaXoh SP7eCKJ6Kqam8hfJKCaKRgeanN7wHAIVnFTr33A03AS5jnaph7alWCuGH3AuJuh1WY8h 7ClSIbQQjZ6Bl7xrRL5zhkbBcd/UjkQcvrhleDWq7KYrjp/l6rX1JYgvd+o+Q9XHxviQ 1N+FBd80hMlDsoc3+wWNQ0yV7+nyBxzwi1zdVTL1GP8McuH/aFaiei4TecLMVzWE4tMD gnYg== X-Gm-Message-State: AIkVDXK7LnkUHsgn7f2VMjmCcJGyJtYwaZ8okDiwbIJFyCJW9sw7XQ4TgODrtchJyT12M6tnhEmGTT30v8TyJw== X-Received: by 10.28.109.156 with SMTP id b28mr15119024wmi.116.1482526777802; Fri, 23 Dec 2016 12:59:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.44.1 with HTTP; Fri, 23 Dec 2016 12:59:37 -0800 (PST) In-Reply-To: <1808755.8aTbUPpvbL@ralph.baldwin.cx> References: <201612230327.uBN3RBC1099442@repo.freebsd.org> <1808755.8aTbUPpvbL@ralph.baldwin.cx> From: Adrian Chadd Date: Fri, 23 Dec 2016 12:59:37 -0800 Message-ID: Subject: Re: svn commit: r310445 - in head/sys/mips: include mips To: John Baldwin Cc: "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Dec 2016 20:59:40 -0000 That's a ... deeep stack. :( How big is each stackframe? -adrian On 22 December 2016 at 19:49, John Baldwin wrote: > On Friday, December 23, 2016 03:27:11 AM John Baldwin wrote: >> Author: jhb >> Date: Fri Dec 23 03:27:11 2016 >> New Revision: 310445 >> URL: https://svnweb.freebsd.org/changeset/base/310445 >> >> Log: >> Teach DDB how to unwind across a kernel stack overflow. >> >> Kernel stack overflows in MIPS call panic() directly from an assembly >> handler after storing the interrupted context's registers in a >> trapframe. Rather than inferring the location of ra, sp, and pc from >> the instruction stream, recognize the pc of a kernel stack overflow >> and pull the registers from the trapframe. >> >> Sponsored by: DARPA / AFRL > > We should possibly use this same logic for other subroutines that use > trapframes. Trying to run a program that core dumps over NFS in my > qemu mips64 machine triggers the stack overflow on a kernel built with > gcc5.3 from ports. The DDB stack trace doesn't get very far: > > panic: kernel stack overflow - trapframe at 0xffffffff80699eb0 > KDB: stack backtrace: > db_trace_self+1c (?,?,?,?) ra ffffffff801620d8 sp ffffffff80699ac0 sz 16 > ffffffff801620ac+2c (?,?,?,?) ra ffffffff80357788 sp ffffffff80699ad0 sz 800 > ffffffff80357744+44 (?,?,?,?) ra ffffffff80304ff4 sp ffffffff80699df0 sz 16 > vpanic+f4 (?,?,?,?) ra ffffffff80305d84 sp ffffffff80699e00 sz 48 > panic+30 (?,ffffffff80699eb0,ffffffffffffffec,ffffffff802c901c) ra ffffffff805388bc sp ffffffff80699e30 sz 96 > MipsTLBInvalidException+360 (?,?,?,?) ra 0 sp ffffffff80699e90 sz 0 > --- Kernel Stack Overflow --- > ffffffff802c9018+4 (?,?,?,?) ra ffffffff805544e4 sp c00000002a5b5fa0 sz 96 > ffffffff80554400+e4 (?,?,?,?) ra ffffffff802c90f4 sp c00000002a5b6000 sz 48 > ffffffff802c9018+dc (?,?,?,?) ra ffffffff805563dc sp c00000002a5b6030 sz 96 > cpu_intr+248 (?,?,?,?) ra ffffffff80538110 sp c00000002a5b6090 sz 80 > MipsKernIntr+188 (?,?,?,?) ra 0 sp c00000002a5b60e0 sz 368 > pid 607 > KDB: enter: panic > > (In particular, we probably should be using the trapframe logic employed > here for MipsKernIntr.) > > I've patched kgdb to add a MIPS kernel target and taught it how to handle > trapframes, etc. (and it can cross-debug, so I used an amd64 host to examine > the dump) and it gives a more useful stack trace. Unfortunately, increasing > the kstack size on MIPS as a workaround for this issue seems to be very > hard as the code in cpu_switch() hard codes exactly two TLB entries to cover > the kernel stack and PCB to avoid TLB faults during a context switch. I think > if you create a kthread with a non-default kstack size on MIPS it's probably > a ticking time bomb in that you might get a TLB miss when accessing the pcb. > > savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 > 171 SAVE_U_PCB_CONTEXT(ra, PCB_REG_PC, a0) > (kgdb) where > #0 savectx () at /usr/home/john/work/git/freebsd/sys/mips/mips/swtch.S:171 > #1 0xffffffff80304dfc in doadump (textdump=textdump@entry=0) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:297 > #2 0xffffffff8015d868 in db_dump (dummy=, > dummy2=, dummy3=, dummy4=) > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:546 > #3 0xffffffff8015e474 in db_command ( > last_cmdp=last_cmdp@entry=0xffffffff8069e778 , > cmd_table=, > cmd_table@entry=0xffffffff8069e730 , dopager=dopager@entry=1) > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:453 > #4 0xffffffff8015eab8 in db_command_loop () > at /usr/home/john/work/git/freebsd/sys/ddb/db_command.c:506 > #5 0xffffffff80162040 in db_trap (type=, code=) > at /usr/home/john/work/git/freebsd/sys/ddb/db_main.c:248 > #6 0xffffffff803583e8 in kdb_trap (type=type@entry=9, code=code@entry=0, > tf=tf@entry=0xffffffff80699ca0 ) > at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:654 > #7 0xffffffff8054ba10 in trap (trapframe=0xffffffff80699ca0 ) > at /usr/home/john/work/git/freebsd/sys/mips/mips/trap.c:828 > #8 > #9 kdb_enter (why=0xffffffff805b6138 "panic", msg=) > at /usr/home/john/work/git/freebsd/sys/kern/subr_kdb.c:444 > #10 0xffffffff8030503c in vpanic (fmt=, > ap=ap@entry=0xffffffff80699e58 ) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:752 > ---Type to continue, or q to quit--- > #11 0xffffffff80305d84 in panic (fmt=) > at /usr/home/john/work/git/freebsd/sys/kern/kern_shutdown.c:690 > #12 > #13 0xffffffff802c901c in intr_event_handle (ie=0x9800000000a42400, > frame=frame@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1397 > #14 0xffffffff805544e4 in gt_pci_intr (v=0x9800000000ab0800) > at /usr/home/john/work/git/freebsd/sys/mips/malta/gt_pci.c:236 > #15 0xffffffff802c90f4 in intr_event_handle (ie=0x9800000000a43900, > frame=frame@entry=0xc00000002a5b6100) > at /usr/home/john/work/git/freebsd/sys/kern/kern_intr.c:1436 > #16 0xffffffff805563dc in cpu_intr (tf=0xc00000002a5b6100) > at /usr/home/john/work/git/freebsd/sys/mips/mips/intr_machdep.c:264 > #17 > #18 le_pci_wrcsr (sc=0x9800000000ae9400, port=, > val=) > at /usr/home/john/work/git/freebsd/sys/dev/le/if_le_pci.c:187 > #19 0xffffffff801893e4 in am79900_start_locked (sc=0x9800000000ae9400) > at /usr/home/john/work/git/freebsd/sys/dev/le/am79900.c:594 > #20 0xffffffff8018b1c0 in lance_start (ifp=) > at /usr/home/john/work/git/freebsd/sys/dev/le/lance.c:243 > #21 0xffffffff804137e4 in if_start (ifp=ifp@entry=0x9800000000ae4800) > at /usr/home/john/work/git/freebsd/sys/net/if.c:3630 > #22 0xffffffff80413998 in if_transmit (ifp=0x9800000000ae4800, > m=) at /usr/home/john/work/git/freebsd/sys/net/if.c:3642 > #23 0xffffffff804174fc in ether_output_frame ( > ---Type to continue, or q to quit--- > ifp=ifp@entry=0x9800000000ae4800, m=0x980000000162d100) > at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:457 > #24 0xffffffff80417b94 in ether_output (ifp=0x9800000000ae4800, > m=0x980000000162d100, dst=0x98000000032878f0, ro=) > at /usr/home/john/work/git/freebsd/sys/net/if_ethersubr.c:429 > #25 0xffffffff8045cf28 in ip_output (m=m@entry=0x980000000162d100, > opt=, ro=0x98000000032878d0, flags=, > imo=, imo@entry=0x0, inp=) > at /usr/home/john/work/git/freebsd/sys/netinet/ip_output.c:664 > #26 0xffffffff80470348 in tcp_output (tp=0x9800000001f4f000) > at /usr/home/john/work/git/freebsd/sys/netinet/tcp_output.c:1432 > #27 0xffffffff80480d1c in tcp_usr_send (so=0x98000000032a8a20, > flags=, m=0x9800000001621a00, nam=0x0, > control=, td=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/netinet/tcp_usrreq.c:956 > #28 0xffffffff803a6ef8 in sosend_generic (so=0x98000000032a8a20, > addr=, uio=0x0, top=0x9800000001621a00, > control=, flags=, td=) > at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1359 > #29 0xffffffff803a7060 in sosend (so=, addr=addr@entry=0x0, > uio=uio@entry=0x0, top=top@entry=0x9800000001621a00, > control=control@entry=0x0, flags=flags@entry=0, td=) > at /usr/home/john/work/git/freebsd/sys/kern/uipc_socket.c:1403 > #30 0xffffffff8049220c in clnt_vc_call (cl=, > ext=0xc00000002a5b69e8, proc=, args=0x980000000162d200, > resultsp=0xc00000002a5b6b40, utimeout=...) > ---Type to continue, or q to quit--- > at /usr/home/john/work/git/freebsd/sys/rpc/clnt_vc.c:398 > #31 0xffffffff804908a4 in clnt_reconnect_call (cl=0x9800000001630400, > ext=0xc00000002a5b69e8, proc=, args=0x980000000162d200, > resultsp=0xc00000002a5b6b40, utimeout=...) > at /usr/home/john/work/git/freebsd/sys/rpc/clnt_rc.c:271 > #32 0xffffffff801dd360 in newnfs_request (nd=0xc00000002a5b6b40, > nmp=0x980000000163b800, clp=clp@entry=0x0, nrp=0x980000000163b928, > vp=0x98000000044aa000, td=td@entry=0x9800000003275a60, > cred=cred@entry=0x9800000001670000, prog=prog@entry=100003, vers=3, > retsum=retsum@entry=0x0, toplevel=toplevel@entry=1, xidp=xidp@entry=0x0, > sep=sep@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfs/nfs_commonkrpc.c:746 > #33 0xffffffff8022c738 in nfscl_request (nd=nd@entry=0xc00000002a5b6b40, > vp=vp@entry=0x98000000044aa000, p=p@entry=0x9800000003275a60, > cred=cred@entry=0x9800000001670000, stuff=stuff@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clport.c:951 > #34 0xffffffff802116a8 in nfsrpc_commit (vp=0x98000000044aa000, > offset=offset@entry=65536, cnt=cnt@entry=65536, > cred=cred@entry=0x9800000001670000, p=p@entry=0x9800000003275a60, > nap=nap@entry=0xc00000002a5b6cc0, > attrflagp=attrflagp@entry=0xc00000002a5b6d88, stuff=stuff@entry=0x0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clrpcops.c:3640 > #35 0xffffffff80220e74 in ncl_commit (vp=vp@entry=0x98000000044aa000, > offset=offset@entry=65536, cnt=, > cred=cred@entry=0x9800000001670000, td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2553 > ---Type to continue, or q to quit--- > #36 0xffffffff8022130c in ncl_flush (vp=0x98000000044aa000, > waitfor=, cred=cred@entry=0x0, td=0x9800000003275a60, > commit=commit@entry=1, > called_from_renewthread=called_from_renewthread@entry=0) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2774 > #37 0xffffffff8022220c in nfs_fsync (ap=) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:2619 > #38 0xffffffff80559508 in VOP_FSYNC_APV (vop=, > a=a@entry=0xc00000002a5b6fa0) at vnode_if.c:1331 > #39 0xffffffff803bcbd4 in VOP_FSYNC (td=, > waitfor=, vp=) at vnode_if.h:549 > #40 bufsync (bo=, waitfor=) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_bio.c:4605 > #41 0xffffffff803e6088 in bufobj_invalbuf (bo=bo@entry=0x98000000044aa0d0, > flags=, slpflag=, slptimeo=) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1633 > #42 0xffffffff803e63a8 in vinvalbuf (vp=vp@entry=0x98000000044aa000, > flags=flags@entry=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_subr.c:1711 > #43 0xffffffff8022df5c in ncl_vinvalbuf (vp=vp@entry=0x98000000044aa000, > flags=flags@entry=1, td=td@entry=0x9800000003275a60, > intrflg=, intrflg@entry=1) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1387 > #44 0xffffffff8021d5ec in nfs_setattr (ap=0xc00000002a5b7298) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clvnops.c:937 > #45 0xffffffff80558564 in VOP_SETATTR_APV (vop=, > ---Type to continue, or q to quit--- > a=a@entry=0xc00000002a5b7298) at vnode_if.c:799 > #46 0xffffffff80230eac in VOP_SETATTR (cred=0x9800000001670000, > vap=0xc00000002a5b71f0, vp=0x98000000044aa000) at vnode_if.h:335 > #47 ncl_write (ap=) > at /usr/home/john/work/git/freebsd/sys/fs/nfsclient/nfs_clbio.c:1280 > #48 0xffffffff80558b70 in VOP_WRITE_APV (vop=, > a=a@entry=0xc00000002a5b74f0) at vnode_if.c:1000 > #49 0xffffffff803f9708 in VOP_WRITE (cred=, > ioflag=, uio=0xc00000002a5b7690, vp=) > at vnode_if.h:413 > #50 vn_io_fault_doio (args=args@entry=0xc00000002a5b76d8, > uio=uio@entry=0xc00000002a5b7690, td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:952 > #51 0xffffffff803f9894 in vn_io_fault1 (vp=vp@entry=0x98000000044aa000, > uio=uio@entry=0xc00000002a5b7690, args=args@entry=0xc00000002a5b76d8, > td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:1060 > #52 0xffffffff803fca30 in vn_rdwr (rw=rw@entry=UIO_WRITE, > vp=vp@entry=0x98000000044aa000, base=base@entry=0x160085000, > len=len@entry=16384, offset=offset@entry=131072, > segflg=segflg@entry=UIO_USERSPACE, ioflg=ioflg@entry=16641, > active_cred=active_cred@entry=0x9800000001670000, > file_cred=file_cred@entry=0x0, aresid=aresid@entry=0xc00000002a5b77a0, > td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:576 > #53 0xffffffff803fcc68 in vn_rdwr_inchunks (rw=rw@entry=UIO_WRITE, > ---Type to continue, or q to quit--- > vp=0x98000000044aa000, base=0x160085000, len=16384, offset=131072, > offset@entry=98304, segflg=segflg@entry=UIO_USERSPACE, > ioflg=ioflg@entry=16641, active_cred=0x9800000001670000, file_cred=0x0, > aresid=aresid@entry=0x0, td=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/vfs_vnops.c:641 > #54 0xffffffff802954f4 in core_write (seg=UIO_USERSPACE, offset=98304, > len=, base=, p=0xc00000002a5b7860) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1229 > #55 core_output (tmpbuf=0x0, p=0xc00000002a5b7860, offset=98304, > len=, base=) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1243 > #56 elf64_coredump (td=, vp=0x98000000044aa000, > limit=9223372036854775807, flags=) > at /usr/home/john/work/git/freebsd/sys/kern/imgact_elf.c:1384 > #57 0xffffffff8030805c in coredump (td=td@entry=0x9800000003275a60) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3477 > #58 0xffffffff8030b818 in sigexit (td=td@entry=0x9800000003275a60, > sig=sig@entry=7) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:3072 > #59 0xffffffff8030c19c in postsig (sig=sig@entry=7) > at /usr/home/john/work/git/freebsd/sys/kern/kern_sig.c:2985 > #60 0xffffffff8036f13c in ast (framep=0xc00000002a5b7d30) > at /usr/home/john/work/git/freebsd/sys/kern/subr_trap.c:314 > #61 > #62 0x0000000120000ff4 in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > -- > John Baldwin >