From owner-freebsd-bugs@FreeBSD.ORG Mon Aug 1 17:00:33 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D540216A41F for ; Mon, 1 Aug 2005 17:00:33 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 84CD443D45 for ; Mon, 1 Aug 2005 17:00:33 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id j71H0XAj064093 for ; Mon, 1 Aug 2005 17:00:33 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id j71H0X6s064092; Mon, 1 Aug 2005 17:00:33 GMT (envelope-from gnats) Date: Mon, 1 Aug 2005 17:00:33 GMT Message-Id: <200508011700.j71H0X6s064092@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org From: Mark Blackman Cc: Subject: Re: kern/84389: 4.11-STABLE stuck in kernel-space? X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mark Blackman List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2005 17:00:34 -0000 The following reply was made to PR kern/84389; it has been noted by GNATS. From: Mark Blackman To: Marc G.Fournier Cc: freebsd-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org Subject: Re: kern/84389: 4.11-STABLE stuck in kernel-space? Date: Mon, 1 Aug 2005 17:57:08 +0100 On 31 Jul 2005, at 22:55, Marc G. Fournier wrote: > > 'k, I don't know how much RAM you have on this machine ... but, on > all of ours, we have 4G, and to fix *our* vnode issue, I have to > build the kernel/world with: 1GB ram. > > CFLAGS= -O -mpentium -pipe -g -DKVA_PAGES=512 > COPTFLAGS= -O -mpentium -pipe -DKVA_PAGES=512 > > and I have /etc/sysctl.conf setup with: > > kern.maxvnodes=522240 > > The fact that its dying at about the same time each day, I'm going > to guess that its something like the 'find' that runs for the > various periodic "security" checks that is sucking back the vnodes > for you ... > > This is all speculation, but this sounds like exactly everythign I > went through when I started to load up machines with jails :) It appears we are in a very similar line of business. :) Is there any data I can provide to pin this down. It didn't happen today, FWIW. From my reading of the core dump, it looks like it might have stuck on a file write in a newly configured jail. It was a postfix 'postdrop' process where the DDB froze it. Assuming it was actually stuck there, then that write was the bad guy. Cheers, Mark > > hope this helps ... > > On Sun, 31 Jul 2005, Mark Blackman wrote: > > >> >> >>> Number: 84389 >>> Category: kern >>> Synopsis: 4.11-STABLE stuck in kernel-space? >>> Confidential: no >>> Severity: serious >>> Priority: medium >>> Responsible: freebsd-bugs >>> State: open >>> Quarter: >>> Keywords: >>> Date-Required: >>> Class: sw-bug >>> Submitter-Id: current-users >>> Arrival-Date: Sun Jul 31 14:00:22 GMT 2005 >>> Closed-Date: >>> Last-Modified: >>> Originator: Mark Blackman >>> Release: 4.11-Stable >>> Organization: >>> Environment: >>> >> FreeBSD varadero.exonetric.net 4.11-STABLE FreeBSD 4.11-STABLE #2: >> Sat Apr 23 22:20:21 BST 2005 root@varadero.exonetric.net:/usr/ >> obj/usr/src/sys/MAIN-NOSMP i386 >> >>> Description: >>> >> System is unresponsive to any network or console userland inputs, >> however it does respond to ping and can break into ddb via serial >> console. I took a panic and rebooted. I have a full debug kernel >> and the full core dump. I've pasted in a full backtrace for >> reference and can answer any question about this core dump. Looks >> to me like some kind of issue >> on vnode file write. As a curious side note, this condition has >> manifested itself at approximately if not exactly the same time >> every day when it has happened (09:20-09:25). >> This is the first reoccurence after several months of dormancy. >> >> The key features of this system are several jails and vnodes (~15 >> each). >> >> backtrace full follows.. >> >> #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 >> error = 0 >> #1 0xc01b940b in boot (howto=256) at /usr/src/sys/kern/ >> kern_shutdown.c:316 >> howto = 256 >> #2 0xc01b9849 in panic (fmt=0xc0330744 "from debugger") at /usr/ >> src/sys/kern/kern_shutdown.c:595 >> fmt = 0xc0330744 "from debugger" >> bootopt = 256 >> buf = "from debugger", '\000' >> #3 0xc0144739 in db_panic (addr=-1070466719, have_addr=0, >> count=-1, modif=0xe3711ab0 "") >> at /usr/src/sys/ddb/db_command.c:435 >> No locals. >> #4 0xc01446d9 in db_command (last_cmdp=0xc0387eb8, >> cmd_table=0xc0387cf8, aux_cmd_tablep=0xc03aeb18) >> at /usr/src/sys/ddb/db_command.c:333 >> cmd_table = (struct command *) 0x0 >> aux_cmd_tablep = (struct command **) 0xc03aeb18 >> cmd = (struct command *) 0xc0330728 >> t = 0 >> modif = "\000\032q?h\f2?\200%\000\000?\003\000\000\000\eq?? >> \032q?h\f2??\003\000\000?\032q?\f2??\003\000\000?\003\000\000\r\000 >> \000\000?\0172?@?h\000\000\eq??\003\000\000\020\eq??\\\024??\f3?$\\ >> \024?h?>?\200\025;??W\024?\200\025;?@\017;?x\000\000\000\200\215}? >> \000\034q?4\eq?" >> addr = -1070466719 >> count = -1 >> have_addr = 0 >> result = 0 >> #5 0xc014479e in db_command_loop () at /usr/src/sys/ddb/ >> db_command.c:457 >> No locals. >> #6 0xc01468db in db_trap (type=3, code=0) at /usr/src/sys/ddb/ >> db_trap.c:71 >> bkpt = 0 >> #7 0xc02fdbae in kdb_trap (type=3, code=0, regs=0xe3711bb8) at / >> usr/src/sys/i386/i386/db_interface.c:158 >> ddb_mode = 1 >> #8 0xc030db40 in trap (frame={tf_fs = -479133680, tf_es = >> -479133680, tf_ds = -1070792688, tf_edi = -811750548, >> tf_esi = -1022605312, tf_ebp = -479126508, tf_isp = >> -479126556, tf_ebx = -811750588, tf_edx = 1017, tf_ecx = 3, >> tf_eax = 0, tf_trapno = 3, tf_err = 0, tf_eip = -1070466719, >> tf_cs = 8, tf_eflags = 70, tf_esp = -811750588, >> tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:592 >> p = (struct proc *) 0xe37d8d80 >> sticks = 13843839841289182128 >> i = 0 >> ucode = 0 >> type = 3 >> code = 0 >> eva = 0 >> #9 0xc031f961 in siointr1 (com=0xc30c4800) at machine/cpufunc.h:67 >> com = (struct com_s *) 0xc30c4800 >> line_status = 249 '?' >> modem_status = 63 '?' >> ioptr = (u_char *) 0x68c040
>> recv_data = 0 '\000' >> int_ctl = 0 '\000' >> int_ctl_new = 0 '\000' >> tc = (struct timecounter *) 0x68c040 >> count = 0 >> #10 0xc031f89b in siointr (arg=0xc30c4800) at /usr/src/sys/isa/ >> sio.c:1947 >> No locals. >> #11 0xc02fee56 in Xfastintr4 () >> No symbol table info available. >> #12 0xc02d16a9 in ufs_vnoperate (ap=0xe3711c88) at /usr/src/sys/ >> ufs/ufs/ufs_vnops.c:2376 >> ap = (struct vop_generic_args *) 0x0 >> #13 0xc01e7b9d in vtruncbuf (vp=0xe195df80, cred=0xc46e5900, >> p=0xe37d8d80, length=196581, blksize=16384) >> at vnode_if.h:1193 >> a = {a_desc = 0xc0382880, a_vp = 0xe195df80, a_bp = >> 0xcf9dab44} >> vp = (struct vnode *) 0x0 >> bp = (struct buf *) 0xcf9dab44 >> blksize = 0 >> bp = (struct buf *) 0xcf9dab44 >> nbp = (struct buf *) 0xcf9dab6c >> s = 0 >> anyfreed = 0 >> trunclbn = 12 >> #14 0xc02c139f in ffs_truncate (vp=0xe195df80, length=196581, >> flags=0, cred=0xc46e5900, p=0xe37d8d80) >> at /usr/src/sys/ufs/ffs/ffs_inode.c:314 >> flags = -510271616 >> ovp = (struct vnode *) 0xe195df80 >> lastblock = 11 >> oip = (struct inode *) 0xc4854300 >> bn = 0 >> lbn = 11 >> lastiblock = {-1, -1, -1} >> indir_lbn = {0, 0, -803029019} >> oldblks = {92528, 92536, 92544, 92552, 92560, 92568, 92576, >> 92584, 92592, 92600, 92608, 92616, 0, 0, 0} >> newblks = {92528, 92536, 92544, 92552, 92560, 92568, 92576, >> 92584, 92592, 92600, 92608, 92616, 0, 0, 0} >> fs = (struct fs *) 0xc462c000 >> bp = (struct buf *) 0xcf9490c8 >> offset = 16357 >> size = -510271616 >> level = -510271616 >> count = 16384 >> nblocks = 32 >> blocksreleased = 0 >> i = 11 >> aflags = 1 >> error = 0 >> allerror = 27 >> osize = 196608 >> #15 0xc02ca002 in ffs_write (ap=0xe3711e64) at /usr/src/sys/ufs/ >> ufs/ufs_readwrite.c:598 >> vp = (struct vnode *) 0xe195df80 >> uio = (struct uio *) 0xe3711ed4 >> ip = (struct inode *) 0xc4854300 >> fs = (struct fs *) 0xc462c000 >> bp = (struct buf *) 0x0 >> p = (struct proc *) 0xe3711ed4 >> lbn = -479125804 >> osize = 196581 >> seqcount = 127 >> blkoffset = 0 >> error = 27 >> extended = 1 >> flags = 2130706433 >> ioflag = 8323073 >> resid = 87 >> size = 0 >> xfersize = 60 >> object = 0xe188b3f4 >> #16 0xc01ef322 in vn_write (fp=0xc431f540, uio=0xe3711ed4, >> cred=0xc46e5900, flags=0, p=0xe37d8d80) at vnode_if.h:363 >> a = {a_desc = 0xc0382100, a_vp = 0xe195df80, a_uio = >> 0xe3711ed4, a_ioflag = 8323073, a_cred = 0xc46e5900} >> vp = (struct vnode *) 0xe195df80 >> uio = (struct uio *) 0xe3711ed4 >> ioflag = 8323073 >> cred = (struct ucred *) 0xc46e5900 >> fp = (struct file *) 0xc431f540 >> vp = (struct vnode *) 0xe195df80 >> error = 8323073 >> ioflag = 8323073 >> #17 0xc01c8899 in dofilewrite (p=0xe37d8d80, fp=0xc431f540, fd=2, >> buf=0x8061e40, nbyte=87, offset=-1, flags=0) >> at /usr/src/sys/sys/file.h:163 >> error = -478311040 >> fp = (struct file *) 0xc431f540 >> cred = (struct ucred *) 0x0 >> p = (struct proc *) 0xe37d8d80 >> fp = (struct file *) 0xc431f540 >> offset = 0 >> auio = {uio_iov = 0xe3711eac, uio_iovcnt = 1, uio_offset = >> 196608, uio_resid = 60, uio_segflg = UIO_USERSPACE, >> uio_rw = UIO_WRITE, uio_procp = 0xe37d8d80} >> aiov = {iov_base = 0x8061e5b "ue_enter: create file >> maildrop/381079.82030: File too large\n", iov_len = 60} >> cnt = 87 >> error = -478311040 >> ktriov = {iov_base = 0x0, iov_len = 0} >> ktruio = {uio_iov = 0xc01bebb0, uio_iovcnt = -479125788, >> uio_offset = 1240175767120, uio_resid = -1070294618, >> uio_segflg = 134656264, uio_rw = UIO_READ, uio_procp = 0x0} >> didktr = 0 >> #18 0xc01c8752 in write (p=0xe37d8d80, uap=0xe3711f80) at /usr/src/ >> sys/kern/sys_generic.c:329 >> p = (struct proc *) 0xe37d8d80 >> uap = (struct write_args *) 0xe3711f80 >> fp = (struct file *) 0xc431f540 >> error = -479125632 >> #19 0xc030e4a1 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = >> 47, tf_edi = 2, tf_esi = 0, tf_ebp = -1077937328, >> tf_isp = -479125548, tf_ebx = 134614320, tf_edx = 134614320, >> tf_ecx = 134618688, tf_eax = 4, tf_trapno = 7, >> tf_err = 2, tf_eip = 672270488, tf_cs = 31, tf_eflags = 659, >> tf_esp = -1077937372, tf_ss = 47}) >> at /usr/src/sys/i386/i386/trap.c:1175 >> params = 0xbfbffb28 "\002" >> i = 0 >> callp = (struct sysent *) 0xc038f9a0 >> p = (struct proc *) 0xe37d8d80 >> orig_tf_eflags = 659 >> sticks = 34 >> error = 0 >> narg = 3 >> args = {2, 134618688, 87, 0, 0, 0, 0, 0} >> have_mplock = 1 >> code = 4 >> #20 0xc02fea85 in Xint0x80_syscall () >> No symbol table info available. >> #21 0x8054550 in ?? () >> No symbol table info available. >> #22 0x8054e80 in ?? () >> No symbol table info available. >> #23 0x805292e in ?? () >> No symbol table info available. >> #24 0x805277f in ?? () >> No symbol table info available. >> #25 0x80526e3 in ?? () >> No symbol table info available. >> #26 0x80524bb in ?? () >> No symbol table info available. >> #27 0x804d3ba in ?? () >> No symbol table info available. >> #28 0x804b07a in ?? () >> No symbol table info available. >> #29 0x8049de2 in ?? () >> No symbol table info available. >> #30 0x80499ea in ?? () >> No symbol table info available. >> >> >> >> >>> How-To-Repeat: >>> >> not known >> >>> Fix: >>> >> not known >> >>> Release-Note: >>> Audit-Trail: >>> Unformatted: >>> >> _______________________________________________ >> freebsd-bugs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-bugs >> To unsubscribe, send any mail to "freebsd-bugs- >> unsubscribe@freebsd.org" >> >> >> > > ---- > Marc G. Fournier Hub.Org Networking Services (http:// > www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: > 7615664 >