From owner-freebsd-mips@FreeBSD.ORG Fri Nov 11 17:31:15 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEED11065673 for ; Fri, 11 Nov 2011 17:31:15 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7CE108FC18 for ; Fri, 11 Nov 2011 17:31:15 +0000 (UTC) Received: by vcbfo14 with SMTP id fo14so5273276vcb.13 for ; Fri, 11 Nov 2011 09:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=HuJEK5lhJST6pn1nvuj0xV7OEvDfBAUyTxwdcbL/awQ=; b=S5GsW9KDTrqSV41HQqGJbuqSjM0B9YvxKhuzc6qLCd+4rJMJjpsjVQL6YwPByma30e 4//vuGzxqfXtmmBtqdksFERopjc9ktz/PEjBSXn8mcgsNYSkoTTRd3snna/Px9EDjFs6 RhRyGm/JKs8niFNEUat9Docq9fet8iEsoy988= MIME-Version: 1.0 Received: by 10.52.97.34 with SMTP id dx2mr22649538vdb.3.1321032674701; Fri, 11 Nov 2011 09:31:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.29.198 with HTTP; Fri, 11 Nov 2011 09:31:14 -0800 (PST) Date: Fri, 11 Nov 2011 09:31:14 -0800 X-Google-Sender-Auth: zDfudIpY8l-8GKD1IBbdUPc6if4 Message-ID: From: Adrian Chadd To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Odd X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2011 17:31:15 -0000 I noticed a UMA use-after-free error whilst tinkering with 11n (and it was one in a $LOTS of time) so I flipped on memguard to see if I could find anything. When I enable memguard + mbufs, I get the below debugging output. I've taken a look at the mbuf allocation/free and mtod dereferencing, also the output of m_print when each mbuf is freed. There's nothing (yet) obvious. I'm not sure whether it's a bug with mbuf handling, or the networking stack doing funny things for alignment and getting it wrong, or VM/UMA bugs - or a mix. But as it's easy to reproduce, I'd like to see if other MIPS hackers can flip this on and reproduce/debug it. add into kernel: options DDB options KDB options DEBUG_MEMGUARD options DEBUG_REDZONE options INVARIANTS INVARIANTS adds the use-after-free sanity checks to uma allocation constructor/destructor calls (sys/vm/uma_dbg.c). MEMGUARD is supposed to add pages before/after each allocation to (better) trap out of bounds access, but it's possible that something is touching the memory before the mbuf (and thus not picked up with the current allocation method.) It hasn't triggered in 5-10 minute tests without memguard flipped on, so it's possible this is just a uma+memguard bug. But it shouldn't be complaining like this, right? ;-) As I said, I did see "modifed after free" occasionally pop up in normal 11n AP behaviour but I have no idea where or what the triggering condition was. Thanks, Adrian # sysctl vm.memguard.desc=mbuf vm.memguard.desc: -> mbuf # ping -q -s 65500 -c 1 127.0.0.1 PING 127.0.0.1 (Memory modified after free 0xc0801f00(256) val=80818283 @ 0xc0801f00 Memory modified after free 0xc0803f00(256) val=80818283 @ 0xc0803f00 Memory modified after free 0xc0805f00(256) val=80818283 @ 0xc0805f00 Memory modified after free 0xc0807f00(256) val=80818283 @ 0xc0807f00 Memory modified after free 0xc0809f00(256) val=80818283 @ 0xc0809f00 Memory modified after free 0xc080bf00(256) val=80818283 @ 0xc080bf00 Memory modified after free 0xc080df00(256) val=80818283 @ 0xc080df00 Memory modified after free 0xc080ff00(256) val=80818283 @ 0xc080ff00 Memory modified after free 0xc0811f00(256) val=80818283 @ 0xc0811f00 Memory modified after free 0xc0813f00(256) val=0 @ 0xc0813f00 Memory modified after free 0xc0815f00(256) val=0 @ 0xc0815f00 Memory modified after free 0xc0817f00(256) val=0 @ 0xc0817f00 Memory modified after free 0xc0819f00(256) val=3d756e6c @ 0xc0819f00 Memory modified after free 0xc081bf00(256) val=6e6c696d @ 0xc081bf00 Memory modified after free 0xc081df00(256) val=e6d0 @ 0xc081df00 Memory modified after free 0xc081ff00(256) val=42360 @ 0xc081ff00 Memory modified after free 0xc0821f00(256) val=70706f72 @ 0xc0821f00 Memory modified after free 0xc0823f00(256) val=726e616d @ 0xc0823f00 Memory modified after free 0xc0825f00(256) val=20002 @ 0xc0825f00 Memory modified after free 0xc0827f00(256) val=2c02021 @ 0xc0827f00 Memory modified after free 0xc0829f00(256) val=608821 @ 0xc0829f00 Memory modified after free 0xc082bf00(256) val=12220017 @ 0xc082bf00 Memory modified after free 0xc082df00(256) val=8e250080 @ 0xc082df00 Memory modified after free 0xc082ff00(256) val=24440004 @ 0xc082ff00 Memory modified after free 0xc0831f00(256) val=399e021 @ 0xc0831f00 Memory modified after free 0xc0833f00(256) val=afb20028 @ 0xc0833f00 Memory modified after free 0xc0835f00(256) val=12000000 @ 0xc0835f00 Memory modified after free 0xc0837f00(256) val=72656542 @ 0xc0837f00 Memory modified after free 0xc0839f00(256) val=0 @ 0xc0839f00 Memory modified after free 0xc083bf00(256) val=afa20010 @ 0xc083bf00 Memory modified after free 0xc083df00(256) val=0 @ 0xc083df00 Memory modified after free 0xc083ff00(256) val=12058 @ 0xc083ff00 Memory modified after free 0xc0841f00(256) val=70646174 @ 0xc0841f00 Memory modified after free 0xc0843f00(256) val=8e420000 @ 0xc0843f00 Memory modified after free 0xc0845f00(256) val=3c1c0004 @ 0xc0845f00 Memory modified after free 0xc0847f00(256) val=8fbf0020 @ 0xc0847f00 Memory modified after free 0xc0849f00(256) val=8fbc0018 @ 0xc0849f00 127.0.0.1): 6550Memory modified after free 0xc084bf00(256) val=8e620000 @ 0xc084bf00 Memory modified after free 0xc084df00(256) val=4dadc0de @ 0xc084df00 Memory modified after free 0xc084ff00(256) val=4dadc0de @ 0xc084ff00 Memory modified after free 0xc0851f00(256) val=4dadc0de @ 0xc0851f00 Memory modified after free 0xc0853f00(256) val=4dadc0de @ 0xc0853f00 Memory modified after free 0xc0855f00(256) val=24020001 @ 0xc0855f00 Memory modified after free 0xc0857f00(256) val=24020003 @ 0xc0857f00 Memory modified after free 0xc0859f00(256) val=24e70010 @ 0xc0859f00 Memory modified after free 0xc085bf00(256) val=320f809 @ 0xc085bf00 Memory modified after free 0xc085df00(256) val=4dadc0de @ 0xc085df00 Memory modified after free 0xc085ff00(256) val=4dadc0de @ 0xc085ff00 Memory modified after free 0xc0861f00(256) val=4dadc0de @ 0xc0861f00 Memory modified after free 0xc0863f00(256) val=4dadc0de @ 0xc0863f00 Memory modified after free 0xc0865f00(256) val=4dadc0de @ 0xc0865f00 Memory modified after free 0xc0867f00(256) val=4dadc0de @ 0xc0867f00 Memory modified after free 0xc0869f00(256) val=4dadc0de @ 0xc0869f00 Memory modified after free 0xc086bf00(256) val=4dadc0de @ 0xc086bf00 Memory modified after free 0xc086df00(256) val=8fbc0018 @ 0xc086df00 Memory modified after free 0xc086ff00(256) val=320f809 @ 0xc086ff00 Memory modified after free 0xc0871f00(256) val=65654253 @ 0xc0871f00 Memory modified after free 0xc0873f00(256) val=0 @ 0xc0873f00 Memory modified after free 0xc0875f00(256) val=15 @ 0xc0875f00 Memory modified after free 0xc0877f00(256) val=21b @ 0xc0877f00 Memory modified after free 0xc0879f00(256) val=72f @ 0xc0879f00 Memory modified after free 0xc087bf00(256) val=d17 @ 0xc087bf00 Memory modified after free 0xc087df00(256) val=4dadc0de @ 0xc087df00 0 data bytes --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 411.724/411.724/411.724/0.000 ms From owner-freebsd-mips@FreeBSD.ORG Sat Nov 12 13:59:20 2011 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FE81106566B; Sat, 12 Nov 2011 13:59:20 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id 421638FC0C; Sat, 12 Nov 2011 13:59:20 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.4) with ESMTP id pACDxJLf073758; Sat, 12 Nov 2011 08:59:19 -0500 (EST) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.4/Submit) id pACDxJ6N073650; Sat, 12 Nov 2011 13:59:19 GMT (envelope-from tinderbox@freebsd.org) Date: Sat, 12 Nov 2011 13:59:19 GMT Message-Id: <201111121359.pACDxJ6N073650@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [head tinderbox] failure on mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Nov 2011 13:59:20 -0000 TB --- 2011-11-12 13:11:23 - tinderbox 2.8 running on freebsd-current.sentex.ca TB --- 2011-11-12 13:11:23 - starting HEAD tinderbox run for mips/mips TB --- 2011-11-12 13:11:23 - cleaning the object tree TB --- 2011-11-12 13:11:32 - cvsupping the source tree TB --- 2011-11-12 13:11:32 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/HEAD/mips/mips/supfile TB --- 2011-11-12 13:11:45 - building world TB --- 2011-11-12 13:11:45 - CROSS_BUILD_TESTING=YES TB --- 2011-11-12 13:11:45 - MAKEOBJDIRPREFIX=/obj TB --- 2011-11-12 13:11:45 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2011-11-12 13:11:45 - SRCCONF=/dev/null TB --- 2011-11-12 13:11:45 - TARGET=mips TB --- 2011-11-12 13:11:45 - TARGET_ARCH=mips TB --- 2011-11-12 13:11:45 - TZ=UTC TB --- 2011-11-12 13:11:45 - __MAKE_CONF=/dev/null TB --- 2011-11-12 13:11:45 - cd /src TB --- 2011-11-12 13:11:45 - /usr/bin/make -B buildworld >>> World build started on Sat Nov 12 13:11:46 UTC 2011 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] cc -fpic -DPIC -O -pipe -G0 -I/src/sbin/geom/class/mountver/../.. -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/sbin/geom/class/mountver/../../misc/subr.c -o subr.So building shared library geom_mountver.so gzip -cn /src/sbin/geom/class/mountver/gmountver.8 > gmountver.8.gz ===> sbin/geom/class/multipath (all) cc -fpic -DPIC -O -pipe -G0 -I/src/sbin/geom/class/multipath/../../../../sys -I/src/sbin/geom/class/multipath/../.. -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/sbin/geom/class/multipath/geom_multipath.c -o geom_multipath.So cc1: warnings being treated as errors /src/sbin/geom/class/multipath/geom_multipath.c: In function 'mp_label': /src/sbin/geom/class/multipath/geom_multipath.c:243: warning: comparison between signed and unsigned *** Error code 1 Stop in /src/sbin/geom/class/multipath. *** Error code 1 Stop in /src/sbin/geom/class. *** Error code 1 Stop in /src/sbin/geom. *** Error code 1 Stop in /src/sbin. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2011-11-12 13:59:19 - WARNING: /usr/bin/make returned exit code 1 TB --- 2011-11-12 13:59:19 - ERROR: failed to build world TB --- 2011-11-12 13:59:19 - 2007.28 user 578.16 system 2875.66 real http://tinderbox.freebsd.org/tinderbox-head-HEAD-mips-mips.full From owner-freebsd-mips@FreeBSD.ORG Sat Nov 12 18:57:19 2011 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 053BF1065673; Sat, 12 Nov 2011 18:57:19 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id B4B248FC0A; Sat, 12 Nov 2011 18:57:18 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.4) with ESMTP id pACIvH68077635; Sat, 12 Nov 2011 13:57:17 -0500 (EST) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.4/Submit) id pACIvHP3077599; Sat, 12 Nov 2011 18:57:17 GMT (envelope-from tinderbox@freebsd.org) Date: Sat, 12 Nov 2011 18:57:17 GMT Message-Id: <201111121857.pACIvHP3077599@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [head tinderbox] failure on mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Nov 2011 18:57:19 -0000 TB --- 2011-11-12 18:06:36 - tinderbox 2.8 running on freebsd-current.sentex.ca TB --- 2011-11-12 18:06:36 - starting HEAD tinderbox run for mips/mips TB --- 2011-11-12 18:06:36 - cleaning the object tree TB --- 2011-11-12 18:06:45 - cvsupping the source tree TB --- 2011-11-12 18:06:45 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/HEAD/mips/mips/supfile TB --- 2011-11-12 18:07:31 - building world TB --- 2011-11-12 18:07:31 - CROSS_BUILD_TESTING=YES TB --- 2011-11-12 18:07:31 - MAKEOBJDIRPREFIX=/obj TB --- 2011-11-12 18:07:31 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2011-11-12 18:07:31 - SRCCONF=/dev/null TB --- 2011-11-12 18:07:31 - TARGET=mips TB --- 2011-11-12 18:07:31 - TARGET_ARCH=mips TB --- 2011-11-12 18:07:31 - TZ=UTC TB --- 2011-11-12 18:07:31 - __MAKE_CONF=/dev/null TB --- 2011-11-12 18:07:31 - cd /src TB --- 2011-11-12 18:07:31 - /usr/bin/make -B buildworld >>> World build started on Sat Nov 12 18:07:32 UTC 2011 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] cc -fpic -DPIC -O -pipe -G0 -I/src/sbin/geom/class/mountver/../.. -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/sbin/geom/class/mountver/../../misc/subr.c -o subr.So building shared library geom_mountver.so gzip -cn /src/sbin/geom/class/mountver/gmountver.8 > gmountver.8.gz ===> sbin/geom/class/multipath (all) cc -fpic -DPIC -O -pipe -G0 -I/src/sbin/geom/class/multipath/../../../../sys -I/src/sbin/geom/class/multipath/../.. -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/sbin/geom/class/multipath/geom_multipath.c -o geom_multipath.So cc1: warnings being treated as errors /src/sbin/geom/class/multipath/geom_multipath.c: In function 'mp_label': /src/sbin/geom/class/multipath/geom_multipath.c:243: warning: comparison between signed and unsigned *** Error code 1 Stop in /src/sbin/geom/class/multipath. *** Error code 1 Stop in /src/sbin/geom/class. *** Error code 1 Stop in /src/sbin/geom. *** Error code 1 Stop in /src/sbin. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2011-11-12 18:57:17 - WARNING: /usr/bin/make returned exit code 1 TB --- 2011-11-12 18:57:17 - ERROR: failed to build world TB --- 2011-11-12 18:57:17 - 2146.34 user 601.64 system 3041.55 real http://tinderbox.freebsd.org/tinderbox-head-HEAD-mips-mips.full