From owner-freebsd-mips@FreeBSD.ORG Fri Nov 11 17:31:15 2011 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEED11065673 for ; Fri, 11 Nov 2011 17:31:15 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7CE108FC18 for ; Fri, 11 Nov 2011 17:31:15 +0000 (UTC) Received: by vcbfo14 with SMTP id fo14so5273276vcb.13 for ; Fri, 11 Nov 2011 09:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=HuJEK5lhJST6pn1nvuj0xV7OEvDfBAUyTxwdcbL/awQ=; b=S5GsW9KDTrqSV41HQqGJbuqSjM0B9YvxKhuzc6qLCd+4rJMJjpsjVQL6YwPByma30e 4//vuGzxqfXtmmBtqdksFERopjc9ktz/PEjBSXn8mcgsNYSkoTTRd3snna/Px9EDjFs6 RhRyGm/JKs8niFNEUat9Docq9fet8iEsoy988= MIME-Version: 1.0 Received: by 10.52.97.34 with SMTP id dx2mr22649538vdb.3.1321032674701; Fri, 11 Nov 2011 09:31:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.29.198 with HTTP; Fri, 11 Nov 2011 09:31:14 -0800 (PST) Date: Fri, 11 Nov 2011 09:31:14 -0800 X-Google-Sender-Auth: zDfudIpY8l-8GKD1IBbdUPc6if4 Message-ID: From: Adrian Chadd To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Odd X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2011 17:31:15 -0000 I noticed a UMA use-after-free error whilst tinkering with 11n (and it was one in a $LOTS of time) so I flipped on memguard to see if I could find anything. When I enable memguard + mbufs, I get the below debugging output. I've taken a look at the mbuf allocation/free and mtod dereferencing, also the output of m_print when each mbuf is freed. There's nothing (yet) obvious. I'm not sure whether it's a bug with mbuf handling, or the networking stack doing funny things for alignment and getting it wrong, or VM/UMA bugs - or a mix. But as it's easy to reproduce, I'd like to see if other MIPS hackers can flip this on and reproduce/debug it. add into kernel: options DDB options KDB options DEBUG_MEMGUARD options DEBUG_REDZONE options INVARIANTS INVARIANTS adds the use-after-free sanity checks to uma allocation constructor/destructor calls (sys/vm/uma_dbg.c). MEMGUARD is supposed to add pages before/after each allocation to (better) trap out of bounds access, but it's possible that something is touching the memory before the mbuf (and thus not picked up with the current allocation method.) It hasn't triggered in 5-10 minute tests without memguard flipped on, so it's possible this is just a uma+memguard bug. But it shouldn't be complaining like this, right? ;-) As I said, I did see "modifed after free" occasionally pop up in normal 11n AP behaviour but I have no idea where or what the triggering condition was. Thanks, Adrian # sysctl vm.memguard.desc=mbuf vm.memguard.desc: -> mbuf # ping -q -s 65500 -c 1 127.0.0.1 PING 127.0.0.1 (Memory modified after free 0xc0801f00(256) val=80818283 @ 0xc0801f00 Memory modified after free 0xc0803f00(256) val=80818283 @ 0xc0803f00 Memory modified after free 0xc0805f00(256) val=80818283 @ 0xc0805f00 Memory modified after free 0xc0807f00(256) val=80818283 @ 0xc0807f00 Memory modified after free 0xc0809f00(256) val=80818283 @ 0xc0809f00 Memory modified after free 0xc080bf00(256) val=80818283 @ 0xc080bf00 Memory modified after free 0xc080df00(256) val=80818283 @ 0xc080df00 Memory modified after free 0xc080ff00(256) val=80818283 @ 0xc080ff00 Memory modified after free 0xc0811f00(256) val=80818283 @ 0xc0811f00 Memory modified after free 0xc0813f00(256) val=0 @ 0xc0813f00 Memory modified after free 0xc0815f00(256) val=0 @ 0xc0815f00 Memory modified after free 0xc0817f00(256) val=0 @ 0xc0817f00 Memory modified after free 0xc0819f00(256) val=3d756e6c @ 0xc0819f00 Memory modified after free 0xc081bf00(256) val=6e6c696d @ 0xc081bf00 Memory modified after free 0xc081df00(256) val=e6d0 @ 0xc081df00 Memory modified after free 0xc081ff00(256) val=42360 @ 0xc081ff00 Memory modified after free 0xc0821f00(256) val=70706f72 @ 0xc0821f00 Memory modified after free 0xc0823f00(256) val=726e616d @ 0xc0823f00 Memory modified after free 0xc0825f00(256) val=20002 @ 0xc0825f00 Memory modified after free 0xc0827f00(256) val=2c02021 @ 0xc0827f00 Memory modified after free 0xc0829f00(256) val=608821 @ 0xc0829f00 Memory modified after free 0xc082bf00(256) val=12220017 @ 0xc082bf00 Memory modified after free 0xc082df00(256) val=8e250080 @ 0xc082df00 Memory modified after free 0xc082ff00(256) val=24440004 @ 0xc082ff00 Memory modified after free 0xc0831f00(256) val=399e021 @ 0xc0831f00 Memory modified after free 0xc0833f00(256) val=afb20028 @ 0xc0833f00 Memory modified after free 0xc0835f00(256) val=12000000 @ 0xc0835f00 Memory modified after free 0xc0837f00(256) val=72656542 @ 0xc0837f00 Memory modified after free 0xc0839f00(256) val=0 @ 0xc0839f00 Memory modified after free 0xc083bf00(256) val=afa20010 @ 0xc083bf00 Memory modified after free 0xc083df00(256) val=0 @ 0xc083df00 Memory modified after free 0xc083ff00(256) val=12058 @ 0xc083ff00 Memory modified after free 0xc0841f00(256) val=70646174 @ 0xc0841f00 Memory modified after free 0xc0843f00(256) val=8e420000 @ 0xc0843f00 Memory modified after free 0xc0845f00(256) val=3c1c0004 @ 0xc0845f00 Memory modified after free 0xc0847f00(256) val=8fbf0020 @ 0xc0847f00 Memory modified after free 0xc0849f00(256) val=8fbc0018 @ 0xc0849f00 127.0.0.1): 6550Memory modified after free 0xc084bf00(256) val=8e620000 @ 0xc084bf00 Memory modified after free 0xc084df00(256) val=4dadc0de @ 0xc084df00 Memory modified after free 0xc084ff00(256) val=4dadc0de @ 0xc084ff00 Memory modified after free 0xc0851f00(256) val=4dadc0de @ 0xc0851f00 Memory modified after free 0xc0853f00(256) val=4dadc0de @ 0xc0853f00 Memory modified after free 0xc0855f00(256) val=24020001 @ 0xc0855f00 Memory modified after free 0xc0857f00(256) val=24020003 @ 0xc0857f00 Memory modified after free 0xc0859f00(256) val=24e70010 @ 0xc0859f00 Memory modified after free 0xc085bf00(256) val=320f809 @ 0xc085bf00 Memory modified after free 0xc085df00(256) val=4dadc0de @ 0xc085df00 Memory modified after free 0xc085ff00(256) val=4dadc0de @ 0xc085ff00 Memory modified after free 0xc0861f00(256) val=4dadc0de @ 0xc0861f00 Memory modified after free 0xc0863f00(256) val=4dadc0de @ 0xc0863f00 Memory modified after free 0xc0865f00(256) val=4dadc0de @ 0xc0865f00 Memory modified after free 0xc0867f00(256) val=4dadc0de @ 0xc0867f00 Memory modified after free 0xc0869f00(256) val=4dadc0de @ 0xc0869f00 Memory modified after free 0xc086bf00(256) val=4dadc0de @ 0xc086bf00 Memory modified after free 0xc086df00(256) val=8fbc0018 @ 0xc086df00 Memory modified after free 0xc086ff00(256) val=320f809 @ 0xc086ff00 Memory modified after free 0xc0871f00(256) val=65654253 @ 0xc0871f00 Memory modified after free 0xc0873f00(256) val=0 @ 0xc0873f00 Memory modified after free 0xc0875f00(256) val=15 @ 0xc0875f00 Memory modified after free 0xc0877f00(256) val=21b @ 0xc0877f00 Memory modified after free 0xc0879f00(256) val=72f @ 0xc0879f00 Memory modified after free 0xc087bf00(256) val=d17 @ 0xc087bf00 Memory modified after free 0xc087df00(256) val=4dadc0de @ 0xc087df00 0 data bytes --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 411.724/411.724/411.724/0.000 ms