From owner-freebsd-stable@FreeBSD.ORG Sun Oct 21 22:09:19 2012 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC5C375E for ; Sun, 21 Oct 2012 22:09:19 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 71BF38FC17 for ; Sun, 21 Oct 2012 22:09:19 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hr7so1549528wib.13 for ; Sun, 21 Oct 2012 15:09:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=ReOsBTDD6AxaIhe2dO3ZC5zwm2x1QqnUaQS3hrWbN54=; b=XoAH+isrK9j7QzzpSO6Gwbey2a0Mi6duLTapZ6RvEbweQtmJndJ0VYF/Hf6Dyt0Z1B hnrmyY09dHeexCDdiQH+g45kPcdau9malLfgqsUpcXZv5y7V1TfUVeqbVpDuJ/+zJ86v IDDWEjrSMhxPsOhdMJYSupEJ9CHN69Mp05BbmZV41L1yCpDDP2tmu0P9cpdV5XU87Ikj UqDcnJLKnwzWLkh0h+pM7duf/xdGJpj3Y8751uTYVHI0d1zZtySFX1QaIvVp8dt8/4LS +w5WGUyWzBTsllDuL//O4j4qDC8cvqqK6xev/t9UFNxJIv72JUovWA92OuLH4X/85Xei iz5w== Received: by 10.180.79.202 with SMTP id l10mr16756637wix.9.1350857358212; Sun, 21 Oct 2012 15:09:18 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPS id ay10sm18345452wib.2.2012.10.21.15.09.17 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 21 Oct 2012 15:09:17 -0700 (PDT) Date: Mon, 22 Oct 2012 00:09:08 +0200 From: Mateusz Guzik To: David Wolfskill Subject: Re: stable/9 @r241776 panic: REDZONE: Buffer underflow detected... Message-ID: <20121021220908.GA20958@dft-labs.eu> References: <20121020141019.GW1817@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20121020141019.GW1817@albert.catwhisker.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 22:09:20 -0000 On Sat, Oct 20, 2012 at 07:10:19AM -0700, David Wolfskill wrote: > This seems ... fairly weird to me. > > Yesterday, I built & booted: > > FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #274 241726M: Fri Oct 19 05:40:05 PDT 2012 root@g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY i386 > > and used the machine all day; nothing unusual (including various > reboots (e.g. when I disembarked the train for the final leg of my > commute home, so I powered the laptop off). > > This morning, I built: > > FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #275 241776M: Sat Oct 20 04:34:45 PDT 2012 root@g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY i386 > > and on first reboot, I got a panic. > [..] > > ... > Starting devd. > REDZONE: Buffer underflow detected. 1 byte corrupted before 0xced40080 (4294966796 bytes allocated). > Allocation backtrace: > #0 0xc0ceac8f at redzone_setup+0xcf > #1 0xc0a5d5c9 at malloc+0x1d9 > ...[about 20 more such lines I didn't record]... > > > bt > Tracing pid 901 tid 100106 td 0xd2b99000 > kdb_enter(...) > panic(...) > free(...) > devread(ce8c2d00,f7274c0c,0,c0b1e4f0,d279e380,...) at devread+0x1a6 > giant_read(...) at giant_read+0x87 > devfs_read(...) at devfs_read+0xc6 > dofileread(...) at dofileread+0x99 > sys_read(...) at sys_read+0x98 > syscall(f7274d08) at syscall+0x387 > This looks a lot like issue you reported a couple of months earlier, even affected buffer address matches. At least part of REDZONE metadata placed directly before the buffer is corrupted. So the idea is to set a watchpoint at a place that is known to contain wrong data (in this case allocation size) and wait for some code to try to modify it. I hacked up the following (really ugly, but should do the job): http://people.freebsd.org/~mjg/patches/watchpoint-hack.diff Note: this assumes that address of affected buffer is always the same. Assuming I didn't mess anything up, instructions are simple: Just try to reproduce the issue, at some point you should be dropped to the debugger. If that happens when dumpdevice is configured, please get a core. Otherwise just a backtrace ("bt" command). Note 2: this code does no clear the watchpoint, so if it fails to catch the offending case, it may catch completely legitimate code later.