From owner-freebsd-net@FreeBSD.ORG Thu Mar 31 18:18:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AFED10656ED; Thu, 31 Mar 2011 18:18:08 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0FBFC8FC08; Thu, 31 Mar 2011 18:18:07 +0000 (UTC) Received: by iyj12 with SMTP id 12so3357226iyj.13 for ; Thu, 31 Mar 2011 11:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=PzhCTNtVq86/xf1SsfG5QImhzqly/GLQJpxjhBPpk2U=; b=j9hcJRmq8PUGCy/wtlQ8vIbjnZZE3bEbFx3DCGU3U6y3gzeTrX8kpNehSSfUklADa6 8zNZgvhjmmb4TYSs8BZC2viqoA7Ey5DyPv2nW5lPOTFoNawAbr944KaCCerls1rRzdp6 J0nTJyE2DAAgdJC5pIRCtKQ7Lk/yGEVRxvYKc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=xgEHmLTG+/0OZsUfw4Zg3h4O2rcgZjMD2K0KLg1HYlg0Ver4wlj1bW+v86MvpX1YP4 0cdsa9nESUPI3IynfL4fmax1sa1Mm98z0d7uKEF3SBzm5cx/vZfsf0HwkuAjgVCVlB5i YownWZUzN+rb4xbPEGRw9hfmc5UfSvyL5XMTg= Received: by 10.43.51.135 with SMTP id vi7mr3256371icb.336.1301595486535; Thu, 31 Mar 2011 11:18:06 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id gx2sm887576ibb.43.2011.03.31.11.18.03 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 31 Mar 2011 11:18:05 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 31 Mar 2011 11:16:52 -0700 From: YongHyeon PYUN Date: Thu, 31 Mar 2011 11:16:52 -0700 To: Yamagi Burmeister Message-ID: <20110331181651.GB11981@michelle.cdnetworks.com> References: <20110330173145.GB8601@michelle.cdnetworks.com> <20110330202858.GC8601@michelle.cdnetworks.com> <20110331171302.GA11981@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="xgyAXRrhYN0wYx8y" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, yongari@freebsd.org Subject: Re: Kernel memory corruption(?) with age(4) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2011 18:18:08 -0000 --xgyAXRrhYN0wYx8y Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Mar 31, 2011 at 08:07:17PM +0200, Yamagi Burmeister wrote: > On Thu, 31 Mar 2011, YongHyeon PYUN wrote: > > >>All boxes are quadcore machines with 8GB RAM, running FreeBSD/amd64. > >>After limiting the memory via hw.physmem to 3GB the problems are gone. > >>The box is running crashfree for more than 6 hours and has served over > >>300GB of data via age(4). > >> > > > >Thanks for testing. Remove the hw.physmem configuration and try > >attached patch and let me know how it goes. > > Thanks for your help, but the patch doesn't work. Another random panic - > this time "page fault in kernel mode" - with nothing age(4) or network > stack related stuff in the backtrace... > > Maybe it'll help to know about a bug fix in the linux atl1 driver, now > replaced by atlx. In git commit 5f08e46b621a769e52a9545a23ab1d5fb2aec1d4 > 64 bit DMA was disabled: > > 64-bit DMA causes data corruption with atl1. We don't know why, and > Atheros is working on it. For now, just use 32-bit DMA. This is a big > hack that is probably wrong, but it stops the bleeding. > > There was no later follow up on it. I think that this can't be problem > on FreeBSD but maybe I'm reading the driver code wrong. The kernel.org > gitweb URL is: > > http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.23.y.git;a=commitdiff;h=5f08e46b621a769e52a9545a23ab1d5fb2aec1d4 > Thanks a lot! It seems the L1 controller has data corruption issue when 64bit DMA addressing is used. Try this one. --xgyAXRrhYN0wYx8y Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="age.dma.diff2" Index: sys/dev/age/if_age.c =================================================================== --- sys/dev/age/if_age.c (revision 220116) +++ sys/dev/age/if_age.c (working copy) @@ -1092,10 +1092,13 @@ * Create Tx/Rx buffer parent tag. * L1 supports full 64bit DMA addressing in Tx/Rx buffers * so it needs separate parent DMA tag. + * XXX + * It seems enabling 64bit DMA causes data corruption. Limit + * DMA address space to 32bit. */ error = bus_dma_tag_create( bus_get_dma_tag(sc->age_dev), /* parent */ - 1, 0, /* alignment, boundary */ + BUS_SPACE_MAXADDR_32BIT, 0, /* alignment, boundary */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ @@ -2452,6 +2455,9 @@ /* Update the consumer index. */ sc->age_cdata.age_rr_cons = rr_cons; + bus_dmamap_sync(sc->age_cdata.age_rx_ring_tag, + sc->age_cdata.age_rx_ring_map, + BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); /* Sync descriptors. */ bus_dmamap_sync(sc->age_cdata.age_rr_ring_tag, sc->age_cdata.age_rr_ring_map, --xgyAXRrhYN0wYx8y--