From owner-freebsd-sparc64@FreeBSD.ORG Sun Dec 23 16:26:25 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8D65FDE3; Sun, 23 Dec 2012 16:26:25 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id 3E3558FC0A; Sun, 23 Dec 2012 16:26:25 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id qBNGQOqo037854; Sun, 23 Dec 2012 16:26:24 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id qBNGQOVC037853; Sun, 23 Dec 2012 16:26:24 GMT (envelope-from tinderbox@freebsd.org) Date: Sun, 23 Dec 2012 16:26:24 GMT Message-Id: <201212231626.qBNGQOVC037853@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [releng_8 tinderbox] failure on sparc64/sparc64 Precedence: bulk X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 16:26:25 -0000 TB --- 2012-12-23 15:35:42 - tinderbox 2.10 running on freebsd-legacy2.sentex.ca TB --- 2012-12-23 15:35:42 - FreeBSD freebsd-legacy2.sentex.ca 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2012-12-23 15:35:42 - starting RELENG_8 tinderbox run for sparc64/sparc64 TB --- 2012-12-23 15:35:42 - cleaning the object tree TB --- 2012-12-23 15:35:42 - checking out /src from svn://svn.freebsd.org/base/stable/8 TB --- 2012-12-23 15:35:42 - cd /tinderbox/RELENG_8/sparc64/sparc64 TB --- 2012-12-23 15:35:42 - /usr/local/bin/svn cleanup /src TB --- 2012-12-23 15:35:49 - /usr/local/bin/svn update /src TB --- 2012-12-23 15:35:55 - building world TB --- 2012-12-23 15:35:55 - CROSS_BUILD_TESTING=YES TB --- 2012-12-23 15:35:55 - MAKEOBJDIRPREFIX=/obj TB --- 2012-12-23 15:35:55 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-12-23 15:35:55 - SRCCONF=/dev/null TB --- 2012-12-23 15:35:55 - TARGET=sparc64 TB --- 2012-12-23 15:35:55 - TARGET_ARCH=sparc64 TB --- 2012-12-23 15:35:55 - TZ=UTC TB --- 2012-12-23 15:35:55 - __MAKE_CONF=/dev/null TB --- 2012-12-23 15:35:55 - cd /src TB --- 2012-12-23 15:35:55 - /usr/bin/make -B buildworld >>> World build started on Sun Dec 23 15:35:55 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything >>> World build completed on Sun Dec 23 16:12:14 UTC 2012 TB --- 2012-12-23 16:12:14 - generating LINT kernel config TB --- 2012-12-23 16:12:14 - cd /src/sys/sparc64/conf TB --- 2012-12-23 16:12:14 - /usr/bin/make -B LINT TB --- 2012-12-23 16:12:14 - cd /src/sys/sparc64/conf TB --- 2012-12-23 16:12:14 - /usr/sbin/config -m LINT TB --- 2012-12-23 16:12:14 - building LINT kernel TB --- 2012-12-23 16:12:14 - CROSS_BUILD_TESTING=YES TB --- 2012-12-23 16:12:14 - MAKEOBJDIRPREFIX=/obj TB --- 2012-12-23 16:12:14 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-12-23 16:12:14 - SRCCONF=/dev/null TB --- 2012-12-23 16:12:14 - TARGET=sparc64 TB --- 2012-12-23 16:12:14 - TARGET_ARCH=sparc64 TB --- 2012-12-23 16:12:14 - TZ=UTC TB --- 2012-12-23 16:12:14 - __MAKE_CONF=/dev/null TB --- 2012-12-23 16:12:14 - cd /src TB --- 2012-12-23 16:12:14 - /usr/bin/make -B buildkernel KERNCONF=LINT >>> Kernel build for LINT started on Sun Dec 23 16:12:14 UTC 2012 >>> stage 1: configuring the kernel >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3.1: making dependencies >>> stage 3.2: building everything [...] cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/refcount.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sha256.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c: In function 'spa_generate_rootconf': /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: 'KM_ZERO' undeclared (first use in this function) /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: (Each undeclared identifier is reported only once /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: for each function it appears in.) *** Error code 1 Stop in /src/sys/modules/zfs. *** Error code 1 Stop in /src/sys/modules. *** Error code 1 Stop in /obj/sparc64/src/sys/LINT. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-12-23 16:26:24 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-12-23 16:26:24 - ERROR: failed to build LINT kernel TB --- 2012-12-23 16:26:24 - 2643.78 user 407.08 system 3042.59 real http://tinderbox.freebsd.org/tinderbox-releng_8-RELENG_8-sparc64-sparc64.full From owner-freebsd-sparc64@FreeBSD.ORG Sun Dec 23 20:07:43 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 85172E09; Sun, 23 Dec 2012 20:07:43 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id 344668FC12; Sun, 23 Dec 2012 20:07:43 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id qBNK7gm4038939; Sun, 23 Dec 2012 20:07:42 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id qBNK7geB038938; Sun, 23 Dec 2012 20:07:42 GMT (envelope-from tinderbox@freebsd.org) Date: Sun, 23 Dec 2012 20:07:42 GMT Message-Id: <201212232007.qBNK7geB038938@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [releng_8 tinderbox] failure on sparc64/sparc64 Precedence: bulk X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 20:07:43 -0000 TB --- 2012-12-23 19:14:34 - tinderbox 2.10 running on freebsd-legacy2.sentex.ca TB --- 2012-12-23 19:14:34 - FreeBSD freebsd-legacy2.sentex.ca 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2012-12-23 19:14:34 - starting RELENG_8 tinderbox run for sparc64/sparc64 TB --- 2012-12-23 19:14:34 - cleaning the object tree TB --- 2012-12-23 19:14:56 - checking out /src from svn://svn.freebsd.org/base/stable/8 TB --- 2012-12-23 19:14:56 - cd /tinderbox/RELENG_8/sparc64/sparc64 TB --- 2012-12-23 19:14:56 - /usr/local/bin/svn cleanup /src TB --- 2012-12-23 19:15:04 - /usr/local/bin/svn update /src TB --- 2012-12-23 19:15:08 - building world TB --- 2012-12-23 19:15:08 - CROSS_BUILD_TESTING=YES TB --- 2012-12-23 19:15:08 - MAKEOBJDIRPREFIX=/obj TB --- 2012-12-23 19:15:08 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-12-23 19:15:08 - SRCCONF=/dev/null TB --- 2012-12-23 19:15:08 - TARGET=sparc64 TB --- 2012-12-23 19:15:08 - TARGET_ARCH=sparc64 TB --- 2012-12-23 19:15:08 - TZ=UTC TB --- 2012-12-23 19:15:08 - __MAKE_CONF=/dev/null TB --- 2012-12-23 19:15:08 - cd /src TB --- 2012-12-23 19:15:08 - /usr/bin/make -B buildworld >>> World build started on Sun Dec 23 19:15:08 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything >>> World build completed on Sun Dec 23 19:53:34 UTC 2012 TB --- 2012-12-23 19:53:34 - generating LINT kernel config TB --- 2012-12-23 19:53:34 - cd /src/sys/sparc64/conf TB --- 2012-12-23 19:53:34 - /usr/bin/make -B LINT TB --- 2012-12-23 19:53:34 - cd /src/sys/sparc64/conf TB --- 2012-12-23 19:53:34 - /usr/sbin/config -m LINT TB --- 2012-12-23 19:53:34 - building LINT kernel TB --- 2012-12-23 19:53:34 - CROSS_BUILD_TESTING=YES TB --- 2012-12-23 19:53:34 - MAKEOBJDIRPREFIX=/obj TB --- 2012-12-23 19:53:34 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-12-23 19:53:34 - SRCCONF=/dev/null TB --- 2012-12-23 19:53:34 - TARGET=sparc64 TB --- 2012-12-23 19:53:34 - TARGET_ARCH=sparc64 TB --- 2012-12-23 19:53:34 - TZ=UTC TB --- 2012-12-23 19:53:34 - __MAKE_CONF=/dev/null TB --- 2012-12-23 19:53:34 - cd /src TB --- 2012-12-23 19:53:34 - /usr/bin/make -B buildkernel KERNCONF=LINT >>> Kernel build for LINT started on Sun Dec 23 19:53:34 UTC 2012 >>> stage 1: configuring the kernel >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3.1: making dependencies >>> stage 3.2: building everything [...] cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/refcount.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sha256.c cc -O2 -pipe -DFREEBSD_NAMECACHE -DBUILDING_ZFS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/src/sys/modules/zfs/../../cddl/compat/opensolaris -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/zmod -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common -I/src/sys/modules/zfs/../.. -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/zfs -I/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common -I/src/sys/modules/zfs/../../../include -DHAVE_KERNEL_OPTION_HEADERS -include /obj/sparc64/src/sys/LINT/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -I/obj/sparc64/src/sys/LINT -mcmodel=medany -msoft-float -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-ar! ith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c: In function 'spa_generate_rootconf': /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: 'KM_ZERO' undeclared (first use in this function) /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: (Each undeclared identifier is reported only once /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3785: error: for each function it appears in.) *** Error code 1 Stop in /src/sys/modules/zfs. *** Error code 1 Stop in /src/sys/modules. *** Error code 1 Stop in /obj/sparc64/src/sys/LINT. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-12-23 20:07:42 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-12-23 20:07:42 - ERROR: failed to build LINT kernel TB --- 2012-12-23 20:07:42 - 2733.95 user 433.50 system 3187.93 real http://tinderbox.freebsd.org/tinderbox-releng_8-RELENG_8-sparc64-sparc64.full From owner-freebsd-sparc64@FreeBSD.ORG Sun Dec 23 21:56:46 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 69E71AA1; Sun, 23 Dec 2012 21:56:46 +0000 (UTC) (envelope-from cross@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id 328CB8FC0A; Sun, 23 Dec 2012 21:56:46 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [IPv6:2001:470:e24c:200:ea06:88ff:feca:960e]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBNLuipj027750 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sun, 23 Dec 2012 16:56:45 -0500 (EST) From: Chris Ross Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Changes to kern.geom.debugflags? Date: Sun, 23 Dec 2012 16:56:44 -0500 Message-Id: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> To: freebsd-questions@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [IPv6:2001:470:e24c:200::ae25]); Sun, 23 Dec 2012 16:56:45 -0500 (EST) Cc: freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 21:56:46 -0000 I had brought up a machine months ago with freebsd-9-stable. I = configured it to boot off of a single disk, with ZFS, expecting I would = likely later attach the other disk to the zpool. I tried to do that = today, but find that I can't write the bootloader to either disk. Google searching shows what I used last time, that if you get a: gpart: /dev/da0a: Operation not permitted you need to run sysctl kern.geom.debugflags=3D0x10 But, that doesn't change anything for me now. I can write the boot = label (using gpart bootcode -p /boot/zfsboot ${disk}) to neither disk, = getting the same error in both cases. Has something changed recently? I'm currently using a Dec 22 9-stable = codebase, built locally with GENERIC kernel. - Chris From owner-freebsd-sparc64@FreeBSD.ORG Mon Dec 24 00:23:02 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B9B01C9 for ; Mon, 24 Dec 2012 00:23:02 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-da0-f45.google.com (mail-da0-f45.google.com [209.85.210.45]) by mx1.freebsd.org (Postfix) with ESMTP id 7BE278FC1C for ; Mon, 24 Dec 2012 00:23:02 +0000 (UTC) Received: by mail-da0-f45.google.com with SMTP id w4so2896438dam.4 for ; Sun, 23 Dec 2012 16:22:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:date:from:x-x-sender:to:cc:subject:in-reply-to :message-id:references:user-agent:mime-version:content-type :x-gm-message-state; bh=MBCwf05DwGqYCM45Cih6/F2x9xScFQ0qDCgMUHgMC8M=; b=I2eP9cJy5Im9iN1z9a9jH6tMiwSG2xWNXdeAA/VZSo0TkZMdQH8DtG1xi5MeWUwpco PMiP3IDw5yxB5vWBNb+2FYgQtRY/z8+6pj8lR7gn6PPyHN75PAI8/fKizTGCLSY4rvp5 /t/ZxtH8L4F+ep8UL1NaQbJWWBNJ6NBrAQC9JoD+tvMQQpxiiB1w+00rCrV9IEcGI9SW TRHc4UoSy7XAp+3HFkwjGPqtF4Wh2oLgTNL79H3N8N8K3YZwaDUBYzEDPUQzdXQiBcyY q/CHNNJ7fhstVOEz8mfidQ7mSI9+s/QG4c7r+3c/PrQcglNkQ+UDWUf7m7QCYgZnjXMD +Q9A== X-Received: by 10.66.83.136 with SMTP id q8mr57987918pay.83.1356308576649; Sun, 23 Dec 2012 16:22:56 -0800 (PST) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by mx.google.com with ESMTPS id pj1sm11235205pbb.71.2012.12.23.16.22.53 (version=SSLv3 cipher=OTHER); Sun, 23 Dec 2012 16:22:55 -0800 (PST) Date: Sun, 23 Dec 2012 14:22:51 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Ian Lepore Subject: Re: Call for testing and review, busdma changes In-Reply-To: <1355085250.87661.345.camel@revolution.hippie.lan> Message-ID: References: <1355077061.87661.320.camel@revolution.hippie.lan> <1355085250.87661.345.camel@revolution.hippie.lan> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Gm-Message-State: ALoCoQkgKRdqsbfit6mqgko07lOFTq/n4XGCGTYAt3QDUg2RPKvxXIpJn8KXOfmyaakTQ16wVDwQ X-Mailman-Approved-At: Mon, 24 Dec 2012 00:39:23 +0000 Cc: powerpc@freebsd.org, marcel@freebsd.org, mips@freebsd.org, mav@freebsd.org, scottl@freebsd.org, attilio@freebsd.org, kib@freebsd.org, sparc64@freebsd.org, arm@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 00:23:02 -0000 On Sun, 9 Dec 2012, Ian Lepore wrote: > On Sun, 2012-12-09 at 08:48 -1000, Jeff Roberson wrote: >> >> On Sun, 9 Dec 2012, Ian Lepore wrote: >> >>> On Sat, 2012-12-08 at 08:51 -1000, Jeff Roberson wrote: >>>> Hello, >>>> >>>> http://people.freebsd.org/~jeff/physbio.diff >>>> >>>> I have a relative large patch that reforms the busdma API so that new >>>> types may be added without modifying every architecture's >>>> busdma_machdep.c. It does this by unifying the bus_dmamap_load_buffer() >>>> routines so that they may be called from MI code. The MD busdma is then >>>> given a chance to do any final processing in the complete() callback. >>>> This patch also contains cam changes to unify the bus_dmamap_load* >>>> handling in cam drivers. >>>> >>>> The arm and mips implementations saw the largest changes since they have >>>> to track virtual addresses for sync(). Previously this was done in a type >>>> specific way. Now it is done in a generic way by recording the list of >>>> virtuals in the map. >>>> >>>> I have verified that this patch passes make universe which includes >>>> several kernel builds from each architecture. I suspect that if I broke >>>> anything your machine simply won't boot or will throw I/O errors. There >>>> is little subtlety, it is mostly refactoring. >>>> > > First an FYI, my replies to this thread aren't making it to the mailing > lists, except when quoted by someone replying to them. They're being > held for moderator approval because they're going to too many addresses. > >>> >>> Testing on armv4/v5 platforms, the patches applied and compiled cleanly, >>> but fault-abort with a NULL deref on what appears to be the first call >>> to bus_dmamap_load(). I'll dig deeper into debugging that after >>> browsing the new code so I can figure out where to start debugging. >> >> Can you give me the file and line of the fault? >> > > Better than that, I can give you patches (attached). There were two > little glitches... the allocated slist wasn't getting attached to the > map, and when building the slist in _bus_dmamap_load_buffer() the slist > needs to participate in the coalescing logic near the end of the loop. I understand the first problem. The second, I'm not sure about. When we're going through bounce pages we use virtual to virtual. Why do you need to flush the cache lines for the source addresses? The device will only do io to the bounce pages so they are the only that need other synchronization. > > I'm not positive the attached quick-fix is perfect, but it allows one of > my arm units here to boot and run and pass some basic IO smoke tests. > I'll be trying it on other arm platforms later today. > >>> >>> Just from an initial quick browsing of the new code for armv4, I notice >>> these things... >>> >>> The new routine __bus_dmamap_mayblock() does (*flags) |= BUS_DMA_WAITOK; >>> which is a no-op because BUS_DMA_WAITOK is zero. I'm not sure what the >>> intent was, but I think any modification of the WAIT-related flags would >>> be conceptually wrong because it overrides the caller's instructions on >>> whether or not to do a deferred load when resources aren't available. >> >> The intent of the original load function seems to be that it is always >> blocking. Most architectures force the flag. The mbuf and uio load >> routines don't support blocking at all because the callback lacks the >> information needed to restart the operation. >> > > The manpage states that bus_dmamap_load() never blocks for any reason. > The mbuf and uio flavors say they are variations of bus_dmapmap_load(); > I take the implication of that to mean that they also are defined as > never blocking. > > The docs then say that WAITOK vs. NOWAIT control the scheduling of a > deferred load, and that NOWAIT is forced in the mbuf and uio cases. > Your refactored code already handles that by adding NOWAIT to the > caller-specified flags for mbufs and uio. > > So the code will do the right thing now (with your patches), but only > because (*flags) |= BUS_DMA_WAITOK is a no-op. This code is problematic. We can only support WAITOK operations for bus_dmamap_load() because the callbacks from the swi when bounce pages are available can only remember one pointer. To fix this for real we'd have to remember the original type and call the appropriate load function again. This could be solved by having a { type, pointer, length } structure that is passed between the front and back-end. This would solve your mbuf problem as well. I think I will do this next. By the way I have an mbuf branch that pushes data and mbuf structure into separate cachelines. It's not only faster on arm and the like but on x86 as well. So someone should pursue this and you'd have way fewer partial line flushes. > >>> >>> The way you've refactored the mbuf-related load routines, the MD code is >>> no longer able to tell that it's an mbuf operation. This comes back to >>> what I said a few days ago... when you look at the current code on >>> various platforms you see a lot of false similarities. The code isn't >>> nearly identical because it really all needs to be the same, it's >>> because it was all copied from the same source, and it doesn't currently >>> work correctly on arm and mips. With your changes, the ability to >>> correct the implementation on those platforms is now gone. Perhaps this >>> can be fixed easily by defining some new flags that the MI routines can >>> OR-in to give the MD routines more info on what's happening. (Correcting >>> the mbuf sync handling on arm and mips requires that we know at sync >>> time that the buffers involved are mbufs, which means we need to know at >>> map-load time so we can set a flag or a type code or something in the >>> map that we can examine at sync time.) >> >> Why do you need to know it is an mbuf if you have a record of the virtual >> addresses? The only type specific information was used to find the >> addresses. See how arm's busdma_machdep-v6 ws implemented. >> > > Because there is a special rule for mbufs on VIVT-cache architectures: > the address of the data buffer is not aligned to a cacheline boundary, > but the sync code is allowed to treat the buffer as if it is aligned and > thus avoid invoking the partial cacheline flush algorithm. That > algorithm was shown a few months ago to be fatally flawed, and we're on > a path (albeit in super-slow-motion) to eliminate it. You can skip the sync because you know the information in the mbuf header is not necessary? > > I have patches to fix the mbuf partial sync and other things related to > partial sync problems, and my next step will be to try to rework them to > fit in with what you've done. Based on what I've seen so far in your > refactoring, I think just passing a flag to say it's an mbuf, from the > MI to the MD mapping routine, will provide enough info. I think I described a better approach above that will solve mutliple problems. Let me know what you think. Jeff > >>> >>>> The next step is to allow for dma loading of physical addresses. This >>>> will permit unmapped I/O. Which is a significant performance optimization >>>> targeted for 10.0. >>> >>> This sounds scary for arm and mips, or more specifically for VIVT cache >>> platforms that can only do a sync based on virtual addresses. Can you >>> talk some more about the plans for this? >> >> It will be for addresses which were never mapped into kva. To support >> unmaapped io. There will only be a need for bounce pages, no syncing. >> > > Can the pages be mapped into any non-kernel vmspaces? If they aren't > mapped into any vmspace at all, then I think there'll be no implications > for VIVT cache architectures. If they appear in any pmap at all then > there may be problems. > > -- Ian > > From owner-freebsd-sparc64@FreeBSD.ORG Mon Dec 24 11:06:50 2012 Return-Path: Delivered-To: freebsd-sparc64@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5087E680 for ; Mon, 24 Dec 2012 11:06:50 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 354978FC1D for ; Mon, 24 Dec 2012 11:06:50 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBOB6oCW066181 for ; Mon, 24 Dec 2012 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBOB6nMO066179 for freebsd-sparc64@FreeBSD.org; Mon, 24 Dec 2012 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Dec 2012 11:06:49 GMT Message-Id: <201212241106.qBOB6nMO066179@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-sparc64@FreeBSD.org Subject: Current problem reports assigned to freebsd-sparc64@FreeBSD.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 11:06:50 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o sparc/170663 sparc64 panics with VIA 6421 SATA150 controller on Blade 1500 o sparc/169669 sparc64 Something seems broken in sparc64 TLS or lang/lua o sparc/164227 sparc64 [boot] Can't boot 9.0-RELEASE/sparc64 on Blade 1500 s sparc/164226 sparc64 [cd] Data corruption on 9.0-RELEASE when reading from o sparc/162513 sparc64 mpt(4), mptutil(8) reports variable, erroneous drive i o sparc/141918 sparc64 [ehci] ehci_interrupt: unrecoverable error, controller s sparc/139134 sparc64 kernel output corruption s sparc/107087 sparc64 [hang] system is hung during boot from CD o sparc/105048 sparc64 [trm] trm(4) panics on sparc64 o sparc/104428 sparc64 [nullfs] nullfs panics on E4500 (but not E420) o sparc/71729 sparc64 printf in kernel thread causes panic on SPARC 11 problems total. From owner-freebsd-sparc64@FreeBSD.ORG Mon Dec 24 20:43:14 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 742BA1A7; Mon, 24 Dec 2012 20:43:14 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 08F738FC0C; Mon, 24 Dec 2012 20:43:07 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id qBOKh0jZ083430; Mon, 24 Dec 2012 13:43:00 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id qBOKgt31071102; Mon, 24 Dec 2012 13:42:55 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: Call for testing and review, busdma changes From: Ian Lepore To: Jeff Roberson In-Reply-To: References: <1355077061.87661.320.camel@revolution.hippie.lan> <1355085250.87661.345.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Dec 2012 13:42:55 -0700 Message-ID: <1356381775.1129.181.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 24 Dec 2012 21:22:01 +0000 Cc: powerpc@freebsd.org, marcel@freebsd.org, mips@freebsd.org, mav@freebsd.org, scottl@freebsd.org, attilio@freebsd.org, kib@freebsd.org, sparc64@freebsd.org, arm@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 20:43:14 -0000 On Sun, 2012-12-23 at 14:22 -1000, Jeff Roberson wrote: > On Sun, 9 Dec 2012, Ian Lepore wrote: > > > On Sun, 2012-12-09 at 08:48 -1000, Jeff Roberson wrote: > >> > >> On Sun, 9 Dec 2012, Ian Lepore wrote: > >> > >>> On Sat, 2012-12-08 at 08:51 -1000, Jeff Roberson wrote: > >>>> Hello, > >>>> > >>>> http://people.freebsd.org/~jeff/physbio.diff > >>>> [...] > >>> > >>> Testing on armv4/v5 platforms, the patches applied and compiled cleanly, > >>> but fault-abort with a NULL deref on what appears to be the first call > >>> to bus_dmamap_load(). I'll dig deeper into debugging that after > >>> browsing the new code so I can figure out where to start debugging. > >> > >> Can you give me the file and line of the fault? > >> > > > > Better than that, I can give you patches (attached). There were two > > little glitches... the allocated slist wasn't getting attached to the > > map, and when building the slist in _bus_dmamap_load_buffer() the slist > > needs to participate in the coalescing logic near the end of the loop. > > I understand the first problem. The second, I'm not sure about. When > we're going through bounce pages we use virtual to virtual. Why do you > need to flush the cache lines for the source addresses? The device will > only do io to the bounce pages so they are the only that need other > synchronization. > As I indicated in my following paragraph, I didn't think my fix for constructing the sync list was perfect, it was just good enough to allow more testing to progress. In particular I don't think it would be right if bouncing were required, but it never is on the arm platforms I have to test with. The logic in your patch didn't work in the normal non-bounce case. The main loop that does the mapping works on a single page at a time. For each page it decides whether to use a bounce page instead, then it either merges the page into the current segment if it can, or starts a new segment. Your patch has it create a new sync list entry for every non-bounced page, but the sync list is sized to maxsegs not maxpages. So your logic writes outside the bounds of the sync list array, and my logic includes the virtual ranges of pages that will be bounced in the sync list (which I think should be harmless anyway, just a bit inefficient). The logic is wrong in both cases, but since bouncing never happens on my hardware I knew my tweak would let me do more testing. In the past, I've never paid close attention to the bounce logic, I've tended to read around it as if it weren't there. Recently I've begun to look at it and the more I look the more I think that to the degree that it works at all, it works by accident. I think mostly the logic just never gets used on the arm platforms that people are working with, because there'd be more problems reported if it were. For example... there is no handling of the PREREAD case when syncing bounce pages. A POSTREAD invalidate is insufficient; if any of the cache lines are dirty at the time the DMA starts, there are several ways those lines can get written back to physical memory while the DMA is in progress, corrupting the data in the IO buffer. Another example... there is an unordered list of bounce pages which are substituted for real pages as needed, one page at a time. No attempt is made to ensure that a run of contiguous physical pages that need to be bounced turns into a corresponding single run of contiguous pages in the bounce zone. In other words, it seems like over time as the list becomes less ordered the bounce logic would increasingly create multi-segment DMA. The reason that seems like a problem is that on every system I've checked (arm and i386) getting to multi-user mode creates about 100 dma tags, and typically only 5 or 6 of those tags allow more than one segment. I understand that this is not "incorrect operation" of the busdma code, but it does seem to be circumstantial evidence that this logic isn't being exercised much, since it appears that few drivers can actually handle multi-segment DMA. While the work I've done so far on arm busdma has concentrated on the buffer allocation and correct operation of the sync routines in the non-bounce cases, I think I'll be looking closely at the map load (currently seems a bit inefficient) and bounce-handling logic soon. > > > > I'm not positive the attached quick-fix is perfect, but it allows one of > > my arm units here to boot and run and pass some basic IO smoke tests. > > I'll be trying it on other arm platforms later today. > > > >>> > >>> Just from an initial quick browsing of the new code for armv4, I notice > >>> these things... > >>> > >>> The new routine __bus_dmamap_mayblock() does (*flags) |= BUS_DMA_WAITOK; > >>> which is a no-op because BUS_DMA_WAITOK is zero. I'm not sure what the > >>> intent was, but I think any modification of the WAIT-related flags would > >>> be conceptually wrong because it overrides the caller's instructions on > >>> whether or not to do a deferred load when resources aren't available. > >> > >> The intent of the original load function seems to be that it is always > >> blocking. Most architectures force the flag. The mbuf and uio load > >> routines don't support blocking at all because the callback lacks the > >> information needed to restart the operation. > >> > > > > The manpage states that bus_dmamap_load() never blocks for any reason. > > The mbuf and uio flavors say they are variations of bus_dmapmap_load(); > > I take the implication of that to mean that they also are defined as > > never blocking. > > > > The docs then say that WAITOK vs. NOWAIT control the scheduling of a > > deferred load, and that NOWAIT is forced in the mbuf and uio cases. > > Your refactored code already handles that by adding NOWAIT to the > > caller-specified flags for mbufs and uio. > > > > So the code will do the right thing now (with your patches), but only > > because (*flags) |= BUS_DMA_WAITOK is a no-op. > > This code is problematic. We can only support WAITOK operations for > bus_dmamap_load() because the callbacks from the swi when bounce pages are > available can only remember one pointer. To fix this for real we'd have > to remember the original type and call the appropriate load function > again. This could be solved by having a { type, pointer, length } > structure that is passed between the front and back-end. This would solve > your mbuf problem as well. I think I will do this next. I'm lost here. I don't understand your point about only being able to remember one pointer. We only ever need to remember one pointer, because a map can only be associated with a single buffer at a time. The fact that a deferred load can't be easily implemented in the case of mbufs and uio might be why the manpage says the NOWAIT flag is implied in those cases. There's no reason why the busdma code can't handle deferred loading for mbufs and uio -- as you say, it seems to only involve storing a bit of extra info for the callback. I think the hard part of the problem is elsewhere. For a transmit mbuf that needs a deferred mapping, does the driver's interface to the network stack allow for the concept of "this call has neither suceeded nor failed, but either outcome is possible, I'll let you know later" (I have no idea what the answer to that is but I can see how it could quickly become complex for drivers and other higher layers to deal with). For a deferred uio mapping, what if the userland pmap is no longer valid when the callback happens? What if userland has freed the memory? I suspect the implied NOWAIT requirement exists to wish away the need to handle such things. My original point was that the WAITOK flag has a value of zero. Your code gives the impression that it forces all callers to accept a deferred mapping, overriding whatever flag they may have passed, but in fact it doesn't do anything at all. Hmm, I just looked at non-arm implementations more closely and I see that some of them have the same logic as your patches, they OR-in a zero in an apparent attempt to override the caller's flags. That's contrary to what the manpage says and IMO contrary to common sense -- if a caller has said that it is unable to handle a deferred mapping callback, it's unlikely that anything is going to work correctly if a deferred callback happens (in fact, it sounds like a crash or panic waiting to happen). I think the only reason the existing code hasn't caused problems is because the flag value is zero and it actually does nothing except mislead when you read it. > By the way I have an mbuf branch that pushes data and mbuf structure into > separate cachelines. It's not only faster on arm and the like but on x86 > as well. So someone should pursue this and you'd have way fewer partial > line flushes. > [...] > You can skip the [mbuf] sync because you know the information in the mbuf header > is not necessary? > Not exactly. In the general case, if the buffer isn't aligned and padded to a cacheline boundary, we don't know anything about the memory before and after the buffer which partially shares a cacheline we're about to operate on, so the existing code attempts to preserve the contents of that unknown memory across the sync operation. (It cannot possibly be reliably sucessful in doing so, but that's another story.) In the case of an mbuf, where the data area following the header is always misaligned to a cacheline, we do know something about the memory before the data buffer: we know that it's an mbuf header which will not be accessed by the cpu while the dma is in progress. Likewise if the length isn't a multiple of the line size we know the data after the mapped length is still part of a buffer which is allocated to a multiple of the line size and the cpu won't touch that memory during the dma. Using that knowledge we can handle a PREREAD or PREWRITE by doing a wbinv on the cacheline (this is no different than the general case), and then we can handle the POSTREAD as if the buffer were aligned. We don't have to attempt to preserve partial cachelines before/after the buffer because we know the cpu hasn't touched that memory during the dma, so no cache line load could have happened. In the changes you mentioned, how do you ensure the mbuf header and data end up in separate cache lines without compile-time knowledge of cache line size? Even if the embedded data buffer in an mbuf becomes cache-aligned, the special mbuf awareness will still be needed to handle the fact that the mapped data length may not be a multiple of the cache line size. We'll still need the special knowledge that it's an mbuf so that we can sync the entire last cacheline without attempting to preserve the part beyond the end of the dma buffer. (If this idea makes you vaguely uneasy, you're not alone. However, I've been assured by Scott and Warner that it's always safe to treat mbufs this way.) > > > > I have patches to fix the mbuf partial sync and other things related to > > partial sync problems, and my next step will be to try to rework them to > > fit in with what you've done. Based on what I've seen so far in your > > refactoring, I think just passing a flag to say it's an mbuf, from the > > MI to the MD mapping routine, will provide enough info. > > I think I described a better approach above that will solve mutliple > problems. Let me know what you think. > > Jeff In the long run we can't just reduce the incidence of partial cacheline flushes, they have to be eliminated completely. We can eliminate them for mbufs because of arbitrary rules for mbufs that are supposedly universally followed in the existing code already. We can also eliminate them for any buffers which are allocated by bus_dmamem_alloc(), basically for reasons similar to mbufs: we can ensure that allocated buffers are always aligned and padded appropriately, and establish a rule that says you must not allow any other access to memory in the allocated buffer during dma. Most especially that means you can't allocate a big buffer and sub-divide it in such a way that there is concurrent cpu and dma access, or multiple concurrent dma operations, within a single buffer returned by bus_dmamem_alloc(). Still unresolved is what to do about the remaining cases -- attempts to do dma in arbitrary buffers not obtained from bus_dmamem_alloc() which are not aligned and padded appropriately. There was some discussion a while back, but no clear resolution. I decided not to get bogged down by that fact and to fix the mbuf and allocated-buffer situations that we know how to deal with for now. -- Ian From owner-freebsd-sparc64@FreeBSD.ORG Tue Dec 25 00:35:58 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CBADAFFE; Tue, 25 Dec 2012 00:35:58 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id 8109E8FC0A; Tue, 25 Dec 2012 00:35:58 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [206.138.151.12]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBP0Zv5k000719 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 24 Dec 2012 19:35:57 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Changes to kern.geom.debugflags? From: Chris Ross In-Reply-To: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> Date: Mon, 24 Dec 2012 19:35:57 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> To: freebsd-questions@freebsd.org X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [206.138.151.250]); Mon, 24 Dec 2012 19:35:57 -0500 (EST) Cc: freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 00:35:58 -0000 On Dec 23, 2012, at 16:56 , Chris Ross wrote: > I had brought up a machine months ago with freebsd-9-stable. I = configured it to boot off of a single disk, with ZFS, expecting I would = likely later attach the other disk to the zpool. I tried to do that = today, but find that I can't write the bootloader to either disk. >=20 > gpart: /dev/da0a: Operation not permitted >=20 > [...] Okay. It occurred to me today what was likely the problem. I was = running, even when single user, off of the zfs pool on the disks I was = trying to write the bootloader to. I tar'd up /boot after my recent install from a Dec 22 9-stable, and = moved it off-host. Then, I booted off of the July stable-9 CD-ROM I = have in the machine, and was able to write bootblocks (with gpart = bootcode) and a bootloader (dd if=3D/boot/zfsloader of=3D/dev/${disk}a = bs=3D512 oseek=3D1024 conv=3Dnotrunc). Now, the new problem. When I try to boot my sparc64 with these bits, = I see: >> FreeBSD/sparc64 ZFS boot block Boot path: /pci@1c,600000/scsi@2/disk@0,0:a Consoles: Open Firmware console =20 ERROR: Last Trap: Division by Zero {1} ok So, does anyone know if something has gone unstable in the sparc64 = zfsboot in recent months? If I boot from the cdrom again and load the = July zfsboot via gpart bootcode, it boots correctly again. Thanks. Any feedback appreciated. - Chris From owner-freebsd-sparc64@FreeBSD.ORG Tue Dec 25 00:02:28 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8DC189EA for ; Tue, 25 Dec 2012 00:02:28 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-da0-f48.google.com (mail-da0-f48.google.com [209.85.210.48]) by mx1.freebsd.org (Postfix) with ESMTP id 4B1B38FC15 for ; Tue, 25 Dec 2012 00:02:28 +0000 (UTC) Received: by mail-da0-f48.google.com with SMTP id k18so3314987dae.35 for ; Mon, 24 Dec 2012 16:02:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:date:from:x-x-sender:to:cc:subject:in-reply-to :message-id:references:user-agent:mime-version:content-type :x-gm-message-state; bh=I0GFDKOg9qlcIkDxXnHwe8OKNN3dwd/o++IWhz6ZEpY=; b=jD0cpe4s2br9Qbrp38M9BFcpJ8qYfgwIByjD7FwCQTngMgrkZz99Nwmfhrn/hhaa+y /KRDz4fyey0bCQpeCvT4zMudxj/yKSy5Cu8riNZbcBRW+i/IS+TeV+9/fz8M04p+I5Jy yKaWrOU2ntxmNhYSN7zP3z4bk2C/wDtbJntM6+lLW313lZPQTwDB5I0wU4UkYVappqZd STAiq/QCf8evwZMPtvLiMHyuRK0e38wxj/80NvpGmIFKzYJHD2YB7NflxvZbleUOBKDq DmrHf0GUax6Ti1yM+Tzpcm4ag8RM+2V7yT8vTblAy1dxynXP8slJ3bZfCXBcfGZlw1Hl lOjg== X-Received: by 10.68.235.40 with SMTP id uj8mr70483044pbc.98.1356393747173; Mon, 24 Dec 2012 16:02:27 -0800 (PST) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by mx.google.com with ESMTPS id oi3sm13055733pbb.1.2012.12.24.16.02.23 (version=SSLv3 cipher=OTHER); Mon, 24 Dec 2012 16:02:26 -0800 (PST) Date: Mon, 24 Dec 2012 14:02:19 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Ian Lepore Subject: Re: Call for testing and review, busdma changes In-Reply-To: <1356390225.1129.217.camel@revolution.hippie.lan> Message-ID: References: <1355077061.87661.320.camel@revolution.hippie.lan> <1355085250.87661.345.camel@revolution.hippie.lan> <1356381775.1129.181.camel@revolution.hippie.lan> <1356390225.1129.217.camel@revolution.hippie.lan> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Gm-Message-State: ALoCoQlm528LU4l8veQ30EoYsZDb6jIExVMt329Dh++vjNflaCWaEH848zEG/2zwHhVbQRq5xh8H X-Mailman-Approved-At: Tue, 25 Dec 2012 01:28:26 +0000 Cc: powerpc@freebsd.org, marcel@freebsd.org, mips@freebsd.org, mav@freebsd.org, scottl@freebsd.org, attilio@freebsd.org, kib@freebsd.org, sparc64@freebsd.org, arm@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 00:02:28 -0000 On Mon, 24 Dec 2012, Ian Lepore wrote: > On Mon, 2012-12-24 at 11:21 -1000, Jeff Roberson wrote: >> On Mon, 24 Dec 2012, Ian Lepore wrote: >> >>> On Sun, 2012-12-23 at 14:22 -1000, Jeff Roberson wrote: >>>> On Sun, 9 Dec 2012, Ian Lepore wrote: >>>> >>>>> On Sun, 2012-12-09 at 08:48 -1000, Jeff Roberson wrote: >>>>>> >>>>>> On Sun, 9 Dec 2012, Ian Lepore wrote: >>>>>> >>>>>>> On Sat, 2012-12-08 at 08:51 -1000, Jeff Roberson wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> http://people.freebsd.org/~jeff/physbio.diff >>>>>>>> >>> [...] >>>>>>> >>>>>>> Testing on armv4/v5 platforms, the patches applied and compiled cleanly, >>>>>>> but fault-abort with a NULL deref on what appears to be the first call >>>>>>> to bus_dmamap_load(). I'll dig deeper into debugging that after >>>>>>> browsing the new code so I can figure out where to start debugging. >>>>>> >>>>>> Can you give me the file and line of the fault? >>>>>> >>>>> >>>>> Better than that, I can give you patches (attached). There were two >>>>> little glitches... the allocated slist wasn't getting attached to the >>>>> map, and when building the slist in _bus_dmamap_load_buffer() the slist >>>>> needs to participate in the coalescing logic near the end of the loop. >>>> >>>> I understand the first problem. The second, I'm not sure about. When >>>> we're going through bounce pages we use virtual to virtual. Why do you >>>> need to flush the cache lines for the source addresses? The device will >>>> only do io to the bounce pages so they are the only that need other >>>> synchronization. >>>> >>> >>> As I indicated in my following paragraph, I didn't think my fix for >>> constructing the sync list was perfect, it was just good enough to allow >>> more testing to progress. In particular I don't think it would be right >>> if bouncing were required, but it never is on the arm platforms I have >>> to test with. >>> >>> The logic in your patch didn't work in the normal non-bounce case. The >>> main loop that does the mapping works on a single page at a time. For >>> each page it decides whether to use a bounce page instead, then it >>> either merges the page into the current segment if it can, or starts a >>> new segment. Your patch has it create a new sync list entry for every >>> non-bounced page, but the sync list is sized to maxsegs not maxpages. >>> >>> So your logic writes outside the bounds of the sync list array, and my >>> logic includes the virtual ranges of pages that will be bounced in the >>> sync list (which I think should be harmless anyway, just a bit >>> inefficient). The logic is wrong in both cases, but since bouncing >>> never happens on my hardware I knew my tweak would let me do more >>> testing. >> >> Ok, so I need to add coalescing and perhaps fix the bounds checking for >> virtual ranges. Correct? I can do that simply enough. I have some other >> changes I will make concurrently and then get another build that I would >> like you to test. Thank you for all the feedback so far. >> > > Okay, but did you see that Olivier committed my busdma changes recently? > It makes for a pretty significant by-hand merge with your work. I've > done that merge -- I think I sent it to you last week -- but what > Olivier committed was a wee bit different than what I merged with then. > I'm going to move this patch further along before I re-merge. I want to get buy in on the next set of API changes I'm going to make. > I think the little patch I sent originally to do the sync list > coalescing based on segments is probably good enough for now. It will > result in syncing virtual ranges that are actually being bounced, but > the arm code has always done that due to bugs (it skips the virtual > range sync only if the entire buffer fits in one bounce page). I will look at this next. mips has the same pattern and I want to make sure it's correct there too. > > I just had a close look at the armv6 implementation of mapping and > syncing for the first time, and... oh my. It's creating (malloc'ing!) a > sync list entry for every individual page of every mapped buffer. I > think it should be doing something more like my quickie-patch that bases > the sync list entries on the segments since it has to sync l2 cache > based on physical ranges for some chips. If you look at my branch I made the two arm busdma implementations identical in this regard. > >>> >>> In the past, I've never paid close attention to the bounce logic, I've >>> tended to read around it as if it weren't there. Recently I've begun to >>> look at it and the more I look the more I think that to the degree that >>> it works at all, it works by accident. I think mostly the logic just >>> never gets used on the arm platforms that people are working with, >>> because there'd be more problems reported if it were. >> >> I think it is even rarely used on x86 anymore. The number of poorly >> behaved DMA engines has been shrinking. >> >>> >>> For example... there is no handling of the PREREAD case when syncing >>> bounce pages. A POSTREAD invalidate is insufficient; if any of the >>> cache lines are dirty at the time the DMA starts, there are several ways >>> those lines can get written back to physical memory while the DMA is in >>> progress, corrupting the data in the IO buffer. >>> >>> Another example... there is an unordered list of bounce pages which are >>> substituted for real pages as needed, one page at a time. No attempt is >>> made to ensure that a run of contiguous physical pages that need to be >>> bounced turns into a corresponding single run of contiguous pages in the >>> bounce zone. In other words, it seems like over time as the list >>> becomes less ordered the bounce logic would increasingly create >>> multi-segment DMA. The reason that seems like a problem is that on >>> every system I've checked (arm and i386) getting to multi-user mode >>> creates about 100 dma tags, and typically only 5 or 6 of those tags >>> allow more than one segment. I understand that this is not "incorrect >>> operation" of the busdma code, but it does seem to be circumstantial >>> evidence that this logic isn't being exercised much, since it appears >>> that few drivers can actually handle multi-segment DMA. >>> >>> While the work I've done so far on arm busdma has concentrated on the >>> buffer allocation and correct operation of the sync routines in the >>> non-bounce cases, I think I'll be looking closely at the map load >>> (currently seems a bit inefficient) and bounce-handling logic soon. >> >> I'm shocked that we're sending I/O of larger than a page size to devices >> that only support 1 sg entry. Rather than memcpying and bouncing these >> drivers should be written to chop up the I/O themselves. For network >> devices they should not advertise jumbo frame support and for disk devices >> they can specify the maximum io size as one page already. The original >> pages that we send down would almost never be contiguous until very >> recently so this wouldn't have worked even without bounce pages. >> >>> >>>>> >>>>> I'm not positive the attached quick-fix is perfect, but it allows one of >>>>> my arm units here to boot and run and pass some basic IO smoke tests. >>>>> I'll be trying it on other arm platforms later today. >>>>> > [...] >>> The fact that a deferred load can't be easily implemented in the case of >>> mbufs and uio might be why the manpage says the NOWAIT flag is implied >>> in those cases. There's no reason why the busdma code can't handle >>> deferred loading for mbufs and uio -- as you say, it seems to only >>> involve storing a bit of extra info for the callback. I think the hard >>> part of the problem is elsewhere. For a transmit mbuf that needs a >>> deferred mapping, does the driver's interface to the network stack allow >>> for the concept of "this call has neither suceeded nor failed, but >>> either outcome is possible, I'll let you know later" (I have no idea >>> what the answer to that is but I can see how it could quickly become >>> complex for drivers and other higher layers to deal with). For a >>> deferred uio mapping, what if the userland pmap is no longer valid when >>> the callback happens? What if userland has freed the memory? I suspect >>> the implied NOWAIT requirement exists to wish away the need to handle >>> such things. >> >> Yes for uio the operation will have to be able to work on something other >> than the current pmap. Many busdma implementations remember the pmap >> because that could also be true when sync is called. Also actual uio >> operations to user pages are probably not initiated by anything in tree. >> Maybe some command interfaces for storage drivers. It is definitely an >> edge case. I would rather not support it but I guess it does make such >> things easier to implement and not require a copy. >> >> Most storage drivers have the logic that you describe above. They watch >> for EINPROGRESS. Then the callback can supply an error if the operation >> fails later. I would like to make this more uniform because cam drivers >> are about to support multiple different memory descriptors. It would be a >> shame if they each had different semantics. So I need deferred load of >> many types to work. >> > > Yeah, I've done some low-level storage driver stuff myself (mmc/sd) and > I can see how easy the deferred load solutions are to implement in that > sort of driver that's already structured to operate asychronously. I'm > not very familiar with how network hardware drivers interface with the > rest of the network stack. I have some idea, I'm just not sure of all > the subtleties involved and whether there are any implications for > something like a deferred load. > > This is one of those situations where I tend to say to myself... the > folks who designed this stuff and imposed the "no deferred load" > restriction on mbufs and uio but not other cases were not stupid or > lazy, so they must have had some other reason. I'd want to know what > that was before I went too far with trying to undo it. In network drivers you just discard the mbuf. Networks are lossy. If you persistently are out of bounce buffers you'll fail to transmit. The flags were manually or'd in for these consumers to simplify the implementation. I intend to change the underlying mechanism to support it but not the mbuf load wrappers. > >>> >>> My original point was that the WAITOK flag has a value of zero. Your >>> code gives the impression that it forces all callers to accept a >>> deferred mapping, overriding whatever flag they may have passed, but in >>> fact it doesn't do anything at all. >> >> Yes I realized that. I just made a commit to eliminate that on my branch. >> Thank you for pointing it out. >> >>> >>> Hmm, I just looked at non-arm implementations more closely and I see >>> that some of them have the same logic as your patches, they OR-in a zero >>> in an apparent attempt to override the caller's flags. That's contrary >>> to what the manpage says and IMO contrary to common sense -- if a caller >>> has said that it is unable to handle a deferred mapping callback, it's >>> unlikely that anything is going to work correctly if a deferred callback >>> happens (in fact, it sounds like a crash or panic waiting to happen). I >>> think the only reason the existing code hasn't caused problems is >>> because the flag value is zero and it actually does nothing except >>> mislead when you read it. >> >> Yes, they all seem to respect NOWAIT as well. So oring it in did nothing. >> >>> >>>> By the way I have an mbuf branch that pushes data and mbuf structure into >>>> separate cachelines. It's not only faster on arm and the like but on x86 >>>> as well. So someone should pursue this and you'd have way fewer partial >>>> line flushes. >>>> >>> [...] >>>> You can skip the [mbuf] sync because you know the information in the mbuf header >>>> is not necessary? >>>> >>> >>> Not exactly. In the general case, if the buffer isn't aligned and >>> padded to a cacheline boundary, we don't know anything about the memory >>> before and after the buffer which partially shares a cacheline we're >>> about to operate on, so the existing code attempts to preserve the >>> contents of that unknown memory across the sync operation. (It cannot >>> possibly be reliably sucessful in doing so, but that's another story.) >>> >>> In the case of an mbuf, where the data area following the header is >>> always misaligned to a cacheline, we do know something about the memory >>> before the data buffer: we know that it's an mbuf header which will not >>> be accessed by the cpu while the dma is in progress. Likewise if the >>> length isn't a multiple of the line size we know the data after the >>> mapped length is still part of a buffer which is allocated to a multiple >>> of the line size and the cpu won't touch that memory during the dma. >>> >>> Using that knowledge we can handle a PREREAD or PREWRITE by doing a >>> wbinv on the cacheline (this is no different than the general case), and >>> then we can handle the POSTREAD as if the buffer were aligned. We don't >>> have to attempt to preserve partial cachelines before/after the buffer >>> because we know the cpu hasn't touched that memory during the dma, so no >>> cache line load could have happened. >>> >>> In the changes you mentioned, how do you ensure the mbuf header and data >>> end up in separate cache lines without compile-time knowledge of cache >>> line size? >>> >>> Even if the embedded data buffer in an mbuf becomes cache-aligned, the >>> special mbuf awareness will still be needed to handle the fact that the >>> mapped data length may not be a multiple of the cache line size. We'll >>> still need the special knowledge that it's an mbuf so that we can sync >>> the entire last cacheline without attempting to preserve the part beyond >>> the end of the dma buffer. (If this idea makes you vaguely uneasy, >>> you're not alone. However, I've been assured by Scott and Warner that >>> it's always safe to treat mbufs this way.) >>> >> >> I understand now. It is safe in the usual cases but not in the general >> case. The mbuf API allows for 'ext bufs' which have reference to >> arbitrary external memory. There are no guarantees about the alignment of >> this memory or what comes before or after it. It could really be >> anything. In practice I think you'll be safe with all code that is in >> tree but I'm not 100% sure. >> > > The ext bufs are exactly the part that makes me vaguely uneasy, but I > figured I'd run with what I was told until it was proven unworkable. > > > [...] >>> >>> Still unresolved is what to do about the remaining cases -- attempts to >>> do dma in arbitrary buffers not obtained from bus_dmamem_alloc() which >>> are not aligned and padded appropriately. There was some discussion a >>> while back, but no clear resolution. I decided not to get bogged down >>> by that fact and to fix the mbuf and allocated-buffer situations that we >>> know how to deal with for now. >> >> It sounds like you're hitting the common cases. It seems ok to keep the >> partial flush logic for the less common cases and deal with the >> performance penalty. >> >> Jeff >> > > I've got to keep stressing... it's not a performance penalty thing, it's > a correct operation thing. Partial cacheline flush logic cannot be > written in a way that's guaranteed to always give correct results, > because the software can't prevent the hardware from doing an eviction > and writing back stale data even while the code that attempts to > preserve the data is running. Some day when all the other code is > right, the place where we currently detect a partial flush and try to > preserve the surrounding data will be a panic call. Until then we'll > keep relying on the partial flush usually working right by accident. I understand now. If you're sharing the line with someone who is active you can't do it safely just by flushing and cleaning before the DMA. They could re-dirty the page. I was just thinking about the sync operation and existing dirty data. Technically you should bounce in this case as well since it can't be made safe. Not bouncing for mbufs is a nice optimization I see. Jeff > > But yeah, taking care of the common cases first was my priority. > > -- Ian > > From owner-freebsd-sparc64@FreeBSD.ORG Tue Dec 25 03:13:19 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EEE7E328 for ; Tue, 25 Dec 2012 03:13:19 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm19.bullet.mail.bf1.yahoo.com (nm19.bullet.mail.bf1.yahoo.com [98.139.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 8A3F28FC17 for ; Tue, 25 Dec 2012 03:13:19 +0000 (UTC) Received: from [98.139.215.140] by nm19.bullet.mail.bf1.yahoo.com with NNFMP; 25 Dec 2012 03:13:12 -0000 Received: from [98.139.213.3] by tm11.bullet.mail.bf1.yahoo.com with NNFMP; 25 Dec 2012 03:13:12 -0000 Received: from [127.0.0.1] by smtp103.mail.bf1.yahoo.com with NNFMP; 25 Dec 2012 03:13:12 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1356405192; bh=ienAAZglLtbBNsor83kO6OPNJsGRWpJRYSiVFT0tQqw=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=wdoP2wMZR800NZ1nu3hDEgMbqcUgrU4R6p1JvMo4WC/4WqzUgf0YjR6BPTcpX+BVwbrF24rMf+rAPg/Kx66E0tVLSTnJbcLMc1uYaWiN8M2bKV1W0jZb9nzik4rjGahChw00xzosT/Hhwqn8DpBZDdMxrE46h/ihvMhbmAvaBmA= X-Yahoo-Newman-Id: 233465.49273.bm@smtp103.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: hDJW8PcVM1l0iy7KkxW1my2pYpqQMHhtseGfw0da88jRdvm gZ1.GIEeIuDLxMoMe6aCGrkTFDyGvMjfolt4MSQYbUHvixI7WwqtgfCbEzlv 4kLqDypbFZ5vpFWVoV7gyYLZblUnYRWwrs2Gjr2V1fyJPcbNsJd8Wg28c_tQ q4K6qDqcysRxLx5T9Wkmr33IxW9rtMUqLTqa9drxawgScADW.UHHqTQ4QLBx HYzNzh.Kj.gC72PUl1dTGcOXxQYgdyx3bZGc6piUmTNddjWRrdzJqYAtJ0x0 hA.2LykQZioWy1OE5732MLw3_Iv1HoKQP43wiYMzo3ty4muqFsvFf6SRvG.A 7R0oygCNlF_sEo.BBPQ96aF83fVlNqD1jZvbVwGbCQScWUg5tHORsjbTudGr j0m8y5OmDN.luvwy0VxK0eXgwKvqFwnYW1IVdtoHRHwVbJySQcYkpoUx_Ze7 RF3WY.CdyYvx7Tg-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Received: from [172.20.10.3] (scott4long@70.193.200.44 with plain) by smtp103.mail.bf1.yahoo.com with SMTP; 24 Dec 2012 19:13:12 -0800 PST Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Call for testing and review, busdma changes From: Scott Long In-Reply-To: <1356390225.1129.217.camel@revolution.hippie.lan> Date: Mon, 24 Dec 2012 22:13:10 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <2D98F70D-4031-4860-BABB-1F4663896234@yahoo.com> References: <1355077061.87661.320.camel@revolution.hippie.lan> <1355085250.87661.345.camel@revolution.hippie.lan> <1356381775.1129.181.camel@revolution.hippie.lan> <1356390225.1129.217.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1499) X-Mailman-Approved-At: Tue, 25 Dec 2012 03:29:07 +0000 Cc: powerpc@freebsd.org, marcel@freebsd.org, mips@freebsd.org, "mav@freebsd.org Motin" , "attilio@FreeBSD.org Rao" , Jeff Roberson , sparc64@freebsd.org, arm@freebsd.org, kib@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 03:13:20 -0000 On Dec 24, 2012, at 6:03 PM, Ian Lepore = wrote: >=20 > Yeah, I've done some low-level storage driver stuff myself (mmc/sd) = and > I can see how easy the deferred load solutions are to implement in = that > sort of driver that's already structured to operate asychronously. = I'm > not very familiar with how network hardware drivers interface with the > rest of the network stack. I have some idea, I'm just not sure of all > the subtleties involved and whether there are any implications for > something like a deferred load. >=20 > This is one of those situations where I tend to say to myself... the > folks who designed this stuff and imposed the "no deferred load" > restriction on mbufs and uio but not other cases were not stupid or > lazy, so they must have had some other reason. I'd want to know what > that was before I went too far with trying to undo it. >=20 Deferring is expensive from a latency standpoint. For disks, this = latency was comparatively small (until recent advances in SSD), so it = didn't matter, but it did matter with network devices. Also, network = drivers already had the concept of dropping mbufs due to resource = shortages, and the strict requirement of guaranteed transactions with = storage didn't apply. Deferring and freezing queues to guarantee = delivery order is a pain in the ass, so the decision was made that it = was cheaper to drop an mbuf on a resource shortage rather than defer. = As for uio's, they're the neglected part of the API and there's really = been no formal direction or master plan put into their evolution. = Anyways, that's my story and I'm sticking to it =3D-) Also, eliminating the concept of deferred load from mbufs then freed us = to look at ways to make the load operation cheaper. There's a lot of = code in _bus_dmamap_load_buffer() that is expensive, but a big one was = the indirect function pointer for the callback in the load wrappers. = The extra storage for filling in the temporary s/g list was also looked = at. Going with direct loads allowed me to remove these and reduce most = of the speed penalties. >=20 >>>=20 >>> Still unresolved is what to do about the remaining cases -- attempts = to >>> do dma in arbitrary buffers not obtained from bus_dmamem_alloc() = which >>> are not aligned and padded appropriately. There was some discussion = a >>> while back, but no clear resolution. I decided not to get bogged = down >>> by that fact and to fix the mbuf and allocated-buffer situations = that we >>> know how to deal with for now. >>=20 Why would these allocations not be handled as normal dynamic buffers = would with bus_dmamap_load()? Scott From owner-freebsd-sparc64@FreeBSD.ORG Tue Dec 25 23:25:21 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8494A317; Tue, 25 Dec 2012 23:25:21 +0000 (UTC) (envelope-from marius@alchemy.franken.de) Received: from alchemy.franken.de (alchemy.franken.de [194.94.249.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0A8A68FC0A; Tue, 25 Dec 2012 23:25:20 +0000 (UTC) Received: from alchemy.franken.de (localhost [127.0.0.1]) by alchemy.franken.de (8.14.5/8.14.5/ALCHEMY.FRANKEN.DE) with ESMTP id qBPNP78j047764; Wed, 26 Dec 2012 00:25:08 +0100 (CET) (envelope-from marius@alchemy.franken.de) Received: (from marius@localhost) by alchemy.franken.de (8.14.5/8.14.5/Submit) id qBPNP7Pv047763; Wed, 26 Dec 2012 00:25:07 +0100 (CET) (envelope-from marius) Date: Wed, 26 Dec 2012 00:25:07 +0100 From: Marius Strobl To: Chris Ross Subject: Re: Changes to kern.geom.debugflags? Message-ID: <20121225232507.GA47735@alchemy.franken.de> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-questions@freebsd.org, freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 23:25:21 -0000 On Mon, Dec 24, 2012 at 07:35:57PM -0500, Chris Ross wrote: > > On Dec 23, 2012, at 16:56 , Chris Ross wrote: > > I had brought up a machine months ago with freebsd-9-stable. I configured it to boot off of a single disk, with ZFS, expecting I would likely later attach the other disk to the zpool. I tried to do that today, but find that I can't write the bootloader to either disk. > > > > gpart: /dev/da0a: Operation not permitted > > > > [...] > > Okay. It occurred to me today what was likely the problem. I was running, even when single user, off of the zfs pool on the disks I was trying to write the bootloader to. > > I tar'd up /boot after my recent install from a Dec 22 9-stable, and moved it off-host. Then, I booted off of the July stable-9 CD-ROM I have in the machine, and was able to write bootblocks (with gpart bootcode) and a bootloader (dd if=/boot/zfsloader of=/dev/${disk}a bs=512 oseek=1024 conv=notrunc). > > Now, the new problem. When I try to boot my sparc64 with these bits, I see: > > >> FreeBSD/sparc64 ZFS boot block > Boot path: /pci@1c,600000/scsi@2/disk@0,0:a > Consoles: Open Firmware console > ERROR: Last Trap: Division by Zero > > {1} ok > > So, does anyone know if something has gone unstable in the sparc64 zfsboot in recent months? If I boot from the cdrom again and load the July zfsboot via gpart bootcode, it boots correctly again. > Please see http://www.freebsd.org/cgi/getmsg.cgi?fetch=0+0+/usr/local/www/db/text/2012/freebsd-sparc64/20121223.freebsd-sparc64 and provide debug information. Marius From owner-freebsd-sparc64@FreeBSD.ORG Wed Dec 26 02:05:36 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A820F96; Wed, 26 Dec 2012 02:05:36 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id D3C898FC13; Wed, 26 Dec 2012 02:05:35 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [206.138.151.12]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBQ25VMn010437 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Tue, 25 Dec 2012 21:05:32 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Changes to kern.geom.debugflags? From: Chris Ross In-Reply-To: <20121225232507.GA47735@alchemy.franken.de> Date: Tue, 25 Dec 2012 21:05:31 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <70B79618-A645-4900-BA92-84B3C3091BF3@distal.com> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> <20121225232507.GA47735@alchemy.franken.de> To: Marius Strobl X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [206.138.151.250]); Tue, 25 Dec 2012 21:05:32 -0500 (EST) Cc: freebsd-questions@freebsd.org, freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2012 02:05:36 -0000 On Dec 25, 2012, at 18:25 , Marius Strobl = wrote: >=20 > Please see > = http://www.freebsd.org/cgi/getmsg.cgi?fetch=3D0+0+/usr/local/www/db/text/2= 012/freebsd-sparc64/20121223.freebsd-sparc64 > and provide debug information. Thank you. I can rebuild everything with DEBUG_FLAGS=3D-g. But do I = need to do that in only boot? And, that message says: > Please use debug versions of the loaders and report the output of > `ctrace` on the boot monitor prompt. Building debug versions is > most easily done via `make DEBUG_FLAGS=3D-g` in path/to/src/boot > when building natively. You can also add the same when cross- > compiling but then you'll end up with debug versions of everything. > In both cases, make sure to not have any old object files in place. But, I'm not sure what "the boot monitor prompt" refers to, where i'm = supposed to enter "ctrace". When the boot crashes, I'm left at the open = prom. Is that where I'm supposed to enter "ctrace"? Apologies if so, I = just haven't known of that OBP command previously. Thank you. I'll try to come up with the debugging output you're = requesting. And more directions would be appreciated. - Chris From owner-freebsd-sparc64@FreeBSD.ORG Thu Dec 27 17:06:19 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0F6A43B5; Thu, 27 Dec 2012 17:06:19 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail1.markmonitor.com (mail1.markmonitor.com [209.66.70.11]) by mx1.freebsd.org (Postfix) with ESMTP id C6EDD8FC0A; Thu, 27 Dec 2012 17:06:18 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail1.markmonitor.com (Postfix) with ESMTP id 0BCAC17A4C; Thu, 27 Dec 2012 10:43:37 -0500 (EST) X-Virus-Scanned: amavisd-new at markmonitor.com Received: from mail1.markmonitor.com ([127.0.0.1]) by localhost (mail1.mm-corp.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id VNKRE5x7SyrV; Thu, 27 Dec 2012 10:43:32 -0500 (EST) Received: from sfoexch3.mm-ads.com (unknown [10.111.51.241]) by mail1.markmonitor.com (Postfix) with ESMTP id 80B0117A28; Thu, 27 Dec 2012 10:43:32 -0500 (EST) Received: from dc-exch4.mm-ads.com ([10.112.0.225]) by sfoexch3.mm-ads.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Dec 2012 07:43:32 -0800 Received: from zalamar.mm-corp.net ([10.112.52.72]) by dc-exch4.mm-ads.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Dec 2012 10:43:30 -0500 Subject: Re: Changes to kern.geom.debugflags? Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Chris Ross In-Reply-To: <20121225232507.GA47735@alchemy.franken.de> Date: Thu, 27 Dec 2012 10:43:27 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <8D01A854-97D9-4F1F-906A-7AB59BF8850B@distal.com> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> <20121225232507.GA47735@alchemy.franken.de> To: Marius Strobl X-Mailer: Apple Mail (2.1283) X-OriginalArrivalTime: 27 Dec 2012 15:43:30.0952 (UTC) FILETIME=[EBDD0080:01CDE448] Cc: freebsd-questions@freebsd.org, freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 17:06:19 -0000 On Dec 25, 2012, at 6:25 PM, Marius Strobl wrote: >> So, does anyone know if something has gone unstable in the sparc64 = zfsboot in recent months? If I boot from the cdrom again and load the = July zfsboot via gpart bootcode, it boots correctly again. >>=20 >=20 > Please see > = http://www.freebsd.org/cgi/getmsg.cgi?fetch=3D0+0+/usr/local/www/db/text/2= 012/freebsd-sparc64/20121223.freebsd-sparc64 > and provide debug information. I built the world with DEBUG_FLAGS=3D-g, and installed new zfsboot and = zfsloader on my sparc64. However, "ctrace" isn't helpful: >> FreeBSD/sparc64 ZFS boot block Boot path: /pci@1c,600000/scsi@2/disk@1,0:a Consoles: Open Firmware console =20 ERROR: Last Trap: Division by Zero {1} ok ctrace No saved state {1} ok=20 Anything else you can suggest to get debugging information out of = zfsloader? - Chris From owner-freebsd-sparc64@FreeBSD.ORG Thu Dec 27 17:30:07 2012 Return-Path: Delivered-To: sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 27150D08; Thu, 27 Dec 2012 17:30:07 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id DA47D8FC14; Thu, 27 Dec 2012 17:30:06 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id qBRHU5Ru086056; Thu, 27 Dec 2012 12:30:05 -0500 (EST) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id qBRHU5l6086053; Thu, 27 Dec 2012 17:30:05 GMT (envelope-from tinderbox@freebsd.org) Date: Thu, 27 Dec 2012 17:30:05 GMT Message-Id: <201212271730.qBRHU5l6086053@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on sparc64/sparc64 Precedence: bulk X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 17:30:07 -0000 TB --- 2012-12-27 17:08:27 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2012-12-27 17:08:27 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2012-12-27 17:08:27 - starting HEAD tinderbox run for sparc64/sparc64 TB --- 2012-12-27 17:08:27 - cleaning the object tree TB --- 2012-12-27 17:08:27 - /usr/local/bin/svn stat /src TB --- 2012-12-27 17:08:30 - At svn revision 244738 TB --- 2012-12-27 17:08:31 - building world TB --- 2012-12-27 17:08:31 - CROSS_BUILD_TESTING=YES TB --- 2012-12-27 17:08:31 - MAKEOBJDIRPREFIX=/obj TB --- 2012-12-27 17:08:31 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-12-27 17:08:31 - SRCCONF=/dev/null TB --- 2012-12-27 17:08:31 - TARGET=sparc64 TB --- 2012-12-27 17:08:31 - TARGET_ARCH=sparc64 TB --- 2012-12-27 17:08:31 - TZ=UTC TB --- 2012-12-27 17:08:31 - __MAKE_CONF=/dev/null TB --- 2012-12-27 17:08:31 - cd /src TB --- 2012-12-27 17:08:31 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Thu Dec 27 17:08:35 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries [...] cc -O2 -pipe -DLIBC_SCCS -DINET6 -I/src/lib/libutil -I/src/lib/libutil/../libc/gen/ -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/lib/libutil/auth.c -o auth.o cc -O2 -pipe -DLIBC_SCCS -DINET6 -I/src/lib/libutil -I/src/lib/libutil/../libc/gen/ -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/lib/libutil/expand_number.c -o expand_number.o cc -O2 -pipe -DLIBC_SCCS -DINET6 -I/src/lib/libutil -I/src/lib/libutil/../libc/gen/ -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/lib/libutil/flopen.c -o flopen.o cc -O2 -pipe -DLIBC_SCCS -DINET6 -I/src/lib/libutil -I/src/lib/libutil/../libc/gen/ -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/lib/libutil/fparseln.c -o fparseln.o cc -O2 -pipe -DLIBC_SCCS -DINET6 -I/src/lib/libutil -I/src/lib/libutil/../libc/gen/ -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /src/lib/libutil/gr_util.c -o gr_util.o cc1: warnings being treated as errors /src/lib/libutil/gr_util.c: In function 'gr_add': /src/lib/libutil/gr_util.c:510: warning: cast discards qualifiers from pointer target type *** [gr_util.o] Error code 1 Stop in /src/lib/libutil. *** [lib/libutil__L] Error code 1 Stop in /src. *** [libraries] Error code 1 Stop in /src. *** [_libraries] Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-12-27 17:30:05 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-12-27 17:30:05 - ERROR: failed to build world TB --- 2012-12-27 17:30:05 - 941.18 user 247.12 system 1298.67 real http://tinderbox.freebsd.org/tinderbox-head-ss-build-HEAD-sparc64-sparc64.full From owner-freebsd-sparc64@FreeBSD.ORG Thu Dec 27 19:20:13 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D78C583 for ; Thu, 27 Dec 2012 19:20:13 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail2.markmonitor.com (mail2.markmonitor.com [64.124.14.95]) by mx1.freebsd.org (Postfix) with ESMTP id 733CF8FC08 for ; Thu, 27 Dec 2012 19:20:13 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail2.markmonitor.com (Postfix) with ESMTP id 2ABA5145C1; Thu, 27 Dec 2012 10:15:28 -0800 (PST) X-Virus-Scanned: amavisd-new at markmonitor.com Received: from mail2.markmonitor.com ([127.0.0.1]) by localhost (mail2.mm-corp.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 03k1oepvUFUs; Thu, 27 Dec 2012 10:15:23 -0800 (PST) Received: from dc-exch2.mm-ads.com (dc-exch2.mm-corp.net [10.112.0.223]) by mail2.markmonitor.com (Postfix) with ESMTP id 87AE013B30; Thu, 27 Dec 2012 10:15:23 -0800 (PST) Received: from dc-exch4.mm-ads.com ([10.112.0.225]) by dc-exch2.mm-ads.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Dec 2012 13:15:23 -0500 Received: from zalamar.mm-corp.net ([10.112.52.72]) by dc-exch4.mm-ads.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Dec 2012 13:15:22 -0500 Subject: Re: Changes to kern.geom.debugflags? Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Chris Ross In-Reply-To: <8D01A854-97D9-4F1F-906A-7AB59BF8850B@distal.com> Date: Thu, 27 Dec 2012 13:15:22 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> <20121225232507.GA47735@alchemy.franken.de> <8D01A854-97D9-4F1F-906A-7AB59BF8850B@distal.com> To: Marius Strobl X-Mailer: Apple Mail (2.1283) X-OriginalArrivalTime: 27 Dec 2012 18:15:22.0701 (UTC) FILETIME=[22E38FD0:01CDE45E] Cc: freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 19:20:13 -0000 On Dec 27, 2012, at 10:43 AM, Chris Ross wrote: >>> FreeBSD/sparc64 ZFS boot block > Boot path: /pci@1c,600000/scsi@2/disk@1,0:a > Consoles: Open Firmware console =20 > ERROR: Last Trap: Division by Zero >=20 > {1} ok ctrace > No saved state > {1} ok=20 >=20 > Anything else you can suggest to get debugging information out of = zfsloader? So, I've started with the tiring process of "printf debugging". I = have gotten out of the loader code, and can show that it's inside of the call to the zfs = devsw dv_init() call where it's failing. >> FreeBSD/sparc64 ZFS boot block Boot path: /pci@1c,600000/scsi@2/disk@1,0:a Consoles: Open Firmware console =20 Goo 1 Goo 2 Looking at dp 0x236970 (block), dv_init of 0x11f320 Calling dv_init() Back from dv_init() Looking at dp 0x234e10 (net), dv_init of 0x1082c0 net0: (net0) Calling dv_init() Back from dv_init() Looking at dp 0x236088 (zfs), dv_init of 0x11cbe0 Calling dv_init() ERROR: Last Trap: Division by Zero {1} ok=20 Next, I guess I'll have to find out where the code for that is, and = work there. If anyone has any ideas or can assist, please let me know. - Chris From owner-freebsd-sparc64@FreeBSD.ORG Fri Dec 28 00:58:15 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D4DBBFFF for ; Fri, 28 Dec 2012 00:58:15 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id 863758FC0C for ; Fri, 28 Dec 2012 00:58:15 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [IPv6:2001:470:e24c:200:ea06:88ff:feca:960e]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBS0wCCW007029 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 27 Dec 2012 19:58:12 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: zfs booting feedback From: Chris Ross In-Reply-To: <20120730113015.GA14735@alchemy.franken.de> Date: Thu, 27 Dec 2012 19:58:11 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <47BA0795-19C3-4910-AC0B-FB8169818633@distal.com> References: <20120708025435.GA12487@pix.net> <20120709140019.GA67276@alchemy.franken.de> <20120710165433.GA98707@pix.net> <20120712172208.GA47484@pix.net> <20120713195807.GU63893@alchemy.franken.de> <20120714004335.GD92944@pix.net> <20120727182558.GH58433@alchemy.franken.de> <20120730113015.GA14735@alchemy.franken.de> To: Marius Strobl X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [IPv6:2001:470:e24c:200::ae25]); Thu, 27 Dec 2012 19:58:13 -0500 (EST) Cc: Kurt Lidl , freebsd-sparc64@freebsd.org X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 00:58:15 -0000 On Thu, Dec 13, 2012 at 01:05:16PM +0300, KOT MATPOCKuH wrote: > I builded world/kernel from stable/9 r244121, installed zfsboot and > zfsloader to disk on Sun Fire V240 with OpenBoot 4.30.4.a. > But boot fails with: > Rebooting with command: boot disk > Boot device: /pci@1c,600000/scsi@2/disk@0,0 File and args: >=20 > >> FreeBSD/sparc64 ZFS boot block > Boot path: /pci@1c,600000/scsi@2/disk@0,0:a > Consoles: Open Firmware console > ERROR: Last Trap: Division by Zero I'm seeing this too, as you may've seen in recent list emails. I'm = trying to track down where the problem lies, but it's not just you. > Also I tried zfsloader builded from sources @ may 2012: > (same zfsboot, but used zfsloader.old) > Boot device: disk File and args: >=20 > >> FreeBSD/sparc64 ZFS boot block > Boot path: /pci@1c,600000/scsi@2/disk@0,0:a > Consoles: Open Firmware console > ofwd_open: Could not open disk1: > ofwd_open: Could not open disk2: > ofwd_open: Could not open disk3: >=20 > FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 > (root@sunspot, Fri Nov 2 08:59:22 MSK 2012) > bootpath=3D"/pci@1c,600000/scsi@2/disk@0,0:a" > ZFS: i/o error - all block copies unavailable > ZFS: i/o error - all block copies unavailable > [...] > Could it a result of crosscompiling? I'm compiling natively, so as others have said, it's likely not = related to cross-compiling. > PS. When writing both zfsloader I got "Invalid argument" message: > # dd if=3D/boot/zfsloader.old of=3D/dev/da0a bs=3D512 oseek=3D1024 > dd: /dev/da0a: Invalid argument > 470+1 records in > 470+0 records out > 240640 bytes transferred in 1.915555 secs (125624 bytes/sec) > Is it okey? I saw this too, and it wasn't initially a problem, but I noticed that = the last couple hundred bytes of the binary weren't being copied out, and hoped that was the problem that was causing the crash. As it happens, it wasn't, but I determined that the error "Invalid argument" is a result of a write to the device of smaller than the 512 byte block- size. If you add a "conv=3Dnotrunc,sync" to the dd command, it will = fill out the last block, and write to the disk device without error. (the notrunc isn't related to the issue, but is in the command I use) Thanks. - Chris From owner-freebsd-sparc64@FreeBSD.ORG Sat Dec 29 20:02:03 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A6691369 for ; Sat, 29 Dec 2012 20:02:03 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id 57E748FC08 for ; Sat, 29 Dec 2012 20:02:03 +0000 (UTC) Received: from [192.168.1.122] (ip98-166-181-220.hr.hr.cox.net [98.166.181.220]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBTK1txP000420 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 29 Dec 2012 15:01:56 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Changes to kern.geom.debugflags? From: Chris Ross In-Reply-To: Date: Sat, 29 Dec 2012 15:01:55 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <6FC4189B-85FA-466F-AA00-C660E9C16367@distal.com> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> <20121225232507.GA47735@alchemy.franken.de> <8D01A854-97D9-4F1F-906A-7AB59BF8850B@distal.com> To: Chris Ross X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [206.138.151.250]); Sat, 29 Dec 2012 15:01:57 -0500 (EST) Cc: freebsd-sparc64@freebsd.org, Marius Strobl X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2012 20:02:03 -0000 On Dec 27, 2012, at 1:15 PM, Chris Ross = wrote: >=20 > On Dec 27, 2012, at 10:43 AM, Chris Ross wrote: >>>> FreeBSD/sparc64 ZFS boot block >> Boot path: /pci@1c,600000/scsi@2/disk@1,0:a >> Consoles: Open Firmware console =20 >> ERROR: Last Trap: Division by Zero >>=20 >> {1} ok ctrace >> No saved state >> {1} ok=20 >>=20 >> Anything else you can suggest to get debugging information out of = zfsloader? >=20 > So, I've started with the tiring process of "printf debugging". I = have gotten out of > the loader code, and can show that it's inside of the call to the zfs = devsw dv_init() > call where it's failing. Okay. Many many iterations, and I found out where it's crashing. In sys/boot/zfs/zfsimpl.c, in dnode_read(), the first line of the while = loop is: uint64_t bn =3D offset / bsize; And, bsize is calculated from: int bsize =3D dnode->dn_datablkszsec << SPA_MINBLOCKSHIFT; When running the code, though, I can confirm that bsize is 0 before = the divide is hit, thus causing the divide by zero trap. I'm going to guess this is a problem with dnode->dn_datablkszsec. Has = anything changed recently in zfs_fmtdev, or more likely zfs_get_root() or = objset_get_dnode(), which is the callchain right before dnode_read() ? - Chris From owner-freebsd-sparc64@FreeBSD.ORG Sat Dec 29 20:21:21 2012 Return-Path: Delivered-To: freebsd-sparc64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 695358D0 for ; Sat, 29 Dec 2012 20:21:21 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id 167E88FC0A for ; Sat, 29 Dec 2012 20:21:20 +0000 (UTC) Received: from [192.168.1.122] (ip98-166-181-220.hr.hr.cox.net [98.166.181.220]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id qBTKLFqd026713 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 29 Dec 2012 15:21:17 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Changes to kern.geom.debugflags? From: Chris Ross In-Reply-To: <6FC4189B-85FA-466F-AA00-C660E9C16367@distal.com> Date: Sat, 29 Dec 2012 15:21:15 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <49AEC210-6BB6-4778-B71C-2E3B86E0902C@distal.com> References: <7AA0B5D0-D49C-4D5A-8FA0-AA57C091C040@distal.com> <6A0C1005-F328-4C4C-BB83-CA463BD85127@distal.com> <20121225232507.GA47735@alchemy.franken.de> <8D01A854-97D9-4F1F-906A-7AB59BF8850B@distal.com> <6FC4189B-85FA-466F-AA00-C660E9C16367@distal.com> To: Chris Ross X-Mailer: Apple Mail (2.1499) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.2 (mail.distal.com [206.138.151.250]); Sat, 29 Dec 2012 15:21:17 -0500 (EST) Cc: freebsd-sparc64@freebsd.org, Marius Strobl X-BeenThere: freebsd-sparc64@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the Sparc List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2012 20:21:21 -0000 On Dec 29, 2012, at 3:01 PM, Chris Ross = wrote: > On Dec 27, 2012, at 1:15 PM, Chris Ross = wrote: >> So, I've started with the tiring process of "printf debugging". I = have gotten out of >> the loader code, and can show that it's inside of the call to the zfs = devsw dv_init() >> call where it's failing. >=20 > Okay. Many many iterations, and I found out where it's crashing. In > sys/boot/zfs/zfsimpl.c, in dnode_read(), the first line of the while = loop > is: >=20 > uint64_t bn =3D offset / bsize; >=20 > And, bsize is calculated from: >=20 > int bsize =3D dnode->dn_datablkszsec << SPA_MINBLOCKSHIFT; >=20 > When running the code, though, I can confirm that bsize is 0 before = the divide > is hit, thus causing the divide by zero trap. >=20 > I'm going to guess this is a problem with dnode->dn_datablkszsec. = Has anything > changed recently in zfs_fmtdev, or more likely zfs_get_root() or = objset_get_dnode(), > which is the callchain right before dnode_read() ? So, the dnode in question is the MOS os_meta_dnode, for the MOS that's = in the SPA for the root. (Mind you, the internals of ZFS are all a black box = to me, so I don't understand what I just said). Anyone have any idea why the dn_datablkszsec would be unset or wrote = for said meta dnode? That seems to be the crux of the problem. Where does = that all get initialized? - Chris