From owner-freebsd-bugs@FreeBSD.ORG Sun May 11 16:20:00 2014 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADCE9713 for ; Sun, 11 May 2014 16:20:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A67B2FA4 for ; Sun, 11 May 2014 16:20:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s4BGK0NZ007639 for ; Sun, 11 May 2014 16:20:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s4BGK0b1007638; Sun, 11 May 2014 16:20:00 GMT (envelope-from gnats) Resent-Date: Sun, 11 May 2014 16:20:00 GMT Resent-Message-Id: <201405111620.s4BGK0b1007638@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, gmc Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFDFF441 for ; Sun, 11 May 2014 16:14:42 +0000 (UTC) Received: from cgiserv.freebsd.org (cgiserv.freebsd.org [IPv6:2001:1900:2254:206a::50:4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A1E1C2F69 for ; Sun, 11 May 2014 16:14:42 +0000 (UTC) Received: from cgiserv.freebsd.org ([127.0.1.6]) by cgiserv.freebsd.org (8.14.8/8.14.8) with ESMTP id s4BGEfnh083568 for ; Sun, 11 May 2014 16:14:41 GMT (envelope-from nobody@cgiserv.freebsd.org) Received: (from nobody@localhost) by cgiserv.freebsd.org (8.14.8/8.14.8/Submit) id s4BGEfa2083567; Sun, 11 May 2014 16:14:41 GMT (envelope-from nobody) Message-Id: <201405111614.s4BGEfa2083567@cgiserv.freebsd.org> Date: Sun, 11 May 2014 16:14:41 GMT From: gmc To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: kern/189673: hast on top of gvinum unstable, freezes system X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2014 16:20:00 -0000 >Number: 189673 >Category: kern >Synopsis: hast on top of gvinum unstable, freezes system >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun May 11 16:20:00 UTC 2014 >Closed-Date: >Last-Modified: >Originator: gmc >Release: 8.4 >Organization: >Environment: FreeBSD hosta 8.4-RELEASE FreeBSD 8.4-RELEASE #0 r265783: Fri May 9 20:24:43 CEST 2014 root@hosta:/usr/obj/usr/src/sys/GENERIC i386 >Description: I have setups where I use gvinum to divide a partition or raw disk into smaller concat devices under /dev/gvinum, that i subsequently use hast on to replicate their contents to a central zfs server for backup purposes. The hast devices get a filesystem, and each contains a single jail. I have observed this issue on a wide range of hardware and FreeBSD versions (ever since 8.0 or earlier). When there is heavy I/O on those volumes, the system that has the primary simply freezes. The secondary shows no issues. In my production environment the secondary is running on top of zfs, but I have observed this in a simple setup where both primary and secondary are on top of gvinum as well. >How-To-Repeat: 1. Create a VM ============== Created a 8GB dynamically allocated disk and set up a VM with all default settings and two network interfaces: the first interface is bridged to my LAN and assigned by dhcp and the second is an internal network. Install FreeBSD 8.4 from disc1 iso. Use the entire 8GB disk to create a slice, and install these partitions: # /dev/ad0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1466368 0 4.2BSD 0 0 0 b: 880600 1466368 swap c: 16777089 0 unused 0 0 # "raw" part, don't edit d: 1048576 2346968 4.2BSD 0 0 0 e: 1048576 3395544 4.2BSD 0 0 0 f: 8333312 4444120 4.2BSD 0 0 0 g: 3999657 12777432 4.2BSD 0 0 0 ad0s1g will be the gvinum partition, the fstab is as follows: # Device Mountpoint FStype Options Dump Pass# /dev/ad0s1b none swap sw 0 0 /dev/ad0s1a / ufs rw 1 1 /dev/ad0s1e /tmp ufs rw 2 2 /dev/ad0s1f /usr ufs rw 2 2 /dev/ad0s1d /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0 The /etc/rc.conf of this vm is: # Created: Fri May 9 20:21:48 2014 # Enable network daemons for user convenience. # Please make all changes to this file, not to /etc/defaults/rc.conf. # This file now contains just the overrides from /etc/defaults/rc.conf. hostname="hosta" ifconfig_em0="dhcp" ifconfig_em1="inet 192.168.13.1 netmask 255.255.255.0" The /etc/hosts is: ::1 localhost localhost.my.domain 127.0.0.1 localhost localhost.my.domain 192.168.13.1 hosta 192.168.13.2 hostb I then pulled the source from svn and compiled the kernel and world: # cd /usr # svn checkout https://svn.eu0.freebsd.org/base/release/8.4.0 src # cd src # make buildkernel && make installkernel # make installkernel # mergemaster -p -iFU # make installworld # mergemaster -iFU # reboot Then set up gvinum with 'gvinum create' and the following config: drive store device /dev/ad0s1g volume demo plex name demo.p0 org concat vol demo sd name demo.p0.s0 drive store len 2097152s driveoffset 265s plex demo.p0 plexoffset 0s Finally, install bonnie (pkg_add -r bonnie). 2. Clone the VM =============== Cloned the VM to create hostb, with rc.conf modified as such: # Created: Fri May 9 20:21:48 2014 # Enable network daemons for user convenience. # Please make all changes to this file, not to /etc/defaults/rc.conf. # This file now contains just the overrides from /etc/defaults/rc.conf. hostname="hostb" ifconfig_em0="dhcp" ifconfig_em1="inet 192.168.13.2 netmask 255.255.255.0" 3. Configure hastd ================== Created /etc/hast.conf on both vm's as follows; resource demo { on hosta { remote 192.168.13.2 local /dev/gvinum/demo } on hostb { remote 192.168.13.1 local /dev/gvinum/demo } } Start & init hast on hosta: root@hosta# /etc/rc.d/hastd onestart root@hosta# hastctl create demo root@hosta# hastctl role primary demo root@hosta# newfs -U /dev/hast/demo and on hostb: root@hostb# /etc/rc.d/hastd onestart root@hostb# hastctl create demo root@hostb# hastctl role secondary demo 4. Mount and start bonnie ========================= root@hosta# mkdir /mnt/demo root@hosta# mount /dev/hast/demo /mnt/demo root@hosta# cd /mnt/demo root@hosta# bonnie & bonnie & bonnie & bonnie & Wait till the system freezes. I normally have gstat running on another pty, but that shouldn't make a difference. >Fix: >Release-Note: >Audit-Trail: >Unformatted: