From owner-freebsd-xen@freebsd.org Mon Jan 11 06:49:05 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F584A69561 for ; Mon, 11 Jan 2016 06:49:05 +0000 (UTC) (envelope-from bounces+73574-0602-freebsd-xen=freebsd.org@sendgrid.net) Received: from o1.l99.sendgrid.net (o1.l99.sendgrid.net [198.37.153.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2654C1B13 for ; Mon, 11 Jan 2016 06:49:04 +0000 (UTC) (envelope-from bounces+73574-0602-freebsd-xen=freebsd.org@sendgrid.net) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=subject:to:references:cc:from:mime-version:in-reply-to:content-type:content-transfer-encoding; s=smtpapi; bh=qL8ngjvWq+Myta8fZKF+GrN9qk4=; b=TDwUcuDLpADdCgtP6n FAVEmoCtXdcNrkyEFvbpcb186Ak6+GGyQlWXTFwRtCZw0+qNUmneTbMgPQIAi8m2 n21usWKW3xNxpWmBdW5zIdaws3xR8rfXB2NbsXKVqDALzvmGgFQ8v8IxwLct6+Nn I4NAi9ZRkSBjgyVG7d83iIJog= Received: by filter0491p1mdw1.sendgrid.net with SMTP id filter0491p1mdw1.21277.5693505826 2016-01-11 06:48:56.366741256 +0000 UTC Received: from mail.tarsnap.com (ec2-54-86-246-204.compute-1.amazonaws.com [54.86.246.204]) by ismtpd0001p1iad1.sendgrid.net (SG) with ESMTP id 92BEOOzdR2W9GhSxRoJEfg for ; Mon, 11 Jan 2016 06:48:56.526 +0000 (UTC) Received: (qmail 90406 invoked from network); 11 Jan 2016 06:46:34 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 11 Jan 2016 06:46:34 -0000 Received: (qmail 75728 invoked from network); 11 Jan 2016 06:48:00 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 11 Jan 2016 06:48:00 -0000 Subject: Re: recent disk-related breakage To: freebsd-xen@freebsd.org, =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <56934B8F.8050503@freebsd.org> Cc: hps@freebsd.org, kib@freebsd.org From: Colin Percival X-Enigmail-Draft-Status: N1110 Message-ID: <5693501F.9060008@freebsd.org> Date: Sun, 10 Jan 2016 22:47:59 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <56934B8F.8050503@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SG-EID: RUbAm5H8PjswBj/QH+sYVehaJogg3iBnZcyVi1bw/Izf1Ps4Y6GPPcXvM4taO7oyHNTNPtjM8S8Njp evM0vtHqJL5voULxGmgWtwZZ3+MMMTgIHP0SNObygagfaPfaG2JgtoNMnE0r5t8yWhT3INH99tgsXT HBeiI5TdIwYxZ0avsxbjgmhakHKMV7slivosnO+EtIj47eHyIgcSk+eBjtsmZ+YvnJMeCsOve1aJDd 4= X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2016 06:49:05 -0000 I'm rather suspicious of r292255 here -- royger, hps, kib, can any of you comment on whether it would be responsible for making non-page-aligned I/Os no longer get split on page boundaries? The commit message is suggestive, but I don't know the code very well. (If I don't hear back I'll determine if it's responsible experimentally, but testing a "system no longer boots" bug in EC2 is painfully slow and it's getting late here.) Colin Percival On 01/10/16 22:28, Colin Percival wrote: > Some time in early December, disk I/O broke. The EC2 image built from r291495 > works fine; the EC2 image built from r292413 is broken. > > Symptoms: > 1. growfs reports "superblock not recognized" > > 2. fsck reports lots of "UNKNOWN FILE TYPE"s and after a few seconds provokes > "panic: XEN disk driver data cannot cross a page boundary" > xbd_mksegarray() at xbd_mksegarray+0x4b/frame 0xfffffe0f4dadb420 > > xbd_queue_cb() at xbd_queue_cb+0x1e8/frame 0xfffffe0f4dadb490 > > bus_dmamap_load_bio() at bus_dmamap_load_bio+0xad/frame 0xfffffe0f4dadb4f0 > > xbd_startio() at xbd_startio+0x194/frame 0xfffffe0f4dadb530 > > xbd_strategy() at xbd_strategy+0x6a/frame 0xfffffe0f4dadb560 > > g_disk_start() at g_disk_start+0x37c/frame 0xfffffe0f4dadb5d0 > > g_io_request() at g_io_request+0x39d/frame 0xfffffe0f4dadb630 > > g_part_start() at g_part_start+0x2b5/frame 0xfffffe0f4dadb6b0 > > g_io_request() at g_io_request+0x39d/frame 0xfffffe0f4dadb710 > > g_io_request() at g_io_request+0x39d/frame 0xfffffe0f4dadb770 > > g_dev_strategy() at g_dev_strategy+0x171/frame 0xfffffe0f4dadb7b0 > > physio() at physio+0x440/frame 0xfffffe0f4dadb850 > > devfs_read_f() at devfs_read_f+0xe7/frame 0xfffffe0f4dadb8b0 > > dofileread() at dofileread+0x98/frame 0xfffffe0f4dadb900 > > kern_readv() at kern_readv+0x68/frame 0xfffffe0f4dadb950 > > sys_read() at sys_read+0x60/frame 0xfffffe0f4dadb9a0 > > > Does anyone remember touching any relevant bits of code in that timeframe? > -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid