From owner-freebsd-current@freebsd.org Tue Jan 5 12:19:17 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DD91A632AD for ; Tue, 5 Jan 2016 12:19:17 +0000 (UTC) (envelope-from pho@holm.cc) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7416A15D7; Tue, 5 Jan 2016 12:19:17 +0000 (UTC) (envelope-from pho@holm.cc) Received: from x2.osted.lan (87-58-223-204-dynamic.dk.customer.tdc.net [87.58.223.204]) by relay01.pair.com (Postfix) with ESMTP id A1D8CD009A1; Tue, 5 Jan 2016 07:12:00 -0500 (EST) Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.9/8.14.9) with ESMTP id u05CBw8m030207 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 5 Jan 2016 13:11:58 +0100 (CET) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.9/8.14.9/Submit) id u05CBw8j030206; Tue, 5 Jan 2016 13:11:58 +0100 (CET) (envelope-from pho) Date: Tue, 5 Jan 2016 13:11:57 +0100 From: Peter Holm To: Konstantin Belousov Cc: Benno Rice , freebsd-current Subject: Re: Possible bug in or around posix_fadvise after r292326 Message-ID: <20160105121157.GA30104@x2.osted.lan> References: <20160105110740.GH3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105110740.GH3625@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jan 2016 12:19:17 -0000 On Tue, Jan 05, 2016 at 01:07:40PM +0200, Konstantin Belousov wrote: > On Mon, Jan 04, 2016 at 10:05:21PM -0800, Benno Rice wrote: > > Hi Konstantin, > > > > I recently updated my dev box to r292962. After doing this I attempted to set up PostgreSQL 9.4. When I ran initdb the last phase hung. Using procstat -kk I found it appeared to be stuck in a loop inside a posix_fadvise syscall. I could not ^C or ^Z the initdb process. I could kill it but a subsequent attempt to rm -rf the /usr/local/pgsql/data directory also got stuck and was unkillable by any means. Rebooting allowed me to remove the directory but the initdb process still hung when I re-ran it. > > > > I tried PostgreSQL 9.3 with similar results. > > > > Looking at the source code for initdb I found that it calls posix_fadvise like so[1]: > > > > /* > > * We do what pg_flush_data() would do in the backend: prefer to use > > * sync_file_range, but fall back to posix_fadvise. We ignore errors > > * because this is only a hint. > > */ > > #if defined(HAVE_SYNC_FILE_RANGE) > > (void) sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WRITE); > > #elif defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED) > > (void) posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); > > #else > > #error PG_FLUSH_DATA_WORKS should not have been defined > > #endif > > > > Looking for recent commits involving POSIX_FADV_DONTNEED I found r292326: > > > > https://svnweb.freebsd.org/changeset/base/292326 > > > > Backing this revision out allowed the initdb process to complete. > > > > My current theory is that some how we???re getting ENOLCK or EAGAIN from the BUF_TIMELOCK call in bnoreuselist: > > > > https://svnweb.freebsd.org/base/head/sys/kern/vfs_subr.c?view=annotate#l1676 > > > > Leading to an infinite loop in vop_stdadvise: > > > > https://svnweb.freebsd.org/base/head/sys/kern/vfs_default.c?annotate=292373#l1083 > > > > I haven???t managed to dig any deeper than that yet. > > > > Is there any other information I could give you to help narrow this down? > > I do not see this issue locally. > I do: (kgdb) f 9 #9 0xffffffff80ac7956 in vop_stdadvise (ap=0xfffffe081dc6d930) at ../../../kern/vfs_default.c:1087 1087 error = bnoreuselist(&bo->bo_dirty, bo, startn, endn); (kgdb) l 1082 endn = ap->a_end / bsize; 1083 for (;;) { 1084 error = bnoreuselist(&bo->bo_clean, bo, startn, endn); 1085 if (error == EAGAIN) 1086 continue; 1087 error = bnoreuselist(&bo->bo_dirty, bo, startn, endn); 1088 if (error == EAGAIN) 1089 continue; 1090 break; 1091 } (kgdb) info loc vp = (struct vnode *) 0xfffff8008bdaa9c0 bo = (struct bufobj *) 0xfffff8008bdaab28 startn = 0x0 endn = 0xffffffffffff start = 0x0 end = 0x8000000000000000 bsize = 0x8000 error = 0x0 (kgdb) https://people.freebsd.org/~pho/stress/log/kostik855.txt - Peter