From owner-freebsd-stable@freebsd.org Sun Jul 3 05:27:22 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABD04B8FEBF for ; Sun, 3 Jul 2016 05:27:22 +0000 (UTC) (envelope-from messenger@webex.com) Received: from sjmda15.webex.com (sjmda15.webex.com [64.68.124.166]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 905982528 for ; Sun, 3 Jul 2016 05:27:22 +0000 (UTC) (envelope-from messenger@webex.com) Received: from rsj1rmd005.webex.com (sjc02-wxp00-lbace03-core-vl120-np10b-6.webex.com [64.68.121.241]) by sjmda15.webex.com (Postfix) with ESMTP id 0E1D946E10 for ; Sun, 3 Jul 2016 04:41:59 +0000 (GMT) Received: from rsj1rmd005.webex.com (localhost [127.0.0.1]) by rsj1rmd005.webex.com (Postfix) with ESMTP id 59A8E20E99 for ; Sun, 3 Jul 2016 04:31:51 +0000 (GMT) Date: Sun, 3 Jul 2016 04:31:51 +0000 (GMT) From: The HD Group Reply-To: uk@fhwds.com To: freebsd-stable@freebsd.org Message-ID: <917623431.4441621467520311365.JavaMail.nobody@rsj1rmd005.webex.com> Subject: Seeking Key Leaders X-Priority: 3 Importance: normal X-WBX-INFO: X-WBX-SID=1018902, X-WBX-CID=1756782900, X-WBX-TID=36939d1c7f1f93bfe0531b06fc0a1690, X-WBX-RID=36939D1C9F3693BFE0531B06FC0A1690, X-WBX-SVC:Event Center, X-WBX-TT:Invitee Invitation for Private Event_html, Date:Sun Jul 03 04:31:51 GMT 2016 reminder-2.3.8-2 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2016 05:27:22 -0000 From owner-freebsd-stable@freebsd.org Mon Jul 4 06:35:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66EFDB907C0 for ; Mon, 4 Jul 2016 06:35:23 +0000 (UTC) (envelope-from www-data@hdh6drsa.cloudapp.net) Received: from hdh6drsa.cloudapp.net (hdh6drsa.cloudapp.net [23.97.203.222]) by mx1.freebsd.org (Postfix) with ESMTP id 066B72E95 for ; Mon, 4 Jul 2016 06:35:23 +0000 (UTC) (envelope-from www-data@hdh6drsa.cloudapp.net) Received: by hdh6drsa.cloudapp.net (Postfix, from userid 33) id 7F25E20388; Mon, 4 Jul 2016 06:35:16 +0000 (UTC) To: freebsd-stable@freebsd.org Subject: Super Oferta TV LED 55 3D 1899,00 X-PHP-Originating-Script: 0:kinoplex2015.php From: 421660 frederick X-Mailer: Microsoft Office Outlook, Build 17.551210 Message-Id: <20160704063516.7F25E20388@hdh6drsa.cloudapp.net> Date: Mon, 4 Jul 2016 06:35:16 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 06:35:23 -0000 From owner-freebsd-stable@freebsd.org Mon Jul 4 10:50:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37B5EB91E47 for ; Mon, 4 Jul 2016 10:50:23 +0000 (UTC) (envelope-from futurestudents@curtin.edu.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 273E026A8 for ; Mon, 4 Jul 2016 10:50:23 +0000 (UTC) (envelope-from futurestudents@curtin.edu.au) Received: by mailman.ysv.freebsd.org (Postfix) id 23240B91E46; Mon, 4 Jul 2016 10:50:23 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2077AB91E45 for ; Mon, 4 Jul 2016 10:50:23 +0000 (UTC) (envelope-from futurestudents@curtin.edu.au) Received: from mail.curtin.edu.au (mail.curtin.edu.au [134.7.171.205]) by mx1.freebsd.org (Postfix) with ESMTP id 749FE26A7 for ; Mon, 4 Jul 2016 10:50:22 +0000 (UTC) (envelope-from futurestudents@curtin.edu.au) Received: from webp1.curtin.edu.au (webp1.curtin.edu.au [134.7.179.12]) by mail.curtin.edu.au (Postfix) with ESMTP id DF02AA025D for ; Mon, 4 Jul 2016 18:19:06 +0800 (AWST) Date: Mon, 4 Jul 2016 18:19:06 +0800 (AWST) From: Curtin University Future Students To: stable@freebsd.org Message-ID: <24067954.327221467627546912.JavaMail.apache@webp1.curtin.edu.au> Subject: =?UTF-8?Q?9=E6=84=A7=E5=8C=AA=E6=8C=BA=E7=94=9F=E6=9D=90_wants_to_show_y?= =?UTF-8?Q?ou_these_Curtin_courses?= X-Mailer: ColdFusion 8 Application Server MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 10:50:23 -0000 From owner-freebsd-stable@freebsd.org Tue Jul 5 05:26:27 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 968C5B855D2 for ; Tue, 5 Jul 2016 05:26:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 76AAA18E3 for ; Tue, 5 Jul 2016 05:26:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id 71F22B855CF; Tue, 5 Jul 2016 05:26:27 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71946B855CE for ; Tue, 5 Jul 2016 05:26:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-it0-x233.google.com (mail-it0-x233.google.com [IPv6:2607:f8b0:4001:c0b::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45C4B18E2 for ; Tue, 5 Jul 2016 05:26:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-it0-x233.google.com with SMTP id h190so76087058ith.1 for ; Mon, 04 Jul 2016 22:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:from:date:message-id:subject:to; bh=cEuLcSck0eUC3ETcsKybUeZuj1h1cE82SGaPpeXsozw=; b=OBUujVrFLBLTktJQpJGrjIQMCw7UscFvYmwt7KbLsJT4p4x11E4lHJWuEQ1Dp7QxC+ oya5gvOmKhwp0EPyuWAMRo/mjSz5Sg8SutyhwpNlLkeUfOrpINIuB8etQgUtY/WhngJF wt6HQWusvyDAtmJh1DTT5RxZSBUHlxo5hSQAdaPvb9n4r9JSZaEtVbdPFjDaSbrpcdOk DPWFPO440lZfqIzOO3sNzT0Vd136u9YmML6E21WvUzUbjgZFu9WebUbUru5trpfCWK9u VeotiqfZW5OtZ8NqSojSf+Z6q/KagZACAxbVorZejJZ2g0HoymUMrjce1FuD1F8kNH6q GQBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=cEuLcSck0eUC3ETcsKybUeZuj1h1cE82SGaPpeXsozw=; b=W2sIdi3JJiumOEFzubC8PUVN9K/tWNCtJQTgwWzhcgw0tA2fHhNWpGeGa6+c6HfrTA J9wCxElaoSnDbI26zhJE7dqTtzpQd2Jf3zSIZhqgh66vZ1NAUp16U7VLXpUHaFBjSWzR uSKoDA5vrnM3pMl/nbEqmUhpXBoB1CmUxYKIOV8yNSym8FL6qmJBbmYffObGXTrPGS0M xt+t+NouaXN3buBHrSMcWf54tbrVwR37rzZQEywioixMzlVJgOsZildrZRHPRdIWzkzj GntPYrmEINzHWLxLZgeR+3AmK+NE2XO4wPDSG4OLYMVPjtAW7DWieVnDgG0sBBWMeWya 0FZA== X-Gm-Message-State: ALyK8tKJkrX5mWYsWjSFbXPFMF0UhmUaYqq/hRW0KUHWisORDV9eJ6OfhTeuLPaUDH9MytuSJtg4lQPA+j8+jIzO X-Received: by 10.36.91.66 with SMTP id g63mr11055580itb.16.1467696386364; Mon, 04 Jul 2016 22:26:26 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.36.59.193 with HTTP; Mon, 4 Jul 2016 22:26:25 -0700 (PDT) From: Maxim Sobolev Date: Mon, 4 Jul 2016 22:26:25 -0700 X-Google-Sender-Auth: DIV2CM4kakl33DY5WwTrB3JuOKg Message-ID: Subject: A faulty program corrupts some its data preventing correct core generation (Failed to write core file for process postgres (error 14)) To: stable@freebsd.org, hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 05:26:27 -0000 Hi all, investigating some random postgresql-9.1.21 server crashes on FreeBSD 10.3, we've started seeing those after upgrading from postgres 9.1.18 on more than one system, so hardware (e.g. RAM issues) are very unlikely. I suspect that postgres is at fault, however I am also curious how could it be that kernel is not capable of generating core file when application does something silly? Is it that some ELF-related data structures got corrupted or something else? Are we protecting the page where ELF header is mapped with R/O flag? I am looking at possibly recreating this by poking around elf header(s), seeing if I can corrupt it in a similar manner reliably, any pointers or suggestions are appreciated. Jun 27 04:10:18 dal12 kernel: Failed to write core file for process postgres (error 14) Jun 27 04:10:18 dal12 kernel: pid 41361 (postgres), uid 70: exited on signal 11 Jul 1 05:21:46 dal12 kernel: Failed to write core file for process postgres (error 14) Jul 1 05:21:46 dal12 kernel: pid 1722 (postgres), uid 70: exited on signal 11 #define EFAULT 14 /* Bad address */ The resulting files are truncated and is not really usable for anything. We've seen the same issue -rw------- 1 pgsql wheel 1310720 Jun 27 04:10 postgres.41361.core -rw------- 1 pgsql wheel 1310720 Jul 1 05:21 postgres.1722.core [ssp-root@dal12 /var/tmp]$ sudo gdb711 postgres postgres.1722.core GNU gdb (GDB) 7.11 [GDB v7.11 for FreeBSD] Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd10.3". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from postgres...(no debugging symbols found)...done. BFD: Warning: /var/tmp/postgres.1722.core is truncated: expected core file size >= 517120000, found: 1310720. [New LWP 100261] Core was generated by `postgres'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 (gdb) where #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 Backtrace stopped: Cannot access memory at address 0x7fffffffdd08 (gdb) q -Max From owner-freebsd-stable@freebsd.org Tue Jul 5 08:38:39 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88515B206F3 for ; Tue, 5 Jul 2016 08:38:39 +0000 (UTC) (envelope-from freebsd@dohd.org) Received: from mx.terantula.net (mx.terantula.net [IPv6:2001:4cb8:1:3111::25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "*.terantula.net", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4AC4A1378 for ; Tue, 5 Jul 2016 08:38:39 +0000 (UTC) (envelope-from freebsd@dohd.org) Received: from mx.terantula.net (mx.terantula.nl [80.255.244.205]) by mx.terantula.net (Postfix) with ESMTP id E686F216F95 for ; Tue, 5 Jul 2016 10:38:26 +0200 (CEST) X-Virus-Scanned: amavisd-new at terantula.nl Received: from mx.terantula.net ([80.255.244.205]) by mx.terantula.net (mx.terantula.net [80.255.244.205]) (amavisd-new, port 10024) with LMTP id yiJOiQ0McYdN for ; Tue, 5 Jul 2016 10:38:24 +0200 (CEST) Received: from nala.dohd.org (nala.xs4all.nl [83.162.240.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx.terantula.net (Postfix) with ESMTPS id C38B5216F94 for ; Tue, 5 Jul 2016 10:38:24 +0200 (CEST) Received: from mailscan.local.dohd.org (mailscan.local.dohd.org [10.0.0.205]) by nala.dohd.org (Postfix) with ESMTP id 823473AC8A for ; Tue, 5 Jul 2016 10:38:24 +0200 (CEST) X-Virus-Scanned: amavisd-new at dohd.org Received: from nala.dohd.org ([10.0.0.5]) by mailscan.local.dohd.org (mailscan.local.dohd.org [10.0.0.205]) (amavisd-new, port 10024) with LMTP id wU3Id-Kd0tY9 for ; Tue, 5 Jul 2016 10:38:23 +0200 (CEST) Received: by nala.dohd.org (Postfix, from userid 1008) id 995723AC83; Tue, 5 Jul 2016 10:38:23 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 nala.dohd.org 995723AC83 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dohd.org; s=_testsel; t=1467707903; bh=G81Vf/gHmGaSKGGKYKzr8N+Bed93f3vX7tQWnkoRs3Y=; h=Date:From:To:Subject:References:In-Reply-To; b=FeiaLoPXqsuLcM/HrpWstcYw8ILu3UIqVcv9L7ebgZP1pjBK0E5Zz0UfwOHHNVCt0 Z3f5UPP+pj59brUSu0iTWjEFciI3deAlRKLO6zCrbKC286KJYOeuYcYnYSz+O2xgmI 45I1JgQJpeLTVLlQKnOC3M7HmKPs4lsYNuxY05oA= Date: Tue, 5 Jul 2016 10:38:23 +0200 From: Mark Huizer To: freebsd-stable@FreeBSD.org Subject: Re: Recent stable: bsnmpd eats up memory and cpu Message-ID: <20160705083823.GA58223@eeyore.local.dohd.org> References: <20160501220107.GA58930@lyxys.ka.sub.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160501220107.GA58930@lyxys.ka.sub.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 08:38:39 -0000 On Mon, May 02, 2016 at 12:01:07AM +0200, Wolfgang Zenker wrote: > Hi, > > after updating some 10-STABLE systems a few days ago, I noticed that on > two of those systems bsnmpd started to use up a lot of cpu time, and the > available memory shrinked until rendering the system unusable. Killing > bsnmpd stops the cpu usage but does not free up memory. > Both affected systems are amd64, one having moved from r297555 to > r298723, the other from r297555 to r298722. Another amd64 system > that went from r297555 to r298722 appears to be not affected. > The two affected systems are on an internal LAN segment and there > is currently no application connecting to snmp on those machines. > > What would be useful debugging data to collect in this case? I saw a resembling thing here after upgrading to 10-stable yesterday. I noticed bsnmp running at 30 to 40% cpu continuously, resembling what I found here: https://forums.freebsd.org/threads/37045/ After ktrace-ing the process I noticed it was doing sysctl stuff all the time and a more close look showed that it was freaking out on the CD device without a medium. So I inserted a CD in the device, and bsnmp went back to normal. "Interesting". So far I didn't notice memory issues. Mark -- Nice testing in little China... From owner-freebsd-stable@freebsd.org Tue Jul 5 08:40:28 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4ADAB20813 for ; Tue, 5 Jul 2016 08:40:28 +0000 (UTC) (envelope-from freebsd@dohd.org) Received: from mx.terantula.net (mx.terantula.net [IPv6:2001:4cb8:1:3111::25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "*.terantula.net", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A6080155A for ; Tue, 5 Jul 2016 08:40:28 +0000 (UTC) (envelope-from freebsd@dohd.org) Received: from mx.terantula.net (mx.terantula.nl [80.255.244.205]) by mx.terantula.net (Postfix) with ESMTP id 2593B216F94 for ; Tue, 5 Jul 2016 10:40:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at terantula.nl Received: from mx.terantula.net ([80.255.244.205]) by mx.terantula.net (mx.terantula.net [80.255.244.205]) (amavisd-new, port 10024) with LMTP id qKIqXRthO13g for ; Tue, 5 Jul 2016 10:40:25 +0200 (CEST) Received: from nala.dohd.org (mail.dohd.org [IPv6:2001:981:f06c:40::25]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx.terantula.net (Postfix) with ESMTPS id 10DB4216F6A for ; Tue, 5 Jul 2016 10:40:25 +0200 (CEST) Received: from mailscan.local.dohd.org (mailscan.local.dohd.org [10.0.0.205]) by nala.dohd.org (Postfix) with ESMTP id AF4AA3AC8A for ; Tue, 5 Jul 2016 10:40:24 +0200 (CEST) X-Virus-Scanned: amavisd-new at dohd.org Received: from nala.dohd.org ([10.0.0.5]) by mailscan.local.dohd.org (mailscan.local.dohd.org [10.0.0.205]) (amavisd-new, port 10024) with LMTP id BpquHQS3BNPe for ; Tue, 5 Jul 2016 10:40:24 +0200 (CEST) Received: by nala.dohd.org (Postfix, from userid 1008) id 0C9BF3AC83; Tue, 5 Jul 2016 10:40:24 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 nala.dohd.org 0C9BF3AC83 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dohd.org; s=_testsel; t=1467708024; bh=wU2ThEKGF1SwHqOiOJfaIC5JFFxkHENmYnwIbfTsSNI=; h=Date:From:To:Subject:References:In-Reply-To; b=aY3Fi1zKgO3B0gm9OMXDfn8kfHj9OY0YQqrMrbyLhsVcDI6PxJfcXXUXLgEUCgsKm yXrRl2J8FWAeJYUaEIKNoNsb1gyxF+iRTGKSx3410iiGu2ntPsRBoqmq6PNsMwiGLa BwkXJGTgOS0dtO1db1eaBhi7iqNratnIib3Ym460= Date: Tue, 5 Jul 2016 10:40:23 +0200 From: Mark Huizer To: freebsd-stable@FreeBSD.org Subject: Re: Recent stable: bsnmpd eats up memory and cpu Message-ID: <20160705084023.GB58223@eeyore.local.dohd.org> References: <20160501220107.GA58930@lyxys.ka.sub.org> <20160705083823.GA58223@eeyore.local.dohd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160705083823.GA58223@eeyore.local.dohd.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 08:40:29 -0000 On Tue, Jul 05, 2016 at 10:38:23AM +0200, Mark Huizer wrote: > On Mon, May 02, 2016 at 12:01:07AM +0200, Wolfgang Zenker wrote: > > Hi, > > > > after updating some 10-STABLE systems a few days ago, I noticed that on > > two of those systems bsnmpd started to use up a lot of cpu time, and the > > available memory shrinked until rendering the system unusable. Killing > > bsnmpd stops the cpu usage but does not free up memory. > > Both affected systems are amd64, one having moved from r297555 to > > r298723, the other from r297555 to r298722. Another amd64 system > > that went from r297555 to r298722 appears to be not affected. > > The two affected systems are on an internal LAN segment and there > > is currently no application connecting to snmp on those machines. > > > > What would be useful debugging data to collect in this case? > > I saw a resembling thing here after upgrading to 10-stable yesterday. > I noticed bsnmp running at 30 to 40% cpu continuously, resembling what I found here: > > https://forums.freebsd.org/threads/37045/ Sorry, I meant https://forums.freebsd.org/threads/56235/ Greetings Mark -- Nice testing in little China... From owner-freebsd-stable@freebsd.org Tue Jul 5 11:48:16 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72CF3B21F9D for ; Tue, 5 Jul 2016 11:48:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5AEC21DFA for ; Tue, 5 Jul 2016 11:48:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 535C5B21F9A; Tue, 5 Jul 2016 11:48:16 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52C25B21F97; Tue, 5 Jul 2016 11:48:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F11141DF8; Tue, 5 Jul 2016 11:48:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u65Bm9b6022894 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 5 Jul 2016 14:48:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u65Bm9b6022894 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u65Bm8AJ022893; Tue, 5 Jul 2016 14:48:08 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 5 Jul 2016 14:48:08 +0300 From: Konstantin Belousov To: Maxim Sobolev Cc: stable@freebsd.org, hackers@freebsd.org Subject: Re: A faulty program corrupts some its data preventing correct core generation (Failed to write core file for process postgres (error 14)) Message-ID: <20160705114808.GN38613@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 11:48:16 -0000 On Mon, Jul 04, 2016 at 10:26:25PM -0700, Maxim Sobolev wrote: > Hi all, investigating some random postgresql-9.1.21 server crashes on > FreeBSD 10.3, we've started seeing those after upgrading from postgres > 9.1.18 on more than one system, so hardware (e.g. RAM issues) are very > unlikely. I suspect that postgres is at fault, however I am also curious > how could it be that kernel is not capable of generating core file when > application does something silly? Is it that some ELF-related data > structures got corrupted or something else? Are we protecting the page > where ELF header is mapped with R/O flag? I am looking at possibly > recreating this by poking around elf header(s), seeing if I can corrupt it > in a similar manner reliably, any pointers or suggestions are appreciated. > > Jun 27 04:10:18 dal12 kernel: Failed to write core file for process > postgres (error 14) > Jun 27 04:10:18 dal12 kernel: pid 41361 (postgres), uid 70: exited on > signal 11 > Jul 1 05:21:46 dal12 kernel: Failed to write core file for process > postgres (error 14) > Jul 1 05:21:46 dal12 kernel: pid 1722 (postgres), uid 70: exited on signal > 11 > > #define EFAULT 14 /* Bad address */ > > The resulting files are truncated and is not really usable for anything. > We've seen the same issue > > -rw------- 1 pgsql wheel 1310720 Jun 27 04:10 postgres.41361.core > -rw------- 1 pgsql wheel 1310720 Jul 1 05:21 postgres.1722.core > > [ssp-root@dal12 /var/tmp]$ sudo gdb711 postgres postgres.1722.core > GNU gdb (GDB) 7.11 [GDB v7.11 for FreeBSD] > Copyright (C) 2016 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-portbld-freebsd10.3". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from postgres...(no debugging symbols found)...done. > BFD: Warning: /var/tmp/postgres.1722.core is truncated: expected core file > size >= 517120000, found: 1310720. > [New LWP 100261] > Core was generated by `postgres'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 > (gdb) where > #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 > Backtrace stopped: Cannot access memory at address 0x7fffffffdd08 > (gdb) q > https://lists.freebsd.org/pipermail/freebsd-stable/2016-June/084877.html From owner-freebsd-stable@freebsd.org Tue Jul 5 14:43:56 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AA7CB73F24 for ; Tue, 5 Jul 2016 14:43:56 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id DB82B16B5 for ; Tue, 5 Jul 2016 14:43:55 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id D7427B73F1F; Tue, 5 Jul 2016 14:43:55 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6E97B73F1E for ; Tue, 5 Jul 2016 14:43:55 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 63F1116B3 for ; Tue, 5 Jul 2016 14:43:55 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-wm0-x22f.google.com with SMTP id r201so155853508wme.1 for ; Tue, 05 Jul 2016 07:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=2UxvES/Nt7XSNT8IovIwmSfj/iLLw0k+Fi0kBHPpE6I=; b=IFLFUwDx8zKlMBfILlOHHuCZXbEzUwnoVgM6s/xCae0Zn6Cs8RctxH8G56cOU3Yl1Q JW4KXOS6chooPaGgE7MMdk9+CfEAzIBIBDn7SVxnLApXv7v1Mje6kXVI4q4FU+4roQjI 2tiJyp2BLf+iRaZ5W+EJYuyY8dGpSjVb8z23HqxR+8QI380vxS62n8oNKH35JC+KtpXO y7pyo9L/86xb/ASgSG6q0ANQRUA5fprNfN4zvu1zWaYT3ZI6AmMzkCEBoUQ9QtPn8kCs vsEBVGGrSBHqGI8/TmgumYS0nn3q/wU4QcArcSM5Uo1OtOEbojuXeuORV33j2YxcBfnb kiaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=2UxvES/Nt7XSNT8IovIwmSfj/iLLw0k+Fi0kBHPpE6I=; b=CcLln/0XMVBjPaqEeyhqTV9AVtVNVCDecqyuPioLKm56A1kEYRlgMUh4V5ut4j7Jdq XaP+IZ7+qJM0YsyVOhjAaQ81hqdkgWADOkcsz1tPsu8g1PdhKY9MphgKTsEf86P6i8kT Hu6ytZRZTfs/ueI63dcsSRnxttrH5x0x6v39IHF3qFNyZ9mYqSswoO9ZXs0ypyYPY85Y x5PcurI6f87OIODzRfZFFMRqnUywbGnwWiz7mhuDbnFrEt9fvd76xrnoy0r73HdQD7lT nqQNV3LUJ5wlf+sA5jYA78eqVhzF6ONc+fydY/P3iQAqMKIYezCae1wc3mfjgLDgjEBu sNBw== X-Gm-Message-State: ALyK8tKb9G7DhLBebeRHA8oI3qwU2jp+76Hm5f9tPq/mpLou+NptOl1OwHGB3VIsrKFc+SFRj1BDVBSSrlWaTlRn X-Received: by 10.194.150.167 with SMTP id uj7mr16004821wjb.168.1467729833520; Tue, 05 Jul 2016 07:43:53 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.194.96.173 with HTTP; Tue, 5 Jul 2016 07:43:52 -0700 (PDT) In-Reply-To: <20160705114808.GN38613@kib.kiev.ua> References: <20160705114808.GN38613@kib.kiev.ua> From: Maxim Sobolev Date: Tue, 5 Jul 2016 07:43:52 -0700 X-Google-Sender-Auth: E-lV_8x0x9_v8XbroxQzrseI7GE Message-ID: Subject: Re: A faulty program corrupts some its data preventing correct core generation (Failed to write core file for process postgres (error 14)) To: Konstantin Belousov Cc: stable@freebsd.org, hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 14:43:56 -0000 Seems like candidate for the MFC into releng/10.3 and appropriate errata entry? -Max On Tue, Jul 5, 2016 at 4:48 AM, Konstantin Belousov wrote: > On Mon, Jul 04, 2016 at 10:26:25PM -0700, Maxim Sobolev wrote: > > Hi all, investigating some random postgresql-9.1.21 server crashes on > > FreeBSD 10.3, we've started seeing those after upgrading from postgres > > 9.1.18 on more than one system, so hardware (e.g. RAM issues) are very > > unlikely. I suspect that postgres is at fault, however I am also curious > > how could it be that kernel is not capable of generating core file when > > application does something silly? Is it that some ELF-related data > > structures got corrupted or something else? Are we protecting the page > > where ELF header is mapped with R/O flag? I am looking at possibly > > recreating this by poking around elf header(s), seeing if I can corrupt > it > > in a similar manner reliably, any pointers or suggestions are > appreciated. > > > > Jun 27 04:10:18 dal12 kernel: Failed to write core file for process > > postgres (error 14) > > Jun 27 04:10:18 dal12 kernel: pid 41361 (postgres), uid 70: exited on > > signal 11 > > Jul 1 05:21:46 dal12 kernel: Failed to write core file for process > > postgres (error 14) > > Jul 1 05:21:46 dal12 kernel: pid 1722 (postgres), uid 70: exited on > signal > > 11 > > > > #define EFAULT 14 /* Bad address */ > > > > The resulting files are truncated and is not really usable for anything. > > We've seen the same issue > > > > -rw------- 1 pgsql wheel 1310720 Jun 27 04:10 > postgres.41361.core > > -rw------- 1 pgsql wheel 1310720 Jul 1 05:21 > postgres.1722.core > > > > [ssp-root@dal12 /var/tmp]$ sudo gdb711 postgres postgres.1722.core > > GNU gdb (GDB) 7.11 [GDB v7.11 for FreeBSD] > > Copyright (C) 2016 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html > > > > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > > and "show warranty" for details. > > This GDB was configured as "x86_64-portbld-freebsd10.3". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > . > > Find the GDB manual and other documentation resources online at: > > . > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from postgres...(no debugging symbols found)...done. > > BFD: Warning: /var/tmp/postgres.1722.core is truncated: expected core > file > > size >= 517120000, found: 1310720. > > [New LWP 100261] > > Core was generated by `postgres'. > > Program terminated with signal SIGSEGV, Segmentation fault. > > #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 > > (gdb) where > > #0 0x0000000800cfba67 in ?? () from /lib/libthr.so.3 > > Backtrace stopped: Cannot access memory at address 0x7fffffffdd08 > > (gdb) q > > > https://lists.freebsd.org/pipermail/freebsd-stable/2016-June/084877.html > > From owner-freebsd-stable@freebsd.org Tue Jul 5 16:59:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DB4FB724C1 for ; Tue, 5 Jul 2016 16:59:31 +0000 (UTC) (envelope-from tundra@tundraware.com) Received: from oceanview.tundraware.com (oceanview.tundraware.com [45.55.60.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "oceanview.tundraware.com", Issuer "oceanview.tundraware.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 534AF1EEC for ; Tue, 5 Jul 2016 16:59:30 +0000 (UTC) (envelope-from tundra@tundraware.com) Received: from slipstream.local (ds1.fai.acsalaska.net [209.193.23.130] (may be forged)) (authenticated bits=0) by oceanview.tundraware.com (8.15.2/8.15.2) with ESMTPSA id u65Gdb5P034312 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 5 Jul 2016 11:39:37 -0500 (CDT) (envelope-from tundra@tundraware.com) To: FreeBSD Stable Maling List From: Tim Daneliuk Subject: Is tar Broken In 10.3-STABLE? Message-ID: <36b73290-0714-d5e5-6740-3318ff1055ec@tundraware.com> Date: Tue, 5 Jul 2016 08:39:34 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (oceanview.tundraware.com [45.55.60.57]); Tue, 05 Jul 2016 11:39:38 -0500 (CDT) X-TundraWare-MailScanner-Information: Please contact the ISP for more information X-TundraWare-MailScanner-ID: u65Gdb5P034312 X-TundraWare-MailScanner: Found to be clean X-TundraWare-MailScanner-From: tundra@tundraware.com X-Spam-Status: No X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 16:59:31 -0000 I just upgraded to r302342 today to verify a problem I saw after a 10.3-STABLE upgrade yesterday. Upgrade was accomplished via makeworld/kernel & installworld/kernel. When using tar with the -T argument to provide a list of backup sources, it blows out with the following error if a source in the file list is missing: tar: INTERNAL ERROR: Function 'archive_read_disk_open' invoked with archive structure in state 'header', should be in state 'new/closed': Unknown error: -1 In the past, tar would make some noise if it was asked to copy a nonexistent file or directory, but it would continue the remainder of the archive operation. Thoughts? From owner-freebsd-stable@freebsd.org Wed Jul 6 01:33:00 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 217B6B20248 for ; Wed, 6 Jul 2016 01:33:00 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail3.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id A6DCE11DB for ; Wed, 6 Jul 2016 01:32:58 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 36833 invoked by uid 907); 6 Jul 2016 01:26:15 -0000 Received: from Unknown (HELO jmmacpro.tmst.com.au) (203.14.245.130) (smtp-auth username janm, mechanism plain) by mail3.transactionware.com (qpsmtpd/0.84) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Wed, 06 Jul 2016 11:26:15 +1000 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Is tar Broken In 10.3-STABLE? From: Jan Mikkelsen In-Reply-To: <36b73290-0714-d5e5-6740-3318ff1055ec@tundraware.com> Date: Wed, 6 Jul 2016 11:26:59 +1000 Cc: FreeBSD Stable Maling List Message-Id: <97B23D8F-B8A3-4460-8AF7-D7393055DA96@transactionware.com> References: <36b73290-0714-d5e5-6740-3318ff1055ec@tundraware.com> To: Tim Daneliuk X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 01:33:00 -0000 Hi, Tar should complain and die if an input path doesn=E2=80=99t exist. So, = no, the behaviour you=E2=80=99re seeing isn=E2=80=99t broken. See also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205358 = This bug has been fixed upstream seems to have been imported into = stable/10 in r302075. =46rom the commit message: - tar and cpio should fail if an input file named on the command line is missing (vendor issue 708) I agree the message could be a bit clearer about what=E2=80=99s going = on! Regards, Jan. > On 6 Jul 2016, at 02:39, Tim Daneliuk wrote: >=20 > I just upgraded to r302342 today to verify a problem I saw=20 > after a 10.3-STABLE upgrade yesterday. Upgrade was=20 > accomplished via makeworld/kernel & installworld/kernel. >=20 >=20 > When using tar with the -T argument to provide a list=20 > of backup sources, it blows out with the following=20 > error if a source in the file list is missing: >=20 > tar: INTERNAL ERROR: Function 'archive_read_disk_open' invoked with = archive structure in state 'header', should be in state 'new/closed': = Unknown error: -1 >=20 > In the past, tar would make some noise if it was asked > to copy a nonexistent file or directory, but it would=20 > continue the remainder of the archive operation. >=20 > Thoughts? > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Wed Jul 6 09:06:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07BEBB21AD2 for ; Wed, 6 Jul 2016 09:06:23 +0000 (UTC) (envelope-from abrancatelli@schema31.it) Received: from benzedrina.milano.schema31.it (benzedrina.schema31.it [151.0.205.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "benzedrina.milano.schema31.it", Issuer "benzedrina.milano.schema31.it" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7680B160E for ; Wed, 6 Jul 2016 09:06:21 +0000 (UTC) (envelope-from abrancatelli@schema31.it) Received: from smtp.schema31.it (localhost [127.0.0.1]) by benzedrina.milano.schema31.it (8.14.9/8.14.9) with ESMTP id u6692bN2099964 for ; Wed, 6 Jul 2016 11:02:37 +0200 (CEST) (envelope-from abrancatelli@schema31.it) MIME-Version: 1.0 Date: Wed, 06 Jul 2016 11:02:37 +0200 From: Andrea Brancatelli To: FreeBSD Stable Maling List Subject: 10.3 update 5 + Inetd problem Organization: Schema31 s.r.l. Message-ID: X-Sender: abrancatelli@schema31.it User-Agent: Roundcube Webmail/1.1.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 09:06:23 -0000 Hello everybody, we're having a curious issue with a machine we recently upgraded to 10.3-RELEASE-p5. The machine is a pretty "stupid" MX with just sendmail running and delivering locally mails to an LMTP instance (dbmail lmtp). For various reasons we don't have a "demonized" LMTPd but we use Inetd to spawn it on demand. After we upgraded the machine from 10.2 to 10.3 all of a sudden the lmtpd process started (randomly) getting (forever) stuck with 100% cpu usage in _"uwait"_ state. On a twin machine with the same setup that is not yet upgraded (10.1-RELEASE-p3), with the same "ports" versions and identical setup the problem never happens. Given that I'm not too much into GDB, Dtrace and all that stuff, what can I do to try to diagnose the problem better? Thank you very much. -- Andrea Brancatelli Schema31 S.p.a. Responsabile IT ROMA - BO - FI - PA ITALY Tel: +39.06.98.358.472 Cell: +39.331.2488468 Fax: +39.055.71.880.466 Società del Gruppo SC31 ITALIA From owner-freebsd-stable@freebsd.org Wed Jul 6 14:51:30 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F8CFB75771; Wed, 6 Jul 2016 14:51:30 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x244.google.com (mail-yw0-x244.google.com [IPv6:2607:f8b0:4002:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E4401869; Wed, 6 Jul 2016 14:51:30 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x244.google.com with SMTP id i12so11388258ywa.0; Wed, 06 Jul 2016 07:51:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=2W18IY4L2vU7WGhVf+mqioV0jxeyxVx64fPGK62GrME=; b=iCgXzL70stA25xEQyFCWP3+nAdvJukcQQs/2wUHO5DkQ9IEcDWjzk84DgGpfVa9Ozn i2d5iawMcB9mh0HySZxJlAot+5dpANnDjQ1pndjmN52XxtGzdq5wdHFoQ1LNs7MtbEmh 2lxYR75dPBYe8OQkfClyr1ab7kihFeQaKc5NPkolgYPnTzH63URwFtu0hx3rHDBSTrvR oMMrm0cuHeuQnCAvwcnrdLnKZl+t/2zmbJh/evJrpK08lZXCX1mgCi1tDhyC97pzJKjb GpcnkFYpqufmUjRo/e8liNrceyO8TWLB6Af10yHnwskeTM1fAtjbe3pKf2uSabCtmOTI +ojg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=2W18IY4L2vU7WGhVf+mqioV0jxeyxVx64fPGK62GrME=; b=S8oHXY36g1Lz8Z2ZnqUng0181qQrcbFuXTc94l4y/xq+dceO/rNEPHJDvOixd26tuy 2ujJ8EeRQrB8+RTLL4Lwl80FXEOU6ho/mTKJXpxyGR9VELfxlJUlV3xxLhrbS5qa6iEI kbPj6zI/h+jsimkqy9Hk4u8xkoHyVG77ZwTuEwcAyvPBYip2c8a3bSnmCSTEviD28Xe3 1lFk8SF3+Pm2Nb1wBoI+d7oiyr389acOYLAIMJDnPtRq3hRulLwn50ZaCpKDCKEzP6er uHAYvfNdghNwceftJ3Z7H3Vgxx0j/PnuFnCfJYcl4BCINHiYSqnJ8VMD8kr1iQDPNLWX lURg== X-Gm-Message-State: ALyK8tI1ddVDAcxk5v/zAHO7K/J42IhO+O7ZMtHS9NnoxJ8u6zcAPmxX1GmqD9yWvPyCfSSYjtn+I67NlXRvKQ== X-Received: by 10.129.82.21 with SMTP id g21mr14762175ywb.66.1467816688930; Wed, 06 Jul 2016 07:51:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Wed, 6 Jul 2016 07:51:28 -0700 (PDT) In-Reply-To: References: From: David Cross Date: Wed, 6 Jul 2016 10:51:28 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 14:51:30 -0000 Ok.. to reply to my own message, I using ktr and debugging printfs I have found the culprit.. but I am still at a loss to 'why', or what the appropriate fix is. Lets go back to the panic (simplified) #0 0xffffffff8043f160 at kdb_backtrace+0x60 #1 0xffffffff80401454 at vpanic+0x124 #2 0xffffffff804014e3 at panic+0x43 #3 0xffffffff8060719a at softdep_deallocate_dependencies+0x6a #4 0xffffffff80499cc1 at brelse+0x151 #5 0xffffffff804979b1 at bufwrite+0x81 #6 0xffffffff80623c80 at ffs_write+0x4b0 #7 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 #8 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 #9 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 #10 0xffffffff80661cc1 at vnode_pager_putpages+0x91 #11 0xffffffff806588e6 at vm_pageout_flush+0x116 #12 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 #13 0xffffffff80651519 at vm_object_page_clean+0x179 #14 0xffffffff80651102 at vm_object_terminate+0xa2 #15 0xffffffff806621a5 at vnode_destroy_vobject+0x85 #16 0xffffffff8062a52f at ufs_reclaim+0x1f #17 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 Via KTR logging I determined that the dangling dependedency was on a freshly allocated buf, *after* vinvalbuf in the vgonel() (so in VOP_RECLAIM itself), called by the vnode lru cleanup process; I further noticed that it was in a newbuf that recycled a bp (unimportant, except it let me narrow down my logging to something managable), from there I get this stacktrace (simplified) #0 0xffffffff8043f160 at kdb_backtrace+0x60 #1 0xffffffff8049c98e at getnewbuf+0x4be #2 0xffffffff804996a0 at getblk+0x830 #3 0xffffffff805fb207 at ffs_balloc_ufs2+0x1327 #4 0xffffffff80623b0b at ffs_write+0x33b #5 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 #6 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 #7 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 #8 0xffffffff80661cc1 at vnode_pager_putpages+0x91 #9 0xffffffff806588e6 at vm_pageout_flush+0x116 #10 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 #11 0xffffffff80651519 at vm_object_page_clean+0x179 #12 0xffffffff80651102 at vm_object_terminate+0xa2 #13 0xffffffff806621a5 at vnode_destroy_vobject+0x85 #14 0xffffffff8062a52f at ufs_reclaim+0x1f #15 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 #16 0xffffffff804b6c6e at vgonel+0x2ee #17 0xffffffff804ba6f5 at vnlru_proc+0x4b5 addr2line on the ffs_balloc_ufs2 gives: /usr/src/sys/ufs/ffs/ffs_balloc.c:778: bp = getblk(vp, lbn, nsize, 0, 0, gbflags); bp->b_blkno = fsbtodb(fs, newb); if (flags & BA_CLRBUF) vfs_bio_clrbuf(bp); if (DOINGSOFTDEP(vp)) softdep_setup_allocdirect(ip, lbn, newb, 0, nsize, 0, bp); Boom, freshly allocated buffer with a dependecy; nothing in VOP_RECLAIM handles this, this is after vinvalbuf is called, it expects that everything is flushed to disk and its just about releasing structures (is my read of the code). Now, perhaps this is a good assumption? the question then is how is this buffer hanging out there surviving a a vinvalbuf. I will note that my test-case that finds this runs and terminates *minutes* before... its not just hanging out there in a race, its surviving background sync, fsync, etc... wtf? Also, I *can* unmount the FS without an error, so that codepath is either ignoring this buffer, or its forcing a sync in a way that doesn't panic? Anyone have next steps? I am making progress here, but its really slow going, this is probably the most complex portion of the kernel and some pointers would be helpful. On Sat, Jul 2, 2016 at 2:31 PM, David Cross wrote: > Ok, I have been trying to trace this down for awhile..I know quite a bit > about it.. but there's a lot I don't know, or I would have a patch. I have > been trying to solve this on my own, but bringing in some outside > assistance will let me move on with my life. > > First up: The stacktrace (from a debugging kernel, with coredump > > #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:298 > #1 0xffffffff8071018a in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:486 > #2 0xffffffff80710afc in vpanic ( > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d", > ap=0xfffffe023ae5cf40) > at /usr/src/sys/kern/kern_shutdown.c:889 > #3 0xffffffff807108c0 in panic ( > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d") > at /usr/src/sys/kern/kern_shutdown.c:818 > #4 0xffffffff80a7c841 in softdep_deallocate_dependencies ( > bp=0xfffffe01f030e148) at /usr/src/sys/ufs/ffs/ffs_softdep.c:14099 > #5 0xffffffff807f793f in buf_deallocate (bp=0xfffffe01f030e148) at > buf.h:428 > #6 0xffffffff807f59c9 in brelse (bp=0xfffffe01f030e148) > at /usr/src/sys/kern/vfs_bio.c:1599 > #7 0xffffffff807f3132 in bufwrite (bp=0xfffffe01f030e148) > at /usr/src/sys/kern/vfs_bio.c:1180 > #8 0xffffffff80ab226a in bwrite (bp=0xfffffe01f030e148) at buf.h:395 > #9 0xffffffff80aafb1b in ffs_write (ap=0xfffffe023ae5d2b8) > at /usr/src/sys/ufs/ffs/ffs_vnops.c:800 > #10 0xffffffff80bdf0ed in VOP_WRITE_APV (vop=0xffffffff80f15480, > a=0xfffffe023ae5d2b8) at vnode_if.c:999 > #11 0xffffffff80b1d02e in VOP_WRITE (vp=0xfffff80077e7a000, > uio=0xfffffe023ae5d378, ioflag=8323232, cred=0xfffff80004235000) > at vnode_if.h:413 > #12 0xffffffff80b1ce97 in vnode_pager_generic_putpages > (vp=0xfffff80077e7a000, > ma=0xfffffe023ae5d660, bytecount=16384, flags=1, > rtvals=0xfffffe023ae5d580) > at /usr/src/sys/vm/vnode_pager.c:1138 > #13 0xffffffff80805a57 in vop_stdputpages (ap=0xfffffe023ae5d478) > at /usr/src/sys/kern/vfs_default.c:760 > #14 0xffffffff80be201e in VOP_PUTPAGES_APV (vop=0xffffffff80f00218, > a=0xfffffe023ae5d478) at vnode_if.c:2861 > #15 0xffffffff80b1d7e3 in VOP_PUTPAGES (vp=0xfffff80077e7a000, > m=0xfffffe023ae5d660, count=16384, sync=1, rtvals=0xfffffe023ae5d580, > offset=0) at vnode_if.h:1189 > #16 0xffffffff80b196f3 in vnode_pager_putpages (object=0xfffff8014a1fce00, > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > at /usr/src/sys/vm/vnode_pager.c:1016 > #17 0xffffffff80b0a605 in vm_pager_put_pages (object=0xfffff8014a1fce00, > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > at vm_pager.h:144 > #18 0xffffffff80b0a18c in vm_pageout_flush (mc=0xfffffe023ae5d660, > count=4, > flags=1, mreq=0, prunlen=0xfffffe023ae5d6f8, eio=0xfffffe023ae5d77c) > at /usr/src/sys/vm/vm_pageout.c:533 > #19 0xffffffff80afec76 in vm_object_page_collect_flush ( > object=0xfffff8014a1fce00, p=0xfffff8023a882370, pagerflags=1, > flags=1, > clearobjflags=0xfffffe023ae5d780, eio=0xfffffe023ae5d77c) > at /usr/src/sys/vm/vm_object.c:971 > #20 0xffffffff80afe91e in vm_object_page_clean (object=0xfffff8014a1fce00, > start=0, end=0, flags=1) at /usr/src/sys/vm/vm_object.c:897 > #21 0xffffffff80afe1fa in vm_object_terminate (object=0xfffff8014a1fce00) > at /usr/src/sys/vm/vm_object.c:735 > #22 0xffffffff80b1a0f1 in vnode_destroy_vobject (vp=0xfffff80077e7a000) > at /usr/src/sys/vm/vnode_pager.c:164 > #23 0xffffffff80abb191 in ufs_prepare_reclaim (vp=0xfffff80077e7a000) > at /usr/src/sys/ufs/ufs/ufs_inode.c:190 > #24 0xffffffff80abb1f9 in ufs_reclaim (ap=0xfffffe023ae5d968) > at /usr/src/sys/ufs/ufs/ufs_inode.c:219 > #25 0xffffffff80be0ade in VOP_RECLAIM_APV (vop=0xffffffff80f15ec0, > a=0xfffffe023ae5d968) at vnode_if.c:2019 > #26 0xffffffff80827849 in VOP_RECLAIM (vp=0xfffff80077e7a000, > td=0xfffff80008931960) at vnode_if.h:830 > #27 0xffffffff808219a9 in vgonel (vp=0xfffff80077e7a000) > at /usr/src/sys/kern/vfs_subr.c:2943 > #28 0xffffffff808294e8 in vlrureclaim (mp=0xfffff80008b2e000) > at /usr/src/sys/kern/vfs_subr.c:882 > #29 0xffffffff80828ea9 in vnlru_proc () at > /usr/src/sys/kern/vfs_subr.c:1000 > #30 0xffffffff806b66c5 in fork_exit (callout=0xffffffff80828c50 > , > arg=0x0, frame=0xfffffe023ae5dc00) at > /usr/src/sys/kern/kern_fork.c:1027 > #31 0xffffffff80b21dce in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:611 > #32 0x0000000000000000 in ?? () > > This is a kernel compiled -O -g, its "almost" GENERIC; the only difference > is some removed drivers, I have reproduced this on a few different kernels, > including a BHYVE one so I can poke at it and not take out the main > machine. The reproduction as it currently stands needs to have jails > running, but I don't believe this is a jail interaction, I think its just > that the process that sets up the problem happens to be running in a jail. > The step is "start jail; run "find /mountpoint -xdev >/dev/null" on the > filesystem, when the vnlru forces the problem vnode out the system panics. > > I made a few modifications to the kernel to spit out information about the > buf that causes the issue, but that is it. > > Information about the buf in question; it has a single softdependency > worklist for direct allocation: > (kgdb) print *bp->b_dep->lh_first > $6 = {wk_list = {le_next = 0x0, le_prev = 0xfffffe01f030e378}, > wk_mp = 0xfffff80008b2e000, wk_type = 4, wk_state = 163841} > > The file that maps to that buffer: > ls -lh MOUNTPOINT/jails/mail/var/imap/db/__db.002 > -rw------- 1 cyrus cyrus 24K Jul 1 20:32 > MOUNTPOINT/jails/mail/var/imap/db/__db.002 > > Any help is appreciated, until then I will keep banging my head against > the proverbial wall on this :) > From owner-freebsd-stable@freebsd.org Wed Jul 6 15:18:36 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C89FB75E29; Wed, 6 Jul 2016 15:18:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A13B21865; Wed, 6 Jul 2016 15:18:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u66FIMY6079547 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 6 Jul 2016 18:18:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u66FIMY6079547 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u66FIM1Y079519; Wed, 6 Jul 2016 18:18:22 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 6 Jul 2016 18:18:22 +0300 From: Konstantin Belousov To: David Cross Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) Message-ID: <20160706151822.GC38613@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 15:18:36 -0000 On Wed, Jul 06, 2016 at 10:51:28AM -0400, David Cross wrote: > Ok.. to reply to my own message, I using ktr and debugging printfs I have > found the culprit.. but I am still at a loss to 'why', or what the > appropriate fix is. > > Lets go back to the panic (simplified) > > #0 0xffffffff8043f160 at kdb_backtrace+0x60 > #1 0xffffffff80401454 at vpanic+0x124 > #2 0xffffffff804014e3 at panic+0x43 > #3 0xffffffff8060719a at softdep_deallocate_dependencies+0x6a > #4 0xffffffff80499cc1 at brelse+0x151 > #5 0xffffffff804979b1 at bufwrite+0x81 > #6 0xffffffff80623c80 at ffs_write+0x4b0 > #7 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 > #8 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 > #9 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 > #10 0xffffffff80661cc1 at vnode_pager_putpages+0x91 > #11 0xffffffff806588e6 at vm_pageout_flush+0x116 > #12 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 > #13 0xffffffff80651519 at vm_object_page_clean+0x179 > #14 0xffffffff80651102 at vm_object_terminate+0xa2 > #15 0xffffffff806621a5 at vnode_destroy_vobject+0x85 > #16 0xffffffff8062a52f at ufs_reclaim+0x1f > #17 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 > > Via KTR logging I determined that the dangling dependedency was on a > freshly allocated buf, *after* vinvalbuf in the vgonel() (so in VOP_RECLAIM > itself), called by the vnode lru cleanup process; I further noticed that it > was in a newbuf that recycled a bp (unimportant, except it let me narrow > down my logging to something managable), from there I get this stacktrace > (simplified) > > #0 0xffffffff8043f160 at kdb_backtrace+0x60 > #1 0xffffffff8049c98e at getnewbuf+0x4be > #2 0xffffffff804996a0 at getblk+0x830 > #3 0xffffffff805fb207 at ffs_balloc_ufs2+0x1327 > #4 0xffffffff80623b0b at ffs_write+0x33b > #5 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 > #6 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 > #7 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 > #8 0xffffffff80661cc1 at vnode_pager_putpages+0x91 > #9 0xffffffff806588e6 at vm_pageout_flush+0x116 > #10 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 > #11 0xffffffff80651519 at vm_object_page_clean+0x179 > #12 0xffffffff80651102 at vm_object_terminate+0xa2 > #13 0xffffffff806621a5 at vnode_destroy_vobject+0x85 > #14 0xffffffff8062a52f at ufs_reclaim+0x1f > #15 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 > #16 0xffffffff804b6c6e at vgonel+0x2ee > #17 0xffffffff804ba6f5 at vnlru_proc+0x4b5 > > addr2line on the ffs_balloc_ufs2 gives: > /usr/src/sys/ufs/ffs/ffs_balloc.c:778: > > bp = getblk(vp, lbn, nsize, 0, 0, gbflags); > bp->b_blkno = fsbtodb(fs, newb); > if (flags & BA_CLRBUF) > vfs_bio_clrbuf(bp); > if (DOINGSOFTDEP(vp)) > softdep_setup_allocdirect(ip, lbn, newb, 0, > nsize, 0, bp); > > > Boom, freshly allocated buffer with a dependecy; nothing in VOP_RECLAIM > handles this, this is after vinvalbuf is called, it expects that everything > is flushed to disk and its just about releasing structures (is my read of > the code). > > Now, perhaps this is a good assumption? the question then is how is this > buffer hanging out there surviving a a vinvalbuf. I will note that my > test-case that finds this runs and terminates *minutes* before... its not > just hanging out there in a race, its surviving background sync, fsync, > etc... wtf? Also, I *can* unmount the FS without an error, so that > codepath is either ignoring this buffer, or its forcing a sync in a way > that doesn't panic? Most typical cause for the buffer dependencies not flushed is a buffer write error. At least you could provide the printout of the buffer to confirm or reject this assumption. Were there any kernel messages right before the panic ? Just in case, did you fsck the volume before using it, after the previous panic ? > > Anyone have next steps? I am making progress here, but its really slow > going, this is probably the most complex portion of the kernel and some > pointers would be helpful. > > On Sat, Jul 2, 2016 at 2:31 PM, David Cross wrote: > > > Ok, I have been trying to trace this down for awhile..I know quite a bit > > about it.. but there's a lot I don't know, or I would have a patch. I have > > been trying to solve this on my own, but bringing in some outside > > assistance will let me move on with my life. > > > > First up: The stacktrace (from a debugging kernel, with coredump > > > > #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:298 > > #1 0xffffffff8071018a in kern_reboot (howto=260) > > at /usr/src/sys/kern/kern_shutdown.c:486 > > #2 0xffffffff80710afc in vpanic ( > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d", > > ap=0xfffffe023ae5cf40) > > at /usr/src/sys/kern/kern_shutdown.c:889 > > #3 0xffffffff807108c0 in panic ( > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d") > > at /usr/src/sys/kern/kern_shutdown.c:818 > > #4 0xffffffff80a7c841 in softdep_deallocate_dependencies ( > > bp=0xfffffe01f030e148) at /usr/src/sys/ufs/ffs/ffs_softdep.c:14099 > > #5 0xffffffff807f793f in buf_deallocate (bp=0xfffffe01f030e148) at > > buf.h:428 > > #6 0xffffffff807f59c9 in brelse (bp=0xfffffe01f030e148) > > at /usr/src/sys/kern/vfs_bio.c:1599 > > #7 0xffffffff807f3132 in bufwrite (bp=0xfffffe01f030e148) > > at /usr/src/sys/kern/vfs_bio.c:1180 > > #8 0xffffffff80ab226a in bwrite (bp=0xfffffe01f030e148) at buf.h:395 > > #9 0xffffffff80aafb1b in ffs_write (ap=0xfffffe023ae5d2b8) > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:800 > > #10 0xffffffff80bdf0ed in VOP_WRITE_APV (vop=0xffffffff80f15480, > > a=0xfffffe023ae5d2b8) at vnode_if.c:999 > > #11 0xffffffff80b1d02e in VOP_WRITE (vp=0xfffff80077e7a000, > > uio=0xfffffe023ae5d378, ioflag=8323232, cred=0xfffff80004235000) > > at vnode_if.h:413 > > #12 0xffffffff80b1ce97 in vnode_pager_generic_putpages > > (vp=0xfffff80077e7a000, > > ma=0xfffffe023ae5d660, bytecount=16384, flags=1, > > rtvals=0xfffffe023ae5d580) > > at /usr/src/sys/vm/vnode_pager.c:1138 > > #13 0xffffffff80805a57 in vop_stdputpages (ap=0xfffffe023ae5d478) > > at /usr/src/sys/kern/vfs_default.c:760 > > #14 0xffffffff80be201e in VOP_PUTPAGES_APV (vop=0xffffffff80f00218, > > a=0xfffffe023ae5d478) at vnode_if.c:2861 > > #15 0xffffffff80b1d7e3 in VOP_PUTPAGES (vp=0xfffff80077e7a000, > > m=0xfffffe023ae5d660, count=16384, sync=1, rtvals=0xfffffe023ae5d580, > > offset=0) at vnode_if.h:1189 > > #16 0xffffffff80b196f3 in vnode_pager_putpages (object=0xfffff8014a1fce00, > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > > at /usr/src/sys/vm/vnode_pager.c:1016 > > #17 0xffffffff80b0a605 in vm_pager_put_pages (object=0xfffff8014a1fce00, > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > > at vm_pager.h:144 > > #18 0xffffffff80b0a18c in vm_pageout_flush (mc=0xfffffe023ae5d660, > > count=4, > > flags=1, mreq=0, prunlen=0xfffffe023ae5d6f8, eio=0xfffffe023ae5d77c) > > at /usr/src/sys/vm/vm_pageout.c:533 > > #19 0xffffffff80afec76 in vm_object_page_collect_flush ( > > object=0xfffff8014a1fce00, p=0xfffff8023a882370, pagerflags=1, > > flags=1, > > clearobjflags=0xfffffe023ae5d780, eio=0xfffffe023ae5d77c) > > at /usr/src/sys/vm/vm_object.c:971 > > #20 0xffffffff80afe91e in vm_object_page_clean (object=0xfffff8014a1fce00, > > start=0, end=0, flags=1) at /usr/src/sys/vm/vm_object.c:897 > > #21 0xffffffff80afe1fa in vm_object_terminate (object=0xfffff8014a1fce00) > > at /usr/src/sys/vm/vm_object.c:735 > > #22 0xffffffff80b1a0f1 in vnode_destroy_vobject (vp=0xfffff80077e7a000) > > at /usr/src/sys/vm/vnode_pager.c:164 > > #23 0xffffffff80abb191 in ufs_prepare_reclaim (vp=0xfffff80077e7a000) > > at /usr/src/sys/ufs/ufs/ufs_inode.c:190 > > #24 0xffffffff80abb1f9 in ufs_reclaim (ap=0xfffffe023ae5d968) > > at /usr/src/sys/ufs/ufs/ufs_inode.c:219 > > #25 0xffffffff80be0ade in VOP_RECLAIM_APV (vop=0xffffffff80f15ec0, > > a=0xfffffe023ae5d968) at vnode_if.c:2019 > > #26 0xffffffff80827849 in VOP_RECLAIM (vp=0xfffff80077e7a000, > > td=0xfffff80008931960) at vnode_if.h:830 > > #27 0xffffffff808219a9 in vgonel (vp=0xfffff80077e7a000) > > at /usr/src/sys/kern/vfs_subr.c:2943 > > #28 0xffffffff808294e8 in vlrureclaim (mp=0xfffff80008b2e000) > > at /usr/src/sys/kern/vfs_subr.c:882 > > #29 0xffffffff80828ea9 in vnlru_proc () at > > /usr/src/sys/kern/vfs_subr.c:1000 > > #30 0xffffffff806b66c5 in fork_exit (callout=0xffffffff80828c50 > > , > > arg=0x0, frame=0xfffffe023ae5dc00) at > > /usr/src/sys/kern/kern_fork.c:1027 > > #31 0xffffffff80b21dce in fork_trampoline () > > at /usr/src/sys/amd64/amd64/exception.S:611 > > #32 0x0000000000000000 in ?? () > > > > This is a kernel compiled -O -g, its "almost" GENERIC; the only difference > > is some removed drivers, I have reproduced this on a few different kernels, > > including a BHYVE one so I can poke at it and not take out the main > > machine. The reproduction as it currently stands needs to have jails > > running, but I don't believe this is a jail interaction, I think its just > > that the process that sets up the problem happens to be running in a jail. > > The step is "start jail; run "find /mountpoint -xdev >/dev/null" on the > > filesystem, when the vnlru forces the problem vnode out the system panics. > > > > I made a few modifications to the kernel to spit out information about the > > buf that causes the issue, but that is it. > > > > Information about the buf in question; it has a single softdependency > > worklist for direct allocation: > > (kgdb) print *bp->b_dep->lh_first > > $6 = {wk_list = {le_next = 0x0, le_prev = 0xfffffe01f030e378}, > > wk_mp = 0xfffff80008b2e000, wk_type = 4, wk_state = 163841} > > > > The file that maps to that buffer: > > ls -lh MOUNTPOINT/jails/mail/var/imap/db/__db.002 > > -rw------- 1 cyrus cyrus 24K Jul 1 20:32 > > MOUNTPOINT/jails/mail/var/imap/db/__db.002 > > > > Any help is appreciated, until then I will keep banging my head against > > the proverbial wall on this :) > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Wed Jul 6 15:30:21 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 334C8B750BB; Wed, 6 Jul 2016 15:30:21 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x233.google.com (mail-yw0-x233.google.com [IPv6:2607:f8b0:4002:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D78D31E15; Wed, 6 Jul 2016 15:30:20 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x233.google.com with SMTP id v77so90432476ywg.0; Wed, 06 Jul 2016 08:30:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=2zIgEdwfw7Z1V6OGww1ImYgIa1uYbU7N6obh6XYYBrc=; b=SmIhSb1Q+w7zEtvd+HQHM5Jf2FdhefWaJtwsUrujd0SJQPQ/GSwrYx2Ej+50rrx0j+ zejoHuQ5OCkdAtQdZnKGZel/vG8ByU4C3DGxkbsl4+l2Iezv91Eiz/AF66KbR2guBvGL o4HPsnmR10zGDnIEx9mm2w4dIeSdxL0FZjObOI2Zbq14fope0PoG+0hxbaG1JU15XTWu MWzDC/KPChrc963y51oiMD3ODba3zJjz17CmEbk7Kezzd5OEvJHy8cWh2jdgRiyDChMs vAGt9CB+bMO2cCX7Kqw11Qu3zPHsaiAgxNV4T1Fwa/3ErRq4nGZbtqi9ym6ZsrgjwlPH ZQrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=2zIgEdwfw7Z1V6OGww1ImYgIa1uYbU7N6obh6XYYBrc=; b=NUF5ViTbPB4FlQLzdJfWKQKzfdUDM/gbs3NTvoLWQH4BjCypvYMhw8+JCJf9fzjq4C M6plTM7uZJHcj9U9Q+zn3/ulilXFsuXf4aPzTklGhXOofPE8pBnJ6IHGZPkKwFDPdrw+ /tesYSccOJEX6VNX55XgzlAnCPXPorx3vkdgMgqIQtpYuIYhE9QjMWNFUcAz7iJaOo+3 wW9dkyIfHN00SK1WrxCrYNfeJQIfAKPDlO76ZZ5QtPb7Fi36w83gsB32RDWkQk00qkXA 1lOs7A+RR5SSUtFAPjH9VYDzMKDJ77SrzpYZFVM+/ej4XJfLCW/9mm25pgn7bBVcgZaW kApA== X-Gm-Message-State: ALyK8tIuoJs71uy2FHLo/dBN0f99pH1lgDg8vOu1OjvbsArcnGfAi0qnvKZs8vC99ZYPpUTOrHE8AKV5B/HxVA== X-Received: by 10.129.5.215 with SMTP id 206mr16078637ywf.210.1467819019999; Wed, 06 Jul 2016 08:30:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Wed, 6 Jul 2016 08:30:19 -0700 (PDT) In-Reply-To: <20160706151822.GC38613@kib.kiev.ua> References: <20160706151822.GC38613@kib.kiev.ua> From: David Cross Date: Wed, 6 Jul 2016 11:30:19 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: Konstantin Belousov Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 15:30:21 -0000 No kernel messages before (if there were I would have written this off a long time ago); And as of right now, this is probably the most fsck-ed filesystem on the planet!.. I have an 'image' that I am going on that is ggate mounted, so I can access it in a bhyve VM to ease debuging so I am not crashing my real machine (with the real filesystem) all the time. One of my initial guesses was that this was a CG allocation error, but a dumpfs seems to show plenty of blocks in the CG to meet this need. Quick note on the testcase, I haven't totally isolated it yet, but the minimal reproduction is a 'ctl_cyrusdb -r", which runs a bdb5 recover op, a ktrace on that shows that it unlinks 3 files, opens them, lseeks then, writes a block, and then mmaps them (but leaves them open). At process termination is munmaps, and then closes. I have tried to write a shorter reproduction that opens, seeks, mmaps (with the same flags), writes the mmaped memory, munmaps, closes and exits, but this has been insufficient to reproduce the issue; There is likely some specific pattern in the bdb5 code tickling this, and behind the mmap-ed interface it is all opaque, and the bdb5 code is pretty complex itself On Wed, Jul 6, 2016 at 11:18 AM, Konstantin Belousov wrote: > On Wed, Jul 06, 2016 at 10:51:28AM -0400, David Cross wrote: > > Ok.. to reply to my own message, I using ktr and debugging printfs I have > > found the culprit.. but I am still at a loss to 'why', or what the > > appropriate fix is. > > > > Lets go back to the panic (simplified) > > > > #0 0xffffffff8043f160 at kdb_backtrace+0x60 > > #1 0xffffffff80401454 at vpanic+0x124 > > #2 0xffffffff804014e3 at panic+0x43 > > #3 0xffffffff8060719a at softdep_deallocate_dependencies+0x6a > > #4 0xffffffff80499cc1 at brelse+0x151 > > #5 0xffffffff804979b1 at bufwrite+0x81 > > #6 0xffffffff80623c80 at ffs_write+0x4b0 > > #7 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 > > #8 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 > > #9 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 > > #10 0xffffffff80661cc1 at vnode_pager_putpages+0x91 > > #11 0xffffffff806588e6 at vm_pageout_flush+0x116 > > #12 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 > > #13 0xffffffff80651519 at vm_object_page_clean+0x179 > > #14 0xffffffff80651102 at vm_object_terminate+0xa2 > > #15 0xffffffff806621a5 at vnode_destroy_vobject+0x85 > > #16 0xffffffff8062a52f at ufs_reclaim+0x1f > > #17 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 > > > > Via KTR logging I determined that the dangling dependedency was on a > > freshly allocated buf, *after* vinvalbuf in the vgonel() (so in > VOP_RECLAIM > > itself), called by the vnode lru cleanup process; I further noticed that > it > > was in a newbuf that recycled a bp (unimportant, except it let me narrow > > down my logging to something managable), from there I get this stacktrace > > (simplified) > > > > #0 0xffffffff8043f160 at kdb_backtrace+0x60 > > #1 0xffffffff8049c98e at getnewbuf+0x4be > > #2 0xffffffff804996a0 at getblk+0x830 > > #3 0xffffffff805fb207 at ffs_balloc_ufs2+0x1327 > > #4 0xffffffff80623b0b at ffs_write+0x33b > > #5 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 > > #6 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 > > #7 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 > > #8 0xffffffff80661cc1 at vnode_pager_putpages+0x91 > > #9 0xffffffff806588e6 at vm_pageout_flush+0x116 > > #10 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 > > #11 0xffffffff80651519 at vm_object_page_clean+0x179 > > #12 0xffffffff80651102 at vm_object_terminate+0xa2 > > #13 0xffffffff806621a5 at vnode_destroy_vobject+0x85 > > #14 0xffffffff8062a52f at ufs_reclaim+0x1f > > #15 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 > > #16 0xffffffff804b6c6e at vgonel+0x2ee > > #17 0xffffffff804ba6f5 at vnlru_proc+0x4b5 > > > > addr2line on the ffs_balloc_ufs2 gives: > > /usr/src/sys/ufs/ffs/ffs_balloc.c:778: > > > > bp = getblk(vp, lbn, nsize, 0, 0, gbflags); > > bp->b_blkno = fsbtodb(fs, newb); > > if (flags & BA_CLRBUF) > > vfs_bio_clrbuf(bp); > > if (DOINGSOFTDEP(vp)) > > softdep_setup_allocdirect(ip, lbn, newb, > 0, > > nsize, 0, bp); > > > > > > Boom, freshly allocated buffer with a dependecy; nothing in VOP_RECLAIM > > handles this, this is after vinvalbuf is called, it expects that > everything > > is flushed to disk and its just about releasing structures (is my read of > > the code). > > > > Now, perhaps this is a good assumption? the question then is how is this > > buffer hanging out there surviving a a vinvalbuf. I will note that my > > test-case that finds this runs and terminates *minutes* before... its not > > just hanging out there in a race, its surviving background sync, fsync, > > etc... wtf? Also, I *can* unmount the FS without an error, so that > > codepath is either ignoring this buffer, or its forcing a sync in a way > > that doesn't panic? > Most typical cause for the buffer dependencies not flushed is a buffer > write error. At least you could provide the printout of the buffer to > confirm or reject this assumption. > > Were there any kernel messages right before the panic ? Just in case, > did you fsck the volume before using it, after the previous panic ? > > > > > Anyone have next steps? I am making progress here, but its really slow > > going, this is probably the most complex portion of the kernel and some > > pointers would be helpful. > > > > On Sat, Jul 2, 2016 at 2:31 PM, David Cross > wrote: > > > > > Ok, I have been trying to trace this down for awhile..I know quite a > bit > > > about it.. but there's a lot I don't know, or I would have a patch. I > have > > > been trying to solve this on my own, but bringing in some outside > > > assistance will let me move on with my life. > > > > > > First up: The stacktrace (from a debugging kernel, with coredump > > > > > > #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:298 > > > #1 0xffffffff8071018a in kern_reboot (howto=260) > > > at /usr/src/sys/kern/kern_shutdown.c:486 > > > #2 0xffffffff80710afc in vpanic ( > > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling > deps > > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d", > > > ap=0xfffffe023ae5cf40) > > > at /usr/src/sys/kern/kern_shutdown.c:889 > > > #3 0xffffffff807108c0 in panic ( > > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling > deps > > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d") > > > at /usr/src/sys/kern/kern_shutdown.c:818 > > > #4 0xffffffff80a7c841 in softdep_deallocate_dependencies ( > > > bp=0xfffffe01f030e148) at /usr/src/sys/ufs/ffs/ffs_softdep.c:14099 > > > #5 0xffffffff807f793f in buf_deallocate (bp=0xfffffe01f030e148) at > > > buf.h:428 > > > #6 0xffffffff807f59c9 in brelse (bp=0xfffffe01f030e148) > > > at /usr/src/sys/kern/vfs_bio.c:1599 > > > #7 0xffffffff807f3132 in bufwrite (bp=0xfffffe01f030e148) > > > at /usr/src/sys/kern/vfs_bio.c:1180 > > > #8 0xffffffff80ab226a in bwrite (bp=0xfffffe01f030e148) at buf.h:395 > > > #9 0xffffffff80aafb1b in ffs_write (ap=0xfffffe023ae5d2b8) > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:800 > > > #10 0xffffffff80bdf0ed in VOP_WRITE_APV (vop=0xffffffff80f15480, > > > a=0xfffffe023ae5d2b8) at vnode_if.c:999 > > > #11 0xffffffff80b1d02e in VOP_WRITE (vp=0xfffff80077e7a000, > > > uio=0xfffffe023ae5d378, ioflag=8323232, cred=0xfffff80004235000) > > > at vnode_if.h:413 > > > #12 0xffffffff80b1ce97 in vnode_pager_generic_putpages > > > (vp=0xfffff80077e7a000, > > > ma=0xfffffe023ae5d660, bytecount=16384, flags=1, > > > rtvals=0xfffffe023ae5d580) > > > at /usr/src/sys/vm/vnode_pager.c:1138 > > > #13 0xffffffff80805a57 in vop_stdputpages (ap=0xfffffe023ae5d478) > > > at /usr/src/sys/kern/vfs_default.c:760 > > > #14 0xffffffff80be201e in VOP_PUTPAGES_APV (vop=0xffffffff80f00218, > > > a=0xfffffe023ae5d478) at vnode_if.c:2861 > > > #15 0xffffffff80b1d7e3 in VOP_PUTPAGES (vp=0xfffff80077e7a000, > > > m=0xfffffe023ae5d660, count=16384, sync=1, > rtvals=0xfffffe023ae5d580, > > > offset=0) at vnode_if.h:1189 > > > #16 0xffffffff80b196f3 in vnode_pager_putpages > (object=0xfffff8014a1fce00, > > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > > > at /usr/src/sys/vm/vnode_pager.c:1016 > > > #17 0xffffffff80b0a605 in vm_pager_put_pages > (object=0xfffff8014a1fce00, > > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) > > > at vm_pager.h:144 > > > #18 0xffffffff80b0a18c in vm_pageout_flush (mc=0xfffffe023ae5d660, > > > count=4, > > > flags=1, mreq=0, prunlen=0xfffffe023ae5d6f8, > eio=0xfffffe023ae5d77c) > > > at /usr/src/sys/vm/vm_pageout.c:533 > > > #19 0xffffffff80afec76 in vm_object_page_collect_flush ( > > > object=0xfffff8014a1fce00, p=0xfffff8023a882370, pagerflags=1, > > > flags=1, > > > clearobjflags=0xfffffe023ae5d780, eio=0xfffffe023ae5d77c) > > > at /usr/src/sys/vm/vm_object.c:971 > > > #20 0xffffffff80afe91e in vm_object_page_clean > (object=0xfffff8014a1fce00, > > > start=0, end=0, flags=1) at /usr/src/sys/vm/vm_object.c:897 > > > #21 0xffffffff80afe1fa in vm_object_terminate > (object=0xfffff8014a1fce00) > > > at /usr/src/sys/vm/vm_object.c:735 > > > #22 0xffffffff80b1a0f1 in vnode_destroy_vobject (vp=0xfffff80077e7a000) > > > at /usr/src/sys/vm/vnode_pager.c:164 > > > #23 0xffffffff80abb191 in ufs_prepare_reclaim (vp=0xfffff80077e7a000) > > > at /usr/src/sys/ufs/ufs/ufs_inode.c:190 > > > #24 0xffffffff80abb1f9 in ufs_reclaim (ap=0xfffffe023ae5d968) > > > at /usr/src/sys/ufs/ufs/ufs_inode.c:219 > > > #25 0xffffffff80be0ade in VOP_RECLAIM_APV (vop=0xffffffff80f15ec0, > > > a=0xfffffe023ae5d968) at vnode_if.c:2019 > > > #26 0xffffffff80827849 in VOP_RECLAIM (vp=0xfffff80077e7a000, > > > td=0xfffff80008931960) at vnode_if.h:830 > > > #27 0xffffffff808219a9 in vgonel (vp=0xfffff80077e7a000) > > > at /usr/src/sys/kern/vfs_subr.c:2943 > > > #28 0xffffffff808294e8 in vlrureclaim (mp=0xfffff80008b2e000) > > > at /usr/src/sys/kern/vfs_subr.c:882 > > > #29 0xffffffff80828ea9 in vnlru_proc () at > > > /usr/src/sys/kern/vfs_subr.c:1000 > > > #30 0xffffffff806b66c5 in fork_exit (callout=0xffffffff80828c50 > > > , > > > arg=0x0, frame=0xfffffe023ae5dc00) at > > > /usr/src/sys/kern/kern_fork.c:1027 > > > #31 0xffffffff80b21dce in fork_trampoline () > > > at /usr/src/sys/amd64/amd64/exception.S:611 > > > #32 0x0000000000000000 in ?? () > > > > > > This is a kernel compiled -O -g, its "almost" GENERIC; the only > difference > > > is some removed drivers, I have reproduced this on a few different > kernels, > > > including a BHYVE one so I can poke at it and not take out the main > > > machine. The reproduction as it currently stands needs to have jails > > > running, but I don't believe this is a jail interaction, I think its > just > > > that the process that sets up the problem happens to be running in a > jail. > > > The step is "start jail; run "find /mountpoint -xdev >/dev/null" on the > > > filesystem, when the vnlru forces the problem vnode out the system > panics. > > > > > > I made a few modifications to the kernel to spit out information about > the > > > buf that causes the issue, but that is it. > > > > > > Information about the buf in question; it has a single softdependency > > > worklist for direct allocation: > > > (kgdb) print *bp->b_dep->lh_first > > > $6 = {wk_list = {le_next = 0x0, le_prev = 0xfffffe01f030e378}, > > > wk_mp = 0xfffff80008b2e000, wk_type = 4, wk_state = 163841} > > > > > > The file that maps to that buffer: > > > ls -lh MOUNTPOINT/jails/mail/var/imap/db/__db.002 > > > -rw------- 1 cyrus cyrus 24K Jul 1 20:32 > > > MOUNTPOINT/jails/mail/var/imap/db/__db.002 > > > > > > Any help is appreciated, until then I will keep banging my head against > > > the proverbial wall on this :) > > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > " > From owner-freebsd-stable@freebsd.org Wed Jul 6 16:02:01 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8688B7591D; Wed, 6 Jul 2016 16:02:01 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x22c.google.com (mail-yw0-x22c.google.com [IPv6:2607:f8b0:4002:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 932C61F1F; Wed, 6 Jul 2016 16:02:01 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x22c.google.com with SMTP id b72so91393961ywa.3; Wed, 06 Jul 2016 09:02:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=UrrFRO5z0hKpEZcOOJfaXV2i/GLH5kmJDlYFWxfHa3I=; b=Aafi8DHqDhuc+7z7gRdCLn2ReKH2MV6Q5RM6C+ZkAkj1O8P69MIv0ZFkd7hm/8irqt SqgBXWU8b8PLkvArerhesHh173Ti+sWa4urHdC4y1M49t+W4bEuMnGplMNKNAXlHKmBW wJVBLG1FjWoP34noF541b/905BK3ncs4ip0vWEWJQ7/Lykf1SNHOiWsEMbEA1I33JE5B Ksxnad2Ri5GN7ENHPWP/rtRilTA5w3wZdIWVVinJsR0Ff0dRmQxkK3CqqOZpda5n+fcB P/fC3dHbbwk22a4zjXxE+msb6Rgfs1V7sN7gVtkkojDvvmxpwe38xFk3MKi0cPUiKroI 2+cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=UrrFRO5z0hKpEZcOOJfaXV2i/GLH5kmJDlYFWxfHa3I=; b=Qm53g9XgoLFPaBGYQcC8U2dV8pnTnkT9ZA3pgLoZzRswVRVgrg7v5fBR+tqdmGIg7w DdyOlTzWxW5HG0ewTRMrNvVXqx6owoh//rYD4tOUgHJZK3l09okrjo1FNNOIe+Bx9yCS W1kpaxFgR0qyQeBsQgCSKFRYg34RsBekvcJJ0QSo4+1gMdSAHJTClInWX0BuvI1AidH0 UwMGaLkztwjnwVPj3/PJ+RC1e2UcsuURmN7zaktzaTqpXAYFdwOzH9KNAIrrCwRrUYBR Uj+rbzzrO9TtdXVSLKETpsseJAlXPh2FobLDvpiur2gw2lNvPALF+cya1CmONLfgGbeb t4Ww== X-Gm-Message-State: ALyK8tLO6d+wAhAfnlW3xCfaX06+1tMks4/1IsIFgXqZQlDEggugt5biuqRApBxGzyB0NkflEqA8fdP21FKnhQ== X-Received: by 10.37.211.136 with SMTP id e130mr15105100ybf.62.1467820920698; Wed, 06 Jul 2016 09:02:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Wed, 6 Jul 2016 09:02:00 -0700 (PDT) In-Reply-To: References: <20160706151822.GC38613@kib.kiev.ua> From: David Cross Date: Wed, 6 Jul 2016 12:02:00 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: Konstantin Belousov Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 16:02:02 -0000 Oh, whoops; how do I printout the buffer? On Wed, Jul 6, 2016 at 11:30 AM, David Cross wrote: > No kernel messages before (if there were I would have written this off a > long time ago); > And as of right now, this is probably the most fsck-ed filesystem on the > planet!.. I have an 'image' that I am going on that is ggate mounted, so I > can access it in a bhyve VM to ease debuging so I am not crashing my real > machine (with the real filesystem) all the time. > > One of my initial guesses was that this was a CG allocation error, but a > dumpfs seems to show plenty of blocks in the CG to meet this need. > > Quick note on the testcase, I haven't totally isolated it yet, but the > minimal reproduction is a 'ctl_cyrusdb -r", which runs a bdb5 recover op, a > ktrace on that shows that it unlinks 3 files, opens them, lseeks then, > writes a block, and then mmaps them (but leaves them open). At process > termination is munmaps, and then closes. I have tried to write a shorter > reproduction that opens, seeks, mmaps (with the same flags), writes the > mmaped memory, munmaps, closes and exits, but this has been insufficient to > reproduce the issue; There is likely some specific pattern in the bdb5 code > tickling this, and behind the mmap-ed interface it is all opaque, and the > bdb5 code is pretty complex itself > > On Wed, Jul 6, 2016 at 11:18 AM, Konstantin Belousov > wrote: > >> On Wed, Jul 06, 2016 at 10:51:28AM -0400, David Cross wrote: >> > Ok.. to reply to my own message, I using ktr and debugging printfs I >> have >> > found the culprit.. but I am still at a loss to 'why', or what the >> > appropriate fix is. >> > >> > Lets go back to the panic (simplified) >> > >> > #0 0xffffffff8043f160 at kdb_backtrace+0x60 >> > #1 0xffffffff80401454 at vpanic+0x124 >> > #2 0xffffffff804014e3 at panic+0x43 >> > #3 0xffffffff8060719a at softdep_deallocate_dependencies+0x6a >> > #4 0xffffffff80499cc1 at brelse+0x151 >> > #5 0xffffffff804979b1 at bufwrite+0x81 >> > #6 0xffffffff80623c80 at ffs_write+0x4b0 >> > #7 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 >> > #8 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 >> > #9 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 >> > #10 0xffffffff80661cc1 at vnode_pager_putpages+0x91 >> > #11 0xffffffff806588e6 at vm_pageout_flush+0x116 >> > #12 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 >> > #13 0xffffffff80651519 at vm_object_page_clean+0x179 >> > #14 0xffffffff80651102 at vm_object_terminate+0xa2 >> > #15 0xffffffff806621a5 at vnode_destroy_vobject+0x85 >> > #16 0xffffffff8062a52f at ufs_reclaim+0x1f >> > #17 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 >> > >> > Via KTR logging I determined that the dangling dependedency was on a >> > freshly allocated buf, *after* vinvalbuf in the vgonel() (so in >> VOP_RECLAIM >> > itself), called by the vnode lru cleanup process; I further noticed >> that it >> > was in a newbuf that recycled a bp (unimportant, except it let me narrow >> > down my logging to something managable), from there I get this >> stacktrace >> > (simplified) >> > >> > #0 0xffffffff8043f160 at kdb_backtrace+0x60 >> > #1 0xffffffff8049c98e at getnewbuf+0x4be >> > #2 0xffffffff804996a0 at getblk+0x830 >> > #3 0xffffffff805fb207 at ffs_balloc_ufs2+0x1327 >> > #4 0xffffffff80623b0b at ffs_write+0x33b >> > #5 0xffffffff806ce9a4 at VOP_WRITE_APV+0x1c4 >> > #6 0xffffffff806639e3 at vnode_pager_generic_putpages+0x293 >> > #7 0xffffffff806d2102 at VOP_PUTPAGES_APV+0x142 >> > #8 0xffffffff80661cc1 at vnode_pager_putpages+0x91 >> > #9 0xffffffff806588e6 at vm_pageout_flush+0x116 >> > #10 0xffffffff806517e2 at vm_object_page_collect_flush+0x1c2 >> > #11 0xffffffff80651519 at vm_object_page_clean+0x179 >> > #12 0xffffffff80651102 at vm_object_terminate+0xa2 >> > #13 0xffffffff806621a5 at vnode_destroy_vobject+0x85 >> > #14 0xffffffff8062a52f at ufs_reclaim+0x1f >> > #15 0xffffffff806d0782 at VOP_RECLAIM_APV+0x142 >> > #16 0xffffffff804b6c6e at vgonel+0x2ee >> > #17 0xffffffff804ba6f5 at vnlru_proc+0x4b5 >> > >> > addr2line on the ffs_balloc_ufs2 gives: >> > /usr/src/sys/ufs/ffs/ffs_balloc.c:778: >> > >> > bp = getblk(vp, lbn, nsize, 0, 0, gbflags); >> > bp->b_blkno = fsbtodb(fs, newb); >> > if (flags & BA_CLRBUF) >> > vfs_bio_clrbuf(bp); >> > if (DOINGSOFTDEP(vp)) >> > softdep_setup_allocdirect(ip, lbn, >> newb, 0, >> > nsize, 0, bp); >> > >> > >> > Boom, freshly allocated buffer with a dependecy; nothing in VOP_RECLAIM >> > handles this, this is after vinvalbuf is called, it expects that >> everything >> > is flushed to disk and its just about releasing structures (is my read >> of >> > the code). >> > >> > Now, perhaps this is a good assumption? the question then is how is >> this >> > buffer hanging out there surviving a a vinvalbuf. I will note that my >> > test-case that finds this runs and terminates *minutes* before... its >> not >> > just hanging out there in a race, its surviving background sync, fsync, >> > etc... wtf? Also, I *can* unmount the FS without an error, so that >> > codepath is either ignoring this buffer, or its forcing a sync in a way >> > that doesn't panic? >> Most typical cause for the buffer dependencies not flushed is a buffer >> write error. At least you could provide the printout of the buffer to >> confirm or reject this assumption. >> >> Were there any kernel messages right before the panic ? Just in case, >> did you fsck the volume before using it, after the previous panic ? >> >> > >> > Anyone have next steps? I am making progress here, but its really slow >> > going, this is probably the most complex portion of the kernel and some >> > pointers would be helpful. >> > >> > On Sat, Jul 2, 2016 at 2:31 PM, David Cross >> wrote: >> > >> > > Ok, I have been trying to trace this down for awhile..I know quite a >> bit >> > > about it.. but there's a lot I don't know, or I would have a patch. >> I have >> > > been trying to solve this on my own, but bringing in some outside >> > > assistance will let me move on with my life. >> > > >> > > First up: The stacktrace (from a debugging kernel, with coredump >> > > >> > > #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:298 >> > > #1 0xffffffff8071018a in kern_reboot (howto=260) >> > > at /usr/src/sys/kern/kern_shutdown.c:486 >> > > #2 0xffffffff80710afc in vpanic ( >> > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling >> deps >> > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d", >> > > ap=0xfffffe023ae5cf40) >> > > at /usr/src/sys/kern/kern_shutdown.c:889 >> > > #3 0xffffffff807108c0 in panic ( >> > > fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling >> deps >> > > b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d") >> > > at /usr/src/sys/kern/kern_shutdown.c:818 >> > > #4 0xffffffff80a7c841 in softdep_deallocate_dependencies ( >> > > bp=0xfffffe01f030e148) at /usr/src/sys/ufs/ffs/ffs_softdep.c:14099 >> > > #5 0xffffffff807f793f in buf_deallocate (bp=0xfffffe01f030e148) at >> > > buf.h:428 >> > > #6 0xffffffff807f59c9 in brelse (bp=0xfffffe01f030e148) >> > > at /usr/src/sys/kern/vfs_bio.c:1599 >> > > #7 0xffffffff807f3132 in bufwrite (bp=0xfffffe01f030e148) >> > > at /usr/src/sys/kern/vfs_bio.c:1180 >> > > #8 0xffffffff80ab226a in bwrite (bp=0xfffffe01f030e148) at buf.h:395 >> > > #9 0xffffffff80aafb1b in ffs_write (ap=0xfffffe023ae5d2b8) >> > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:800 >> > > #10 0xffffffff80bdf0ed in VOP_WRITE_APV (vop=0xffffffff80f15480, >> > > a=0xfffffe023ae5d2b8) at vnode_if.c:999 >> > > #11 0xffffffff80b1d02e in VOP_WRITE (vp=0xfffff80077e7a000, >> > > uio=0xfffffe023ae5d378, ioflag=8323232, cred=0xfffff80004235000) >> > > at vnode_if.h:413 >> > > #12 0xffffffff80b1ce97 in vnode_pager_generic_putpages >> > > (vp=0xfffff80077e7a000, >> > > ma=0xfffffe023ae5d660, bytecount=16384, flags=1, >> > > rtvals=0xfffffe023ae5d580) >> > > at /usr/src/sys/vm/vnode_pager.c:1138 >> > > #13 0xffffffff80805a57 in vop_stdputpages (ap=0xfffffe023ae5d478) >> > > at /usr/src/sys/kern/vfs_default.c:760 >> > > #14 0xffffffff80be201e in VOP_PUTPAGES_APV (vop=0xffffffff80f00218, >> > > a=0xfffffe023ae5d478) at vnode_if.c:2861 >> > > #15 0xffffffff80b1d7e3 in VOP_PUTPAGES (vp=0xfffff80077e7a000, >> > > m=0xfffffe023ae5d660, count=16384, sync=1, >> rtvals=0xfffffe023ae5d580, >> > > offset=0) at vnode_if.h:1189 >> > > #16 0xffffffff80b196f3 in vnode_pager_putpages >> (object=0xfffff8014a1fce00, >> > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) >> > > at /usr/src/sys/vm/vnode_pager.c:1016 >> > > #17 0xffffffff80b0a605 in vm_pager_put_pages >> (object=0xfffff8014a1fce00, >> > > m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) >> > > at vm_pager.h:144 >> > > #18 0xffffffff80b0a18c in vm_pageout_flush (mc=0xfffffe023ae5d660, >> > > count=4, >> > > flags=1, mreq=0, prunlen=0xfffffe023ae5d6f8, >> eio=0xfffffe023ae5d77c) >> > > at /usr/src/sys/vm/vm_pageout.c:533 >> > > #19 0xffffffff80afec76 in vm_object_page_collect_flush ( >> > > object=0xfffff8014a1fce00, p=0xfffff8023a882370, pagerflags=1, >> > > flags=1, >> > > clearobjflags=0xfffffe023ae5d780, eio=0xfffffe023ae5d77c) >> > > at /usr/src/sys/vm/vm_object.c:971 >> > > #20 0xffffffff80afe91e in vm_object_page_clean >> (object=0xfffff8014a1fce00, >> > > start=0, end=0, flags=1) at /usr/src/sys/vm/vm_object.c:897 >> > > #21 0xffffffff80afe1fa in vm_object_terminate >> (object=0xfffff8014a1fce00) >> > > at /usr/src/sys/vm/vm_object.c:735 >> > > #22 0xffffffff80b1a0f1 in vnode_destroy_vobject >> (vp=0xfffff80077e7a000) >> > > at /usr/src/sys/vm/vnode_pager.c:164 >> > > #23 0xffffffff80abb191 in ufs_prepare_reclaim (vp=0xfffff80077e7a000) >> > > at /usr/src/sys/ufs/ufs/ufs_inode.c:190 >> > > #24 0xffffffff80abb1f9 in ufs_reclaim (ap=0xfffffe023ae5d968) >> > > at /usr/src/sys/ufs/ufs/ufs_inode.c:219 >> > > #25 0xffffffff80be0ade in VOP_RECLAIM_APV (vop=0xffffffff80f15ec0, >> > > a=0xfffffe023ae5d968) at vnode_if.c:2019 >> > > #26 0xffffffff80827849 in VOP_RECLAIM (vp=0xfffff80077e7a000, >> > > td=0xfffff80008931960) at vnode_if.h:830 >> > > #27 0xffffffff808219a9 in vgonel (vp=0xfffff80077e7a000) >> > > at /usr/src/sys/kern/vfs_subr.c:2943 >> > > #28 0xffffffff808294e8 in vlrureclaim (mp=0xfffff80008b2e000) >> > > at /usr/src/sys/kern/vfs_subr.c:882 >> > > #29 0xffffffff80828ea9 in vnlru_proc () at >> > > /usr/src/sys/kern/vfs_subr.c:1000 >> > > #30 0xffffffff806b66c5 in fork_exit (callout=0xffffffff80828c50 >> > > , >> > > arg=0x0, frame=0xfffffe023ae5dc00) at >> > > /usr/src/sys/kern/kern_fork.c:1027 >> > > #31 0xffffffff80b21dce in fork_trampoline () >> > > at /usr/src/sys/amd64/amd64/exception.S:611 >> > > #32 0x0000000000000000 in ?? () >> > > >> > > This is a kernel compiled -O -g, its "almost" GENERIC; the only >> difference >> > > is some removed drivers, I have reproduced this on a few different >> kernels, >> > > including a BHYVE one so I can poke at it and not take out the main >> > > machine. The reproduction as it currently stands needs to have jails >> > > running, but I don't believe this is a jail interaction, I think its >> just >> > > that the process that sets up the problem happens to be running in a >> jail. >> > > The step is "start jail; run "find /mountpoint -xdev >/dev/null" on >> the >> > > filesystem, when the vnlru forces the problem vnode out the system >> panics. >> > > >> > > I made a few modifications to the kernel to spit out information >> about the >> > > buf that causes the issue, but that is it. >> > > >> > > Information about the buf in question; it has a single softdependency >> > > worklist for direct allocation: >> > > (kgdb) print *bp->b_dep->lh_first >> > > $6 = {wk_list = {le_next = 0x0, le_prev = 0xfffffe01f030e378}, >> > > wk_mp = 0xfffff80008b2e000, wk_type = 4, wk_state = 163841} >> > > >> > > The file that maps to that buffer: >> > > ls -lh MOUNTPOINT/jails/mail/var/imap/db/__db.002 >> > > -rw------- 1 cyrus cyrus 24K Jul 1 20:32 >> > > MOUNTPOINT/jails/mail/var/imap/db/__db.002 >> > > >> > > Any help is appreciated, until then I will keep banging my head >> against >> > > the proverbial wall on this :) >> > > >> > _______________________________________________ >> > freebsd-stable@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> > To unsubscribe, send any mail to " >> freebsd-stable-unsubscribe@freebsd.org" >> > > From owner-freebsd-stable@freebsd.org Wed Jul 6 17:38:04 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 429DBB75D61; Wed, 6 Jul 2016 17:38:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF32B151D; Wed, 6 Jul 2016 17:38:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u66Hbwgs063744 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 6 Jul 2016 20:37:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u66Hbwgs063744 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u66HbwNk063743; Wed, 6 Jul 2016 20:37:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 6 Jul 2016 20:37:58 +0300 From: Konstantin Belousov To: David Cross Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) Message-ID: <20160706173758.GF38613@kib.kiev.ua> References: <20160706151822.GC38613@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 17:38:04 -0000 On Wed, Jul 06, 2016 at 12:02:00PM -0400, David Cross wrote: > Oh, whoops; how do I printout the buffer? In kgdb, p/x *(struct buf *)address From owner-freebsd-stable@freebsd.org Wed Jul 6 18:21:22 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21921B758E1; Wed, 6 Jul 2016 18:21:22 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x22b.google.com (mail-yw0-x22b.google.com [IPv6:2607:f8b0:4002:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0B001E4D; Wed, 6 Jul 2016 18:21:21 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x22b.google.com with SMTP id i12so93695696ywa.1; Wed, 06 Jul 2016 11:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ikdf9Rq5KobUSGbfr4AZiaSUJp18yozYm7mOPTwnuUY=; b=DEqHrWCXbjcIYFI1pGyosuciVRwkrQH/ArI4mdJjdasWdIr0xdtZkpcO2QXD+o0gnS WJAnoNND6plO8Njl2BVvxNjasYKWt986YwtpFF6GCHvzNSwWo1zqeW7EzQcnd6qmMHq9 YXG7IKYGyh1g0GtgfahKUwnUzPX5c6T7dI9+E23Og7/cj3VnXCtrGc8Q3P0UHcrq/Y5T kFAO0StGgoskh2HGWPezYJYTCCcwoYdraqtEoCWw60/ej6cGN9J7ptl+44SP0gP4Qsxk AH/WWryVPoJhrweV9ctEkk7Rr8hsUyOB3yRYwCLAlSarduyTrsm0BuUYF5k+ra111xp1 Yw5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ikdf9Rq5KobUSGbfr4AZiaSUJp18yozYm7mOPTwnuUY=; b=Lk7j6Afh5UxY+3bMfhdFN1pnRP12mP0tDr46r4X182IMQQcmzIrGXERSXOOprYAtnn omdBR/azu5KChkAoKsJKhJFg0zjVYCMxeiW7QQ4Ht2i9DVNAXDAVwpgjcxf2H0NuXwff 8Dplat+eaD4PJn2Fh4aca+X8oGDVmj8BHKaa7U/jXZ0PnE9Db8hFFazTDJ9g6kBq1Ocl lkTkOdE4kPzkqYj5dHZCmi8NECuanjCTVqRO0VVJpmV0189E5/d1/iw70FeOG9ijh40S 444rcMhrfz+p9V0amPdNyMdC5qF55zYb8VPczP++N8L2ZzQF1fuso8dgMeHG+29QlhYS 3Opw== X-Gm-Message-State: ALyK8tIr8B4SmAoSIsmDAFIZfEaoMsXeq1XVYe3MzxjanVoR6pZWrN0u6Q8gz3QCxRffJO4rwwMiHuCp5JuTQQ== X-Received: by 10.129.102.195 with SMTP id a186mr15678812ywc.76.1467829281073; Wed, 06 Jul 2016 11:21:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Wed, 6 Jul 2016 11:21:20 -0700 (PDT) In-Reply-To: <20160706173758.GF38613@kib.kiev.ua> References: <20160706151822.GC38613@kib.kiev.ua> <20160706173758.GF38613@kib.kiev.ua> From: David Cross Date: Wed, 6 Jul 2016 14:21:20 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: Konstantin Belousov Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 18:21:22 -0000 (kgdb) up 5 #5 0xffffffff804aafa1 in brelse (bp=0xfffffe00f77457d0) at buf.h:428 428 (*bioops.io_deallocate)(bp); Current language: auto; currently minimal (kgdb) p/x *(struct buf *)0xfffffe00f77457d0 $1 = {b_bufobj = 0xfffff80002e88480, b_bcount = 0x4000, b_caller1 = 0x0, b_data = 0xfffffe00f857b000, b_error = 0x0, b_iocmd = 0x0, b_ioflags = 0x0, b_iooffset = 0x0, b_resid = 0x0, b_iodone = 0x0, b_blkno = 0x115d6400, b_offset = 0x0, b_bobufs = {tqe_next = 0x0, tqe_prev = 0xfffff80002e884d0}, b_vflags = 0x0, b_freelist = {tqe_next = 0xfffffe00f7745a28, tqe_prev = 0xffffffff80c2afc0}, b_qindex = 0x0, b_flags = 0x20402800, b_xflags = 0x2, b_lock = {lock_object = {lo_name = 0xffffffff8075030b, lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xfffffe0000602f00}, lk_lock = 0xfffff800022e8000, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60}, b_bufsize = 0x4000, b_runningbufspace = 0x0, b_kvabase = 0xfffffe00f857b000, b_kvaalloc = 0x0, b_kvasize = 0x4000, b_lblkno = 0x0, b_vp = 0xfffff80002e883b0, b_dirtyoff = 0x0, b_dirtyend = 0x0, b_rcred = 0x0, b_wcred = 0x0, b_saveaddr = 0x0, b_pager = { pg_reqpage = 0x0}, b_cluster = {cluster_head = {tqh_first = 0x0, tqh_last = 0x0}, cluster_entry = {tqe_next = 0x0, tqe_prev = 0x0}}, b_pages = {0xfffff800b99b30b0, 0xfffff800b99b3118, 0xfffff800b99b3180, 0xfffff800b99b31e8, 0x0 }, b_npages = 0x4, b_dep = { lh_first = 0xfffff800023d8c00}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0, b_pin_count = 0x0} This is the freshly allocated buf that causes the panic; is this what is needed? I "know" which vnode will cause the panic on vnlru cleanup, but I don't know how to walk the memory list without a 'hook'.. as in, i can setup the kernel in a state that I know will panic when the vnode is cleaned up, I can force a panic 'early' (kill -9 1), and then I could get that vnode.. if I could get the vnode list to walk. On Wed, Jul 6, 2016 at 1:37 PM, Konstantin Belousov wrote: > On Wed, Jul 06, 2016 at 12:02:00PM -0400, David Cross wrote: > > Oh, whoops; how do I printout the buffer? > > In kgdb, p/x *(struct buf *)address > From owner-freebsd-stable@freebsd.org Wed Jul 6 21:08:53 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD6FBB75ED0 for ; Wed, 6 Jul 2016 21:08:53 +0000 (UTC) (envelope-from s-4sedi7a3kz5vmkuap2tph4t3139mv14fekry8w7236urnc0d78raadlu@bounce.linkedin.com) Received: from maild-fe.linkedin.com (maild-fe.linkedin.com [IPv6:2620:109:c00d:104::87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.linkedin.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 889101377 for ; Wed, 6 Jul 2016 21:08:53 +0000 (UTC) (envelope-from s-4sedi7a3kz5vmkuap2tph4t3139mv14fekry8w7236urnc0d78raadlu@bounce.linkedin.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linkedin.com; s=proddkim1024; t=1467839324; bh=5tp1Y3mquNipnL9YCSIuyDrpuJzEbZZywl1Ox5bZBfg=; h=From:Subject:MIME-Version:Content-Type:To:Date:X-LinkedIn-Class: X-LinkedIn-Template:X-LinkedIn-fbl; b=tocZzDEfk75vWNz3lSNRynrf2GsiqRACU1RQ8Icxv8A04+7Yg/qBehK4WKVRvAF6M HSvqk8VjXvgwIY6c9z3j+fE2U7E5LSWZtWKOhl45PGHi8WWuC2hy7YUgZidd63oyH5 XI55nxUjjsDYyr6q7Vxf96wlmf8naA5aoZiWDGvg= From: Giorgio Lago via LinkedIn Message-ID: <1009064032.2811332.1467839324186.JavaMail.app@lva1-app4794.prod.linkedin.com> Subject: =?UTF-8?Q?O_convite_de_Giorgio_Lago_est=C3=A1_aguardando_sua_resposta?= MIME-Version: 1.0 To: Date: Wed, 6 Jul 2016 21:08:44 +0000 (UTC) X-LinkedIn-Class: INVITE-REMIND-GUEST X-LinkedIn-Template: second_guest_reminder_01 X-LinkedIn-fbl: m2-aszs32ecrgpvyz930zqm5sim5ft81njczf0np342hq9q6tzipmhzzrjzjz8j9nqn21u2kj99kwdnkwmifvxekhah4eg9q1zd7f100m X-LinkedIn-Id: -hwzlvm-iqb4171m-ge Content-Type: text/plain;charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 21:08:53 -0000 Giorgio Lago would like to connect on LinkedIn. How would you like to respo= nd? Accept: https://www.linkedin.com/e/v2?e=3D-hwzlvm-iqb4171m-ge&t=3Dssuw&ek= =3Dsecond_guest_reminder_01&li=3D13&m=3Dhero&ts=3Daccept_text&sharedKey=3Dd= lXI--ld&invitationId=3D6151415342305464320 View Giorgio Lago's profile: https://www.linkedin.com/e/v2?e=3D-hwzlvm-= iqb4171m-ge&t=3Dssuw&ek=3Dsecond_guest_reminder_01&li=3D3&m=3Dhero&ts=3Dpro= file_text&sharedKey=3DdlXI--ld&invitationId=3D6151415342305464320 Eu gostaria de adicion=C3=A1-lo =C3=A0 minha rede profissional no LinkedIn. -Giorgio Voc=C3=AA recebeu um convite para se conectar. O LinkedIn utiliza seu ender= e=C3=A7o de e-mail para fazer sugest=C3=B5es a nossos usu=C3=A1rios em recu= rsos como Pessoas que talvez voc=C3=AA conhe=C3=A7a. Cancelar inscri=C3=A7= =C3=A3o: https://www.linkedin.com/e/v2?e=3D-hwzlvm-iqb4171m-ge&t=3Dlun&midT= oken=3DAQFmrFVhGEd36Q&ek=3Dsecond_guest_reminder_01&li=3D15&m=3Dunsub&ts=3D= HTML&eid=3D-hwzlvm-iqb4171m-ge&loid=3DAQEBxxkC5Zj2AgAAAVXCCp_8QtS1iDRa0yUyQ= RztSvf98jhLoL-W3mo31rM3KJ1C7LIvMyA9BzfdJHx5upnkYC8A Este e-mail foi enviado para freebsd-stable@freebsd.org. If you need assistance or have questions, please contact LinkedIn Customer = Service: https://www.linkedin.com/e/v2?e=3D-hwzlvm-iqb4171m-ge&a=3Dcustomer= ServiceUrl&ek=3Dsecond_guest_reminder_01 © 2016 LinkedIn Corporation, 2029 Stierlin Court, Mountain View, CA 94= 043. LinkedIn e a logomarca do LinkedIn s=C3=A3o marcas registradas da Link= edIn. From owner-freebsd-stable@freebsd.org Thu Jul 7 00:12:29 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4B63B758D1; Thu, 7 Jul 2016 00:12:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 857D5126D; Thu, 7 Jul 2016 00:12:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u670CLT1011655 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 7 Jul 2016 03:12:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u670CLT1011655 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u670CIZE011654; Thu, 7 Jul 2016 03:12:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 7 Jul 2016 03:12:18 +0300 From: Konstantin Belousov To: David Cross Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) Message-ID: <20160707001218.GI38613@kib.kiev.ua> References: <20160706151822.GC38613@kib.kiev.ua> <20160706173758.GF38613@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 00:12:30 -0000 On Wed, Jul 06, 2016 at 02:21:20PM -0400, David Cross wrote: > (kgdb) up 5 > #5 0xffffffff804aafa1 in brelse (bp=0xfffffe00f77457d0) at buf.h:428 > 428 (*bioops.io_deallocate)(bp); > Current language: auto; currently minimal > (kgdb) p/x *(struct buf *)0xfffffe00f77457d0 > $1 = {b_bufobj = 0xfffff80002e88480, b_bcount = 0x4000, b_caller1 = 0x0, > b_data = 0xfffffe00f857b000, b_error = 0x0, b_iocmd = 0x0, b_ioflags = > 0x0, > b_iooffset = 0x0, b_resid = 0x0, b_iodone = 0x0, b_blkno = 0x115d6400, > b_offset = 0x0, b_bobufs = {tqe_next = 0x0, tqe_prev = > 0xfffff80002e884d0}, > b_vflags = 0x0, b_freelist = {tqe_next = 0xfffffe00f7745a28, > tqe_prev = 0xffffffff80c2afc0}, b_qindex = 0x0, b_flags = 0x20402800, > b_xflags = 0x2, b_lock = {lock_object = {lo_name = 0xffffffff8075030b, > lo_flags = 0x6730000, lo_data = 0x0, lo_witness = > 0xfffffe0000602f00}, > lk_lock = 0xfffff800022e8000, lk_exslpfail = 0x0, lk_timo = 0x0, > lk_pri = 0x60}, b_bufsize = 0x4000, b_runningbufspace = 0x0, > b_kvabase = 0xfffffe00f857b000, b_kvaalloc = 0x0, b_kvasize = 0x4000, > b_lblkno = 0x0, b_vp = 0xfffff80002e883b0, b_dirtyoff = 0x0, > b_dirtyend = 0x0, b_rcred = 0x0, b_wcred = 0x0, b_saveaddr = 0x0, b_pager > = { > pg_reqpage = 0x0}, b_cluster = {cluster_head = {tqh_first = 0x0, > tqh_last = 0x0}, cluster_entry = {tqe_next = 0x0, tqe_prev = 0x0}}, > b_pages = {0xfffff800b99b30b0, 0xfffff800b99b3118, 0xfffff800b99b3180, > 0xfffff800b99b31e8, 0x0 }, b_npages = 0x4, b_dep = { > lh_first = 0xfffff800023d8c00}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, > b_fsprivate3 = 0x0, b_pin_count = 0x0} > > > This is the freshly allocated buf that causes the panic; is this what is > needed? I "know" which vnode will cause the panic on vnlru cleanup, but I > don't know how to walk the memory list without a 'hook'.. as in, i can > setup the kernel in a state that I know will panic when the vnode is > cleaned up, I can force a panic 'early' (kill -9 1), and then I could get > that vnode.. if I could get the vnode list to walk. Was the state printed after the panic occured ? What is strange is that buffer was not even tried for i/o, AFAIS. Apart from empty b_error/b_iocmd, the b_lblkno is zero, which means that the buffer was never allocated on the disk. The b_blkno looks strangely high. Can you print *(bp->b_vp) ? If it is UFS vnode, do p *(struct inode)(->v_data). I am esp. interested in the vnode size. Can you reproduce the problem on HEAD ? From owner-freebsd-stable@freebsd.org Thu Jul 7 09:29:21 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83463B746E9 for ; Thu, 7 Jul 2016 09:29:21 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 760AA196F for ; Thu, 7 Jul 2016 09:29:20 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from mr185083 (mr185083.univ-rennes1.fr [129.20.185.83]) by mx.zohomail.com with SMTPS id 1467883758620651.7898755646409; Thu, 7 Jul 2016 02:29:18 -0700 (PDT) Date: Thu, 7 Jul 2016 11:29:14 +0200 From: Patrick Lamaiziere To: Konstantin Belousov Cc: freebsd-stable@freebsd.org Subject: Re: 10.3-RELEASE amd64 segmentation faults in wc, sh... Message-ID: <20160707112914.40fd2069@mr185083> In-Reply-To: <20160630125204.GO38613@kib.kiev.ua> References: <20160630135732.76f27305@mr185083> <20160630125204.GO38613@kib.kiev.ua> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd10.3) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 09:29:21 -0000 Le Thu, 30 Jun 2016 15:52:04 +0300, Konstantin Belousov a écrit : Hello, > On Thu, Jun 30, 2016 at 01:57:32PM +0200, Patrick Lamaiziere wrote: > > Hello, > > > > I'm building a pair of firewall with 10.3 and I see some rare > > segmentation faults (5 in a week) in processes like wc, sh or > > ifstated. ... > This is most likely the problems, reported and fixed in r300758, > PR 204764 r302063, and PR 204426 r302236. First and second commits > are already in stable/10, the third one will be merged in several > days. For the record, it looks like the first and second commits fixed this problem here. I've not yet tested the third one (don't know if it is already in 10/stable). Thanks again Konstantin. Regards, From owner-freebsd-stable@freebsd.org Thu Jul 7 14:26:13 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACE4CB74F6D; Thu, 7 Jul 2016 14:26:13 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x22b.google.com (mail-yw0-x22b.google.com [IPv6:2607:f8b0:4002:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 716B61809; Thu, 7 Jul 2016 14:26:13 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x22b.google.com with SMTP id l125so15266370ywb.2; Thu, 07 Jul 2016 07:26:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=WNSjvTdBSDtMESYgVMuHn1mA6cAnDXoYPjRyL8goSd4=; b=GL+VREkdaiKLGsvPl8zpVaZaXmUaKFXxYpCn8E4fdIkr3JlL5zLb8TAR/pUxqTDZXH Notas4vUkj/dB295/9pjV/WERr/L0uvMbpybND3kYgsRA5sxokWwnp6Y7gy2qeHT32Ss DVajPHTjTO5wZ7mpZgBcbDlmO8M9mMK4KRFZXdJHghz8K+Pfsw37bz349qbif5Nuu0Do iaXeLmzrwjXQCHjeJiuDmTONWyphzxSp8bmcrAR5W4JLcD5sBbA7PJUBOgj9ce1ANSk1 cc4rFfPgfmHEUjoiF8HdgPeoC6n25CzTeFEWGq/TdEeQnmOilnIIC98bsSbbD7CpgLaR +nDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=WNSjvTdBSDtMESYgVMuHn1mA6cAnDXoYPjRyL8goSd4=; b=T7pxCx+NDTnizlaGUjVTOLysqIz+gCZwH2lxj285vu5dWy5E8j2jyvoiO9MRuEOzz5 7CZSV6i8xRUvvg4wq8Kc8Rw2Y8S7ehQ/WTnSI7skkvB5gWgjGySG2eLVna7RxjLEpz/V 5EpVl9JxPySytasUwZjSU3ikrjBCSq11Ixmx6VuaJaTBXC+dClXqzkQi5ygWYgjPZmXu pSmNmE2VDOO8LJjD5CKV6pl/OQR7zNaafIXUOF2vI64MSgUt/HIFLW2lVQob4s/Hse5H xx43hEp8yK1LdrNP8N+rE3VSP9ucqHX4sndMB7H5wJPAnUx0QRYXLl0DZe+KVYxYkIIc 7xYQ== X-Gm-Message-State: ALyK8tK0GxE+KvzNG6H4aHsewnYkA34HvMo2wofG60dSuY/bBsFIdfSTDrCE4Sn03hpKYBhMxJ9Gvg1H6OdozA== X-Received: by 10.37.205.130 with SMTP id d124mr408294ybf.181.1467901572542; Thu, 07 Jul 2016 07:26:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Thu, 7 Jul 2016 07:26:10 -0700 (PDT) In-Reply-To: <20160707001218.GI38613@kib.kiev.ua> References: <20160706151822.GC38613@kib.kiev.ua> <20160706173758.GF38613@kib.kiev.ua> <20160707001218.GI38613@kib.kiev.ua> From: David Cross Date: Thu, 7 Jul 2016 10:26:10 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: Konstantin Belousov Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 14:26:13 -0000 The state was printed after the panic, yes. If I understand the idea of softupdates correctly, I don't think its odd this buffer wasn't even attempted to be written, it has b_dep defined, that means those blocks should be written first, right? Also, I was just able to reproduce this on 11.0-ALPHA6, I did a fresh fsck on the filesystem to ensure it was clean (I typically don't fsck between reprouction runs, since it takes so long, and when I do need a 'clean' slate I just restore the snapshot, its faster than fsck). The panic from 11.0-ALPHA6 is: root@bhyve103:~ # panic: softdep_deallocate_dependencies: dangling deps cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe011b3861b0 vpanic() at vpanic+0x182/frame 0xfffffe011b386230 panic() at panic+0x43/frame 0xfffffe011b386290 softdep_deallocate_dependencies() at softdep_deallocate_dependencies+0x71/frame 0xfffffe011b3862b0 brelse() at brelse+0x162/frame 0xfffffe011b386310 bufwrite() at bufwrite+0x206/frame 0xfffffe011b386360 ffs_write() at ffs_write+0x3ed/frame 0xfffffe011b386410 VOP_WRITE_APV() at VOP_WRITE_APV+0x16f/frame 0xfffffe011b386520 vnode_pager_generic_putpages() at vnode_pager_generic_putpages+0x2d5/frame 0xffffe011b3865f0 VOP_PUTPAGES_APV() at VOP_PUTPAGES_APV+0xda/frame 0xfffffe011b386620 vnode_pager_putpages() at vnode_pager_putpages+0x89/frame 0xfffffe011b386690 vm_pageout_flush() at vm_pageout_flush+0x12d/frame 0xfffffe011b386720 vm_object_page_collect_flush() at vm_object_page_collect_flush+0x23a/frame 0xffffe011b386820 vm_object_page_clean() at vm_object_page_clean+0x1be/frame 0xfffffe011b3868a0 vm_object_terminate() at vm_object_terminate+0xa5/frame 0xfffffe011b3868e0 vnode_destroy_vobject() at vnode_destroy_vobject+0x63/frame 0xfffffe011b386910 ufs_reclaim() at ufs_reclaim+0x1f/frame 0xfffffe011b386940 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xda/frame 0xfffffe011b386970 vgonel() at vgonel+0x204/frame 0xfffffe011b3869e0 vnlru_proc() at vnlru_proc+0x577/frame 0xfffffe011b386a70 fork_exit() at fork_exit+0x84/frame 0xfffffe011b386ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011b386ab0 Pardon the machine name, I have a setup script for bhyve VMs, and I didn't tweak the name, just the install location: root@bhyve103:~ # uname -a FreeBSD bhyve103.priv.dcrosstech.com 11.0-ALPHA6 FreeBSD 11.0-ALPHA6 #0 r302303: Fri Jul 1 03:32:49 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 On the 10.3 kernel I was also able to walk the mnt_nvnodes list before the FS panic and I have the vnode * saved from before the vnlru attempted reclaim. print *((struct vnode *)0xfffff80002dc2760) $6 = {v_tag = 0xffffffff8072b891 "ufs", v_op = 0xffffffff80a13c40, v_data = 0xfffff8006a20b160, v_mount = 0xfffff800024e9cc0, v_nmntvnodes = { tqe_next = 0xfffff80002dc2588, tqe_prev = 0xfffff80002dc2958}, v_un = { vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xfffffe0000932ef8}, v_cache_src = { lh_first = 0x0}, v_cache_dst = {tqh_first = 0xfffff8006a18ce00, tqh_last = 0xfffff8006a18ce20}, v_cache_dd = 0x0, v_lock = {lock_object = { lo_name = 0xffffffff8072b891 "ufs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000607280}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = { lo_name = 0xffffffff8074a89f "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe00005fd680}, mtx_lock = 4}, v_vnlock = 0xfffff80002dc27c8, v_actfreelist = { tqe_next = 0xfffff80002dc2938, tqe_prev = 0xfffff80002dc2648}, v_bufobj = { bo_lock = {lock_object = {lo_name = 0xffffffff80754d34 "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000605700}, rw_lock = 1}, bo_ops = 0xffffffff809e97c0, bo_object = 0xfffff80002c1b400, bo_synclist = { le_next = 0xfffff80002dc2a08, le_prev = 0xfffff80002dc2688}, bo_private = 0xfffff80002dc2760, __bo_vnode = 0xfffff80002dc2760, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80002dc2880}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = { tqh_first = 0xfffffe00f7ae8658, tqh_last = 0xfffffe00f7ae86a8}, bv_root = {pt_root = 18446741878841706297}, bv_cnt = 1}, bo_numoutput = 0, bo_flag = 1, bo_bsize = 16384}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80002dc28e8}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 2, v_usecount = 0, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 18236560, v_type = VREG} I think what is wanted is the buffer and their dependency lists.. I am not sure where those are under all of this.. bo_*? On Wed, Jul 6, 2016 at 8:12 PM, Konstantin Belousov wrote: > On Wed, Jul 06, 2016 at 02:21:20PM -0400, David Cross wrote: > > (kgdb) up 5 > > #5 0xffffffff804aafa1 in brelse (bp=0xfffffe00f77457d0) at buf.h:428 > > 428 (*bioops.io_deallocate)(bp); > > Current language: auto; currently minimal > > (kgdb) p/x *(struct buf *)0xfffffe00f77457d0 > > $1 = {b_bufobj = 0xfffff80002e88480, b_bcount = 0x4000, b_caller1 = 0x0, > > b_data = 0xfffffe00f857b000, b_error = 0x0, b_iocmd = 0x0, b_ioflags = > > 0x0, > > b_iooffset = 0x0, b_resid = 0x0, b_iodone = 0x0, b_blkno = 0x115d6400, > > b_offset = 0x0, b_bobufs = {tqe_next = 0x0, tqe_prev = > > 0xfffff80002e884d0}, > > b_vflags = 0x0, b_freelist = {tqe_next = 0xfffffe00f7745a28, > > tqe_prev = 0xffffffff80c2afc0}, b_qindex = 0x0, b_flags = 0x20402800, > > b_xflags = 0x2, b_lock = {lock_object = {lo_name = 0xffffffff8075030b, > > lo_flags = 0x6730000, lo_data = 0x0, lo_witness = > > 0xfffffe0000602f00}, > > lk_lock = 0xfffff800022e8000, lk_exslpfail = 0x0, lk_timo = 0x0, > > lk_pri = 0x60}, b_bufsize = 0x4000, b_runningbufspace = 0x0, > > b_kvabase = 0xfffffe00f857b000, b_kvaalloc = 0x0, b_kvasize = 0x4000, > > b_lblkno = 0x0, b_vp = 0xfffff80002e883b0, b_dirtyoff = 0x0, > > b_dirtyend = 0x0, b_rcred = 0x0, b_wcred = 0x0, b_saveaddr = 0x0, > b_pager > > = { > > pg_reqpage = 0x0}, b_cluster = {cluster_head = {tqh_first = 0x0, > > tqh_last = 0x0}, cluster_entry = {tqe_next = 0x0, tqe_prev = 0x0}}, > > b_pages = {0xfffff800b99b30b0, 0xfffff800b99b3118, 0xfffff800b99b3180, > > 0xfffff800b99b31e8, 0x0 }, b_npages = 0x4, b_dep = > { > > lh_first = 0xfffff800023d8c00}, b_fsprivate1 = 0x0, b_fsprivate2 = > 0x0, > > b_fsprivate3 = 0x0, b_pin_count = 0x0} > > > > > > This is the freshly allocated buf that causes the panic; is this what is > > needed? I "know" which vnode will cause the panic on vnlru cleanup, but > I > > don't know how to walk the memory list without a 'hook'.. as in, i can > > setup the kernel in a state that I know will panic when the vnode is > > cleaned up, I can force a panic 'early' (kill -9 1), and then I could get > > that vnode.. if I could get the vnode list to walk. > > Was the state printed after the panic occured ? What is strange is that > buffer was not even tried for i/o, AFAIS. Apart from empty > b_error/b_iocmd, > the b_lblkno is zero, which means that the buffer was never allocated on > the disk. > > The b_blkno looks strangely high. Can you print *(bp->b_vp) ? If it is > UFS vnode, do p *(struct inode)(->v_data). I am esp. interested > in the vnode size. > > Can you reproduce the problem on HEAD ? > From owner-freebsd-stable@freebsd.org Fri Jul 8 09:24:38 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EB6CB76328 for ; Fri, 8 Jul 2016 09:24:38 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-211-154.reflexion.net [208.70.211.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33BE51CC2 for ; Fri, 8 Jul 2016 09:24:37 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 19170 invoked from network); 8 Jul 2016 09:25:14 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 8 Jul 2016 09:25:14 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v7.90.3) with SMTP; Fri, 08 Jul 2016 05:24:41 -0400 (EDT) Received: (qmail 15158 invoked from network); 8 Jul 2016 09:24:40 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 8 Jul 2016 09:24:40 -0000 X-No-Relay: not in my network Received: from [192.168.0.105] (ip70-189-131-151.lv.lv.cox.net [70.189.131.151]) by iron2.pdx.net (Postfix) with ESMTPSA id 1F0A71C405F for ; Fri, 8 Jul 2016 02:24:21 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: 11.0: amd64 -> armv6 -r302331 -> -r302412 re-cross-build (update): got "sh: ./make_keys: Exec format error" again for init_ketry.h in ncursesw From: Mark Millard In-Reply-To: <0394F424-484D-43B3-9C7D-8C6A8E8709F0@dsl-only.net> Date: Fri, 8 Jul 2016 02:24:34 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <99A8A423-1EC4-4B3A-922D-9CF37CD2DD86@dsl-only.net> References: <0394F424-484D-43B3-9C7D-8C6A8E8709F0@dsl-only.net> To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 09:24:38 -0000 [Sending to freebsd-stable as well this time, adding the 11.0 reference = to the subject too.] [Before the below cross build/update attempt I updated my amd64 from = -r302331 -> -r302412.] Summary: It appears that WITHOUT_META_MODE=3D still needs to be forced = for cross compiles at least sometimes in order to avoid "Exec format = error". man src.conf only mentions WITHOUT_META_MODE=3D in one place: > WITH_DIRDEPS_BUILD . . . > WITH_META_MODE (unless WITHOUT_META_MODE is set = explicitly) . . . > This must be set in the environment, make command line, or > /etc/src-env.conf, not /etc/src.conf. In attempting to update my cross build (amd64 -> armv6) from -r302331 to = -r302412 it failed with: > --- init_keytry.h --- > sh: ./make_keys: Exec format error > *** [init_keytry.h] Error code 126 >=20 > make[4]: stopped in /usr/src/lib/ncurses/ncursesw > .ERROR_TARGET=3D'init_keytry.h' > = .ERROR_META_FILE=3D'/usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/= init_keytry.h.meta' > .MAKE.LEVEL=3D'4' > MAKEFILE=3D'' > .MAKE.MODE=3D'meta missing-filemon=3Dyes missing-meta=3Dyes silent=3Dyes= verbose' > .CURDIR=3D'/usr/src/lib/ncurses/ncursesw' > .MAKE=3D'make' > .OBJDIR=3D'/usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw' > .TARGETS=3D'all' > DESTDIR=3D'/usr/obj/clang/arm.armv6/usr/src/tmp' > LD_LIBRARY_PATH=3D'' > MACHINE=3D'arm' > MACHINE_ARCH=3D'armv6' > MAKEOBJDIRPREFIX=3D'/usr/obj/clang/arm.armv6' > MAKESYSPATH=3D'/usr/src/share/mk' > MAKE_VERSION=3D'20160606' > = PATH=3D'/usr/obj/clang/arm.armv6/usr/src/tmp/legacy/usr/sbin:/usr/obj/clan= g/arm.armv6/usr/src/tmp/legacy/usr/bin:/usr/obj/clang/arm.armv6/usr/src/tm= p/legacy/bin:/usr/obj/clang/arm.armv6/usr/src/tmp/usr/sbin:/usr/obj/clang/= arm.armv6/usr/src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin' > SRCTOP=3D'/usr/src' > OBJTOP=3D'/usr/obj/clang/arm.armv6/usr/src' > .MAKE.MAKEFILES=3D'/usr/src/share/mk/sys.mk = /usr/src/share/mk/local.sys.env.mk /usr/src/share/mk/src.sys.env.mk = /root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host = /usr/src/share/mk/bsd.mkopt.mk /root/src.configs/make.conf = /usr/src/share/mk/local.sys.mk /usr/src/share/mk/src.sys.mk = /etc/src.conf /usr/src/lib/ncurses/ncursesw/Makefile = /usr/src/lib/ncurses/ncursesw/../ncurses/Makefile = /usr/src/share/mk/src.opts.mk /usr/src/share/mk/bsd.own.mk = /usr/src/share/mk/bsd.opts.mk /usr/src/share/mk/bsd.cpu.mk = /usr/src/share/mk/bsd.compiler.mk /usr/src/share/mk/bsd.compiler.mk = /usr/src/lib/ncurses/ncursesw/../config.mk /usr/src/share/mk/bsd.lib.mk = /usr/src/share/mk/bsd.init.mk /usr/src/share/mk/local.init.mk = /usr/src/share/mk/src.init.mk = /usr/src/lib/ncurses/ncursesw/../Makefile.inc = /usr/src/lib/ncurses/ncursesw/../../Makefile.inc = /usr/src/share/mk/bsd.libnames.mk /usr/src/share/mk/src.libnames.mk = /usr/src/share/mk/bsd.symver.mk /usr/src/share/mk/bsd.nls.mk = /usr/src/share/mk/bsd.files.mk /usr/src/share/mk/bsd.incs.mk = /usr/src/share/mk/bsd.confs.mk /usr/src/share/mk/bsd.links.mk = /usr/src/share/mk/bsd.dep.mk /usr/src/share/mk/bsd.clang-analyze.mk = /usr/src/share/mk/bsd.obj.mk /usr/src/share/mk/bsd.subdir.mk = /usr/src/share/mk/bsd.sys.mk' > .PATH=3D'. /usr/src/lib/ncurses/ncursesw = /usr/src/lib/ncurses/ncursesw/../ncurses = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/include = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/base = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/tinfo = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/tty = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/widechar = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/ncurses/trace = /usr/src/lib/ncurses/ncursesw/../../../contrib/ncurses/man' > 1 error again. This was based on: > # more = ~/sys_build_scripts.amd64-host/make_rpi2_nodebug_clang_bootstrap-amd64-hos= t.sh=20 > kldload -n filemon && \ > script = ~/sys_typescripts/typescript_make_rpi2_nodebug_clang_bootstrap-amd64-host-= $(date +%Y-%m-%d:%H:%M:%S) \ > env __MAKE_CONF=3D"/root/src.configs/make.conf" = SRC_ENV_CONF=3D"/root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host= " \ > WITH_META_MODE=3Dyes \ > MAKEOBJDIRPREFIX=3D"/usr/obj/clang" \ > make $* and. . . > # more ~/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host=20 > TO_TYPE=3Darmv6 > # > KERNCONF=3DRPI2-NODBG > TARGET=3Darm > .if ${.MAKE.LEVEL} =3D=3D 0 > TARGET_ARCH=3D${TO_TYPE} > .export TARGET_ARCH > .endif > # > WITH_CROSS_COMPILER=3D > WITHOUT_SYSTEM_COMPILER=3D > # > #CPUTYPE=3Dsoft > WITH_LIBCPLUSPLUS=3D > WITH_BINUTILS_BOOTSTRAP=3D > WITH_CLANG_BOOTSTRAP=3D > WITH_CLANG=3D > WITH_CLANG_IS_CC=3D > WITH_CLANG_FULL=3D > WITH_CLANG_EXTRAS=3D > WITH_LLDB=3D > # > WITH_BOOT=3D > WITHOUT_LIB32=3D > WITHOUT_LIBSOFT=3D > # > WITHOUT_ELFTOOLCHAIN_BOOTSTRAP=3D > WITHOUT_GCC_BOOTSTRAP=3D > WITHOUT_GCC=3D > WITHOUT_GCC_IS_CC=3D > WITHOUT_GNUCXX=3D > # > NO_WERROR=3D > #WERROR=3D > MALLOC_PRODUCTION=3D > # > WITH_DEBUG_FILES=3D > # > XCFLAGS+=3D -march=3Darmv7-a -mcpu=3Dcortex-a7 > XCXXFLAGS+=3D -march=3Darmv7-a -mcpu=3Dcortex-a7 make.conf was empty. The earlier -r302331 cross build had WITH_LIBSOFT=3D in use. -r302412 is = my first testing of WITHOUT_LIBSOFT=3D after rebuilding all ports to = avoid libsoft. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-stable@freebsd.org Fri Jul 8 10:28:00 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD523B76F8F for ; Fri, 8 Jul 2016 10:28:00 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-211-153.reflexion.net [208.70.211.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7844D1E3A for ; Fri, 8 Jul 2016 10:27:59 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 16827 invoked from network); 8 Jul 2016 10:27:59 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 8 Jul 2016 10:27:59 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v7.90.3) with SMTP; Fri, 08 Jul 2016 06:28:47 -0400 (EDT) Received: (qmail 30822 invoked from network); 8 Jul 2016 10:28:46 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 8 Jul 2016 10:28:46 -0000 X-No-Relay: not in my network X-No-Relay: not in my network X-No-Relay: not in my network X-No-Relay: not in my network Received: from [192.168.0.105] (ip70-189-131-151.lv.lv.cox.net [70.189.131.151]) by iron2.pdx.net (Postfix) with ESMTPSA id 521E51C405F; Fri, 8 Jul 2016 03:27:43 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: 11.0 -r302412: check-old does not list older /usr/libsoft/. . . materials; delete-old and delete-old-libs do not delete them Message-Id: Date: Fri, 8 Jul 2016 03:27:56 -0700 To: freebsd-arm , freebsd-stable@freebsd.org, FreeBSD Toolchain , FreeBSD Current Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 10:28:01 -0000 I just did a amd64 -> armv6 cross-build/update for -r302331 -> -r302412. -r302331's build had WITH_LIBSOFT=3Dyes -r302412's build was WITHOUT_LIBSOFT (by not specifying WITH_LIBSOFT=3Dyes= ) (I had rebuilt all the installed ports [-r417989 update] under = -r302331.) The check-old stage did not report anything about /usr/libsoft materials = as to be deleted. Nor did delete-old or delete-old-libs remove any of them. Looking in the DESTDIR tree shows that the libsoft files are still there = and are old: libsoft was not rebuilt. -r302412's build was WITHOUT_META_MODE because of a different issue when = WITH_META_MODE was attempted. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-stable@freebsd.org Fri Jul 8 12:17:33 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2306B8289A for ; Fri, 8 Jul 2016 12:17:33 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CD2C31283 for ; Fri, 8 Jul 2016 12:17:33 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id C8DC3B82899; Fri, 8 Jul 2016 12:17:33 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8824B82898 for ; Fri, 8 Jul 2016 12:17:33 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F5AA127F for ; Fri, 8 Jul 2016 12:17:32 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id u68CHQ93067250 for ; Fri, 8 Jul 2016 12:17:26 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id u68CHQGc067249 for stable@freebsd.org; Fri, 8 Jul 2016 05:17:26 -0700 (PDT) (envelope-from david) Date: Fri, 8 Jul 2016 05:17:26 -0700 From: David Wolfskill To: stable@freebsd.org Subject: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 Message-ID: <20160708121726.GL1265@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="RSXYIiHEeqBIa1aP" Content-Disposition: inline User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 12:17:34 -0000 --RSXYIiHEeqBIa1aP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable After having built & smoke-tested: FreeBSD freebeast.catwhisker.org 11.0-ALPHA6 FreeBSD 11.0-ALPHA6 #36 r3023= 88M/302389:1100120: Thu Jul 7 04:18:53 PDT 2016 root@freebeast.catwhis= ker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 yesterday, I had "cloned" that slice to slice 3, then: svn switch ^/stable/11 /S3/usr/src to update slice 3's src to "Relative URL: ^/stable/11", then started the usual build world & kernel, install kernel & world, mergemaster, &c. from src/UPDATING. All went well, through "mergemaster," but the next step "make delete-old" failed for me (showing last bit of "mergemaster" run for context): =2E.. *** You installed a services file, so make sure that you run '/usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services' to rebuild your services database Would you like to run it now? y or n [n] y Running /usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services Friday, July 8, 2016 at 05:05:23 AM PDT bmake[1]: "/usr/src/ObsoleteFiles.inc" line 8019: Malformed conditional (${= TARGET_ARCH} !=3D "aarch64" && ${TARGET_CPUARCH} !=3D "arm" && ${TARGET_AR= CH} !=3D "powerpc" && ${TARGET_ARCH} !=3D "powerpc64" && ${TARGET_ARCH} != =3D "sparc64") bmake[1]: Fatal errors encountered -- cannot continue bmake[1]: stopped in /usr/src *** Error code 1 Stop. make: stopped in /usr/src =2EERROR_TARGET=3D'delete-old' =2EERROR_META_FILE=3D'' =2EMAKE.LEVEL=3D'0' MAKEFILE=3D'' =2EMAKE.MODE=3D'normal' =2ECURDIR=3D'/usr/src' =2EMAKE=3D'make' =2EOBJDIR=3D'/usr/obj/usr/src' =2ETARGETS=3D'delete-old' DESTDIR=3D'' LD_LIBRARY_PATH=3D'' MACHINE=3D'amd64' MACHINE_ARCH=3D'amd64' MAKEOBJDIRPREFIX=3D'/usr/obj' MAKESYSPATH=3D'/usr/src/share/mk' MAKE_VERSION=3D'20160606' PATH=3D'/sbin:/bin:/usr/sbin:/usr/bin' SRCTOP=3D'/usr/src' OBJTOP=3D'/usr/obj/usr/src' freebeast(11.0)[2]=20 svn info says: freebeast(11.0)[8] svn info /usr/src Path: /usr/src Working Copy Root Path: /usr/src URL: file:///svn/freebsd/src/base/stable/11 Relative URL: ^/stable/11 Repository Root: file:///svn/freebsd/src/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 302426 Node Kind: directory Schedule: normal Last Changed Author: gjb Last Changed Rev: 302412 Last Changed Date: 2016-07-07 17:26:38 -0700 (Thu, 07 Jul 2016) freebeast(11.0)[9]=20 I'll go ahead & reboot for a smoke test, see how far I get (and probably go on to build & smoke-test today's head). Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --RSXYIiHEeqBIa1aP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJXf5nWXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDQ0I3Q0VGOTE3QTgwMUY0MzA2NEQ3N0Ix NTM5Q0M0MEEwNDlFRTE3AAoJEBU5zECgSe4XLh4H/R5Ve/j4JI51k5VfBnD8kw21 flAw9ttiX+GmBFRPwNUxh3zwaJgPfynuhbmMo1dTvPd4rPGNGGErR8YQGaRvbxF3 jTV9kb+Jh0h5Wn6AB9XqqEaAQ5vOcfyINAsMtGX7EsGnViOSdbwg8ZmDoQM0RQga WC+7SfpNO3kdn/1itGsoFglZjJpv/j7OsTIK/bnFDYgK3+0HyJ4H6/f9+D6BNfNF wlVnA4TwdN9E/SiIDiFaOMPQoAxrbLzCRXPXwOOlCj3DYySKPcx9qoTGBRhXyoqz 4PzXSCu2izFLPQJb/HT9JjRLosBQxs0D73HFIBbKIbTy1AQ4PT3UI5i6fGa0bJU= =4/t3 -----END PGP SIGNATURE----- --RSXYIiHEeqBIa1aP-- From owner-freebsd-stable@freebsd.org Fri Jul 8 13:19:22 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 819CAB83A5B for ; Fri, 8 Jul 2016 13:19:22 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-pa0-x242.google.com (mail-pa0-x242.google.com [IPv6:2607:f8b0:400e:c03::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50E941965 for ; Fri, 8 Jul 2016 13:19:22 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-pa0-x242.google.com with SMTP id ib6so6695919pad.3 for ; Fri, 08 Jul 2016 06:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=Ya+QFqSgy71xUqbrU2mMYKm5wnHHxOJ4b1FND/h+R9g=; b=zAUwZi97CAN7u1EZeIOscCNaBnbTeOcHvWG6jbqIbRYfkyV6CDyYtjyEPjZ9Tyd3oH KMCSSUECgnImKKvaMIPJ5EOW4vu7XTdwdQhqy+qPmLzTvTxcbvQCsyd+vx4LcGTBHCmH riLR+nT/zKaMDjY5dSoLVoTfGgxHrXqGX2VyF6vhFxCtAwhgMBIfpMP+gOXEsNNfdYPi p0fKq0VXkUBk5MlkL2xEYUiCtenk7rPU1BZI9O7LYi/09BV+2A60VG360uVv/lkPW4Do KLTnE2Uit06BFbcfJIFUYkO36Y0l1UJMB2jErtlY3lZMGUajI7t8DeuU9awJzY52PjAu GCZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=Ya+QFqSgy71xUqbrU2mMYKm5wnHHxOJ4b1FND/h+R9g=; b=GfqxkIlkLK1zuVjv3l00893SaPSwlfOFeJtWvuFmUAWRuxovW/nboeEjpEYhfE6lk4 fBT/oQaYitJ4DLXTToD8usa0A7RHjp/5QvTDdCA24OZPfjcsM6v0lJqiB5ks6juDbBt+ PXL5YSA9OSOY7XSCMQH9DoW+cciC8L1OsUSTU3I/fIM2gGDXEOwYabJP8htMyk7g82ym RRx+v6lQSXAkX2IhCl7CZ7N/8AV/rCDmoVpO6w1t4g9IwW61SFbDP2pWtMG4aac4ItF8 4Yvzy3M9ev+cd9L8KD97s1zw7pOaZLEXPRDBphhC8sPMdWlvYrSccmO2pDvFK1pJTgfT 7HIA== X-Gm-Message-State: ALyK8tJDwo1MpaUZfyG2SEWfskhAjFfOsbTMtpetE0fwWW/Cg3c1eoWuSZF+oSssSO5YoA== X-Received: by 10.194.66.101 with SMTP id e5mr5844682wjt.143.1467983961530; Fri, 08 Jul 2016 06:19:21 -0700 (PDT) Received: from Johans-MacBook-Air-2.local (92-111-79-242.static.chello.nl. [92.111.79.242]) by smtp.googlemail.com with ESMTPSA id f73sm8327149wmg.1.2016.07.08.06.19.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 08 Jul 2016 06:19:20 -0700 (PDT) Subject: Re: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 To: freebsd-stable@freebsd.org References: <20160708121726.GL1265@albert.catwhisker.org> From: Johan Hendriks Message-ID: Date: Fri, 8 Jul 2016 15:19:19 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160708121726.GL1265@albert.catwhisker.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 13:19:22 -0000 Op 08/07/16 om 14:17 schreef David Wolfskill: > After having built & smoke-tested: > FreeBSD freebeast.catwhisker.org 11.0-ALPHA6 FreeBSD 11.0-ALPHA6 #36 r302388M/302389:1100120: Thu Jul 7 04:18:53 PDT 2016 root@freebeast.catwhisker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 > > yesterday, I had "cloned" that slice to slice 3, then: > svn switch ^/stable/11 /S3/usr/src > > to update slice 3's src to "Relative URL: ^/stable/11", then started > the usual build world & kernel, install kernel & world, mergemaster, > &c. from src/UPDATING. All went well, through "mergemaster," but > the next step "make delete-old" failed for me (showing last bit of > "mergemaster" run for context): > > ... > *** You installed a services file, so make sure that you run > '/usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services' > to rebuild your services database > > Would you like to run it now? y or n [n] y > Running /usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services > > > Friday, July 8, 2016 at 05:05:23 AM PDT > bmake[1]: "/usr/src/ObsoleteFiles.inc" line 8019: Malformed conditional (${TARGET_ARCH} != "aarch64" && ${TARGET_CPUARCH} != "arm" && ${TARGET_ARCH} != "powerpc" && ${TARGET_ARCH} != "powerpc64" && ${TARGET_ARCH} != "sparc64") > bmake[1]: Fatal errors encountered -- cannot continue > bmake[1]: stopped in /usr/src > *** Error code 1 > > Stop. > make: stopped in /usr/src > .ERROR_TARGET='delete-old' > .ERROR_META_FILE='' > .MAKE.LEVEL='0' > MAKEFILE='' > .MAKE.MODE='normal' > .CURDIR='/usr/src' > .MAKE='make' > .OBJDIR='/usr/obj/usr/src' > .TARGETS='delete-old' > DESTDIR='' > LD_LIBRARY_PATH='' > MACHINE='amd64' > MACHINE_ARCH='amd64' > MAKEOBJDIRPREFIX='/usr/obj' > MAKESYSPATH='/usr/src/share/mk' > MAKE_VERSION='20160606' > PATH='/sbin:/bin:/usr/sbin:/usr/bin' > SRCTOP='/usr/src' > OBJTOP='/usr/obj/usr/src' > freebeast(11.0)[2] > > svn info says: > freebeast(11.0)[8] svn info /usr/src > Path: /usr/src > Working Copy Root Path: /usr/src > URL: file:///svn/freebsd/src/base/stable/11 > Relative URL: ^/stable/11 > Repository Root: file:///svn/freebsd/src/base > Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f > Revision: 302426 > Node Kind: directory > Schedule: normal > Last Changed Author: gjb > Last Changed Rev: 302412 > Last Changed Date: 2016-07-07 17:26:38 -0700 (Thu, 07 Jul 2016) > > freebeast(11.0)[9] > > > I'll go ahead & reboot for a smoke test, see how far I get (and probably > go on to build & smoke-test today's head). > > Peace, > david Just a me too, was running HEAD, switched to stable/11. BTW i deleted the old /usr/src and installed a new source tree with a new checkout. after make builworld cycle and a reboot, a make delete-old gives the following error root@desk:~ # cd /usr/src/ root@desk:/usr/src # make delete-old make[1]: "/usr/src/ObsoleteFiles.inc" line 8019: Malformed conditional (${TARGET_ARCH} != "aarch64" && ${TARGET_CPUARCH} != "arm" && ${TARGET_ARCH} != "powerpc" && ${TARGET_ARCH} != "powerpc64" && ${TARGET_ARCH} != "sparc64") make[1]: Fatal errors encountered -- cannot continue make[1]: stopped in /usr/src *** Error code 1 Stop. make: stopped in /usr/src regards Johan From owner-freebsd-stable@freebsd.org Fri Jul 8 13:23:55 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FE80B83CDB for ; Fri, 8 Jul 2016 13:23:55 +0000 (UTC) (envelope-from imb@protected-networks.net) Received: from mail.auburn.protected-networks.net (mail.auburn.protected-networks.net [IPv6:2001:470:1f07:4e1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.auburn.protected-networks.net", Issuer "Protected Networks Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 38CDB1EE2 for ; Fri, 8 Jul 2016 13:23:55 +0000 (UTC) (envelope-from imb@protected-networks.net) Received: from toshi.auburn.protected-networks.net (toshi.auburn.protected-networks.net [192.168.1.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: imb@mail.auburn.protected-networks.net) by mail.auburn.protected-networks.net (Postfix) with ESMTPSA id F2D571CC26; Fri, 8 Jul 2016 09:23:44 -0400 (EDT) Subject: Re: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 To: Johan Hendriks , freebsd-stable@freebsd.org References: <20160708121726.GL1265@albert.catwhisker.org> From: Michael Butler Openpgp: id=6F63E6399DCC8E3E94D60F0642FF6BAE0442D492 Message-ID: <8a88a250-6212-eb80-b942-90926ee1511c@protected-networks.net> Date: Fri, 8 Jul 2016 09:23:43 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 13:23:55 -0000 Seems to be a typo .. TARGET_CPUARCH should be TARGET_ARCH .. On 07/08/16 09:19, Johan Hendriks wrote: > > > Op 08/07/16 om 14:17 schreef David Wolfskill: >> After having built & smoke-tested: >> FreeBSD freebeast.catwhisker.org 11.0-ALPHA6 FreeBSD 11.0-ALPHA6 #36 r302388M/302389:1100120: Thu Jul 7 04:18:53 PDT 2016 root@freebeast.catwhisker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 >> >> yesterday, I had "cloned" that slice to slice 3, then: >> svn switch ^/stable/11 /S3/usr/src >> >> to update slice 3's src to "Relative URL: ^/stable/11", then started >> the usual build world & kernel, install kernel & world, mergemaster, >> &c. from src/UPDATING. All went well, through "mergemaster," but >> the next step "make delete-old" failed for me (showing last bit of >> "mergemaster" run for context): >> >> ... >> *** You installed a services file, so make sure that you run >> '/usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services' >> to rebuild your services database >> >> Would you like to run it now? y or n [n] y >> Running /usr/sbin/services_mkdb -q -o /var/db/services.db /etc/services >> >> >> Friday, July 8, 2016 at 05:05:23 AM PDT >> bmake[1]: "/usr/src/ObsoleteFiles.inc" line 8019: Malformed conditional (${TARGET_ARCH} != "aarch64" && ${TARGET_CPUARCH} != "arm" && ${TARGET_ARCH} != "powerpc" && ${TARGET_ARCH} != "powerpc64" && ${TARGET_ARCH} != "sparc64") >> bmake[1]: Fatal errors encountered -- cannot continue >> bmake[1]: stopped in /usr/src >> *** Error code 1 >> >> Stop. >> make: stopped in /usr/src >> .ERROR_TARGET='delete-old' >> .ERROR_META_FILE='' >> .MAKE.LEVEL='0' >> MAKEFILE='' >> .MAKE.MODE='normal' >> .CURDIR='/usr/src' >> .MAKE='make' >> .OBJDIR='/usr/obj/usr/src' >> .TARGETS='delete-old' >> DESTDIR='' >> LD_LIBRARY_PATH='' >> MACHINE='amd64' >> MACHINE_ARCH='amd64' >> MAKEOBJDIRPREFIX='/usr/obj' >> MAKESYSPATH='/usr/src/share/mk' >> MAKE_VERSION='20160606' >> PATH='/sbin:/bin:/usr/sbin:/usr/bin' >> SRCTOP='/usr/src' >> OBJTOP='/usr/obj/usr/src' >> freebeast(11.0)[2] >> >> svn info says: >> freebeast(11.0)[8] svn info /usr/src >> Path: /usr/src >> Working Copy Root Path: /usr/src >> URL: file:///svn/freebsd/src/base/stable/11 >> Relative URL: ^/stable/11 >> Repository Root: file:///svn/freebsd/src/base >> Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f >> Revision: 302426 >> Node Kind: directory >> Schedule: normal >> Last Changed Author: gjb >> Last Changed Rev: 302412 >> Last Changed Date: 2016-07-07 17:26:38 -0700 (Thu, 07 Jul 2016) >> >> freebeast(11.0)[9] >> >> >> I'll go ahead & reboot for a smoke test, see how far I get (and probably >> go on to build & smoke-test today's head). >> >> Peace, >> david > > Just a me too, was running HEAD, switched to stable/11. > BTW i deleted the old /usr/src and installed a new source tree with a > new checkout. > > after make builworld cycle and a reboot, a make delete-old gives the > following error > > root@desk:~ # cd /usr/src/ > root@desk:/usr/src # make delete-old > make[1]: "/usr/src/ObsoleteFiles.inc" line 8019: Malformed conditional > (${TARGET_ARCH} != "aarch64" && ${TARGET_CPUARCH} != "arm" && > ${TARGET_ARCH} != "powerpc" && ${TARGET_ARCH} != "powerpc64" && > ${TARGET_ARCH} != "sparc64") > make[1]: Fatal errors encountered -- cannot continue > make[1]: stopped in /usr/src > *** Error code 1 > > Stop. > make: stopped in /usr/src > > regards > Johan > > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Fri Jul 8 13:32:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63F77B84070 for ; Fri, 8 Jul 2016 13:32:08 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3215D185E for ; Fri, 8 Jul 2016 13:32:07 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id u68DW6Ng069016; Fri, 8 Jul 2016 13:32:06 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id u68DW65s069015; Fri, 8 Jul 2016 06:32:06 -0700 (PDT) (envelope-from david) Date: Fri, 8 Jul 2016 06:32:06 -0700 From: David Wolfskill To: Michael Butler Cc: Johan Hendriks , freebsd-stable@freebsd.org Subject: Re: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 Message-ID: <20160708133206.GP1265@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , Michael Butler , Johan Hendriks , freebsd-stable@freebsd.org References: <20160708121726.GL1265@albert.catwhisker.org> <8a88a250-6212-eb80-b942-90926ee1511c@protected-networks.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="pzaoPuhJruh3gUb+" Content-Disposition: inline In-Reply-To: <8a88a250-6212-eb80-b942-90926ee1511c@protected-networks.net> User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 13:32:08 -0000 --pzaoPuhJruh3gUb+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 08, 2016 at 09:23:43AM -0400, Michael Butler wrote: > Seems to be a typo .. TARGET_CPUARCH should be TARGET_ARCH .. > .... Yes; making that change eliminated the problem for me (on head; I'll replicate that on stable/11 shortly). Thanks! Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --pzaoPuhJruh3gUb+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJXf6tWXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDQ0I3Q0VGOTE3QTgwMUY0MzA2NEQ3N0Ix NTM5Q0M0MEEwNDlFRTE3AAoJEBU5zECgSe4XWBsH/299b4rNgPkk1LbTkXn1Shz2 o2bKL9i6WfOL7u80CnahCkxan9NuUbWSVKmjXUSs+C1Ax5myN3Mqq9S/2bF/tJqj 3IHEhnXdWu48ASMAaK89ZNB0CpKe4EfT9rZk1fbuYnUvezjMnklm8bogFsudJASo 38cjDSWa2FIniGgp7A+B6L960GwUckSnNgea97S2HvBLnhrSa4gx8WR39SEFG600 mtkK2bpGJcyUoWt6cV7TkaxW7zfojPV6Ao2vyZH7+97vuVd1DSC90DTceikJFuIt M/1oaG+sjKVb+qI9Y8mI88PNQwyXfLlzP2g9ngd4vJ2iVf/QAfv/0NDCAZ8+RMI= =RlyZ -----END PGP SIGNATURE----- --pzaoPuhJruh3gUb+-- From owner-freebsd-stable@freebsd.org Fri Jul 8 16:06:57 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8FAEB82D16 for ; Fri, 8 Jul 2016 16:06:57 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm42-vm4.bullet.mail.bf1.yahoo.com (nm42-vm4.bullet.mail.bf1.yahoo.com [216.109.114.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6432C1591 for ; Fri, 8 Jul 2016 16:06:57 +0000 (UTC) (envelope-from pfg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1467993835; bh=M5TNSlZFFSRDOGpcpMI0nGWvMXgfJB+Of3n6r9+sTWk=; h=From:Subject:To:Date:From:Subject; b=pyOipE9GFPOrY1UBHhyJiAurtDera3OSUuico5+j3Zkb6vt5xikLY+QYZIGzM8Q8kIhofMh7hoyNAp9MGPdDCHPtcv7uvt9DnEFT2Ahb8SxPI3/UEHDXbZH99nFysJ6ljUV7XfHY9aofL3KO5Hpe9MRVLJweX+EPxW5BGUobCEeQ4FN6Er2xm0RZmIK2SxNKXFkNc2fSa29Gs4rqPAtM9Y/atNZmckZq/B4I0PqbKVBXjLGZiAv3r1Q732nfT0viN/w4cBOzDpMvOmWFTufEAwXX9OalgChD+efzOkEhMTSamh7wSU4qZxZD/sG5NYCz1QtsTt3sqIU1cSfxowR2uw== Received: from [98.139.215.141] by nm42.bullet.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 16:03:55 -0000 Received: from [98.139.211.207] by tm12.bullet.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 16:03:55 -0000 Received: from [127.0.0.1] by smtp216.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 16:03:55 -0000 X-Yahoo-Newman-Id: 783518.65233.bm@smtp216.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: mQv4c7cVM1k2DHKKSMWPIpCx7VzMErNe6ZpS9C0SftH7wEk Ku7QHuSkUh2OKT5LjoU79KLDsagEpoYYtUk3Bf6F1Lj5P2JIa737iE.07IK8 In0r1Szzdh6QGX3o63d2tGZO3F0xwz7uY8xQ8ZAmWZlMwY3PJ_MiSoudmNmQ pqCP9PZuOo0M5WaWreEDjeSKnkPF1wEJMn.aNgYpIi4k_WkZ5lmcsf1ndaY6 yBGOGaJUmg_1nznUVzqi7Okgecqr94H5inxW1iKBUK_41WfIDUvJF_WxtdkF wqRlzUygJsHP2Lz7wpSjnpltcuYHCsqNENrEvUfo_QsfUIjdmyRVdOdwoO4u LeYk8YfHadb8FdxSthZxnQwdI.a3PapsskoM1V3jwOGdME7e.FKhr6aEg.5q 963AVsaic612rpDIQpz.QVn8aHTsOlp3RboVZyuIJpvoC.g5Lrbj8nxrFDDx qamOOPQ.9xMtAEHOpas8GjzHBhstnWsbASc4PHpEUUG70UBwywDTG9LskJtX eWzAKqeczvoe73UKiT5AWCJg_SyI_ftPdZmTVigVkJGylag-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf From: Pedro Giffuni Subject: Problem building stable-11 on FreeBSD stable-10 To: freebsd-stable@FreeBSD.org Message-ID: <1b69a8dd-27a1-bde9-132f-4419a26b6555@FreeBSD.org> Date: Fri, 8 Jul 2016 11:04:06 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 16:06:57 -0000 Hello; Sorry if this is a naive question, I just tried to build 11-stable on 10-stable and I got this: _____ ... cc -O2 -pipe -march=core2 -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/usr/src/sys/GENERIC/opt_global.h -I. -I/usr/src/sys -fno-common -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/obj/usr/src/sys/GENERIC -MD -MP -MF.depend.aac.o -MTaac.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -mno-aes -mno-avx -std=iso9899:1999 -c /usr/src/sys/modules/aac/../../dev/aac/aac.c -o aac.o --- all_subdir_aacraid --- /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2510:50: error: invalid conversion specifier 'b' [-Werror,-Wformat-invalid-specifier] device_printf(sc->aac_dev, "Supported Options=%b\n", ~^ /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2512:10: error: data argument not used by format string [-Werror,-Wformat-extra-args] "\20" ^ 2 errors generated. *** [aacraid.o] Error code 1 make[4]: stopped in /usr/src/sys/modules/aacraid 1 error ... _____ So, while here, is there a "blessed" way to upgrade 10-stable to 11-stable, without wiping out my disks? Pedro. From owner-freebsd-stable@freebsd.org Fri Jul 8 16:16:48 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F6D4B83209 for ; Fri, 8 Jul 2016 16:16:48 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 103D71DE9; Fri, 8 Jul 2016 16:16:48 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 85E6AD1; Fri, 8 Jul 2016 16:16:48 +0000 (UTC) Date: Fri, 8 Jul 2016 16:16:48 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org Message-ID: <390763160.6.1467994608488.JavaMail.jenkins@jenkins-9.freebsd.org> In-Reply-To: <582582014.121.1466550319308.JavaMail.jenkins@jenkins-9.freebsd.org> References: <582582014.121.1466550319308.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Build failed in Jenkins: FreeBSD_stable_10 #299 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_stable_10 X-Jenkins-Result: FAILURE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 16:16:48 -0000 See ------------------------------------------ Started by an SCM change > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/freebsd/freebsd-ci.git # timeout=10 Fetching upstream changes from https://github.com/freebsd/freebsd-ci.git > git --version # timeout=10 > git -c core.askpass=true fetch --tags --progress https://github.com/freebsd/freebsd-ci.git +refs/heads/*:refs/remotes/origin/* --depth=1 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 2226af5363e9e6a0937e559f3f75dfff22184745 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 2226af5363e9e6a0937e559f3f75dfff22184745 > git rev-list 2226af5363e9e6a0937e559f3f75dfff22184745 # timeout=10 [Pipeline] node Still waiting to schedule task Waiting for next available executor on jenkins-10.freebsd.org Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Resuming build Aborted by lwhsu [Pipeline] // node [Pipeline] node Running on master in /usr/local/jenkins/workspace/FreeBSD_stable_10 [Pipeline] { From owner-freebsd-stable@freebsd.org Fri Jul 8 16:36:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 878CAB83B12 for ; Fri, 8 Jul 2016 16:36:31 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: from mail-it0-x242.google.com (mail-it0-x242.google.com [IPv6:2607:f8b0:4001:c0b::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5164E1290 for ; Fri, 8 Jul 2016 16:36:31 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: by mail-it0-x242.google.com with SMTP id y93so5616584ita.0 for ; Fri, 08 Jul 2016 09:36:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=lmW0CR3imCn+C2XHTAXYQTCYNvcKr8g5+NYcehL92Hs=; b=gRJVqk+aFqC0A5N1XRW8Tv4wg+K0Aq2fge6T++e3MeBN0PGwFvVQL6SVku8bJa0ExR L6+SVvjl1C5p7VNEkJYv8Q6MaXoUpAUEyB8Qu1ec99zt2jaX4MxqnSIF5N8TBru3Mz3I 9xsa2yKgCWbGVJlWGshWyE+Ja5aNTQ3VwxN0iHoUma2LshzX2bVj29CHcsJWFeRNfwNq 69kGMMu06U+/BE8cqOEY5mKSMmUgtrS04lK0ZrjpNzZjS+JhhxuTNdtBsPc/y32ZhQPv bD6TV15di4GL7Igpq0viMVfRLo5JtQfUuKV0Ca3Tj1gLy8kusq8KuloGIMEj/9Iep5MI kCVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=lmW0CR3imCn+C2XHTAXYQTCYNvcKr8g5+NYcehL92Hs=; b=hhjd8TxeJhi1KbrX9qU4Dg3OYFTMnNzSrLIqVEu2B8pISEW6Rd0otaQPdxxVLBUsuD EZ6a+warIuYrsZ0QDIgtg2oxRDjqndc11vv8RaPK23MzY5nKQstS9wOSc1au2GJkOnh/ Nek4vh+d4rDuhIkWqVlMKDA+jfnrFFJkoYyNnMYHVtHnqV5mVEJtYELIw9UevcPeUB3Z Dj3oMKQb4Efnt1RUMjrIebpAJ9BzGo2+Bka5UNjK9MA6Tsv6NU3R+3rRSfL3fsv5MWI5 d7RdltcpvODxkiuqkuH99LAdPrQ1IfCcdsUZxDN+tz55zbybYt+fioelPxDDeQKSoSML qj/w== X-Gm-Message-State: ALyK8tKOeexMct7uxnYPDEwVFyYHZhFiaa5DLfwop4JpnYZ7awmc/Jho3J1lSKwOup5oYA== X-Received: by 10.36.91.80 with SMTP id g77mr4440108itb.46.1467995790375; Fri, 08 Jul 2016 09:36:30 -0700 (PDT) Received: from [172.22.132.55] ([192.119.231.58]) by smtp.gmail.com with ESMTPSA id d189sm3190939iod.3.2016.07.08.09.36.29 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jul 2016 09:36:29 -0700 (PDT) From: Guy Helmer Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: 9-STABLE: Panic when destroying gmirror that is synchronizing Message-Id: Date: Fri, 8 Jul 2016 11:36:26 -0500 To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 16:36:31 -0000 Hi, I=E2=80=99m able to replicate a problem where destroying a gmirror that = is synchronizing causes a panic in 9-STABLE (^/stable/9 rev 302430) on = amd64. I=E2=80=99ve forced a gmirror into an inconsistent state by = forcing a reset, and then issued =E2=80=9Cgmirror destroy -f mirror1=E2=80= =9D after the reboot and the system is synchronizing the two disks in = the mirror. As a workaround, I can use =E2=80=9Cgeom mirror clear=E2=80=9D= on the providers before geom_mirror.ko is loaded. It=E2=80=99s a rare issue but I wanted to document it in case it can be = fixed. Kernel stack trace and code snippets follow. If anything other = information would be useful, please let me know. Guy Unread portion of the kernel message buffer: GEOM_MIRROR: Device mirror1: provider mirror/mirror1 destroyed. GEOM_MIRROR: Device mirror1: rebuilding provider = gptid/48c59e09-3c94-11e6-8928-000c29ce3c97 stopped. Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x98 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff806df6ff stack pointer =3D 0x28:0xffffff800024eb30 frame pointer =3D 0x28:0xffffff800024eb40 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 13 (g_up) trap number =3D 12 panic: page fault cpuid =3D 0 KDB: stack backtrace: #0 0xffffffff8072e336 at kdb_backtrace+0x66 #1 0xffffffff806f3d5e at panic+0x1ce #2 0xffffffff80910ae7 at trap_fatal+0x277 #3 0xffffffff80910e31 at trap_pfault+0x211 #4 0xffffffff809113f9 at trap+0x329 #5 0xffffffff808fa311 at calltrap+0x8 #6 0xffffffff81412593 at g_mirror_sync_done+0x53 #7 0xffffffff8077439e at biodone+0xae #8 0xffffffff806576ac at g_io_schedule_up+0xac #9 0xffffffff80657e0c at g_up_procbody+0x5c #10 0xffffffff806c0b4f at fork_exit+0x11f #11 0xffffffff808fa83e at fork_trampoline+0xe Uptime: 1m54s GEOM_MIRROR: Device mirror0: rebuilding provider = gptid/48ac38bf-3c94-11e6-8928-000c29ce3c97 stopped. Dumping 87 out of 238 MB:..19%..37%..55%..74%..92% Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from = /boot/kernel/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_mirror.ko #0 doadump (textdump=3D) at pcpu.h:235 235 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D) at pcpu.h:235 #1 0xffffffff806f3836 in kern_reboot (howto=3D260) at ../../../kern/kern_shutdown.c:454 #2 0xffffffff806f3d37 in panic (fmt=3D0x1
) at ../../../kern/kern_shutdown.c:642 #3 0xffffffff80910ae7 in trap_fatal (frame=3D0xc, eva=3D) at ../../../amd64/amd64/trap.c:876 #4 0xffffffff80910e31 in trap_pfault (frame=3D0xffffff800024ea80, = usermode=3D0) at ../../../amd64/amd64/trap.c:798 #5 0xffffffff809113f9 in trap (frame=3D0xffffff800024ea80) at ../../../amd64/amd64/trap.c:462 #6 0xffffffff808fa311 in calltrap () at = ../../../amd64/amd64/exception.S:238 #7 0xffffffff806df6ff in _mtx_lock_flags (m=3D0x80, opts=3D0,=20 file=3D0xffffffff8141d0d8 = "/usr/src/sys/modules/geom/geom_mirror/../../../geom/mirror/g_mirror.c", = line=3D990) at atomic.h:164 #8 0xffffffff81412593 in g_mirror_sync_done (bp=3D0xfffffe0004c963e0) at = /usr/src/sys/modules/geom/geom_mirror/../../../geom/mirror/g_mirror.c:990 #9 0xffffffff8077439e in biodone (bp=3D0xfffffe0004c963e0) at ../../../kern/vfs_bio.c:3667 #10 0xffffffff806576ac in g_io_schedule_up (tp=3D) at ../../../geom/geom_io.c:808 #11 0xffffffff80657e0c in g_up_procbody (arg=3D) at ../../../geom/geom_kern.c:97 #12 0xffffffff806c0b4f in fork_exit ( callout=3D0xffffffff80657db0 , arg=3D0x0,=20 frame=3D0xffffff800024ec40) at ../../../kern/kern_fork.c:1000 #13 0xffffffff808fa83e in fork_trampoline () at ../../../amd64/amd64/exception.S:613 #14 0x0000000000000000 in ?? () (kgdb)=20 geom/mirror/g_mirror.c:990: static void g_mirror_sync_done(struct bio *bp) { struct g_mirror_softc *sc; G_MIRROR_LOGREQ(3, bp, "Synchronization request delivered."); sc =3D bp->bio_from->geom->softc; bp->bio_cflags =3D G_MIRROR_BIO_FLAG_SYNC; mtx_lock(&sc->sc_queue_mtx); <--- bioq_insert_tail(&sc->sc_queue, bp); mtx_unlock(&sc->sc_queue_mtx); wakeup(sc); } kern/vfs_vio.c:3667: done =3D bp->bio_done; if (done =3D=3D NULL) wakeup(bp); mtx_unlock(mtxp); if (done !=3D NULL) done(bp); <--- if (transient) { pmap_qremove(start, OFF_TO_IDX(end - start)); vm_map_remove(bio_transient_map, start, end); atomic_add_int(&inflight_transient_maps, -1); } } geom/geom_io.c:808: bp =3D g_bioq_first(&g_bio_run_up); if (bp !=3D NULL) { g_bioq_unlock(&g_bio_run_up); THREAD_NO_SLEEPING(); CTR4(KTR_GEOM, "g_up biodone bp %p provider %s = off " "%jd len %ld", bp, bp->bio_to->name, bp->bio_offset, bp->bio_length); biodone(bp); <--- THREAD_SLEEPING_OK(); continue; } CTR0(KTR_GEOM, "g_up going to sleep"); geom/geom_kern.c:97: static void g_up_procbody(void *arg) { mtx_assert(&Giant, MA_NOTOWNED); thread_lock(g_up_td); sched_prio(g_up_td, PRIBIO); thread_unlock(g_up_td); for(;;) { g_io_schedule_up(g_up_td); <--- } } From owner-freebsd-stable@freebsd.org Fri Jul 8 17:16:51 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24878B84669 for ; Fri, 8 Jul 2016 17:16:51 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm1-vm1.bullet.mail.bf1.yahoo.com (nm1-vm1.bullet.mail.bf1.yahoo.com [98.139.213.163]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D173F1BFB for ; Fri, 8 Jul 2016 17:16:50 +0000 (UTC) (envelope-from pfg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1467998209; bh=AGLlAgj7Sdwcs1lyHo0FmwnTksFotW36k3IPjCGYnrg=; h=Subject:To:References:From:Date:In-Reply-To:From:Subject; b=oEJ0nypwguKdrApHr8mWvZGmEh0vhv0jOoNy0UAdeeifkVrTi79U6Cepb+nljKdG4zr4pEoTBuK8LRysIyLzt1CYOeAi8FKtfgUsA/Ao0PK+Ensmzpc/ovs0E84XEvS6z8vIk1wCsVr5CBFRNtNKJEeg90KvFkpsjwZyoRMqG8OIiT2rn/B7Aj2ouDaB+Oe0nRU/eeowERzDTzrUxJpMjPSzbBWoxU1BDa1IBuJDIYV7UB7TJailGrb1shxzwzFanVY5TpIT9FkwsVsORYHR+6mc7P+TCcBiARNXAEWZYp1ZrtV38cAdUkd5MDgQeO2Xb5GKTquFPPcxPcnfxwsZvw== Received: from [98.139.215.140] by nm1.bullet.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 17:16:49 -0000 Received: from [98.139.211.199] by tm11.bullet.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 17:16:49 -0000 Received: from [127.0.0.1] by smtp208.mail.bf1.yahoo.com with NNFMP; 08 Jul 2016 17:16:49 -0000 X-Yahoo-Newman-Id: 455888.9142.bm@smtp208.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 7ImIgNQVM1kmZDj7KZunWW0ub2D2cJccxUxw.Hbf4ynZab. FWUs6YVUiDJ2t3qM7ds6BnhzSWVDThbVAJM3Udgokbqmx95mAxKHcLAKG.hB Lr1.iQPPhhfSOObOrAw9FdM0Ee7tDhLLiPmQS1WM7HtS3EiQcGeAwAP08Yov 8wlPbTZbKsnNFqvvJ12ccc6zU0ElSq9s4Flf3XxLas4hINvtvSeg8HNnGenD ITdxf.62DiSwKP_6aDxN_EiKk43jwLNpVuR0wIxyPWi1LCf6IofF6qNnDTna ohmQKOR7IlNjPXsQCIDwe6PevdA254KwOuPhQkrfcpjnXBtzw0QHgYGgk0sP sbQav7CZmQEy_mMh36p3UCX0m6UJDJOcZK7FoksjOM2RvXUJuOLdKxzeXUrm jSge9bzfZRF4ZNk4efS4bL8HWUmNsbqN5_GQuMd1wfWWyyDz0WdlDq_sAQ4a sRmniQfSLDKxr03Kte_J8bXnlPAxgSbsWqjbli5OFNAXK9gqBVIBmTP9djsk lOoTN583Gla4qTbOEf53cZkhIbJPzgP.WhLkIo3mPAEi72A-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Subject: Re: Problem building stable-11 on FreeBSD stable-10 To: freebsd-stable@FreeBSD.org References: <1b69a8dd-27a1-bde9-132f-4419a26b6555@FreeBSD.org> From: Pedro Giffuni Message-ID: Date: Fri, 8 Jul 2016 12:16:59 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <1b69a8dd-27a1-bde9-132f-4419a26b6555@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 17:16:51 -0000 On 07/08/16 11:04, Pedro Giffuni wrote: > Hello; > > Sorry if this is a naive question, I just tried to build 11-stable on > 10-stable and I got this: > > _____ > ... > cc -O2 -pipe -march=core2 -fno-strict-aliasing -Werror -D_KERNEL > -DKLD_MODULE -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include > /usr/obj/usr/src/sys/GENERIC/opt_global.h -I. -I/usr/src/sys -fno-common > -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer > -I/usr/obj/usr/src/sys/GENERIC -MD -MP -MF.depend.aac.o -MTaac.o > -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector > -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef > -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option > -Wno-unknown-pragmas -Wno-error-tautological-compare > -Wno-error-empty-body -Wno-error-parentheses-equality > -Wno-error-unused-function -Wno-error-pointer-sign -mno-aes -mno-avx > -std=iso9899:1999 -c /usr/src/sys/modules/aac/../../dev/aac/aac.c -o aac.o > --- all_subdir_aacraid --- > /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2510:50: error: > invalid conversion specifier 'b' [-Werror,-Wformat-invalid-specifier] > device_printf(sc->aac_dev, "Supported Options=%b\n", > ~^ > /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2512:10: error: > data argument not used by format string [-Werror,-Wformat-extra-args] > "\20" > ^ > 2 errors generated. > *** [aacraid.o] Error code 1 > > make[4]: stopped in /usr/src/sys/modules/aacraid > 1 error > ... > _____ > For the record, I got around this by building the kernel-toolchain target. Pedro. From owner-freebsd-stable@freebsd.org Fri Jul 8 18:05:26 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F28DFB8543F for ; Fri, 8 Jul 2016 18:05:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC73117BD for ; Fri, 8 Jul 2016 18:05:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22c.google.com with SMTP id h190so15522637ith.1 for ; Fri, 08 Jul 2016 11:05:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ZhqYtsn5DzroktQP4l6ztGbpPxXaYPzpqXyyy+oswPw=; b=fywqhEvAkU+8WcTN7u37fcEnBzSzefasysz2O2EXWanLUUdrdteYZhb4MOJY5EBlvZ 13cCQhnH3e9lvFrWWhDQs6WUktgJStuvuTHyuT5b5ayXc37LKR53gAZ/zmnMXUr7XMeG mSfubRD86OGiRVJo8TZBmnTvtgdO7+rxakMcUlmLGLfSkZ4X5XheGtVfzeLqmV3hMbcA VS7shyao1zyOUkIfAlRAgDZvNj/n3tRmvMpTSXmnm0pxy6QRIyn89Lz7gU8mJVjdYNXe e9LS1y7NlJWk/XsUmQBXi0eGt0o7NHjynfW/arq/2EQHhZ1zw4TiA5+7DBs2E7C7bqu/ 4GkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ZhqYtsn5DzroktQP4l6ztGbpPxXaYPzpqXyyy+oswPw=; b=GkfBNdYIHvxzG7bwiJ+7VnS7Y7PMBLPpW7MpWkDW8zdBRIXGC6Xnh88quRaMlFvVm6 IcuJhRE7qqOyTl9qw8KQb/2ZOoPfqJnU7ZsLidkI2P6OvzbAY1p/7REbdUiupVX47Bw8 v/73hcbmUd7VDUmaJysOYfyc4xP2M+byHNSoVdFhqEQYYCbsIBBrWqhelm7f6NLEScgM dgW+2quGv6wUUDudwoPkPsw7YEUAA8ALKpTFJZSZ5JZmUtgD/jbGZtdXsbD4kTVQGOij hkcC3Hm3GguwUs4e4YAtIhnITmkFkFxu35Icn1RE9W0/NTK3UCfVeQK76e4KykyR+YCV TwYw== X-Gm-Message-State: ALyK8tKnrL85PWVLH6c3fprpL1fwBL+WMOm8U2aCyvbjy7mzPLQe0EJCYzy0jW7EgN6twlN6ZQoIaSQ4VA3PCw== X-Received: by 10.36.41.16 with SMTP id p16mr4948940itp.60.1468001126178; Fri, 08 Jul 2016 11:05:26 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.137.131 with HTTP; Fri, 8 Jul 2016 11:05:25 -0700 (PDT) X-Originating-IP: [69.53.245.200] In-Reply-To: References: <1b69a8dd-27a1-bde9-132f-4419a26b6555@FreeBSD.org> From: Warner Losh Date: Fri, 8 Jul 2016 12:05:25 -0600 X-Google-Sender-Auth: 7KsY0ihLgs6uHO6p1zxpkU_LRwY Message-ID: Subject: Re: Problem building stable-11 on FreeBSD stable-10 To: Pedro Giffuni Cc: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 18:05:27 -0000 That's the official supported way. Warner On Fri, Jul 8, 2016 at 11:16 AM, Pedro Giffuni wrote: > > > On 07/08/16 11:04, Pedro Giffuni wrote: >> >> Hello; >> >> Sorry if this is a naive question, I just tried to build 11-stable on >> 10-stable and I got this: >> >> _____ >> ... >> cc -O2 -pipe -march=core2 -fno-strict-aliasing -Werror -D_KERNEL >> -DKLD_MODULE -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include >> /usr/obj/usr/src/sys/GENERIC/opt_global.h -I. -I/usr/src/sys -fno-common >> -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer >> -I/usr/obj/usr/src/sys/GENERIC -MD -MP -MF.depend.aac.o -MTaac.o >> -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float >> -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector >> -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef >> -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option >> -Wno-unknown-pragmas -Wno-error-tautological-compare >> -Wno-error-empty-body -Wno-error-parentheses-equality >> -Wno-error-unused-function -Wno-error-pointer-sign -mno-aes -mno-avx >> -std=iso9899:1999 -c /usr/src/sys/modules/aac/../../dev/aac/aac.c -o aac.o >> --- all_subdir_aacraid --- >> /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2510:50: error: >> invalid conversion specifier 'b' [-Werror,-Wformat-invalid-specifier] >> device_printf(sc->aac_dev, "Supported Options=%b\n", >> ~^ >> /usr/src/sys/modules/aacraid/../../dev/aacraid/aacraid.c:2512:10: error: >> data argument not used by format string [-Werror,-Wformat-extra-args] >> "\20" >> ^ >> 2 errors generated. >> *** [aacraid.o] Error code 1 >> >> make[4]: stopped in /usr/src/sys/modules/aacraid >> 1 error >> ... >> _____ >> > > For the record, I got around this by building the kernel-toolchain > target. > > > Pedro. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Fri Jul 8 18:59:43 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 929CAB831A3 for ; Fri, 8 Jul 2016 18:59:43 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 77B8F15C1; Fri, 8 Jul 2016 18:59:43 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 99EB7D7; Fri, 8 Jul 2016 18:59:43 +0000 (UTC) Date: Fri, 8 Jul 2016 18:59:43 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org Message-ID: <1650114875.10.1468004383566.JavaMail.jenkins@jenkins-9.freebsd.org> In-Reply-To: <390763160.6.1467994608488.JavaMail.jenkins@jenkins-9.freebsd.org> References: <390763160.6.1467994608488.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is unstable: FreeBSD_stable_10 #300 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_stable_10 X-Jenkins-Result: UNSTABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 18:59:43 -0000 See From owner-freebsd-stable@freebsd.org Fri Jul 8 21:14:52 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3ECBB85B2C for ; Fri, 8 Jul 2016 21:14:52 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-211-154.reflexion.net [208.70.211.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F95B1DA7 for ; Fri, 8 Jul 2016 21:14:51 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 402 invoked from network); 8 Jul 2016 21:14:45 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 8 Jul 2016 21:14:45 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v7.90.3) with SMTP; Fri, 08 Jul 2016 17:14:55 -0400 (EDT) Received: (qmail 17450 invoked from network); 8 Jul 2016 21:14:55 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 8 Jul 2016 21:14:55 -0000 X-No-Relay: not in my network X-No-Relay: not in my network Received: from [192.168.0.105] (ip70-189-131-151.lv.lv.cox.net [70.189.131.151]) by iron2.pdx.net (Postfix) with ESMTPSA id 48D831C4079; Fri, 8 Jul 2016 14:14:32 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 Message-Id: Date: Fri, 8 Jul 2016 14:14:48 -0700 To: freebsd-stable@freebsd.org, FreeBSD Current Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 21:14:52 -0000 On Fri, Jul 08, 2016 at 09:23:43AM -0400, Michael Butler wrote: > Seems = to be a typo .. TARGET_CPUARCH should be TARGET_ARCH ..=20 > ....=20 ${TARGET_CPUARCH} !=3D "arm" apparently was an attempt to not explicitly = test for each specific arm variant (such as armv=C2=A7). ${TARGET_ARCH} !=3D "arm" does not appear to achieve the original test's = intent and will mishandle things list armv6 as far as I can tell. (It = certainly does avoid the malformed conditional problem that stops the = build.) =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-stable@freebsd.org Fri Jul 8 21:18:57 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4638B85DC9; Fri, 8 Jul 2016 21:18:57 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id D527313B6; Fri, 8 Jul 2016 21:18:57 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by freefall.freebsd.org (Postfix) with ESMTP id 8BEF21279; Fri, 8 Jul 2016 21:18:57 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Date: Fri, 8 Jul 2016 21:18:56 +0000 From: Glen Barber To: Mark Millard Cc: freebsd-stable@freebsd.org, FreeBSD Current Subject: Re: "make delete-old" failed: 11.0-ALPHA6 @r302388 -> stable/11 @r302412 Message-ID: <20160708211856.GH25877@FreeBSD.org> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Enx9fNJ0XV5HaWRu" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event X-PEKBAC-Definition: Problem Exists, Keyboard Between Admin/Computer User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 21:18:58 -0000 --Enx9fNJ0XV5HaWRu Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 08, 2016 at 02:14:48PM -0700, Mark Millard wrote: > On Fri, Jul 08, 2016 at 09:23:43AM -0400, Michael Butler wrote: > Seems t= o be a typo .. TARGET_CPUARCH should be TARGET_ARCH ..=20 > > ....=20 >=20 > ${TARGET_CPUARCH} !=3D "arm" apparently was an attempt to not explicitly = test for each specific arm variant (such as armv=A7). >=20 > ${TARGET_ARCH} !=3D "arm" does not appear to achieve the original test's = intent and will mishandle things list armv6 as far as I can tell. (It certa= inly does avoid the malformed conditional problem that stops the build.) >=20 We're aware of this, and a fix is pending approval from re@. Glen --Enx9fNJ0XV5HaWRu Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXgBjAAAoJEAMUWKVHj+KT41EP/2qWGckHJNAAzhxl/vjZNuTh LchTHeSQxp6pTh5+OxTRZrZth4y/AWGVB4ZCXcwAUNXd8uQdsOHqagUFVpVeH3/I TX1ksO5nmLfbiSSUBm4bw4LqW/z7i/35kwBt7LrOF/s4mls+pDFXmLyO/R8ujn87 aJEHblriYqy4YKvP8pG719RayeHiYRX9JiZ9S5yNreeN73WnRpGnym2Bg6LQfmPA zIzAREfQsMwn8b2VcxkFivoXiBSiLuMlO95fVaNbOaGrAm1qU9mpILoV65Z9L0wG /T5EqhvgR8YHLgUWlCW1YUybDIGfnj6FOS4mETvoPEd7DV/cBd+WG6n1T/rofRq4 81g2RU3zrGimwtce22keA6sua0ig/rTq4X66syDdShVRjcdExXIQEN11N9kynr83 +p+Ln34qu/2VgWnI55cqvq9sgFwjoPUpRnNSWEBnlyfxmhRxAMTIW0X19s56cFR2 Gisv3kio7CBvXEKwuhSbz/Oa99kJqv2pmzutgepI6Xe+xlSkLMAVgCME2Mhiy7MV 25JdDIzB1MeEhLZMwh1qp4Y459ylyGgWLOi6vhGqzA14rUuudBPGrPZVdwm6QRre XwtLEmWQ+2CLU/Nsbhi1Bb6dnJDrKM+9cExcqTt/ovmv+in96XU8axzfa8t3mCDS J10E8YR2sP6QVezW8oTI =iQDK -----END PGP SIGNATURE----- --Enx9fNJ0XV5HaWRu-- From owner-freebsd-stable@freebsd.org Fri Jul 8 22:57:32 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBAB8B84671 for ; Fri, 8 Jul 2016 22:57:32 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id C08F613D9; Fri, 8 Jul 2016 22:57:32 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id E950CF7; Fri, 8 Jul 2016 22:57:32 +0000 (UTC) Date: Fri, 8 Jul 2016 22:57:32 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org Message-ID: <62698937.11.1468018652840.JavaMail.jenkins@jenkins-9.freebsd.org> In-Reply-To: <1650114875.10.1468004383566.JavaMail.jenkins@jenkins-9.freebsd.org> References: <1650114875.10.1468004383566.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is still unstable: FreeBSD_stable_10 #301 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_stable_10 X-Jenkins-Result: UNSTABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 22:57:33 -0000 See From owner-freebsd-stable@freebsd.org Sat Jul 9 01:43:14 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A645FB83E1D; Sat, 9 Jul 2016 01:43:14 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x22a.google.com (mail-yw0-x22a.google.com [IPv6:2607:f8b0:4002:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73B8C18B9; Sat, 9 Jul 2016 01:43:14 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x22a.google.com with SMTP id l125so50839438ywb.2; Fri, 08 Jul 2016 18:43:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:cc; bh=s9wFMRb5nSXLWOxXSbco6K4+d90kJ7McKN32ouFL5Cw=; b=Jre5cmm+10apMmStyyjiCkew3eGvH5AppI+CpYf74E0Ei2/x+WZJN2nR2L/hviMbCd 315YEYB0ufUlbEuCHxuEf1DWrmyKYhcBP5KpuoFw9G8dhdsYmowY6A7FPsKv1vep/PW3 P76DYlb4eCa9xzllyxrLb6YaO11jW69DlUmOWQ/xe1qjGWlLHLpZzFoqv0JgXhM3i9Rc jI/rT54BQQdglWE5Nz1Uhu1QjKe+maTV6zH0j4N8OnTKK+f7TwrFz0InVV0FZMjahC0S 85L3dxqYjuCeG2p6Cf6ao/Vu57Qo7YvoZKilE2sXYuDnnsn3OK3FJPA9bZWYAX5o6O+K h3iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=s9wFMRb5nSXLWOxXSbco6K4+d90kJ7McKN32ouFL5Cw=; b=On6E2EAJdOGwkgVsZLqZAs5wREV41ReW5Fqv6pINIHNRkv1APBiiaxgMx9r9xHTVrF qePcn2qPcLaSnxu0QPuyB2dpb/ZOupFX9ndLN81zXr5sYjjvKzlmdYsi8zXtCUsaSUaK NxtyCZ0HIeiW9K+P7Zq+5ASve9CzovfPch1IrcJNmBSS6PR3492JjzTV8YIc5qzcxm+X jyyOzh61tKhZ9V2X6XumPz/eG8kjO74wZQ9gJ4GsSLanOT0MHJ6RZjx4y8/kxnQSf0YN JT7UIOGwPwF6/j7YgU2yCiqKc8Bjn5lvkvSgjcsqikNdOCsGNrkYrClvfA2HwnN4/H65 Mfpw== X-Gm-Message-State: ALyK8tLjYGaFn4fg4A4RzLheSqw5T/aKTklr34TdDzp2cZjrkSWuVm6NlujkdTbmLsiaalVEAAhxcTCtWgHwag== X-Received: by 10.13.217.20 with SMTP id b20mr115656ywe.44.1468028592896; Fri, 08 Jul 2016 18:43:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Fri, 8 Jul 2016 18:43:12 -0700 (PDT) From: David Cross Date: Fri, 8 Jul 2016 21:43:12 -0400 Message-ID: Subject: Re: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) FOUND IT, including reproduction steps To: Konstantin Belousov Cc: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 01:43:14 -0000 Ok... I found it. All of the writes go through ffs_write (including VOP_RECLAIM, so my statement that VOP_RECLAIM couldn't handle things that vinvalbuf left behind is obviously incorrect). Sometimes it worked, sometimes it paniced, I started putting more deugging into it and I noticed the following: The problem file would balloc twice as follows: attempting to balloc inode 18237205 softdep_setup_allocdirect(18237205, 1, 72834400, 0, 8192, 0, 0xfffffe00f76a6d88) balloc at 291337600, flags: 50000 attempting to balloc inode 18237205 softdep_setup_allocdirect(18237205, 0, 72834448, 0, 16384, 0, 0xfffffe00f7749970) balloc at 291337792, flags: 7f040080 panic: softdep_deallocate_dependencies: dangling deps Furthrer reading of ffs_write to figure out why it worked sometimes and not others pointed me at the IO_SYNC flag, if passed in ffs_write dispatches to bwrite.. which gives the panic, otherwise it goes to bawrite which does not. However the problem is in ufs_balloc, around line 778 (which I saw in the earlier newbuf dump); There NO call to any write method for that buffer. If we compare this to the other calls to softdep_setup_allocdirect in that function (lines: 148, 264, 708, 828) we see that each of them has some call to bwrite, bdwrite, bawrite following it (a number of the other calls do not make any direct calls to b*writes either, I do not know nearly enough to say if those are correct or incorrect; I tried adding bwrite arround those lines with a conditional on IO_SYNC and I only made it panic earlier. I just don't know what the semantics of this enough. That being said, I was finally able to isolate a set of reproduction steps that anyone can run. As it stands it relies on a set of filesystem options that are no longer standard (but were, not that long ago), but I definitely believe they could be trivially modified to work on *any* UFS1/UFS2 filesystem... To that extent I am NOT including them, I will reply individually with the exploit code an instructions to reproduce; if you want, and you have an appropriate commit history or other credentials I will forward it on. Thanks, and I eagerly look forward to the patch, or assisting where I can in development.