From owner-freebsd-bugs@freebsd.org Fri Jan 11 04:20:25 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 484C814A458A for ; Fri, 11 Jan 2019 04:20:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D5B3387382 for ; Fri, 11 Jan 2019 04:20:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 971C414A4589; Fri, 11 Jan 2019 04:20:24 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A58214A4588 for ; Fri, 11 Jan 2019 04:20:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E6FD08737E for ; Fri, 11 Jan 2019 04:20:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 2E7AA633E for ; Fri, 11 Jan 2019 04:20:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x0B4KNTx041707 for ; Fri, 11 Jan 2019 04:20:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x0B4KNA6041706 for bugs@FreeBSD.org; Fri, 11 Jan 2019 04:20:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234838] ena drop-outs on 12.0-RELEASE Date: Fri, 11 Jan 2019 04:20:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: leif@ofWilsonCreek.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2019 04:20:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234838 --- Comment #2 from Leif Pedersen --- Of course! Thanks for looking! It's our standby MySQL database, so there's a light but steady stream of network IO for the DB replication. Its filesystems are ZFS. Once an hour, s= ix other machines (developer sandboxes) pull the latest ZFS snapshot of the DB= (in parallel) so our developers can clone any recent hourly snapshot for testin= g. These messages happen about three times per day at the top of the hour when this hourly pull would be running. They happen sometimes at other odd times: once on Jan 4, and once on Dec 19, looking at logs going back to Dec 12. So= it probably isn't just the 6 concurrent `zfs send`s that cause it. Also at the top of the hour, it runs a mysqldump cron which takes ~10 minut= es. That has nothing to do with the network; just full disclosure that it incre= ases CPU load significantly. There are batch jobs that send a lot of transactions through the DB replica= tion stream, but they don't seem to correlate. I think those cause only minimal network IO but high CPU & disk load. The machine doesn't do anything else. Hopefully that's a helpful idea of the work load. It's the only machine I've upgraded to 12. Our other machines are running 11.1. This one is my canary;= I dare not upgrade the more important machines with this symptom. Also, occasionally it will panic with the message "Fatal double fault" and = no backtrace. I can get you more on that if you want. Sometimes this panic hap= pens when the machine is booting, before it has even configured the network. Related? I dunno. Just giving you anything that might be useful. --=20 You are receiving this mail because: You are the assignee for the bug.=