From owner-freebsd-questions@freebsd.org Sun Sep 25 23:59:20 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FCA5BEA09B for ; Sun, 25 Sep 2016 23:59:20 +0000 (UTC) (envelope-from chiller@driftfun.com) Received: from mail-ext5.uio.no (mail-ext5.uio.no [IPv6:2001:700:100:10::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F12810AF for ; Sun, 25 Sep 2016 23:59:19 +0000 (UTC) (envelope-from chiller@driftfun.com) Received: from mail-mx1.uio.no ([129.240.10.29]) by mail-out5.uio.no with esmtp (Exim 4.80.1) (envelope-from ) id 1boJKF-0008BR-Dc; Mon, 26 Sep 2016 01:59:15 +0200 Received: from putsch.kolbu.ws ([158.36.191.193]) by mail-mx1.uio.no with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.80) (envelope-from ) id 1boJKE-0002bp-IE; Mon, 26 Sep 2016 01:59:15 +0200 Received: from chiller by putsch.kolbu.ws with local (Exim 4.87 (FreeBSD)) (envelope-from ) id 1boJKD-0002kf-JI; Mon, 26 Sep 2016 01:59:13 +0200 Date: Mon, 26 Sep 2016 01:59:13 +0200 From: =?utf-8?B?U3TDpWxl?= Kristoffersen To: Anton Yuzhaninov Cc: "freebsd-questions@freebsd.org" Subject: Re: Server gets a high load, but no CPU use, and then later stops respond on the network Message-ID: <20160925235913.GA9921@putsch.kolbu.ws> References: <20160913232351.GA36091@putsch.kolbu.ws> <68f553b9-8546-7707-df86-88851b3283f8@citrin.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <68f553b9-8546-7707-df86-88851b3283f8@citrin.ru> User-Agent: Mutt/1.6.1 (2016-04-27) X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO) X-UiO-Scanned: F77DCDC39838E818C73FD147CB994ADCEFAFA42A X-UiO-SPAM-Test: remote_host: 158.36.191.193 spam_score: -49 maxlevel 80 minaction 2 bait 0 mail/h: 1 total 34866 max/h 377 blacklist 0 greylist 0 ratelimit 0 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Sep 2016 23:59:20 -0000 On 2016-09-20 at 16:57, Anton Yuzhaninov wrote: > In top output you should look at WCPU and STATE for kernel threads and > for unresponding network daemons. It had a problem last night, and beneath is the last output in the logfile. (Also available here: http://pastebin.com/G1t3ayPd ) Unfortunately it did not save anything after that, even if it did echo back text written in an already opened ssh connection. No errors were printed on the ipmi kvm console. 2016-09-25 02:27:00 last pid: 86980; load averages: 0.09, 0.07, 0.08 up 4+02:13:16 02:27:00 627 processes: 24 running, 565 sleeping, 38 waiting Mem: 4824K Active, 475M Inact, 30G Wired, 58M Cache, 801M Free ARC: 27G Total, 20G MFU, 5978M MRU, 232K Anon, 152M Header, 1026M Other Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 384K CPU13 13 97.0H 100.00% [idle{idle: cpu13}] 11 root 155 ki31 0K 384K CPU0 0 96.9H 100.00% [idle{idle: cpu0}] 11 root 155 ki31 0K 384K CPU7 7 96.7H 100.00% [idle{idle: cpu7}] 11 root 155 ki31 0K 384K CPU5 5 96.7H 100.00% [idle{idle: cpu5}] 11 root 155 ki31 0K 384K RUN 4 96.7H 100.00% [idle{idle: cpu4}] 11 root 155 ki31 0K 384K CPU3 3 96.7H 100.00% [idle{idle: cpu3}] 11 root 155 ki31 0K 384K CPU1 1 96.7H 100.00% [idle{idle: cpu1}] 11 root 155 ki31 0K 384K CPU9 9 96.7H 100.00% [idle{idle: cpu9}] 11 root 155 ki31 0K 384K CPU6 6 96.7H 100.00% [idle{idle: cpu6}] 11 root 155 ki31 0K 384K CPU10 10 96.7H 100.00% [idle{idle: cpu10}] 11 root 155 ki31 0K 384K CPU2 2 96.6H 100.00% [idle{idle: cpu2}] 11 root 155 ki31 0K 384K CPU17 17 96.6H 100.00% [idle{idle: cpu17}] 11 root 155 ki31 0K 384K CPU19 19 96.6H 100.00% [idle{idle: cpu19}] 11 root 155 ki31 0K 384K CPU16 16 96.6H 100.00% [idle{idle: cpu16}] 11 root 155 ki31 0K 384K CPU15 15 96.6H 100.00% [idle{idle: cpu15}] 11 root 155 ki31 0K 384K CPU18 18 96.6H 100.00% [idle{idle: cpu18}] 11 root 155 ki31 0K 384K CPU23 23 96.6H 100.00% [idle{idle: cpu23}] 11 root 155 ki31 0K 384K CPU12 12 96.6H 100.00% [idle{idle: cpu12}] 11 root 155 ki31 0K 384K CPU22 22 96.6H 100.00% [idle{idle: cpu22}] 11 root 155 ki31 0K 384K CPU11 11 96.2H 100.00% [idle{idle: cpu11}] 11 root 155 ki31 0K 384K CPU8 8 96.7H 99.27% [idle{idle: cpu8}] 11 root 155 ki31 0K 384K CPU21 21 96.6H 99.27% [idle{idle: cpu21}] 11 root 155 ki31 0K 384K CPU14 14 96.6H 99.27% [idle{idle: cpu14}] 11 root 155 ki31 0K 384K CPU20 20 96.6H 91.99% [idle{idle: cpu20}] 931 root 28 0 306M 14036K select 15 20:36 12.60% /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf{smbd} 0 root -92 - 0K 6512K - 11 62:27 0.10% [kernel{igb0 taskq}] 5 root -16 - 0K 16K ipmire 11 193:27 0.00% [ipmi0: kcs] 7 root -16 - 0K 32K psleep 0 39:15 0.00% [pagedaemon{pagedaemon}] 1807 root 20 0 298M 12796K select 0 17:12 0.00% /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf 771 plex 20 0 484M 288M uwait 7 11:15 0.00% /usr/local/share/plexmediaserver-plexpass/Plex_Media_Server (Plex Media Server){Plex Media Serve} 771 plex 20 0 484M 288M kqread 19 11:08 0.00% /usr/local/share/plexmediaserver-plexpass/Plex_Media_Server (Plex Media Server){Plex Media Serve} 17 root 16 - 0K 16K syncer 21 5:25 0.00% [syncer] 12 root -88 - 0K 608K WAIT 8 4:54 0.00% [intr{irq19: uhci2 uhc}] 2 root -16 - 0K 96K - 12 3:17 0.00% [cam{doneq0}] 14 root -16 - 0K 16K - 16 3:04 0.00% [rand_harvestq] 18 root -16 - 0K 16K vlruwt 17 2:19 0.00% [vnlru] 12 root -88 - 0K 608K WAIT 13 2:18 0.00% [intr{irq258: mps0}] 2 root -16 - 0K 96K - 10 2:18 0.00% [cam{doneq3}] 2 root -16 - 0K 96K - 15 2:17 0.00% [cam{doneq1}] 771 plex 20 0 484M 288M uwait 13 2:15 0.00% /usr/local/share/plexmediaserver-plexpass/Plex_Media_Server (Plex Media Server){Plex Media Serve} 0 root -16 - 0K 6512K swapin 8 1:48 0.00% [kernel{swapper}] 3 root -8 - 0K 176K tx->tx 22 1:32 0.00% [zfskern{txg_thread_enter}] 843 plex 35 15 339M 93664K select 12 1:31 0.00% [Plex Script Host{Plex Script Host}] 12 root -60 - 0K 608K WAIT 0 1:15 0.00% [intr{swi4: clock}] 2 root -16 - 0K 96K - 11 1:05 0.00% [cam{doneq2}] 0 root -16 - 0K 6512K - 13 1:04 0.00% [kernel{zio_write_intr_3}] 0 root -16 - 0K 6512K - 23 1:04 0.00% [kernel{zio_write_intr_2}] 0 root -16 - 0K 6512K - 17 1:04 0.00% [kernel{zio_write_intr_4}] 0 root -16 - 0K 6512K - 20 1:04 0.00% [kernel{zio_write_intr_0}] 0 root -16 - 0K 6512K - 3 1:04 0.00% [kernel{zio_write_intr_7}] 0 root -16 - 0K 6512K - 5 1:04 0.00% [kernel{zio_write_intr_5}] 0 root -16 - 0K 6512K - 19 1:04 0.00% [kernel{zio_write_intr_1}] 0 root -16 - 0K 6512K - 21 1:04 0.00% [kernel{zio_write_intr_6}] 0 root -16 - 0K 6512K - 3 1:02 0.00% [kernel{zio_read_intr_5}] 0 root -16 - 0K 6512K - 22 1:02 0.00% [kernel{zio_read_intr_11}] 0 root -16 - 0K 6512K - 11 1:02 0.00% [kernel{zio_read_intr_12}] 0 root -16 - 0K 6512K - 15 1:02 0.00% [kernel{zio_read_intr_6}] 0 root -16 - 0K 6512K - 6 1:02 0.00% [kernel{zio_read_intr_10}] 0 root -16 - 0K 6512K - 6 1:02 0.00% [kernel{zio_read_intr_17}] 0 root -16 - 0K 6512K - 23 1:02 0.00% [kernel{zio_read_intr_0}] 0 root -16 - 0K 6512K - 15 1:02 0.00% [kernel{zio_read_intr_2}] 0 root -16 - 0K 6512K - 6 1:02 0.00% [kernel{zio_read_intr_9}] 0 root -16 - 0K 6512K - 20 1:02 0.00% [kernel{zio_read_intr_13}] 0 root -16 - 0K 6512K - 4 1:02 0.00% [kernel{zio_read_intr_15}] 0 root -16 - 0K 6512K - 19 1:02 0.00% [kernel{zio_read_intr_1}] 0 root -16 - 0K 6512K - 7 1:02 0.00% [kernel{zio_read_intr_7}] 0 root -16 - 0K 6512K - 4 1:02 0.00% [kernel{zio_read_intr_8}] 0 root -16 - 0K 6512K - 8 1:02 0.00% [kernel{zio_read_intr_16}] 0 root -16 - 0K 6512K - 11 1:02 0.00% [kernel{zio_read_intr_4}] 0 root -16 - 0K 6512K - 3 1:02 0.00% [kernel{zio_read_intr_14}] 0 root -16 - 0K 6512K - 0 1:01 0.00% [kernel{zio_read_intr_3}] 746 root 20 0 14500K 2032K select 0 0:40 0.00% /usr/sbin/powerd -a hiadaptive 0 root -17 - 0K 6512K - 0 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 13 0:40 0.00% [kernel{zio_write_intr_3}] 0 root -17 - 0K 6512K - 11 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 19 0:40 0.00% [kernel{zio_write_intr_7}] 0 root -17 - 0K 6512K - 7 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 2 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 9 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 4 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 15 0:40 0.00% [kernel{zio_write_intr_1}] 0 root -17 - 0K 6512K - 2 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 18 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 1 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 13 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 14 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 0 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 13 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 21 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 9 0:40 0.00% [kernel{zio_write_intr_5}] 0 root -17 - 0K 6512K - 17 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 9 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 2 0:40 0.00% [kernel{zio_write_intr_4}] 0 root -17 - 0K 6512K - 7 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 11 0:40 0.00% [kernel{zio_write_intr_2}] 0 root -17 - 0K 6512K - 14 0:40 0.00% [kernel{zio_write_issue_}] 0 root -16 - 0K 6512K - 15 0:40 0.00% [kernel{zio_write_intr_6}] 0 root -16 - 0K 6512K - 0 0:40 0.00% [kernel{zio_write_intr_0}] 0 root -17 - 0K 6512K - 17 0:40 0.00% [kernel{zio_write_issue_}] 0 root -17 - 0K 6512K - 10 0:40 0.00% [kernel{zio_write_issue_}] ------------------------------------------------------------------------------- -- Ståle Kristoffersen