From owner-freebsd-fs@freebsd.org Fri Feb 19 11:07:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F24ABAAEF24 for ; Fri, 19 Feb 2016 11:07:03 +0000 (UTC) (envelope-from n.corvini@gmail.com) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C16FB1AF7 for ; Fri, 19 Feb 2016 11:07:03 +0000 (UTC) (envelope-from n.corvini@gmail.com) Received: by mail-ig0-x22f.google.com with SMTP id y8so36060840igp.0 for ; Fri, 19 Feb 2016 03:07:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to; bh=QKy5xG3ZUvwvmCPriA0DxQxSlh28x8FsNYUlFDH5SFw=; b=n/CLQgXo70cPDuFW5r+l9AUoyxPzORXONSCh6Vn+WN+TZmIDvXu7TM/h0g/oTr8ZaZ VoZBkd/an4K7UDtKoBZ6GUwK3xJ0N5QO4kbSzEfdXCEPnLVGVeEA755WfRdzoHrBPkPS RzmwgLMQPntrHYY7rfVojz89OflLWunzoBS68sC3WqVeFQVXl2qu6FhDbcKwG5Y6Tz1K ravDX7T5zCtqw77L0ixmVHScuJ0koGiMWIgE5+800+pzRNBWCX86NcfcXD5rcrdcUNfM IByrt7PAqrO3G/xVNg63Jgx5nLi2n0IMHs5l6eo5ROddXfscpHylEDJFneKRfpBoECUk aTdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=QKy5xG3ZUvwvmCPriA0DxQxSlh28x8FsNYUlFDH5SFw=; b=hVun1X5qE6Z32r4kbuFiP0UcKT8rpBO0m7qbFCmMf4z95vhn6yPKn7COElYqzbAMsh W1uDf+Z/Z7MiZKkV2hYZBCdJKPkMp86emcSJYWLil1lS3pcXwUAqLRRPqRQdFC6pBqJQ y+CGWtv/GG1m5WyIfNiPDAE7uouSOrPQGIg+lTy+P/2eputK4qY0q4dGd+RnXiQAOsQK jZdyivc63TqEJBAqCToWRo2aKRYb9fvkTfd2xfebsm9usy12nZs4k7u2lIEaZm4tYcjI sXYyk4L8P2Xm/ioGGU+il86JTZaKt4X6dnzGXTIz95OK+n/YK/u69IJjMvSD9e9PwQs/ qAZA== X-Gm-Message-State: AG10YOQ2QbGaABBllgwCLO5CCJb331gQOi68Z0YyiqqE3ExPGH4q3hPn40B/JxtQoXuvQEygiK1K/MVG3HpTpA== MIME-Version: 1.0 X-Received: by 10.50.131.201 with SMTP id oo9mr8345808igb.68.1455880023205; Fri, 19 Feb 2016 03:07:03 -0800 (PST) Received: by 10.107.136.166 with HTTP; Fri, 19 Feb 2016 03:07:03 -0800 (PST) Date: Fri, 19 Feb 2016 12:07:03 +0100 Message-ID: Subject: Zfs heavy io writing | zfskern txg_thread_enter From: =?UTF-8?Q?Niccol=C3=B2_Corvini?= To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2016 11:07:04 -0000 Hi, first time here! We are having a problem with a server running FreeBsd 9.1 with ZFS on a single sata drive. Since a few days ago, in the morning the system becomes really slow due of a really heavy io writing. We investigated and we think it might start at night, maybe correlated to to crondaily (standard) but we are not sure. After a few hours the situation returns to normal. Any help is much appreciated The machine is a Intel Xeon E5-2620 with 36GB of RAM, the HDD is a 2TB an is half full. gstat output: L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 13 135 21 641 256.7 108 6410 41.4 128.8| ada0 0 0 0 0 0.0 0 0 0.0 0.0| ada0p1 13 135 21 641 256.7 108 6410 41.7 128.8| ada0p2 0 0 0 0 0.0 0 0 0.0 0.0| cd0 0 0 0 0 0.0 0 0 0.0 0.0| gptid/3c0de011-4f37-11e5-8217-3085a91c3292 0 0 0 0 0.0 0 0 0.0 0.0| zvol/zroot/swap 13 135 21 641 256.7 108 6410 41.7 128.9| gpt/disk1 Using top -m io shows that the responsible is [zfskern{txg_thread_enter}] top -m io output: PID JID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 3 0 root 14 1 0 37 0 37 30.33% [zfskern{txg_thread_enter}] 49866 215 7070 26 2 0 5 0 5 4.10% postgres: stats collector process (postgres) 99901 5 70 42 0 0 4 0 4 3.28% postgres: promeditec promeditec.osr.test 192.168.0.246(278 24820 199 www 10 0 7 0 0 7 5.74% [jsvc{jsvc}] 33869 212 88 19 2 0 2 0 2 1.64% [mysqld{mysqld}] 93400 0 root 13 0 10 0 0 10 8.20% [find] 89407 215 7070 10 0 0 1 0 1 0.82% postgres: alfresco alfconservazione.dotcom.ts.it 192.168.0 15776 5 70 11 0 0 4 0 4 3.28% postgres: stats collector process (postgres) 33869 212 88 10 0 0 3 0 3 2.46% [mysqld{mysqld}] 33869 212 88 2 0 0 11 0 11 9.02% [mysqld{mysqld}] 18685 198 root 5 0 0 2 0 2 1.64% /usr/sbin/syslogd -s 15852 214 70 4 1 0 1 0 1 0.82% postgres: alfresco alfcomunets.dotcom.ts.it 192.168.0.212( 98335 120 root 11 0 29 0 0 29 23.77% find /var/log -name messages.* -mtime -2 16128 214 70 8 0 0 1 0 1 0.82% postgres: alfresco alfaxErre8 192.168.0.208(50558) (postg 1116 198 root 10 0 0 1 0 1 0.82% sendmail: ./u1J9k90d001112 local: client DATA status (send 1120 198 root 7 0 0 4 0 4 3.28% mail.local -l Using procstat -kk on the zfskern pid shows: PID TID COMM TDNAME KSTACK 3 100129 zfskern arc_reclaim_thre mi_switch sleepq_timedwait _cv_timedwait arc_reclaim_thread fork_exit fork_trampoline 3 100130 zfskern l2arc_feed_threa mi_switch sleepq_timedwait _cv_timedwait l2arc_feed_thread fork_exit fork_trampoline 3 100504 zfskern txg_thread_enter mi_switch sleepq_wait _cv_wait txg_thread_wait txg_quiesce_thread fork_exit fork_trampoline 3 100505 zfskern txg_thread_enter mi_switch sleepq_wait _cv_wait zio_wait dsl_pool_sync spa_sync txg_sync_thread fork_exit fork_trampoline 3 100506 zfskern zvol zroot/swap mi_switch sleepq_wait _sleep zvol_geom_worker fork_exit fork_trampoline systat -vmstat 7 users Load 0.50 0.62 1.46 Feb 19 11:46 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 41318k 199492 1251183k 588716 2254748 count All 46844k 229048 -1968M 892088 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 2836 total 4k 8516 393 12k 236 1445 11k 11281 zfod atkbd0 1 ozfod acpi0 9 4.4%Sys 0.0%Intr 0.3%User 0.0%Nice 95.4%Idle %ozfod ehci0 17 | | | | | | | | | | | daefr ehci1 23 == 11171 prcfr 79 cpu0:timer dtbuf 11265 totfr isci0 264 Namei Name-cache Dir-cache 1095774 desvn react 24 em0:rx 0 Calls hits % hits % 409282 numvn pdwak 16 em0:tx 0 78 41 53 273943 frevn pdpgs em0:link intrn 196 ahci0 278 Disks ada0 cd0 pass0 pass1 20445132 wire 267 cpu21:time KB/t 2.55 0.00 0.00 0.00 37317552 act 55 cpu13:time tps 223 0 0 0 4948708 inact 86 cpu5:timer MB/s 0.56 0.00 0.00 0.00 884804 cache 24 cpu12:time %busy 94 0 0 0 1370012 free 63 cpu10:time buf 306 cpu19:time 63 cpu11:time 55 cpu14:time 86 cpu9:timer 71 cpu18:time 86 cpu3:timer 47 cpu23:time 55 cpu6:timer 55 cpu22:time 71 cpu2:timer 39 cpu20:time zpool status: pool: zroot state: ONLINE scan: scrub repaired 0 in 3h46m with 0 errors on Wed Nov 4 21:54:44 2015 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors