From owner-freebsd-fs@freebsd.org Sun Mar 21 21:00:09 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2A81F5AC530 for ; Sun, 21 Mar 2021 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F3VQ10JbYz4QyV for ; Sun, 21 Mar 2021 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 07E105AC685; Sun, 21 Mar 2021 21:00:09 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0782A5AC52F for ; Sun, 21 Mar 2021 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F3VQ05rwLz3wXn for ; Sun, 21 Mar 2021 21:00:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B921A214F9 for ; Sun, 21 Mar 2021 21:00:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12LL08jE073760 for ; Sun, 21 Mar 2021 21:00:08 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12LL08xB073759 for fs@FreeBSD.org; Sun, 21 Mar 2021 21:00:08 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202103212100.12LL08xB073759@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 21 Mar 2021 21:00:08 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Mar 2021 21:00:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 244899 | [PATCH] zfs: xattr on a symlink target > 136 caus Open | 251035 | [zfs] Allow 64 bit ZFS to support 32 bit ioctls ( 11 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Mar 22 10:17:46 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E53D05A8B50; Mon, 22 Mar 2021 10:17:46 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F3r6K5GXqz58C1; Mon, 22 Mar 2021 10:17:44 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay11.mail.gandi.net (Postfix) with ESMTPSA id E87CD100003; Mon, 22 Mar 2021 10:17:42 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 97AFB7203B9D5; Mon, 22 Mar 2021 11:17:42 +0100 (CET) Date: Mon, 22 Mar 2021 11:17:42 +0100 From: Mathieu Chouquet-Stringer To: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: 4F3r6K5GXqz58C1 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of mchouque@thi.eu.com has no SPF policy when checking 217.70.178.231) smtp.mailfrom=mchouque@thi.eu.com X-Spamd-Result: default: False [-1.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[me@mathieu.digital,mchouque@thi.eu.com]; RCVD_IN_DNSWL_LOW(-0.10)[217.70.178.231:from]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.70.178.231:from]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR]; FROM_NEQ_ENVFROM(0.00)[me@mathieu.digital,mchouque@thi.eu.com]; TAGGED_FROM(0.00)[freebsd]; RWL_MAILSPIKE_NEUTRAL(0.00)[217.70.178.231:from]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[mathieu.digital]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[217.70.178.231:from:127.0.2.255]; R_SPF_NA(0.00)[no SPF record]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current,freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Mar 2021 10:17:47 -0000 Hello, I'm running vanilla 13.0-RC3 at the moment but experienced the same issue with RC2 and RC1. This instance of FreeBSD is running in a Linux kvm and it has 16 threads and 32GB of RAM: hw.model: Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS) hw.machine: amd64 hw.ncpu: 16 hw.physmem: 34318278656 I have 2 zpools, 1 is for the root filesystems the other one is just my storage array. NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 29T 24.5T 4.54T - - 11% 84% 1.00x ONLINE - zroot 11.9G 8.76G 3.18G - - 62% 73% 1.00x ONLINE - Zroot is on vtbd0 device backed by ssd and I used to be able to scrub it reaching SATA2 bandwidth limits, close to 300MB/s. Now when I scrub it, I get half that at best. But that's not the worst part as it's really small so it's still quick enough. The worst part is my data zpool. It's raidz2 array with 8 devices, a log mirror on SSDs and 2 caches devices on SSD. The sas/sata controllers (2 x SAS3008 in IT mode) are dedicated to FreeBSD so it has full control of them (through vfio pci). pool: data state: ONLINE scan: scrub in progress since Sun Mar 21 23:47:40 2021 2.82T scanned at 77.8M/s, 2.24T issued at 62.0M/s, 24.5T total 0B repaired, 9.17% done, 4 days 08:23:32 to go config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da7 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da6 ONLINE 0 0 0 da9 ONLINE 0 0 0 da2 ONLINE 0 0 0 da10 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 da5p1 ONLINE 0 0 0 cache da0p2 ONLINE 0 0 0 da5p2 ONLINE 0 0 0 spares da8 AVAIL I used to be able to scrub the whole pool in a day or so but not anymore. I started scrubbing it almost 11 hours ago and I've only done 10% of it. iostat shows devices aren't that busy: extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b vtbd0 9 2 264.1 14.3 0 0 2 0 0 0 da0 0 71 10.9 9106.4 0 0 0 0 0 3 da1 1443 29 8632.4 1095.7 0 0 79 0 0 34 da2 1442 29 8631.3 1095.7 0 0 82 0 0 31 da3 1434 28 8637.7 1096.1 0 0 84 0 1 36 da4 1434 29 8631.8 1095.7 0 0 79 0 0 33 da5 0 71 11.7 9130.4 0 0 0 0 0 3 da6 1438 29 8634.1 1095.6 0 0 82 0 0 31 da7 1441 29 8636.2 1095.6 0 0 84 0 1 33 da8 0 0 0.0 0.0 1 2 0 1 0 0 da9 1441 29 8630.8 1095.8 0 0 81 0 0 32 da10 1438 29 8633.7 1095.5 0 0 82 0 0 32 pass1 0 0 0.0 0.0 0 0 0 0 0 0 pass2 0 0 0.0 0.0 43 17 36 41 0 0 pass3 0 0 0.0 0.0 43 15 35 40 0 0 pass4 0 0 0.0 0.0 44 12 42 43 0 0 pass5 0 0 0.0 0.0 45 13 32 41 0 0 pass6 0 0 0.0 0.0 0 0 0 0 0 0 pass7 0 0 0.0 0.0 47 23 44 46 0 0 pass8 0 0 0.0 0.0 44 15 35 42 0 0 pass9 0 0 0.0 0.0 33 0 198 71 0 0 pass10 0 0 0.0 0.0 43 16 33 40 0 0 pass11 0 0 0.0 0.0 41 9 32 39 0 0 Here's what I have in my sysctl.conf: vfs.zfs.l2arc_write_max=314572800 vfs.zfs.min_auto_ashift=12 vfs.zfs.vdev.async_read_max_active=10 I tried playing with scan_mem_lim_fact (I went through the whole openzfs doc yesterday) but it didn't make any difference: vfs.zfs.scan_mem_lim_fact=5 With or without vfs.zfs.scan_mem_lim_fact, I often see I have 1 and sometimes 2 threads pegged at 100% of CPU in kernel mode... 1[|| 6.8%] 5[|| 8.8%] 9[| 4.7%] 13[|| 9.4%] 2[|| 8.3%] 6[| 4.2%] 10[|| 8.9%] 14[| 5.7%] 3[|| 8.4%] 7[|||||||||100.0%] 11[||| 14.1%] 15[| 5.2%] 4[| 5.7%] 8[|| 9.4%] 12[|| 13.0%] 16[| 4.2%] Mem[||||||||||||||||||||||||||9.33G/32.0G] Tasks: 45, 0 thr; 2 running Swp[| 130M/8.00G] Load average: 2.51 2.63 2.46 Uptime: 10:47:36 PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command 0 root -16 0 0 12976 S 22.5 0.0 9h45:36 kernel 1 root 24 0 11824 32 S 0.0 0.0 0:00.36 /sbin/init 3453 root 20 0 17668 2224 S 0.0 0.0 0:00.48 /usr/local/sbin/smartd -A /usr/local/ 3974 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv0 4973 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv1 5618 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv2 5974 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv3 6320 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv4 6386 root 20 0 39188 2932 S 0.0 0.0 0:00.65 /usr/local/sbin/nmbd --daemon --confi Any idea how I could help debug that? Cheers, -- Mathieu Chouquet-Stringer me+freebsd@mathieu.digital The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Mon Mar 22 15:15:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2C7CA5B2386 for ; Mon, 22 Mar 2021 15:15:29 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mail.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 4F3yjr1yQnz3px8 for ; Mon, 22 Mar 2021 15:15:27 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 12MFFIUc058103; Mon, 22 Mar 2021 15:15:18 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 12MFFHva015191; Mon, 22 Mar 2021 15:15:17 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 12MFFHRK015188; Mon, 22 Mar 2021 15:15:17 GMT Date: Mon, 22 Mar 2021 15:15:17 GMT Message-Id: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> From: Martin Simmons To: Mathieu Chouquet-Stringer CC: freebsd-fs@freebsd.org In-reply-to: (message from Mathieu Chouquet-Stringer on Mon, 22 Mar 2021 11:17:42 +0100) Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) References: X-Rspamd-Queue-Id: 4F3yjr1yQnz3px8 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[46.17.166.21:from]; FREEFALL_USER(0.00)[martin]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[46.17.166.21:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.996]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Mar 2021 15:15:29 -0000 >>>>> On Mon, 22 Mar 2021 11:17:42 +0100, Mathieu Chouquet-Stringer said: > > PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command > 0 root -16 0 0 12976 S 22.5 0.0 9h45:36 kernel > 1 root 24 0 11824 32 S 0.0 0.0 0:00.36 /sbin/init > 3453 root 20 0 17668 2224 S 0.0 0.0 0:00.48 /usr/local/sbin/smartd -A /usr/local/ > 3974 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv0 > 4973 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv1 > 5618 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv2 > 5974 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv3 > 6320 root 52 0 12892 48 S 0.0 0.0 0:00.00 /usr/libexec/getty Pc ttyv4 > 6386 root 20 0 39188 2932 S 0.0 0.0 0:00.65 /usr/local/sbin/nmbd --daemon --confi > > Any idea how I could help debug that? You could try: procstat -kk 0 to get an overview of what the kernel threads are doing. __Martin From owner-freebsd-fs@freebsd.org Mon Mar 22 15:39:32 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B61665B2A6C for ; Mon, 22 Mar 2021 15:39:32 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F3zFb40y4z3qmY for ; Mon, 22 Mar 2021 15:39:31 +0000 (UTC) (envelope-from mchouque@thi.eu.com) X-Originating-IP: 62.210.143.248 Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 28B23C000D; Mon, 22 Mar 2021 15:39:27 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id C58977203B9D5; Mon, 22 Mar 2021 16:39:27 +0100 (CET) Date: Mon, 22 Mar 2021 16:39:27 +0100 From: Mathieu Chouquet-Stringer To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Mar 2021 15:39:32 -0000 Hello Martin, Thanks for taking the time to reply to me. On Mon, Mar 22, 2021 at 03:15:17PM +0000, Martin Simmons wrote: > You could try: > > procstat -kk 0 > > to get an overview of what the kernel threads are doing. If I grep -v sleep, here's what I see most when running procstat: PID TID COMM TDNAME KSTACK 0 101321 kernel z_rd_int_2 nanouptime+0x43 vdev_queue_io_done+0x175 zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe But I had to run it a couple of times to actually see that: if I loop with the following one liner, I get mostly nothing... while :; do date; sudo procstat -kk 0 | grep -v sleep; echo; done I ran a different a similar loop for 3 minutes and got the following (note timestamp comes after the output): 0 101274 kernel z_rd_int_3 binuptime+0x48 xpt_run_devq+0x41b xpt_action_default+0x7de dastart+0x2ef xpt_run_allocq+0x83 dastrategy+0x82 g_disk_start+0x2f0 g_io_request+0x1d9 zio_vdev_io_start+0x204 zio_execute+0x3c vdev_queue_io_done+0x212 zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe 0 101386 kernel z_wr_int_2 _sx_xlock_hard+0x189 vdev_queue_io_done+0x28c zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:32:09 CET 2021 0 101374 kernel z_wr_int_2 _sx_xlock_hard+0x189 vdev_queue_io_done+0x28c zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:32:50 CET 2021 0 101281 kernel z_rd_int_10 nanouptime+0x43 spa_taskq_dispatch_ent+0x53 zio_done+0xe3a zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:33:22 CET 2021 0 100469 kernel z_wr_int_11 nanouptime+0x43 spa_taskq_dispatch_ent+0x53 zio_done+0xe3a zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe 0 100487 kernel z_wr_int_5 nanouptime+0x43 kstat_runq_exit+0x15 vdev_queue_io_done+0x11a zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:34:07 CET 2021 0 100425 kernel z_wr_iss_8 vdev_mirror_io_start+0xb1 zio_vdev_io_start+0x22f zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:34:58 CET 2021 0 101424 kernel z_wr_int_4 _sx_xlock_hard+0x189 vdev_queue_io_done+0x28c zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe Mon Mar 22 16:35:51 CET 2021 Here's the complete output though: PID TID COMM TDNAME KSTACK 0 100000 kernel swapper mi_switch+0xc1 sleepq_timedwait+0x2f _sleep+0x1ab swapper+0x69 btext+0x2c 0 100019 kernel softirq_0 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100020 kernel softirq_1 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100021 kernel softirq_2 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100022 kernel softirq_3 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100023 kernel softirq_4 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100024 kernel softirq_5 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100025 kernel softirq_6 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100026 kernel softirq_7 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100027 kernel softirq_8 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100028 kernel softirq_9 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100029 kernel softirq_10 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100030 kernel softirq_11 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100031 kernel softirq_12 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100032 kernel softirq_13 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100033 kernel softirq_14 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100034 kernel softirq_15 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100035 kernel if_io_tqg_0 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100036 kernel if_io_tqg_1 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100037 kernel if_io_tqg_2 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100038 kernel if_io_tqg_3 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100039 kernel if_io_tqg_4 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100040 kernel if_io_tqg_5 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100041 kernel if_io_tqg_6 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100042 kernel if_io_tqg_7 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100043 kernel if_io_tqg_8 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100044 kernel if_io_tqg_9 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100045 kernel if_io_tqg_10 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100046 kernel if_io_tqg_11 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100047 kernel if_io_tqg_12 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100048 kernel if_io_tqg_13 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100049 kernel if_io_tqg_14 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100050 kernel if_io_tqg_15 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100051 kernel if_config_tqg_0 mi_switch+0xc1 msleep_spin_sbt+0x194 gtaskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100053 kernel thread taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100054 kernel in6m_free taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100055 kernel inm_free taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100057 kernel linuxkpi_irq_wq mi_switch+0xc1 msleep_spin_sbt+0x194 taskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100058 kernel kqueue_ctx taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100059 kernel aiod_kick taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100061 kernel linuxkpi_short_wq_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100062 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100063 kernel linuxkpi_short_wq_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100064 kernel linuxkpi_short_wq_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100065 kernel linuxkpi_short_wq_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100066 kernel linuxkpi_short_wq_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100067 kernel linuxkpi_short_wq_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100068 kernel linuxkpi_short_wq_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100069 kernel linuxkpi_short_wq_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100070 kernel linuxkpi_short_wq_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100071 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100072 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100073 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100074 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100075 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100076 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100077 kernel linuxkpi_short_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100078 kernel linuxkpi_long_wq_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100079 kernel linuxkpi_long_wq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100080 kernel linuxkpi_long_wq_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100081 kernel linuxkpi_long_wq_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100082 kernel linuxkpi_long_wq_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100083 kernel linuxkpi_long_wq_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100084 kernel linuxkpi_long_wq_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100085 kernel linuxkpi_long_wq_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100086 kernel linuxkpi_long_wq_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100087 kernel linuxkpi_long_wq_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100088 kernel linuxkpi_long_wq_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100089 kernel linuxkpi_long_wq_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100090 kernel linuxkpi_long_wq_12 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100091 kernel linuxkpi_long_wq_13 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100092 kernel linuxkpi_long_wq_14 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100093 kernel linuxkpi_long_wq_15 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100094 kernel linuxkpi_long_wq_16 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100144 kernel firmware taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100148 kernel crypto_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100149 kernel crypto_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100150 kernel crypto_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100151 kernel crypto_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100152 kernel crypto_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100153 kernel crypto_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100154 kernel crypto_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100155 kernel crypto_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100156 kernel crypto_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100157 kernel crypto_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100158 kernel crypto_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100159 kernel crypto_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100160 kernel crypto_12 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100161 kernel crypto_13 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100162 kernel crypto_14 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100163 kernel crypto_15 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100190 kernel vtnet0 rxq 0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100191 kernel vtnet0 txq 0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100200 kernel virtio_balloon mi_switch+0xc1 _sleep+0x1cb vtballoon_thread+0xbd fork_exit+0x7e fork_trampoline+0xe 0 100201 kernel mpr0 taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100203 kernel mpr1 taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100210 kernel vtnet1 rxq 0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100211 kernel vtnet1 txq 0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100215 kernel mca taskq mi_switch+0xc1 msleep_spin_sbt+0x194 taskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100216 kernel system_taskq_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100217 kernel system_taskq_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100218 kernel system_taskq_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100219 kernel system_taskq_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100220 kernel system_taskq_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100221 kernel system_taskq_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100222 kernel system_taskq_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100223 kernel system_taskq_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100224 kernel system_taskq_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100225 kernel system_taskq_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100226 kernel system_taskq_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100227 kernel system_taskq_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100228 kernel system_taskq_12 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100229 kernel system_taskq_13 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100230 kernel system_taskq_14 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100231 kernel system_taskq_15 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100232 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100233 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100234 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100235 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100236 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100237 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100238 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100239 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100240 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100241 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100242 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100243 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100244 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100245 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100246 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100247 kernel system_delay_taskq_ mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100252 kernel arc_prune_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100253 kernel arc_prune_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100254 kernel arc_prune_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100255 kernel arc_prune_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100256 kernel arc_prune_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100257 kernel arc_prune_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100258 kernel arc_prune_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100259 kernel arc_prune_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100260 kernel arc_prune_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100261 kernel arc_prune_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100262 kernel arc_prune_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100263 kernel arc_prune_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100264 kernel arc_prune_12 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100265 kernel arc_prune_13 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100266 kernel arc_prune_14 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100267 kernel arc_prune_15 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100292 kernel acpi_task_0 mi_switch+0xc1 msleep_spin_sbt+0x194 taskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100293 kernel acpi_task_1 mi_switch+0xc1 msleep_spin_sbt+0x194 taskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100294 kernel acpi_task_2 mi_switch+0xc1 msleep_spin_sbt+0x194 taskqueue_thread_loop+0x107 fork_exit+0x7e fork_trampoline+0xe 0 100295 kernel CAM taskq mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100311 kernel z_null_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100312 kernel z_null_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100313 kernel z_rd_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100314 kernel z_rd_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100315 kernel z_rd_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100316 kernel z_rd_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100317 kernel z_rd_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100318 kernel z_rd_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100319 kernel z_rd_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100320 kernel z_rd_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100321 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100322 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100323 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100324 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100325 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100326 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100327 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100328 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100329 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100330 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100331 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100332 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100333 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100334 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100335 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100336 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100337 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100338 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100339 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100340 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100341 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100342 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100343 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100344 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100345 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100346 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100347 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100348 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100349 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100350 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100351 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100352 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100353 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100354 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100355 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100356 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100357 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100358 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100359 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100360 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100361 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100362 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100363 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100364 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100365 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100366 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100367 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100368 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100369 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100370 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100371 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100372 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100373 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100374 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100375 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100376 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100377 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100378 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100379 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100380 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100381 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100382 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100383 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100384 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100385 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100386 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100387 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100388 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100389 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100390 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100391 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100392 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100393 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100394 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100395 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100396 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100397 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100398 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100399 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100400 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100401 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100402 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100403 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100404 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100405 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100406 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100407 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100408 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100409 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100410 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100411 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100412 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100413 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100414 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100415 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100416 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100417 kernel z_wr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100418 kernel z_wr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100419 kernel z_wr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100420 kernel z_wr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100421 kernel z_wr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100422 kernel z_wr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100423 kernel z_wr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100424 kernel z_wr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100425 kernel z_wr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100426 kernel z_wr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100427 kernel z_wr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100428 kernel z_wr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100429 kernel z_wr_iss_h_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100430 kernel z_wr_iss_h_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100431 kernel z_wr_iss_h_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100432 kernel z_wr_iss_h_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100433 kernel z_wr_iss_h_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100434 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100435 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100436 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100437 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100438 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100439 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100440 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100441 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100442 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100443 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100444 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100445 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100446 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100447 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100448 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100449 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100450 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100451 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100452 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100453 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100454 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100455 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100456 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100457 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100458 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100459 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100460 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100461 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100462 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100463 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100464 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100465 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100466 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100467 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100468 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100469 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100470 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100471 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100472 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100473 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100474 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100475 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100476 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100477 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100478 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100479 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100480 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100481 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100482 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100483 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100484 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100485 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100486 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100487 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100488 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100489 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100490 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100491 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100492 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100493 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100494 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100495 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100496 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100497 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100498 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100499 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100500 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100501 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100502 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100503 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100504 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100505 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100506 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100507 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100508 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100509 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100510 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100511 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100512 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100513 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100514 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100515 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100516 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100517 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100518 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100519 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100520 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100521 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100522 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100523 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100524 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100525 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100526 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100527 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100528 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100529 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100530 kernel z_wr_int_h_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100531 kernel z_wr_int_h_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100532 kernel z_wr_int_h_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100533 kernel z_wr_int_h_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100534 kernel z_wr_int_h_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100535 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100536 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100537 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100538 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100539 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100540 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100541 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100542 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100543 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100544 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100545 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100546 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100547 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100548 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100549 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100550 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100551 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100552 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100553 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100554 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100555 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100556 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100557 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100558 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100559 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100560 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100561 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100562 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100563 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100564 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100565 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100566 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100567 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100568 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100569 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100570 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100571 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100572 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100573 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100574 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100575 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100576 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100577 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100578 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100579 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100580 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100581 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100582 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100583 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100584 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100585 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100586 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100587 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100588 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100589 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100590 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100591 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100592 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100593 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100594 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100595 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100596 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100597 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100598 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100599 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100600 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100601 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100602 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100603 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100604 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100605 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100606 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100607 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100608 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100609 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100610 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100611 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100612 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100613 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100614 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100615 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100616 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100617 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100618 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100619 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100620 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100621 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100622 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100623 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100624 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100625 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100626 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100627 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100628 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100629 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100630 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100631 kernel z_fr_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100632 kernel z_cl_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100633 kernel z_cl_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100634 kernel z_ioctl_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100635 kernel z_ioctl_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100636 kernel z_trim_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100637 kernel z_trim_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100638 kernel z_trim_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100639 kernel z_trim_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 100640 kernel z_trim_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101237 kernel z_null_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101238 kernel z_null_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101239 kernel z_rd_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101240 kernel z_rd_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101241 kernel z_rd_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101242 kernel z_rd_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101243 kernel z_rd_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101244 kernel z_rd_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101245 kernel z_rd_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101246 kernel z_rd_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101247 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101248 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101249 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101250 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101251 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101252 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101253 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101254 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101255 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101256 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101257 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101258 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101259 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101260 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101261 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101262 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101263 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101264 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101265 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101266 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101267 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101268 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101269 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101270 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101271 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101272 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101273 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101274 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101275 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101276 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101277 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101278 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101279 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101280 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101281 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101282 kernel z_rd_int_11 nanouptime+0x43 vdev_queue_io_done+0x198 zio_vdev_io_done+0x14c zio_execute+0x3c taskqueue_run_locked+0x181 taskqueue_thread_loop+0xac fork_exit+0x7e fork_trampoline+0xe 0 101283 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101284 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101285 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101286 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101287 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101288 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101289 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101290 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101291 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101292 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101293 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101294 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101295 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101296 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101297 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101298 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101299 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101300 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101301 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101302 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101303 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101304 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101305 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101306 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101307 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101308 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101309 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101310 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101311 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101312 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101313 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101314 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101315 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101316 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101317 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101318 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101319 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101320 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101321 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101322 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101323 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101324 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101325 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101326 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101327 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101328 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101329 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101330 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101331 kernel z_rd_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101332 kernel z_rd_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101333 kernel z_rd_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101334 kernel z_rd_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101335 kernel z_rd_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101336 kernel z_rd_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101337 kernel z_rd_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101338 kernel z_rd_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101339 kernel z_rd_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101340 kernel z_rd_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101341 kernel z_rd_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101342 kernel z_rd_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101343 kernel z_wr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101344 kernel z_wr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101345 kernel z_wr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101346 kernel z_wr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101347 kernel z_wr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101348 kernel z_wr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101349 kernel z_wr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101350 kernel z_wr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101351 kernel z_wr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101352 kernel z_wr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101353 kernel z_wr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101354 kernel z_wr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101355 kernel z_wr_iss_h_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101356 kernel z_wr_iss_h_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101357 kernel z_wr_iss_h_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101358 kernel z_wr_iss_h_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101359 kernel z_wr_iss_h_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101360 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101361 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101362 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101363 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101364 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101365 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101366 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101367 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101368 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101369 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101370 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101371 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101372 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101373 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101374 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101375 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101376 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101377 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101378 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101379 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101380 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101381 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101382 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101383 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101384 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101385 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101386 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101387 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101388 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101389 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101390 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101391 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101392 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101393 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101394 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101395 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101396 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101397 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101398 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101399 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101400 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101401 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101402 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101403 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101404 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101405 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101406 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101407 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101408 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101409 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101410 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101411 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101412 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101413 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101414 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101415 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101416 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101417 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101418 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101419 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101420 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101421 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101422 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101423 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101424 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101425 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101426 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101427 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101428 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101429 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101430 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101431 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101432 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101433 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101434 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101435 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101436 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101437 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101438 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101439 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101440 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101441 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101442 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101443 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101444 kernel z_wr_int_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101445 kernel z_wr_int_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101446 kernel z_wr_int_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101447 kernel z_wr_int_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101448 kernel z_wr_int_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101449 kernel z_wr_int_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101450 kernel z_wr_int_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101451 kernel z_wr_int_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101452 kernel z_wr_int_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101453 kernel z_wr_int_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101454 kernel z_wr_int_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101455 kernel z_wr_int_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101456 kernel z_wr_int_h_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101457 kernel z_wr_int_h_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101458 kernel z_wr_int_h_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101459 kernel z_wr_int_h_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101460 kernel z_wr_int_h_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101461 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101462 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101463 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101464 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101465 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101466 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101467 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101468 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101469 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101470 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101471 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101472 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101473 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101474 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101475 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101476 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101477 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101478 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101479 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101480 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101481 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101482 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101483 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101484 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101485 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101486 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101487 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101488 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101489 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101490 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101491 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101492 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101493 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101494 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101495 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101496 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101497 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101498 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101499 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101500 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101501 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101502 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101503 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101504 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101505 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101506 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101507 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101508 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101509 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101510 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101511 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101512 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101513 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101514 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101515 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101516 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101517 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101518 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101519 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101520 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101521 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101522 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101523 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101524 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101525 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101526 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101527 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101528 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101529 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101530 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101531 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101532 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101533 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101534 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101535 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101536 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101537 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101538 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101539 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101540 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101541 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101542 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101543 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101544 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101545 kernel z_fr_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101546 kernel z_fr_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101547 kernel z_fr_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101548 kernel z_fr_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101549 kernel z_fr_iss_4 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101550 kernel z_fr_iss_5 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101551 kernel z_fr_iss_6 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101552 kernel z_fr_iss_7 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101553 kernel z_fr_iss_8 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101554 kernel z_fr_iss_9 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101555 kernel z_fr_iss_10 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101556 kernel z_fr_iss_11 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101557 kernel z_fr_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101558 kernel z_cl_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101559 kernel z_cl_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101560 kernel z_ioctl_iss mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101561 kernel z_ioctl_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101562 kernel z_trim_iss_0 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101563 kernel z_trim_iss_1 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101564 kernel z_trim_iss_2 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101565 kernel z_trim_iss_3 mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe 0 101566 kernel z_trim_int mi_switch+0xc1 _sleep+0x1cb taskqueue_thread_loop+0xdf fork_exit+0x7e fork_trampoline+0xe -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Tue Mar 23 00:27:04 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 927155BF201 for ; Tue, 23 Mar 2021 00:27:04 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F4ByH54n5z3Ff5 for ; Tue, 23 Mar 2021 00:27:02 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 685F5240005; Tue, 23 Mar 2021 00:27:00 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 1AC327203B9D5; Tue, 23 Mar 2021 01:27:00 +0100 (CET) Date: Tue, 23 Mar 2021 01:27:00 +0100 From: Mathieu Chouquet-Stringer To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2021 00:27:04 -0000 On Mon, Mar 22, 2021 at 04:39:27PM +0100, Mathieu Chouquet-Stringer wrote: > Hello Martin, > > Thanks for taking the time to reply to me. > > On Mon, Mar 22, 2021 at 03:15:17PM +0000, Martin Simmons wrote: > > You could try: > > > > procstat -kk 0 > > > > to get an overview of what the kernel threads are doing. Exploring some more with top -SH, it seems that's my thread: 31 root -8 - 0B 3984K CPU7 7 38:14 99.92% zfskern{dsl_scan_iss_0} That's the only busy thing on this OS and I know it's the one I see in htop because I see it ping-ponging on different CPUs. Seeing dsl_scan_iss led me to there: https://github.com/openzfs/zfs/blob/64e0fe14fff465604952d576051ca80c2f1be898/module/zfs/dsl_scan.c And that led me to reading about vfs.zfs.scan_legacy: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html?highlight=scan_mem_lim_fact#zfs-scan-legacy I killed the current scrub, put vfs.zfs.scan_legacy=1 in sysctl.conf, rebooted and restarted it, it's already telling me it'll finish quicker than the one which was running and wich had already done 10%. It's been running for 22 minutes now. scan: scrub in progress since Tue Mar 23 01:03:09 2021 63.0G scanned at 87.5M/s, 63.0G issued at 87.5M/s, 24.5T total 0B repaired, 0.25% done, 3 days 09:14:15 to go I'll let it run overnight and I'll check back tomorrow: I mean you can't trust the progress bar... -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Tue Mar 23 09:38:17 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3AEAE5A8372 for ; Tue, 23 Mar 2021 09:38:17 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F4RBJ1LWpz4Zqx for ; Tue, 23 Mar 2021 09:38:15 +0000 (UTC) (envelope-from mchouque@thi.eu.com) X-Originating-IP: 62.210.143.248 Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id A227C60004; Tue, 23 Mar 2021 09:38:13 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 640B27203B9D5; Tue, 23 Mar 2021 10:38:13 +0100 (CET) Date: Tue, 23 Mar 2021 10:38:13 +0100 From: Mathieu Chouquet-Stringer To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2021 09:38:17 -0000 On Tue, Mar 23, 2021 at 01:27:00AM +0100, Mathieu Chouquet-Stringer wrote: > scan: scrub in progress since Tue Mar 23 01:03:09 2021 > 63.0G scanned at 87.5M/s, 63.0G issued at 87.5M/s, 24.5T total > 0B repaired, 0.25% done, 3 days 09:14:15 to go > > I'll let it run overnight and I'll check back tomorrow: I mean you > can't trust the progress bar... So it's better but still not great, it's been running for 9.5 hours now: scan: scrub in progress since Tue Mar 23 01:03:09 2021 3.90T scanned at 121M/s, 3.90T issued at 121M/s, 24.5T total 0B repaired, 15.93% done, 2 days 01:24:58 to go For reference I had the following after 12 hours with the new scrub algorithm: scan: scrub in progress since Sun Mar 21 23:47:40 2021 2.82T scanned at 77.8M/s, 2.24T issued at 62.0M/s, 24.5T total 0B repaired, 9.17% done, 4 days 08:23:32 to go -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Tue Mar 23 09:40:50 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1AD645A8A52 for ; Tue, 23 Mar 2021 09:40:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F4RFG0543z4b2t for ; Tue, 23 Mar 2021 09:40:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 02F565A85DF; Tue, 23 Mar 2021 09:40:50 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02C185A8743 for ; Tue, 23 Mar 2021 09:40:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F4RFF6dpBz4Zrt for ; Tue, 23 Mar 2021 09:40:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D58091E77D for ; Tue, 23 Mar 2021 09:40:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12N9endr012362 for ; Tue, 23 Mar 2021 09:40:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12N9entU012361 for fs@FreeBSD.org; Tue, 23 Mar 2021 09:40:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 249115] The zfs command in the installer fails to log a set command Date: Tue, 23 Mar 2021 09:40:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dclarke@blastwave.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2021 09:40:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D249115 --- Comment #5 from Dennis Clarke --- Testing with 13.0 RC3 and not able to reproduce ... yet. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Mar 24 12:31:04 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F2C515B46D5 for ; Wed, 24 Mar 2021 12:31:03 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mail.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 4F56zC03zXz4RKt for ; Wed, 24 Mar 2021 12:31:02 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 12OCUqMx044651; Wed, 24 Mar 2021 12:30:52 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 12OCUq4n030004; Wed, 24 Mar 2021 12:30:52 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 12OCUqur030001; Wed, 24 Mar 2021 12:30:52 GMT Date: Wed, 24 Mar 2021 12:30:52 GMT Message-Id: <202103241230.12OCUqur030001@higson.cam.lispworks.com> From: Martin Simmons To: Mathieu Chouquet-Stringer CC: freebsd-fs@freebsd.org In-reply-to: (message from Mathieu Chouquet-Stringer on Tue, 23 Mar 2021 10:38:13 +0100) Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> X-Rspamd-Queue-Id: 4F56zC03zXz4RKt X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [0.15 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[46.17.166.21:from]; FREEFALL_USER(0.00)[martin]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[46.17.166.21:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.85)[-0.850]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 12:31:04 -0000 >>>>> On Tue, 23 Mar 2021 10:38:13 +0100, Mathieu Chouquet-Stringer said: > > So it's better but still not great, it's been running for 9.5 hours now: > > scan: scrub in progress since Tue Mar 23 01:03:09 2021 > 3.90T scanned at 121M/s, 3.90T issued at 121M/s, 24.5T total > 0B repaired, 15.93% done, 2 days 01:24:58 to go > > For reference I had the following after 12 hours with the new scrub algorithm: > > scan: scrub in progress since Sun Mar 21 23:47:40 2021 > 2.82T scanned at 77.8M/s, 2.24T issued at 62.0M/s, 24.5T total > 0B repaired, 9.17% done, 4 days 08:23:32 to go FWIW, I think the new scrub algorithm was added in 2018. You could try to profile it: https://wiki.freebsd.org/DTrace/Tutorial#Lesson_9._Profile_On-CPU_Kernel_Stacks __Martin From owner-freebsd-fs@freebsd.org Wed Mar 24 12:58:56 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 61D675B52D2 for ; Wed, 24 Mar 2021 12:58:56 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F57bM3KKmz4Sfg for ; Wed, 24 Mar 2021 12:58:54 +0000 (UTC) (envelope-from mchouque@thi.eu.com) X-Originating-IP: 62.210.143.248 Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 9EB7BE0010; Wed, 24 Mar 2021 12:58:52 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 502CC7203B9D5; Wed, 24 Mar 2021 13:58:52 +0100 (CET) Date: Wed, 24 Mar 2021 13:58:52 +0100 From: Mathieu Chouquet-Stringer To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202103241230.12OCUqur030001@higson.cam.lispworks.com> X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 12:58:56 -0000 On Wed, Mar 24, 2021 at 12:30:52PM +0000, Martin Simmons wrote: > FWIW, I think the new scrub algorithm was added in 2018. You're right, git says Wed Nov 15 20:27:01 2017 -0500 > You could try to profile it: > > https://wiki.freebsd.org/DTrace/Tutorial#Lesson_9._Profile_On-CPU_Kernel_Stacks dtrace, hello old friend... :-) That's the next step... -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Thu Mar 25 00:15:59 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B039857C5C8 for ; Thu, 25 Mar 2021 00:15:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5Qcb2wVdz3jVH for ; Thu, 25 Mar 2021 00:15:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 23B1857C996; Thu, 25 Mar 2021 00:15:59 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 223EC57C8AD for ; Thu, 25 Mar 2021 00:15:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5QcZ0x3Cz3jfX for ; Thu, 25 Mar 2021 00:15:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0E45E1D8A7 for ; Thu, 25 Mar 2021 00:15:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12P0FvhD087541 for ; Thu, 25 Mar 2021 00:15:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12P0Fvq9087540 for fs@FreeBSD.org; Thu, 25 Mar 2021 00:15:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 00:15:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 00:15:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 00:17:04 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1B91057C5DE for ; Thu, 25 Mar 2021 00:17:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5Qdr06Z5z3jYC for ; Thu, 25 Mar 2021 00:17:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 03E5E57CB22; Thu, 25 Mar 2021 00:17:04 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 03AC757C652 for ; Thu, 25 Mar 2021 00:17:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5Qdq6ccxz3jqB for ; Thu, 25 Mar 2021 00:17:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D68361D90C for ; Thu, 25 Mar 2021 00:17:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12P0H3Yc087706 for ; Thu, 25 Mar 2021 00:17:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12P0H3Qi087705 for fs@FreeBSD.org; Thu, 25 Mar 2021 00:17:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 159356] [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-specific Date: Thu, 25 Mar 2021 00:17:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 00:17:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D159356 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 00:36:47 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8746A57CD71 for ; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5R4b3F3Bz3l1h for ; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 6CEED57CD70; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6CB6C57D287 for ; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5R4b2Zkmz3kyD for ; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 47A171DC0F for ; Thu, 25 Mar 2021 00:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12P0al0x097598 for ; Thu, 25 Mar 2021 00:36:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12P0alew097597 for fs@FreeBSD.org; Thu, 25 Mar 2021 00:36:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 00:36:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 00:36:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |asomers@FreeBSD.org --- Comment #1 from Alan Somers --- Three questions: 1) Have you tested this on 14.0-CURRENT or on one of the 13.0 pre releases?= =20 The ZFS codebase is substantially different in FreBSD 13+ . 2) Have you tested this on stable/12 after r368491? That revision fixed a related bug for file systems. I never tested it for zvols. 3) The error you pasted looks suspicious. Are you sure you copy/pasted it correctly? There is no trailing slash in the name as you pasted it. root@lrrr:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1 cannot open 'zroot/test/disk1/': trailing slash in name --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 01:50:26 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 92B295A8FCE for ; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5SjZ3L9Hz3rm0 for ; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 723A55A903E; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 71F9D5A9115 for ; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5SjZ2fq7z3rbV for ; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4E47E1EA4C for ; Thu, 25 Mar 2021 01:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12P1oQ2i032249 for ; Thu, 25 Mar 2021 01:50:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12P1oQ99032248 for fs@FreeBSD.org; Thu, 25 Mar 2021 01:50:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254410] newfs creates .snap with world readable permissions 0777 Date: Thu, 25 Mar 2021 01:50:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: swills@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 01:50:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254410 Steve Wills changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 08:55:23 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D23935B4B94 for ; Thu, 25 Mar 2021 08:55:23 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5f7t5tVQz4lHP for ; Thu, 25 Mar 2021 08:55:22 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id D088923842B; Thu, 25 Mar 2021 08:55:13 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1616662513; bh=0U+egdwVKNMYHvN1EylhMriiIHzetRJOsUpmRdGM8wQ=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=YvCFu7o09IcuzAGhuufWwU/1k3XG8+WPFvDjkDs/JXdxAgaJd9Ga/Iap4cDlN1fRf hXSWCamzmvfGFZilFpqeBNfCGVk8pBoq8dheo+18yij05h/cNHNg01xbfFCfKNreCU lZ4y2gYZNhV1H7HcE/5ioAm788HC2ZwHYf4vh6wA= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 25 Mar 2021 08:55:13 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.1497.012; Thu, 25 Mar 2021 08:55:13 +0000 From: Matt Churchyard To: Mathieu Chouquet-Stringer CC: "freebsd-fs@freebsd.org" Subject: RE: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Thread-Topic: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Thread-Index: AQHXHwShBASzm+41FEaFBtPr0Iszi6qQHmJQgAAGqYCAAJNmAIAAmgKAgAHCrO+AAAe4AIABSsvg Date: Thu, 25 Mar 2021 08:55:12 +0000 Message-ID: <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 4F5f7t5tVQz4lHP X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=YvCFu7o0; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-1.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; DMARC_NA(0.00)[userve.net]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 08:55:23 -0000 -----Original Message----- From: owner-freebsd-fs@freebsd.org On Behalf= Of Mathieu Chouquet-Stringer Sent: 24 March 2021 12:59 To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) On Wed, Mar 24, 2021 at 12:30:52PM +0000, Martin Simmons wrote: > FWIW, I think the new scrub algorithm was added in 2018. You're right, git says Wed Nov 15 20:27:01 2017 -0500 > You could try to profile it: >=20 > https://wiki.freebsd.org/DTrace/Tutorial#Lesson_9._Profile_On-CPU_Kernel_= Stacks > dtrace, hello old friend... :-) > That's the next step... Just an a aside, I did post a message a few weeks ago with a similar proble= m on 13 (as well as snapshot issues). Scrub seemed ok for a short while, bu= t then ground to a halt. It would take 10+ minutes to go 0.01%, with everyt= hing appearing fairly idle. I finally gave up and stopped it after about 20= hours. Moving to 12.2 and rebuilding the pool, the system scrubbed the sam= e data in an hour, and I've just scrubbed the same system after a month of = use with about 4 times the data in 3 hours 20. As far as I'm aware, both sh= ould be using effectively the same "new" scrub code. Will be interesting if you find a cause as I didn't get any response to wha= t for me was a complete showstopper for moving to 13. Matt --=20 Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Mar 25 10:08:07 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 435D15B7838 for ; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5glq1Frpz4qs5 for ; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 2B0C45B776E; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2AD755B7837 for ; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5glq0frWz4qlK for ; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 03C26255B1 for ; Thu, 25 Mar 2021 10:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PA86DR077312 for ; Thu, 25 Mar 2021 10:08:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PA86Dm077311 for fs@FreeBSD.org; Thu, 25 Mar 2021 10:08:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 10:08:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: patrik@hildingsson.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 10:08:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #2 from Patrik Hildingsson --- (In reply to Alan Somers from comment #1) 1) Please see the below copy/paste from a 13.0-RC3 system where it seems to work just fine. root@testbsd:~ # sysctl -w vfs.zfs.vol.mode=3D2 root@testbsd:~ # zfs create zroot/test root@testbsd:~ # zfs create -o volmode=3Ddev -V 50M zroot/test/disk0 root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 1.016171 secs (51594487 bytes/sec) root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 0.833823 secs (62877584 bytes/sec) root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok-2 root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 0.444901 secs (117843859 bytes/sec) root@testbsd:~ # zfs create -V 52428800 -o 'reservation=3Dnone' -o 'volblocksize=3D8192' -o 'checksum=3Don' -o 'compression=3Dlz4' -o 'readonl= y=3Doff' -o 'copies=3D1' -o 'logbias=3Dthroughput' -o 'dedup=3Doff' -o 'sync=3Dstandard= ' -o 'volmode=3Ddev' -o 'snapshot_limit=3Dnone' -o 'redundant_metadata=3Dmost' zroot/test/disk1 root@testbsd:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1 root@testbsd:~ # uname -K 1300139 root@testbsd:~ # uname -U 1300139 2) No, I did not try 12-STABLE. I have only tried 12.2-RELEASE-p4. I have reproduced the error on two different installations of 12.2-RELEASE-p4. 3) The paste indeed looks suspicious. I forgot to trim away the prepending "root@lrrr:~ ", other than that it is correct. I did not leave out a traili= ng forward slash. It isn't supposed to be there. ZFS incorrectly prompts that there is a trailing forward slash. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 13:31:56 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EAE535BD342 for ; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5mH068cfz3LG3 for ; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id D33CB5BD341; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D30685BD75B for ; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5mH05b7Yz3LJD for ; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B310227DB0 for ; Thu, 25 Mar 2021 13:31:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PDVuCO076155 for ; Thu, 25 Mar 2021 13:31:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PDVusn076154 for fs@FreeBSD.org; Thu, 25 Mar 2021 13:31:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 13:31:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 13:31:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #3 from Alan Somers --- Please do try stable/12. I think r368491 most likely fixed this problem. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 14:43:43 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 648635BF6FC for ; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5nsq2Dhdz3R97 for ; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4AEED5BF6FB; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4AB9C5BF72D for ; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5nsq16KQz3QtJ for ; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 18AF1FDC for ; Thu, 25 Mar 2021 14:43:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PEhhWM013671 for ; Thu, 25 Mar 2021 14:43:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PEhh7J013670 for fs@FreeBSD.org; Thu, 25 Mar 2021 14:43:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 14:43:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: patrik@hildingsson.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 14:43:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #4 from Patrik Hildingsson --- (In reply to Alan Somers from comment #3) The error occurs even in FreeBSD 12.2-STABLE r369477 root@testbsd:~ # zfs create zroot/test root@testbsd:~ # zfs create -o volmode=3Ddev -V 50M zroot/test/disk0 root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 1.582797 secs (33124142 bytes/sec) root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 0.926790 secs (56570297 bytes/sec) root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok-2 root@testbsd:~ # dd if=3D/dev/random of=3D/dev/zvol/zroot/test/disk0 dd: /dev/zvol/zroot/test/disk0: end of device 102401+0 records in 102400+0 records out 52428800 bytes transferred in 1.235393 secs (42438953 bytes/sec) root@testbsd:~ # zfs create -V 52428800 -o 'reservation=3Dnone' -o 'volblocksize=3D8192' -o 'checksum=3Don' -o 'compression=3Dlz4' -o 'readonl= y=3Doff' -o 'copies=3D1' -o 'logbias=3Dthroughput' -o 'dedup=3Doff' -o 'sync=3Dstandard= ' -o 'volmode=3Ddev' -o 'snap shot_limit=3Dnone' -o 'redundant_metadata=3Dmost' zroot/test/disk1 root@testbsd:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1 cannot open 'zroot/test/disk1/': trailing slash in name warning: cannot send 'zroot/test/disk0@test-now_ok': signal received root@testbsd:~ # uname -K 1202505 root@testbsd:~ # uname -U 1202505 root@testbsd:~ # freebsd-version 12.2-STABLE --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 15:34:51 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D072A5C19ED for ; Thu, 25 Mar 2021 15:34:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5q0q2vXpz3lxM for ; Thu, 25 Mar 2021 15:34:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) X-Originating-IP: 195.64.148.76 Received: from [192.168.0.88] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 5B8CBC0006; Thu, 25 Mar 2021 15:34:48 +0000 (UTC) Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) To: Matt Churchyard , Mathieu Chouquet-Stringer Cc: "freebsd-fs@freebsd.org" References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> From: Andriy Gapon Message-ID: <13825648-7965-8a95-67a2-262cb50c2d71@FreeBSD.org> Date: Thu, 25 Mar 2021 17:34:47 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F5q0q2vXpz3lxM X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; TAGGED_RCPT(0.00)[freebsd]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 15:34:51 -0000 On 25/03/2021 10:55, Matt Churchyard wrote: > Just an a aside, I did post a message a few weeks ago with a similar problem > on 13 (as well as snapshot issues). Scrub seemed ok for a short while, but > then ground to a halt. It would take 10+ minutes to go 0.01%, with everything > appearing fairly idle. I finally gave up and stopped it after about 20 hours. > Moving to 12.2 and > rebuilding the pool, This could make a huge difference, though. Scrub performance significantly depends on the data layout. The more sequential it is the better. So, older (and more fragmented) pools are at disadvantage. > the system scrubbed the same data in > an hour, and I've just scrubbed the same system after a month of use with > about 4 times the data in 3 hours 20. As far as I'm aware, both should be > using effectively the same "new" scrub code. > > Will be interesting if you find a cause as I didn't get any response to what > for me was a complete showstopper for moving to 13. -- Andriy Gapon -- Andriy Gapon From owner-freebsd-fs@freebsd.org Thu Mar 25 15:50:47 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2C7175C1FFC for ; Thu, 25 Mar 2021 15:50:47 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5qM96FlBz3mcD; Thu, 25 Mar 2021 15:50:45 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id EE2422383AE; Thu, 25 Mar 2021 15:50:42 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1616687443; bh=yfBOHLjgPbZ7WS93/oEfcPFu5uSTD6bkcCKI2erEbcc=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=JejJSMYGIrkaPrIBldu3arFsw/sgE7jFIz4TmmAKLdAvL7al0puBnd3fPa97oaIvI xWj5iTlzQAJ7Yq5lrkniUfLZKApTXKOQgSANn9YNO4ki/zv0TQQz2EgR/wX3Zh3a85 gruHECmtHq+O9HAOBQGRnxmnxXbKLKOL4NrenMJ4= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 25 Mar 2021 15:50:42 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.1497.012; Thu, 25 Mar 2021 15:50:42 +0000 From: Matt Churchyard To: Andriy Gapon , Mathieu Chouquet-Stringer CC: "freebsd-fs@freebsd.org" Subject: RE: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Thread-Topic: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Thread-Index: AQHXHwShBASzm+41FEaFBtPr0Iszi6qQHmJQgAAGqYCAAJNmAIAAmgKAgAHCrO+AAAe4AIABSsvggABzGoCAAAAwUA== Date: Thu, 25 Mar 2021 15:50:41 +0000 Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> <13825648-7965-8a95-67a2-262cb50c2d71@FreeBSD.org> In-Reply-To: <13825648-7965-8a95-67a2-262cb50c2d71@FreeBSD.org> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4F5qM96FlBz3mcD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=JejJSMYG; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-2.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; MIME_BASE64_TEXT_BOGUS(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[userve.net:+]; MIME_BASE64_TEXT(0.10)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TAGGED_RCPT(0.00)[freebsd]; DMARC_NA(0.00)[userve.net]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 15:50:47 -0000 LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEFuZHJpeSBHYXBvbiA8YXZnQEZyZWVC U0Qub3JnPiANClNlbnQ6IDI1IE1hcmNoIDIwMjEgMTU6MzUNClRvOiBNYXR0IENodXJjaHlhcmQg PG1hdHQuY2h1cmNoeWFyZEB1c2VydmUubmV0PjsgTWF0aGlldSBDaG91cXVldC1TdHJpbmdlciA8 bWUrZnJlZWJzZEBtYXRoaWV1LmRpZ2l0YWw+DQpDYzogZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw0K U3ViamVjdDogUmU6IFNjcnViIGluY3JlZGlibHkgc2xvdyB3aXRoIDEzLjAtUkMzIChhcyB3ZWxs IGFzIFJDMSAmIDIpDQoNCk9uIDI1LzAzLzIwMjEgMTA6NTUsIE1hdHQgQ2h1cmNoeWFyZCB3cm90 ZToNCj4gSnVzdCBhbiBhIGFzaWRlLCBJIGRpZCBwb3N0IGEgbWVzc2FnZSBhIGZldyB3ZWVrcyBh Z28gd2l0aCBhIHNpbWlsYXIgDQo+IHByb2JsZW0gb24gMTMgKGFzIHdlbGwgYXMgc25hcHNob3Qg aXNzdWVzKS4gU2NydWIgc2VlbWVkIG9rIGZvciBhIA0KPiBzaG9ydCB3aGlsZSwgYnV0IHRoZW4g Z3JvdW5kIHRvIGEgaGFsdC4gSXQgd291bGQgdGFrZSAxMCsgbWludXRlcyB0byANCj4gZ28gMC4w MSUsIHdpdGggZXZlcnl0aGluZyBhcHBlYXJpbmcgZmFpcmx5IGlkbGUuIEkgZmluYWxseSBnYXZl IHVwIGFuZCBzdG9wcGVkIGl0IGFmdGVyIGFib3V0IDIwIGhvdXJzLg0KPiBNb3ZpbmcgdG8gMTIu MiBhbmQNCj4gcmVidWlsZGluZyB0aGUgcG9vbCwNCg0KPlRoaXMgY291bGQgbWFrZSBhIGh1Z2Ug ZGlmZmVyZW5jZSwgdGhvdWdoLg0KPlNjcnViIHBlcmZvcm1hbmNlIHNpZ25pZmljYW50bHkgZGVw ZW5kcyBvbiB0aGUgZGF0YSBsYXlvdXQuDQo+VGhlIG1vcmUgc2VxdWVudGlhbCBpdCBpcyB0aGUg YmV0dGVyLg0KPlNvLCBvbGRlciAoYW5kIG1vcmUgZnJhZ21lbnRlZCkgcG9vbHMgYXJlIGF0IGRp c2FkdmFudGFnZS4NCg0KSXQgY291bGQsIGJ1dCB0aGVuIHRoaXMgd2FzIGEgYnJhbmQgbmV3IGlu c3RhbGwgZm9yIGEgYmFja3VwIHNlcnZlciBhbmQgaGFkIGFib3V0IDEwIGRhdGFzZXRzIHNlbnQg dG8gaXQgdmlhIHNlbmQvcmVjdi4gQWJzb2x1dGVseSBubyBvdGhlciBmaWxlc3lzdGVtIHVzYWdl LiBJdCB3YXMgcmVhbGx5IGEgYmVzdCBjYXNlIGZvciBzY3J1YiBhdCB0aGlzIHBvaW50Lg0KV2hl biBJIHNheSBpdCB3YXMgb2sgZm9yIGEgd2hpbGUsIEkgZG9uJ3QgbWVhbiBhIGZ1bGwgc2NydWIg Y29tcGxldGVkIGZpbmUgZm9yIGRheXMvd2Vla3MsIHRoZW4gc3RhcnRlZCBzbG93aW5nIGRvd24u IEkgbWVhbiB0aGUgZmlyc3Qgc2NydWIsIGRheSAxLCBvZiB1bnRvdWNoZWQgbmV3IGRhdGEgc2Vl bWVkIHRvIHJ1biBvayB1bnRpbCBhcm91bmQgMTYlLCB0aGVuIGp1c3Qgc3RvcHBlZCBkZWFkIGZv ciBhYm91dCA2IGhvdXJzLCBtb3ZpbmcgMC4wMSUgZXZlcnkgbm93IGFuZCB0aGVuLg0KDQpUaGUg b25seSByZWFsIGRpZmZlcmVuY2UgaXMgdGhhdCBJIHdhcyBob3BpbmcgdG8gdXNlIGRhdGFzZXQg ZW5jcnlwdGlvbiwgd2hpY2ggd2FzIGVuYWJsZWQsIGJ1dCB0aGVuIHRoaXMgc2hvdWxkbid0IHJl YWxseSBhZmZlY3Qgc2NydWIgYXMgdGhlIHdob2xlIHBvaW50IG9mIHRoZSBvcGVuemZzIGltcGxl bWVudGF0aW9uIGlzIHRoYXQgZW5jcnlwdGlvbiBkb2Vzbid0IGFmZmVjdCBhbnl0aGluZyBzY3J1 YiBuZWVkcyB0byBkbyBpdHMgam9iLg0KDQpUaGUgMTIuMiBpbnN0YWxsIHdhcyBkb25lIGluIGV4 YWN0bHkgdGhlIHNhbWUgd2F5LiBBIG5ldyBwb29sIHdpdGggdGhlIHNhbWUgZGF0YXNldHMgc2Vu dCB0byBpdC4gQXMgaW4gbXkgcHJldmlvdXMgbWVzc2FnZSwgaXQgc3RpbGwgc2NydWJzIGluIGp1 c3Qgb3ZlciAzIGhvdXJzIGV2ZW4gdGhvdWdoIGl0IGhhcyBub3cgYmVlbiByZWNlaXZpbmcgc25h cHNob3RzIG9mIGFib3V0IDQwIGRhdGFzZXRzLCBjb250YWluaW5nIGFib3V0IDQgdGltZXMgbW9y ZSBkYXRhLCBkYWlseSBmb3Igb3ZlciBhIG1vbnRoLiBXaGF0ZXZlciB3YXMgaGFwcGVuaW5nIG9u IDEzIGNsZWFybHkgd2Fzbid0IGhhcmR3YXJlIHJlbGF0ZWQuDQoNCk1hdHQNCg0KPiB0aGUgc3lz dGVtIHNjcnViYmVkIHRoZSBzYW1lIGRhdGEgaW4NCj4gYW4gaG91ciwgYW5kIEkndmUganVzdCBz Y3J1YmJlZCB0aGUgc2FtZSBzeXN0ZW0gYWZ0ZXIgYSBtb250aCBvZiB1c2UgDQo+IHdpdGggYWJv dXQgNCB0aW1lcyB0aGUgZGF0YSBpbiAzIGhvdXJzIDIwLiBBcyBmYXIgYXMgSSdtIGF3YXJlLCBi b3RoIA0KPiBzaG91bGQgYmUgdXNpbmcgZWZmZWN0aXZlbHkgdGhlIHNhbWUgIm5ldyIgc2NydWIg Y29kZS4NCj4gDQo+IFdpbGwgYmUgaW50ZXJlc3RpbmcgaWYgeW91IGZpbmQgYSBjYXVzZSBhcyBJ IGRpZG4ndCBnZXQgYW55IHJlc3BvbnNlIA0KPiB0byB3aGF0IGZvciBtZSB3YXMgYSBjb21wbGV0 ZSBzaG93c3RvcHBlciBmb3IgbW92aW5nIHRvIDEzLg0KDQoNCi0tDQpBbmRyaXkgR2Fwb24NCg0K LS0NCkFuZHJpeSBHYXBvbg0K From owner-freebsd-fs@freebsd.org Thu Mar 25 16:42:14 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 363FB5C3D97 for ; Thu, 25 Mar 2021 16:42:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5rVZ0swrz3rdw for ; Thu, 25 Mar 2021 16:42:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1DE555C3D96; Thu, 25 Mar 2021 16:42:14 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1DB395C3BB7 for ; Thu, 25 Mar 2021 16:42:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5rVZ0JZKz3rkx for ; Thu, 25 Mar 2021 16:42:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F1CAE24CE for ; Thu, 25 Mar 2021 16:42:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PGgDlb073096 for ; Thu, 25 Mar 2021 16:42:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PGgDgt073095 for fs@FreeBSD.org; Thu, 25 Mar 2021 16:42:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 16:42:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 16:42:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #5 from Alan Somers --- Well darn. Are you interested to try debugging this yourself, Patrick? I'm guessing that the problem lies in libzfs, not in the kernel. I would focus= on that "trailing slash in name" message. Try to figure out where that comes from. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 16:54:29 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38ACC5C3BF9 for ; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5rmj0y2pz3s0g for ; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 209B65C3BF8; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 206535C438B for ; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5rmj0Mhtz3sKn for ; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 004322ABB for ; Thu, 25 Mar 2021 16:54:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PGsS6E077572 for ; Thu, 25 Mar 2021 16:54:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PGsSpp077571 for fs@FreeBSD.org; Thu, 25 Mar 2021 16:54:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 16:54:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: patrik@hildingsson.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 16:54:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #6 from Patrik Hildingsson --- (In reply to Alan Somers from comment #5) To be honest, not really. I plan to move to 13.0-RELEASE as soon as you rel= ease it. I was just trying to put the issue in the spotlight, in case someone else r= uns into the same issue in the future. Do you think it's worth the effort to debug and find the issue? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 25 17:46:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5902C5C6596 for ; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5sx81s3hz4Rfs for ; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 3F91D5C62CD; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3F5C45C64D4 for ; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5sx81CMHz4Rfr for ; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1C44B33EF for ; Thu, 25 Mar 2021 17:46:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 12PHkqTJ099718 for ; Thu, 25 Mar 2021 17:46:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 12PHkqFf099717 for fs@FreeBSD.org; Thu, 25 Mar 2021 17:46:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 254530] ZFS (base) receive fails to receive zvol dataset if dataset already exist Date: Thu, 25 Mar 2021 17:46:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Mar 2021 17:46:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254530 --- Comment #7 from Alan Somers --- It isn't worth the effort for me, but then I'm not the one affected. I or another developer would certainly review a patch if you produce one. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Mar 26 09:37:52 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B2B245B932E; Fri, 26 Mar 2021 09:37:52 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6H2R3yfyz4S1C; Fri, 26 Mar 2021 09:37:50 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 2B85B240016; Fri, 26 Mar 2021 09:37:47 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id D14847203B9D5; Fri, 26 Mar 2021 10:37:47 +0100 (CET) Date: Fri, 26 Mar 2021 10:37:47 +0100 From: Mathieu Chouquet-Stringer To: Matt Churchyard Cc: "freebsd-fs@freebsd.org" , current@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> MIME-Version: 1.0 In-Reply-To: <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2021 09:37:52 -0000 On Thu, Mar 25, 2021 at 08:55:12AM +0000, Matt Churchyard wrote: > Just an a aside, I did post a message a few weeks ago with a similar > problem on 13 (as well as snapshot issues). Scrub seemed ok for a > short while, but then ground to a halt. It would take 10+ minutes to > go 0.01%, with everything appearing fairly idle. I finally gave up and > stopped it after about 20 hours. Moving to 12.2 and rebuilding the > pool, the system scrubbed the same data in an hour, and I've just > scrubbed the same system after a month of use with about 4 times the > data in 3 hours 20. As far as I'm aware, both should be using > effectively the same "new" scrub code. > > Will be interesting if you find a cause as I didn't get any response > to what for me was a complete showstopper for moving to 13. Bear with me, I'm slowly resilvering now... But same thing, it's not even maxing out my slow drives... Looks like it'll take 2 days... I did some flame graphs using dtrace. The first one is just the output of that: dtrace -x stackframes=100 -n 'profile-99 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }' Clearly my machine is not busy at all. And the second is the output of pretty much the same thing except I'm only capturing pid 31 which is the one busy. dtrace -x stackframes=100 -n 'profile-99 /arg0 && pid == 31/ { @[stack()] = count(); } tick-60s { exit(0); }' One striking thing is how many times hpet_get_timecount is present... -- Mathieu Chouquet-Stringer mchouque@free.fr The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Fri Mar 26 12:41:12 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 30E205BE26A; Fri, 26 Mar 2021 12:41:12 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6M5z62wpz4d1v; Fri, 26 Mar 2021 12:41:11 +0000 (UTC) (envelope-from mchouque@thi.eu.com) X-Originating-IP: 62.210.143.248 Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 9959860002; Fri, 26 Mar 2021 12:41:08 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 48BB17203B9D5; Fri, 26 Mar 2021 13:41:08 +0100 (CET) Date: Fri, 26 Mar 2021 13:41:08 +0100 From: Mathieu Chouquet-Stringer To: Michael Gmelin Cc: Matt Churchyard , "freebsd-fs@freebsd.org" , current@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> <20210326132945.3274687e@bsd64.grem.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210326132945.3274687e@bsd64.grem.de> X-Rspamd-Queue-Id: 4F6M5z62wpz4d1v X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_FROM(0.00)[freebsd]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2021 12:41:12 -0000 On Fri, Mar 26, 2021 at 01:29:45PM +0100, Michael Gmelin wrote: > Does tuning of > > - vfs.zfs.scrub_delay > - vfs.zfs.resilver_min_time_ms > - vfs.zfs.resilver_delay > > make a difference? Only vfs.zfs.resilver_min_time_ms exists in 13 it seems. The default values is 3000 but doubling it to 6000 doesn't change anything. And that could make sense because it seems I'm often cpu bound as I see the following in top (-SH): PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 31 root -8 - 0B 3984K CPU12 12 758:07 99.65% zfskern{dsl_scan_iss_0} And that's on an Intel(R) Xeon(R) CPU E5-2696 v2 @ 2.50GHz with a turbo @ 3.50GHz... Disks are busy roughly between 30 to 50% of the time: extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b da2 1520 0 34974.1 0.0 0 0 0 0 1 31 da3 1469 0 35030.3 0.0 0 0 0 0 0 32 da4 1317 0 35106.6 0.0 0 0 0 0 0 42 da6 1293 0 35094.5 0.0 0 0 0 0 0 36 da7 1466 0 35032.3 0.0 0 0 0 0 0 31 da8 0 1303 0.0 34917.9 0 0 0 0 0 43 da9 1396 0 35046.4 0.0 0 0 0 0 1 34 da10 1393 0 35062.4 0.0 0 0 0 0 0 32 da8 is the disk being resilvered. -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Fri Mar 26 12:30:23 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5DB3C5BDFD8; Fri, 26 Mar 2021 12:30:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6LsT5lqTz4chd; Fri, 26 Mar 2021 12:30:21 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id b1a7a5c9; Fri, 26 Mar 2021 12:30:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=20180501; bh=kAm37C2Z dSf5xxuaEZyc71zmh18=; b=PP1N54l0qo2FcZKwQr3YGIi+i8ZASjezWu8Xdru2 oe6lYfeeiX41DxRLUZ9tul9iN8HIsHzlP2WYNF5XalRzZyvuCV5Nac+1r/WIh6yl Vdk643mu51KXX4TY5kmSNcnLnCgQD/M6W21nIBw6EoskoyWhU8nOxUg3yvE8Eev4 hHombsZ48t9BvDSR+hMs+itE0Gn39+oOS3xyRCiqgPqxk5ajl2X1vXY2+GVHJg3H w+S7GONkf4Qr+MJe3sDlFWhG3iiIFYyxZwK9VbU15fdrNI+nC8gDlzU6SkbPAWtL dJMGaTYsObe61N9xJAti9TrWL5ukWLkpWHEf7kSJxYoN3A== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=20180501; b=OO e8b6n30Zzzx3UpQm0CZKFhKsSYO7BCBcOW5xVxJHqmOR0a8Rv8DW99jYlPEpjVEA L13OKVP+HeKI/Aqr35aeAFNgsMtsaWwv43Q8yDAhR0cMCigObCG9WC4uY9o796FN yYlAiucF2nghs0loBhDEkrOobZZ8th3BG/AXf34UCQy8DdM7Z0D7RQPTgHPMDVhj 8uyVmGytxps9nIOD6EWuQEMl7En8+XragBPxM8QiI2LJozIbaXhCBgMao9CT2Shg KjBtYM0/9ReAvkohyqRMJfjeVvmmIW2PrjUa+2o8hOmc+P9EJW3j9f7vm3l9s4eP 1iVuGxHlfboZtwO/aFFQ== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id c7d67f02 (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Fri, 26 Mar 2021 12:30:09 +0000 (UTC) Date: Fri, 26 Mar 2021 13:29:45 +0100 From: Michael Gmelin To: Mathieu Chouquet-Stringer Cc: Matt Churchyard , "freebsd-fs@freebsd.org" , current@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: <20210326132945.3274687e@bsd64.grem.de> In-Reply-To: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F6LsT5lqTz4chd X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=grem.de header.s=20180501 header.b=PP1N54l0; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@grem.de designates 213.239.217.29 as permitted sender) smtp.mailfrom=freebsd@grem.de X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[grem.de:s=20180501]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:213.239.217.29/32]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grem.de]; SPAMHAUS_ZRD(0.00)[213.239.217.29:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[grem.de:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[213.239.217.29:from]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs,current] X-Mailman-Approved-At: Sat, 27 Mar 2021 08:15:33 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2021 12:30:23 -0000 On Fri, 26 Mar 2021 10:37:47 +0100 Mathieu Chouquet-Stringer wrote: > On Thu, Mar 25, 2021 at 08:55:12AM +0000, Matt Churchyard wrote: > > Just an a aside, I did post a message a few weeks ago with a similar > > problem on 13 (as well as snapshot issues). Scrub seemed ok for a > > short while, but then ground to a halt. It would take 10+ minutes to > > go 0.01%, with everything appearing fairly idle. I finally gave up > > and stopped it after about 20 hours. Moving to 12.2 and rebuilding > > the pool, the system scrubbed the same data in an hour, and I've > > just scrubbed the same system after a month of use with about 4 > > times the data in 3 hours 20. As far as I'm aware, both should be > > using effectively the same "new" scrub code. > > > > Will be interesting if you find a cause as I didn't get any response > > to what for me was a complete showstopper for moving to 13. > > Bear with me, I'm slowly resilvering now... But same thing, it's not > even maxing out my slow drives... Looks like it'll take 2 days... > > I did some flame graphs using dtrace. The first one is just the output > of that: > dtrace -x stackframes=100 -n 'profile-99 /arg0/ { @[stack()] = > count(); } tick-60s { exit(0); }' > > Clearly my machine is not busy at all. > And the second is the output of pretty much the same thing except I'm > only capturing pid 31 which is the one busy. > dtrace -x stackframes=100 -n 'profile-99 /arg0 && pid == 31/ { > @[stack()] = count(); } tick-60s { exit(0); }' > > One striking thing is how many times hpet_get_timecount is present... Does tuning of - vfs.zfs.scrub_delay - vfs.zfs.resilver_min_time_ms - vfs.zfs.resilver_delay make a difference? Best, Michael -- Michael Gmelin From owner-freebsd-fs@freebsd.org Sat Mar 27 10:22:06 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 859BD5BF278 for ; Sat, 27 Mar 2021 10:22:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F6vz23QN8z3Gml for ; Sat, 27 Mar 2021 10:22:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 737825BF808; Sat, 27 Mar 2021 10:22:06 +0000 (UTC) Delivered-To: fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 733EE5BF4F2 for ; Sat, 27 Mar 2021 10:22:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6vz172PLz3GZY for ; Sat, 27 Mar 2021 10:22:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 6B5CB240008 for ; Sat, 27 Mar 2021 10:22:03 +0000 (UTC) To: freebsd-fs From: Andriy Gapon Subject: stuck reboot, zfs, paging in progress Message-ID: <095cc2b5-80e6-9cb3-e905-571f9e1ebc04@FreeBSD.org> Date: Sat, 27 Mar 2021 12:22:02 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F6vz172PLz3GZY X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR]; local_wl_from(0.00)[FreeBSD.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Mar 2021 10:22:06 -0000 Has anyone else seen anything like this? Seems to be related to mmap + zfs. This is on stable/13, f165641df4da17 from a couple of weeks ago. #0 sched_switch (td=td@entry=0xfffffe00357f8ac0, flags=, flags@entry=260) at /usr/devel/git/trant/sys/kern/sched_ule.c:2147 #1 0xffffffff80847246 in mi_switch (flags=flags@entry=260) at /usr/devel/git/trant/sys/kern/kern_synch.c:542 #2 0xffffffff8088b9e6 in sleepq_switch (wchan=wchan@entry=0xfffff80681a9ccf0, pri=pri@entry=84) at /usr/devel/git/trant/sys/kern/subr_sleepqueue.c:608 #3 0xffffffff8088b88c in sleepq_wait (wchan=0xfffff80681a9ccf0, pri=pri@entry=84) at /usr/devel/git/trant/sys/kern/subr_sleepqueue.c:659 #4 0xffffffff80846f1f in _blockcount_sleep (bc=bc@entry=0xfffff80681a9ccf0, lock=lock@entry=0x0, wmesg=wmesg@entry=0xffffffff80c05803 "bovlbx", prio=prio@entry=84) at /usr/devel/git/trant/sys/kern/kern_synch.c:461 #5 0xffffffff80a9ac53 in _blockcount_wait (bc=bc@entry=0xfffff80681a9ccf0, lo=lo@entry=0x0, wmesg=wmesg@entry=0xffffffff80c05803 "bovlbx", prio=84) at /usr/devel/git/trant/sys/sys/blockcount.h:85 #6 0xffffffff80a9aca8 in vm_object_pip_wait_unlocked (object=, waitid=) at /usr/devel/git/trant/sys/vm/vm_object.c:405 #7 0xffffffff80902d9a in bufobj_invalbuf (bo=bo@entry=0xfffff806a60c6a70, flags=flags@entry=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) at /usr/devel/git/trant/sys/kern/vfs_subr.c:1978 #8 0xffffffff809031ea in vinvalbuf (vp=, vp@entry=0xfffff806a60c6988, flags=flags@entry=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) at /usr/devel/git/trant/sys/kern/vfs_subr.c:2021 #9 0xffffffff80ab4dc1 in vnode_destroy_vobject (vp=vp@entry=0xfffff806a60c6988) at /usr/devel/git/trant/sys/vm/vnode_pager.c:213 #10 0xffffffff80906421 in vgonel (vp=vp@entry=0xfffff806a60c6988) at /usr/devel/git/trant/sys/kern/vfs_subr.c:3978 #11 0xffffffff80905c4a in vflush (mp=mp@entry=0xfffffe01bc462ac0, rootrefs=rootrefs@entry=0, flags=flags@entry=2, td=, td@entry=0xfffffe00357f8ac0) at /usr/devel/git/trant/sys/kern/vfs_subr.c:3722 #12 0xffffffff80378136 in zfs_umount (vfsp=0xfffffe01bc462ac0, vfsp@entry=, fflag=, fflag@entry=) at /usr/devel/git/trant/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c:1662 #13 0xffffffff808fbacf in dounmount (mp=, mp@entry=0xfffffe01bc462ac0, flags=flags@entry=524288, td=0xfffffe00357f8ac0) at /usr/devel/git/trant/sys/kern/vfs_mount.c:1843 #14 0xffffffff80906c5f in unmount_or_warn (mp=mp@entry=0xfffffe01bc462ac0) at /usr/devel/git/trant/sys/kern/vfs_subr.c:4640 #15 0xffffffff80906c3c in vfs_unmountall () at /usr/devel/git/trant/sys/kern/vfs_subr.c:4674 #16 0xffffffff808df142 in bufshutdown (show_busybufs=0) at /usr/devel/git/trant/sys/kern/vfs_bio.c:1443 #17 0xffffffff80839c75 in kern_reboot (howto=0) at /usr/devel/git/trant/sys/kern/kern_shutdown.c:472 #18 0xffffffff808396f2 in sys_reboot (td=, uap=0xfffffe00357f8ea8) at /usr/devel/git/trant/sys/kern/kern_shutdown.c:313 #19 0xffffffff80b22409 in syscallenter (td=0xfffffe00357f8ac0) at /usr/devel/git/trant/sys/amd64/amd64/../../kern/subr_syscall.c:189 #20 0xffffffff80b220e5 in amd64_syscall (td=0xfffffe00357f8ac0, traced=0) at /usr/devel/git/trant/sys/amd64/amd64/trap.c:1156 (kgdb) p *vp $13 = {v_type = VREG, v_irflag = 1, v_seqc = 1, v_nchash = 3207342140, v_op = 0xffffffff80e592c0 , v_data = 0xfffff805410f6ce8, v_mount = 0xfffffe01bc462ac0, v_nmntvnodes = {tqe_next = 0xfffff8057f14dc00, tqe_prev = 0xfffffe01bc462b48}, {v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffff806a60c69e0}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff80c4edb4 "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0x0}, lk_lock = 18446741875583847104, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 0xffffffff80cf0f55 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, v_vnlock = 0xfffff806a60c69f8, v_vnodelist = { tqe_next = 0xfffff80732d603d0, tqe_prev = 0xfffff801cde180c8}, v_lazylist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80cb36ff "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops = 0xffffffff80ee28a0 , bo_object = 0xfffff80681a9cc60, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff806a60c6988, bo_clean = { bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806a60c6ab8}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806a60c6ad8}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_domain = 3, bo_bsize = 1048576}, v_pollinfo = 0xfffff804d3647700, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff806a60c6b20}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 2, v_usecount = 0, v_iflag = 0, v_vflag = 0, v_mflag = 0, v_dbatchcpu = -1, v_writecount = 0, v_seqc_users = 1, v_hash = 111545449} (kgdb) p vp->v_bufobj $14 = {bo_lock = {lock_object = {lo_name = 0xffffffff80cb36ff "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops = 0xffffffff80ee28a0 , bo_object = 0xfffff80681a9cc60, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff806a60c6988, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806a60c6ab8}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806a60c6ad8}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_domain = 3, bo_bsize = 1048576} (kgdb) p *vp->v_bufobj->bo_object $15 = {lock = {lock_object = {lo_name = 0xffffffff80c0f804 "vm object", lo_flags = 627245056, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, object_list = {tqe_next = 0xfffff80681a9cd68, tqe_prev = 0xfffff80681a9cb78}, shadow_head = {lh_first = 0x0}, shadow_list = {le_next = 0x0, le_prev = 0xfffff80019a69030}, memq = {tqh_first = 0xfffffe0012d02360, tqh_last = 0xfffffe001c379ed0}, rtree = {rt_root = 18446735311032427072}, size = 798661, domain = {dr_policy = 0x0, dr_iter = 0}, generation = 997761, cleangeneration = 1, ref_count = 0, shadow_count = 0, memattr = 6 '\006', type = 2 '\002', flags = 4104, pg_color = 0, paging_in_progress = {__count = 2147483777}, busy = {__count = 0}, resident_page_count = 518525, backing_object = 0x0, backing_object_offset = 0, pager_object_list = {tqe_next = 0x0, tqe_prev = 0x0}, rvq = {lh_first = 0xfffff807f9d6c9e0}, handle = 0xfffff806a60c6988, un_pager = {vnp = {vnp_size = 3271314029, writemappings = 0}, devp = {devp_pglist = {tqh_first = 0xc2fc4a6d, tqh_last = 0x0}, ops = 0x0, dev = 0x0}, sgp = {sgp_pglist = {tqh_first = 0xc2fc4a6d, tqh_last = 0x0}}, swp = { swp_tmpfs = 0xc2fc4a6d, swp_blks = {pt_root = 0}, writemappings = 0}, phys = {ops = 0xc2fc4a6d, {data_ptr = 0x0, data_val = 0}}}, cred = 0x0, charge = 0, umtx_data = 0x0} (kgdb) p/x $15.paging_in_progress $16 = {__count = 0x80000081} -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Mar 27 11:59:48 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1DAC65C1CEB for ; Sat, 27 Mar 2021 11:59:48 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6y7l4Fqlz3Nlk for ; Sat, 27 Mar 2021 11:59:47 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 523BC1C3E for ; Sat, 27 Mar 2021 07:59:46 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Sat, 27 Mar 2021 07:59:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=HZK+nYecQc98v8oY8x/nDyqIdsv 6MPBWneprBzxjnA4=; b=UnWA5rc4GYqR/3J6oDeaT1MBInBYHYZwFvP3lCMIhP5 ObkxyvA7knMnU71bBFBV/uzZ9YhDG7Rz3iBG69lXUFLBQcLx1Agn5/TiNjmADo31 57ojDCkj1Ib0HHDmcUfqPDMZQDM+reNOvvHQoxdVQc7kdTqe0NctP58xGSWhsxrv JpSHikgZWreSMAnrGSELApLT0IJqVsj10oyM+/AKLR/0FVEXEYR6SKxALp+8o+/F wj8DK+uU2eJlMKwgs99LPwyVAs14Mz88IhX69pBeKxgXT9M7pyQ6WnH/pr8XwaCY lZgfZ603fWcYKqZD+eF8bOFEeXHQUPKzJUNsNjkG4EQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=HZK+nY ecQc98v8oY8x/nDyqIdsv6MPBWneprBzxjnA4=; b=V+U8kLtEHrDFefkYsBxn2F /2nGFIp1DTnSFsV6pJFO+dgCDXPpJ4nDJWnzjomTn/NPsKpm+b8tt/R0C599Q027 QIZLBhF/DufJy9st3AYVK3InElCQ/pVdXGrVUJhx5uhYWrDd0cjOwXrXMPDt0Tsh gok7FkblLo+MRNKC/MQMK5J7mO/H01i+j5oJNnhV/FYfPwwVaZxcS+A8xxP+caYn +BhOqaLsTwOy9vyaX7KMrut9bYuZTm7gleS0nEAczu5yF20Fw9AqB/d/jzIEsa3R DoRqY8oQsaMU/DkRrmTheapt2y8e+DeLD3Ha30MjEVSC8BilrdUG+cCoEvGiw7Tw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudehgedgfeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucggtffrrghtthgvrhhnpedtheeigfdvudefkeekvddtfedvte dttdekuddvgeevlefftdekffdujedvhfduteenucfkphepkedvrdejtddrledurddutddt necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthgvtg hhqdhlihhsthhsseiihiigshhtrdhnvght X-ME-Proxy: Received: from ceres.zyxst.net (ceres.zyxst.net [82.70.91.100]) by mail.messagingengine.com (Postfix) with ESMTPA id 800591080057 for ; Sat, 27 Mar 2021 07:59:45 -0400 (EDT) Date: Sat, 27 Mar 2021 11:59:43 +0000 From: tech-lists To: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="+rKNgfZrojsotj+o" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4F6y7l4Fqlz3Nlk X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=UnWA5rc4; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=V+U8kLtE; dmarc=none; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.21 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-5.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[64.147.123.21:from]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Mar 2021 11:59:48 -0000 --+rKNgfZrojsotj+o Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Have you tried the following sysctl: vfs.read_max=3D128 ? --=20 J. --+rKNgfZrojsotj+o Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmBfHi8ACgkQs8o7QhFz NAUaPQ//f3MKDpeyDmEg/9+6bItKEFE8JaoWwXmyDC1wEi/Q2PYDr468X9IGG6Mp JetPG3y9I77T6TuoJu8q20CBkGR2Nhdz2dcSZMHWODzFRdBKdi73h7Xgp1Ohcy4u q4kgoQD8KUBvQqGdYMqsSbyVU/aCvVbmuFMxnDRgbxdrq3YH1gWJvJaU/jk58uam cwZ4lD2ZXiDNkwQM6gl9dzwfTzoAWbRKw8SS0cZgnRR/BMjljVfSimSy1cH/q9bD VM9/d6Gg97N/GEEx5SfMMRkFCEhDeXBxO5uWHOY6knY3OLkCRQIcKapHQpE/0zPu lD89PaANI4ZB7XPWtP0xnPTHK8Pfdh/32kJurKKZhMIsXAs76pfygLNfqC954ntN kOn2aLbNIUGYCEdpJ9909FYi92fMPmJcvQof7UMoueUjNaLkGr0Wup+b+T4VR2+d AHFBq6nRI+kMx+gsud8VxF2NS1w14RZ7BnuU3l6Y8mnHNOQe6R79dcTvhnMLEl8R X0SyRsN5rvx/1sXRG+qbIosxLWXzRsFSAae3j+YDGOl33RCXdnieX99qHq8s3qQ/ P9hvlExgAsNpQd9rk2gHJayZ5gYOywdyDruIUYefu6ohuvUmxYzqFGDp3QJoldNA Xb6YMeotqPu2sKzAWvtgYUCkkysHLkY0fmKdjT37MLv3rOUUzgc= =giXR -----END PGP SIGNATURE----- --+rKNgfZrojsotj+o-- From owner-freebsd-fs@freebsd.org Sat Mar 27 16:06:48 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C4C4E5787F9 for ; Sat, 27 Mar 2021 16:06:48 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F73cl6BSlz3tt3 for ; Sat, 27 Mar 2021 16:06:46 +0000 (UTC) (envelope-from mchouque@thi.eu.com) X-Originating-IP: 62.210.143.248 Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 599F91C0002; Sat, 27 Mar 2021 16:06:43 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 1B31A7203B9D5; Sat, 27 Mar 2021 17:06:43 +0100 (CET) Date: Sat, 27 Mar 2021 17:06:43 +0100 From: Mathieu Chouquet-Stringer To: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4F73cl6BSlz3tt3 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of mchouque@thi.eu.com has no SPF policy when checking 217.70.183.197) smtp.mailfrom=mchouque@thi.eu.com X-Spamd-Result: default: False [-1.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[me@mathieu.digital,mchouque@thi.eu.com]; RCVD_IN_DNSWL_LOW(-0.10)[217.70.183.197:from]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.70.183.197:from]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd]; FROM_NEQ_ENVFROM(0.00)[me@mathieu.digital,mchouque@thi.eu.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[mathieu.digital]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[217.70.183.197:from:127.0.2.255]; R_SPF_NA(0.00)[no SPF record]; RWL_MAILSPIKE_POSSIBLE(0.00)[217.70.183.197:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Mar 2021 16:06:48 -0000 Hello, TL;DR: read_max makes no difference, don't use hpet with KVM, use scan_legacy. On Sat, Mar 27, 2021 at 11:59:43AM +0000, tech-lists wrote: > Have you tried the following sysctl: > > vfs.read_max=128 It's something I haven't so I gave it a shot. Note that I first ran my tests on zroot which is on a virtio controller (the image file is on mirror made of two SSDs). This pool is small with only 12.1GB allocated out of 21.9GB: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 21.9G 12.1G 9.88G - - 37% 54% 1.00x ONLINE - Scrubbing is hard to bench because the status seems stuck forever: scan: scrub in progress since Sat Mar 27 15:16:27 2021 11.8G scanned at 146M/s, 1.23M issued at 15.2K/s, 12.1G total 0B repaired, 0.01% done, no estimated completion time scan: scrub in progress since Sat Mar 27 15:16:27 2021 12.0G scanned at 145M/s, 3.58M issued at 43.2K/s, 12.1G total 0B repaired, 0.03% done, no estimated completion time scan: scrub in progress since Sat Mar 27 15:16:27 2021 12.1G scanned at 142M/s, 409M issued at 4.70M/s, 12.1G total 0B repaired, 3.31% done, no estimated completion time Look how it already read 12.0G so pretty much 100% of the allocated space in the pool and it still shows 0.03% done... And with a bs of 128k (zfs record size), dd with iflags=direct tells me I can read this thing at: 23266197504 bytes (23 GB, 22 GiB) transferred 23.042s, 1010 MB/s With a bs of 32k I still get 310MB/s. So please find below scrubbing duration with vfs.read_max 512, 256, 128 and 64: it makes no difference. Please not it seems it's completely cpu bound again as I see this process using 100% of the CPU the whole time scrub is running: 31 root -8 - 0B 3952K CPU0 0 20:05 100.00% zfskern{txg_thread_enter} - vfs.read_max=512 vfs.read_max: 64 -> 512 - Scrub 1 scan: scrub repaired 0B in 00:02:53 with 0 errors on Sat Mar 27 15:10:41 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:13:33 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:54 with 0 errors on Sat Mar 27 15:16:27 2021 - vfs.read_max=256 vfs.read_max: 512 -> 256 - Scrub 1 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:19:19 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:53 with 0 errors on Sat Mar 27 15:22:12 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:25:04 2021 - vfs.read_max=128 vfs.read_max: 256 -> 128 - Scrub 1 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:27:56 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:53 with 0 errors on Sat Mar 27 15:30:49 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:33:41 2021 - vfs.read_max=64 vfs.read_max: 128 -> 64 - Scrub 1 scan: scrub repaired 0B in 00:02:53 with 0 errors on Sat Mar 27 15:36:34 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:52 with 0 errors on Sat Mar 27 15:39:26 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:53 with 0 errors on Sat Mar 27 15:42:19 2021 I am attaching a flamegraph for that busy process: again a lot of hpet_get_timecount... So I went and disabled hpet in kvm: I gained 30 seconds and again, playing with vfs.read_max made no difference but then again it's a ssd. And given I still am cpu bound I'm not sure I'll gain anything by playing with block sizes... But the first lesson is to not use hpet with KVM. - vfs.read_max=512 vfs.read_max: 64 -> 512 - Scrub 1 scan: scrub repaired 0B in 00:02:19 with 0 errors on Sat Mar 27 16:14:25 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:19 with 0 errors on Sat Mar 27 16:16:44 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:19:02 2021 - vfs.read_max=256 vfs.read_max: 512 -> 256 - Scrub 1 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:21:20 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:19 with 0 errors on Sat Mar 27 16:23:39 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:25:57 2021 - vfs.read_max=128 vfs.read_max: 256 -> 128 - Scrub 1 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:28:15 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:30:33 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:32:51 2021 - vfs.read_max=64 vfs.read_max: 128 -> 64 - Scrub 1 scan: scrub repaired 0B in 00:02:19 with 0 errors on Sat Mar 27 16:35:10 2021 - Scrub 2 scan: scrub repaired 0B in 00:02:18 with 0 errors on Sat Mar 27 16:37:28 2021 - Scrub 3 scan: scrub repaired 0B in 00:02:19 with 0 errors on Sat Mar 27 16:39:47 2021 An interesting thing is vfs.zfs.scan_legacy=1 makes the progress start right away. See how I read almost 50% of the disk and I'm already 49% done while with the non legacy scan, progress only moves after reading all the allocated space. scan: scrub in progress since Sat Mar 27 16:41:33 2021 5.85G scanned at 133M/s, 5.85G issued at 133M/s, 12.1G total 0B repaired, 48.50% done, 00:00:47 to go It's altogether way quicker (I gain 54 seconds on the no hpet test!!!) and totally reproducible: scan: scrub repaired 0B in 00:01:35 with 0 errors on Sat Mar 27 16:43:08 2021 scan: scrub repaired 0B in 00:01:35 with 0 errors on Sat Mar 27 16:46:10 2021 I will start a scrub of my zraid2 zpool and tweak read_max to see if it makes a difference but at this point I doubt it. And scan_legacy could be quicker because I see as many busy threads as there are vdevs while if I start a scan with the non legacy scan, I only see one from the get go. PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -16 - 0B 13M CPU0 0 8:40 99.17% kernel{z_rd_iss_4} 0 root -16 - 0B 13M CPU9 9 8:40 99.15% kernel{z_rd_iss_6} 0 root -16 - 0B 13M CPU7 7 8:40 98.97% kernel{z_rd_iss_5} 0 root -16 - 0B 13M CPU4 4 8:40 98.96% kernel{z_rd_iss_7} 0 root -16 - 0B 13M CPU6 6 8:40 98.59% kernel{z_rd_iss_1} 0 root -16 - 0B 13M CPU5 5 8:40 98.56% kernel{z_rd_iss_2} 0 root -16 - 0B 13M CPU14 14 8:40 98.33% kernel{z_rd_iss_3} 0 root -16 - 0B 13M CPU11 11 8:40 97.89% kernel{z_rd_iss_0} -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare -- From owner-freebsd-fs@freebsd.org Sat Mar 27 16:36:10 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0B13B579532 for ; Sat, 27 Mar 2021 16:36:10 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F74Gc6lXRz3vtf for ; Sat, 27 Mar 2021 16:36:07 +0000 (UTC) (envelope-from mchouque@thi.eu.com) Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu [62.210.143.248]) (Authenticated sender: m@thi.eu.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 6D2E9200005; Sat, 27 Mar 2021 16:36:05 +0000 (UTC) Received: by weirdfishes.localdomain (Postfix, from userid 1000) id 2AF3D7203B9D5; Sat, 27 Mar 2021 17:36:05 +0100 (CET) Date: Sat, 27 Mar 2021 17:36:05 +0100 From: Mathieu Chouquet-Stringer To: freebsd-fs@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: References: MIME-Version: 1.0 In-Reply-To: X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Mar 2021 16:36:10 -0000 On Sat, Mar 27, 2021 at 05:06:43PM +0100, Mathieu Chouquet-Stringer wrote: > I am attaching a flamegraph for that busy process: again a lot of > hpet_get_timecount... So I went and disabled hpet in kvm: I gained 30 > seconds and again, playing with vfs.read_max made no difference but then > again it's a ssd. And given I still am cpu bound I'm not sure I'll gain > anything by playing with block sizes... But the first lesson is to not > use hpet with KVM. Forgot to attache the svg file... -- Mathieu Chouquet-Stringer The sun itself sees not till heaven clears. -- William Shakespeare --