Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 14 Mar 2016 14:12:03 +0300
From:      DemIS <demis@yandex.ru>
To:        Steven Hartland <killing@multiplay.co.uk>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Subject:   Re: Kernel panic zio.c, line: 270 FreeBSD 10.2 (or 10.3)
Message-ID:  <7814671457953923@web29j.yandex.ru>
In-Reply-To: <56E5E96B.8040209@multiplay.co.uk>
References:  <1061671455578760@web3g.yandex.ru> <56C2655F.9010809@multiplay.co.uk> <1076701455583595@web2g.yandex.ru> <56C2782E.2010404@multiplay.co.uk> <2311371455610101@web11j.yandex.ru> <56C2E7A6.9090004@multiplay.co.uk> <4332271457902960@web8o.yandex.ru> <56E5E96B.8040209@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help 
Hi, Steven,

 more information for you:
 I wrote a script component.
 He gets all the directory inode and checks them through the program zdb -A -dddddd inode.
 And this is not without causing the server is restarted, only the crash zdb program.

 Over the past few weeks (so much time working script, one inode for about three seconds) unable to verify 163272 folder.
 I got 55 inode folders with errors, an example of the error:
zdb -A -dddddd hdd/usr/wf 7962880

Dataset hdd/usr/wf [ZPL], ID 44, cr_txg 25, 3.82T, 2548316 objects, rootbp DVA[0]=<0:621dba8000:3000> DVA[1]=<0:3a4f9aa0000:3000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=19128707L/19128707P fill=2548316 cksum=d2d2f7fd9:fb64a627f1e:bb115b7f9840d:672c8b8b245ae81

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
   7962880    1    16K    512  28.5K    512  100.00  ZFS directory (K=inherit) (Z=inherit)
                                        304   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED SPILL_BLKPTR
        dnode maxblkid: 0
        path    /_๓ิีฤษั_็๐/ํ๏_๏ิามฤฮลฮำหฯล ็๐/04-็๐ํ๏_๐้-02/07-ไาีวฯล
        uid     10010
        gid     10001
        atime   Mon Mar 14 13:33:12 2016
        mtime   Fri May 22 12:52:21 2015
        ctime   Thu Dec 10 10:32:40 2015
        crtime  Thu Dec 10 10:32:40 2015
        gen     17690948
        mode    40000
        size    2
        parent  7962875
        links   2
        pflags  40800000042
Assertion failed: db->db.db_size >= dn->dn_bonuslen (0x0 >= 0x130), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c, line 339.
Assertion failed: (!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 3293.
Assertion failed: size > 0 (0x0 > 0x0), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 1536.
Assertion failed: (bytes > 0), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 2789.
Assertion failed: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 270.
Bus error (core dumped)

I plan to do the same for node files.
 But:
 1. This is a very very long time.
 2. every time when there is a crash of the entire system has to run the script again, because log processing greatly damaged.
 And almost no chance that the script files for my time to work through.

 Working copies of damaged files and folders I have in backups.
 But the problem is that I can not replace a broken folder, because rm folder or rm file leads to a crash.


14.03.2016, 01:27, "Steven Hartland" <killing@multiplay.co.uk>:
> Thanks DemIS, size = 0 is definitely bad however I don't see bp listed
> from frame #8.
>
> šFrom the debugger:
> š> frame 8
> š> print bp
>
> On 13/03/2016 21:02, DemIS wrote:
>> šHi Steven ,
>>
>> šFinally I was able to get the data that you have requested.
>> šThe following quote from the dump to
>> šššsetfacl -b -m <someAccount> /hdd/usr/wf/some/long/path
>>
>> škgdb /usr/obj/usr/src/sys/TEO/kernel.debug /usr/crash/vmcore.last
>> šGNU gdb 6.1.1 [FreeBSD]
>> šCopyright 2004 Free Software Foundation, Inc.
>> šGDB is free software, covered by the GNU General Public License, and you are
>> šwelcome to change it and/or distribute copies of it under certain conditions.
>> šType "show copying" to see the conditions.
>> šThere is absolutely no warranty for GDB. Type "show warranty" for details.
>> šThis GDB was configured as "amd64-marcel-freebsd"...
>>
>> šUnread portion of the kernel message buffer:
>> španic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270
>> šcpuid = 2
>> šKDB: stack backtrace:
>> šdb_trace_self_wrapper() at db_trace_self_wrapper+0x2e/frame 0xfffffe0677ebfbe0
>> škdb_backtrace() at kdb_backtrace+0x58/frame 0xfffffe0677ebfc90
>> švpanic() at vpanic+0x1df/frame 0xfffffe0677ebfd00
>> švpanic() at vpanic/frame 0xfffffe0677ebfd60
>> šzfs_kmem_alloc() at zfs_kmem_alloc/frame 0xfffffe0677ebfdb0
>> šzio_buf_alloc() at zio_buf_alloc+0x84/frame 0xfffffe0677ebfdf0
>> šarc_get_data_buf() at arc_get_data_buf+0x7a/frame 0xfffffe0677ebfe60
>> šarc_buf_alloc() at arc_buf_alloc+0xe0/frame 0xfffffe0677ebfeb0
>> šarc_read() at arc_read+0x65d/frame 0xfffffe0677ec0040
>> šdbuf_read_impl() at dbuf_read_impl+0x4af/frame 0xfffffe0677ec00e0
>> šdbuf_read() at dbuf_read+0x33b/frame 0xfffffe0677ec0190
>> šdmu_spill_hold_by_dnode() at dmu_spill_hold_by_dnode+0xf7/frame 0xfffffe0677ec01e0
>> šdmu_spill_hold_existing() at dmu_spill_hold_existing+0x16b/frame 0xfffffe0677ec0240
>> šsa_get_spill() at sa_get_spill+0x3c/frame 0xfffffe0677ec0260
>> šsa_modify_attrs() at sa_modify_attrs+0x118/frame 0xfffffe0677ec0330
>> šsa_attr_op() at sa_attr_op+0x9db/frame 0xfffffe0677ec03a0
>> šsa_bulk_update_impl() at sa_bulk_update_impl+0x74/frame 0xfffffe0677ec03f0
>> šsa_bulk_update() at sa_bulk_update+0x45/frame 0xfffffe0677ec0430
>> šzfs_aclset_common() at zfs_aclset_common+0x95f/frame 0xfffffe0677ec0620
>> šzfs_setacl() at zfs_setacl+0x451/frame 0xfffffe0677ec06c0
>> šzfs_setsecattr() at zfs_setsecattr+0x10d/frame 0xfffffe0677ec0720
>> šzfs_freebsd_setacl() at zfs_freebsd_setacl+0x15c/frame 0xfffffe0677ec0790
>> šVOP_SETACL_APV() at VOP_SETACL_APV+0x174/frame 0xfffffe0677ec07f0
>> šVOP_SETACL() at VOP_SETACL+0x5a/frame 0xfffffe0677ec0860
>> švacl_set_acl() at vacl_set_acl+0xf1/frame 0xfffffe0677ec08c0
>> šsys___acl_set_file() at sys___acl_set_file+0x91/frame 0xfffffe0677ec09d0
>> šsyscallenter() at syscallenter+0x754/frame 0xfffffe0677ec0af0
>> šamd64_syscall() at amd64_syscall+0x1f/frame 0xfffffe0677ec0bf0
>> šXfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0677ec0bf0
>> š--- syscall (348, FreeBSD ELF64, sys___acl_set_file), rip = 0x80092a17a, rsp = 0x7fffffffe2e8, rbp = 0x7fffffffe7c0 ---
>> šUptime: 2d11h24m58s
>> šDumping 5431 out of 24533 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> šReading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/if_lagg.ko.symbols
>> šReading symbols from /boot/kernel/aio.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/aio.ko.symbols
>> šReading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/ichsmb.ko.symbols
>> šReading symbols from /boot/kernel/ipmi.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/ipmi.ko.symbols
>> šReading symbols from /boot/kernel/zfs.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/zfs.ko.symbols
>> šReading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/opensolaris.ko.symbols
>> šReading symbols from /boot/kernel/ums.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/ums.ko.symbols
>> šReading symbols from /boot/kernel/smbfs.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/smbfs.ko.symbols
>> šReading symbols from /boot/kernel/libiconv.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/libiconv.ko.symbols
>> šReading symbols from /boot/kernel/libmchain.ko.symbols...done.
>> šLoaded symbols for /boot/kernel/libmchain.ko.symbols
>> š#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:263
>> š263 dumptid = curthread->td_tid;
>> š(kgdb) bt full
>> š#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:263
>> ššššššššššcoredump = 1
>> š#1 0xffffffff80e21a6a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
>> ššššššššššfirst_buf_printf = 1
>> š#2 0xffffffff80e2240c in vpanic (fmt=0xffffffff8268f154 "solaris assert: %s (0x%jx %s 0x%jx), file: %s, line: %d", ap=0xfffffe0677ebfd40)
>> ššššššat /usr/src/sys/kern/kern_shutdown.c:758
>> ššššššššššbuf = "solaris assert: c < (1ULL << 24) >> 9 (0x7", 'f' <repeats 13 times>, " < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270", '\0' <repeats 92 times>
>> ššššššššššother_cpus = {__bits = {65531}}
>> šššššššššštd = (struct thread *) 0xfffff800713124a0
>> ššššššššššbootopt = 260
>> ššššššššššnewpanic = 1
>> š#3 0xffffffff80e221d0 in panic (fmt=0xffffffff8268f154 "solaris assert: %s (0x%jx %s 0x%jx), file: %s, line: %d")
>> ššššššat /usr/src/sys/kern/kern_shutdown.c:687
>> ššššššššššap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0xfffffe0677ebfd78, reg_save_area = 0xfffffe0677ebfd10}}
>> š#4 0xffffffff8268e470 in assfail3 (a=0xffffffff825cbcb9 "c < (1ULL << 24) >> 9", lv=36028797018963967, op=0xffffffff825cbccf "<", rv=32768,
>> ššššššf=0xffffffff825cbcd1 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c", l=270)
>> ššššššat /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>> šNo locals.
>> š#5 0xffffffff8250e414 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:270
>> šššššššššš__left = 36028797018963967
>> šššššššššš__right = 32768
>> ššššššššššc = 36028797018963967
>> ššššššššššflags = 0
>> š#6 0xffffffff8243dc1a in arc_get_data_buf (buf=0xfffff804795ddc18)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>> ššššššššššstate = (arc_state_t *) 0xffffffff825e8700
>> ššššššššššsize = 0
>> šššššššššštype = ARC_BUFC_METADATA
>> š#7 0xffffffff8243db60 in arc_buf_alloc (spa=0xfffff800719d1000, size=0, tag=0xfffff803b1e8de00, type=ARC_BUFC_METADATA)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1551
>> ššššššššššhdr = (arc_buf_hdr_t *) 0xfffff8025cc22288
>> ššššššššššbuf = (arc_buf_t *) 0xfffff804795ddc18
>> š#8 0xffffffff824419bd in arc_read (pio=0xfffff80071def398, spa=0xfffff800719d1000, bp=0xfffffe001589e180, done=0xffffffff82455380 <dbuf_read_done>,
>> ššššššprivate=0xfffff803b1e8de00, priority=ZIO_PRIORITY_SYNC_READ, zio_flags=128, arc_flags=0xfffffe0677ec0094, zb=0xfffffe0677ec0098)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3389
>> ššššššššššexists = (arc_buf_hdr_t *) 0x0
>> šššššššššštype = ARC_BUFC_METADATA
>> ššššššššššvd = (vdev_t *) 0x0
>> ššššššššššdevw = 0
>> ššššššššššb_asize = 0
>> ššššššššššsize = 0
>> ššššššššššacb = (arc_callback_t *) 0xfffff800713124a0
>> ššššššššššaddr = 0
>> ššššššššššb_compress = ZIO_COMPRESS_OFF
>> ššššššššššsdt_sdt___arc__hit = {{version = 1287817688, prov = 0x30c48348fffff834, probe_entry = {tqe_next = 0x2e6666666666c35d, tqe_prev = 0x841f0f},
>> ššššššargtype_list = {tqh_first = 0x20ec8348e5894855, tqh_last = 0xf0758948f87d8948},
>> ššššššmod = 0xf87d8b48e8558948 <Address 0xf87d8b48e8558948 out of bounds>, func = 0xe8558b48f0758b48 <Address 0xe8558b48f0758b48 out of bounds>,
>> ššššššname = 0x12b8c189c031 <Address 0x12b8c189c031 out of bounds>, id = 5531134888965130240, n_args = -387955317, sdtp_lf = 0x9066c35d20c48348}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___arc__hit = (const void * const) 0x58f0f000000
>> ššššššššššsdta_sdt___arc__hit0 = {{ndx = -443987883, type = 0x4c984d894c000001 <Address 0x4c984d894c000001 out of bounds>,
>> ššššššxtype = 0x48884d8948904589 <Address 0x48884d8948904589 out of bounds>, argtype_entry = {tqe_next = 0xb8d04d8d48805589, tqe_prev = 0xba4100000001},
>> ššššššprobe = 0x7589f87d89480000}}
>> š---Type <return> to continue, or q <return> to quit---
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___arc__hit0 = (const void * const) 0x1518e0f000000
>> ššššššššššsdt_sdt___arc__miss = {{version = -193623052, prov = 0xfe810000, probe_entry = {tqe_next = 0x41db8445c3950f41, tqe_prev = 0x45c7c44589c2440f},
>> ššššššargtype_list = {tqh_first = 0x958d4800000000c0, tqh_last = 0x10518948ffffff70}, mod = 0x851894810558d48 <Address 0x851894810558d48 out of bounds>,
>> ššššššfunc = 0xc7000000300441c7 <Address 0xc7000000300441c7 out of bounds>, name = 0x8100b00000001001 <Address 0x8100b00000001001 out of bounds>,
>> ššššššid = -8824803469832437635, n_args = -145, sdtp_lf = 0x4d8bd0458d480000}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___arc__miss = (const void * const) 0x2b800000174e900
>> ššššššššššsdta_sdt___arc__miss0 = {{ndx = 687440336, type = 0x8d89ffffff608589 <Address 0x8d89ffffff608589 out of bounds>,
>> ššššššxtype = 0x2b870fffffff5c <Address 0x2b870fffffff5c out of bounds>, argtype_entry = {tqe_next = 0xffff60858b480000, tqe_prev = 0x5c958b10488b48ff},
>> ššššššprobe = 0x148f26348ffffff}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___arc__miss0 = (const void * const) 0x8be04589e4458b00
>> ššššššššššsdta_sdt___arc__miss1 = {{ndx = 146964977, type = 0xffffff508d894810 <Address 0xffffff508d894810 out of bounds>,
>> ššššššxtype = 0x858b4800000020e9 <Address 0x858b4800000020e9 out of bounds>, argtype_entry = {tqe_next = 0x8488b48ffffff60, tqe_prev = 0x8c18148ca8948},
>> ššššššprobe = 0x8948084889480000}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___arc__miss1 = (const void * const) 0x1f8d0ff4453be045
>> ššššššššššsdta_sdt___arc__miss2 = {{ndx = -44907, type = 0x48008b48ffffff50 <Address 0x48008b48ffffff50 out of bounds>,
>> ššššššxtype = 0x3d48c84589 <Address 0x3d48c84589 out of bounds>, argtype_entry = {tqe_next = 0xff6f8d88c1950f00, tqe_prev = 0xffffff6f858affff},
>> ššššššprobe = 0x5850f01a8}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___arc__miss2 = (const void * const) 0x452be0458b000001
>> ššššššššššsdta_sdt___arc__miss3 = {{ndx = 186601, type = 0x28f981d04d8bd0 <Address 0x28f981d04d8bd0 out of bounds>,
>> ššššššxtype = 0xffff488589480000 <Address 0xffff488589480000 out of bounds>, argtype_entry = {tqe_next = 0xfffffff448d89ff,
>> šššššššštqe_prev = 0x858b480000002b87}, probe = 0x10488b48ffffff48}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___arc__miss3 = (const void * const) 0xdc7d81dc4589e4
>> ššššššššššsdt_sdt___l2arc__hit = {{version = -12282485, prov = 0x8c281f10148f2, probe_entry = {tqe_next = 0x388d894810890000, tqe_prev = 0x20e9ffffff},
>> ššššššargtype_list = {tqh_first = 0x48ffffff48858b48, tqh_last = 0x8148ca894808488b},
>> ššššššmod = 0x48894800000008c1 <Address 0x48894800000008c1 out of bounds>, func = 0xffffff3895894808 <Address 0xffffff3895894808 out of bounds>,
>> ššššššname = 0x8bffffff38858b48 <Address 0x8bffffff38858b48 out of bounds>, id = 5587026407265569032, n_args = -1982953028, sdtp_lf = 0x89ffffff309589ff}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___l2arc__hit = (const void * const) 0x4589f44d8b000000
>> ššššššššššsdta_sdt___l2arc__hit0 = {{ndx = -54091, type = 0xec03fcb8000002 <Address 0xec03fcb8000002 out of bounds>,
>> ššššššxtype = 0xa30fffffff348d8b <Address 0xa30fffffff348d8b out of bounds>, argtype_entry = {tqe_next = 0xb800000058820fc8,
>> šššššššštqe_prev = 0xff348d8b0713fc00}, probe = 0xdb820fc8a30fffff}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___l2arc__hit0 = (const void * const) 0xf48c0f000000
>> ššššššššššsdt_sdt___l2arc__read = {{version = -1962934272, prov = 0xff28858901e883ff, probe_entry = {tqe_next = 0x5840fffff,
>> šššššššštqe_prev = 0x7d8b48000001d3e9}, argtype_list = {tqh_first = 0xbc558bc8758b48f8, tqh_last = 0x208d8948c189c031},
>> ššššššmod = 0xff20858b4cffffff <Address 0xff20858b4cffffff out of bounds>, func = 0x89fffff5c1e8ffff <Address 0x89fffff5c1e8ffff out of bounds>,
>> ššššššname = 0x48000001b0e9c045 <Address 0x48000001b0e9c045 out of bounds>, id = -467863854793079411, n_args = 40, sdtp_lf = 0xffff148d89ffffff}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___l2arc__read = (const void * const) 0xf7c04d8b99c889c0
>> ššššššššššsdta_sdt___l2arc__read0 = {{ndx = 730271743, type = 0x8b48ffffff18858b <Address 0x8b48ffffff18858b out of bounds>,
>> ššššššxtype = 0xffffff14958b1048 <Address 0xffffff14958b1048 out of bounds>, argtype_entry = {tqe_next = 0xc281f10148f26348,
>> šššššššštqe_prev = 0x8948108900000008}, probe = 0x20e9ffffff088d}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___l2arc__read0 = (const void * const) 0x4d8b48e8458b4800
>> ššššššššššsdta_sdt___l2arc__read1 = {{ndx = -1958215680, type = 0xca894808488b48ff <Address 0xca894808488b48ff out of bounds>,
>> ššššššxtype = 0x4800000008c18148 <Address 0x4800000008c18148 out of bounds>, argtype_entry = {tqe_next = 0xff08958948084889,
>> šššššššštqe_prev = 0xffff08858b48ffff}, probe = 0xb0458948008b48ff}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___l2arc__read1 = (const void * const) 0xf055af0fdc558bf8
>> ššššššššššsdt_sdt___l2arc__miss = {{version = -125990072, prov = 0x31b0458b4cbc558b, probe_entry = {tqe_next = 0x4589fffff52ae8c9,
>> šššššššštqe_prev = 0x8d4800000119e9c0}, argtype_list = {tqh_first = 0x28f981d04d8bd045, tqh_last = 0xff00858948000000},
>> ššššššmod = 0xfffffefc8d89ffff <Address 0xfffffefc8d89ffff out of bounds>, func = 0x8b480000002b870f <Address 0x8b480000002b870f out of bounds>,
>> ššššššname = 0x488b48ffffff0085 <Address 0x488b48ffffff0085 out of bounds>, id = 5260204360416463632, n_args = 21557859, sdtp_lf = 0x8d89481089000000}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___l2arc__miss = (const void * const) 0xe47d81e44589f9
>> ššššššššššsdta_sdt___l2arc__miss0 = {{ndx = -272, type = 0xffffff00858b4800 <Address 0xffffff00858b4800 out of bounds>,
>> ššššššxtype = 0x48ca894808488b48 <Address 0x48ca894808488b48 out of bounds>, argtype_entry = {tqe_next = 0x894800000008c181,
>> šššššššštqe_prev = 0xfffef09589480848}, probe = 0xfffffef0858b48ff}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___l2arc__miss0 = (const void * const) 0x8b48f10148f26348
>> š---Type <return> to continue, or q <return> to quit---
>> ššššššššššsdt_sdt___l2arc__miss = {{version = -800223928, prov = 0xfa81d0558bb04589, probe_entry = {tqe_next = 0xe88d894800000028,
>> šššššššštqe_prev = 0xfffee49589fffffe}, argtype_list = {tqh_first = 0x480000002b870fff, tqh_last = 0x8b48fffffee8858b},
>> ššššššmod = 0xfffffee4958b1048 <Address 0xfffffee4958b1048 out of bounds>, func = 0xc281f10148f26348 <Address 0xc281f10148f26348 out of bounds>,
>> ššššššname = 0x8948108900000008 <Address 0x8948108900000008 out of bounds>, id = 9264484975564941, n_args = -1958215680, sdtp_lf = 0xca894808488b48ff}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___l2arc__miss = (const void * const) 0x1518e0f000000
>> ššššššššššsdta_sdt___l2arc__miss0 = {{ndx = 146899272, type = 0xfed8958948084889 <Address 0xfed8958948084889 out of bounds>,
>> ššššššxtype = 0xfffed8858b48ffff <Address 0xfffed8858b48ffff out of bounds>, argtype_entry = {tqe_next = 0xa8458948008b48ff,
>> šššššššštqe_prev = 0xc8758b48f87d8b48}, probe = 0x4ca84d8b48bc558b}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___l2arc__miss0 = (const void * const) 0xe45503dc558bf875
>> ššššššššššhdr = (arc_buf_hdr_t *) 0x0
>> ššššššššššbuf = (arc_buf_t *) 0x0
>> ššššššššššhash_lock = (kmutex_t *) 0x0
>> ššššššššššrzio = (zio_t *) 0xfffffe0677ebfff0
>> ššššššššššguid = 18385013180861733512
>> š#9 0xffffffff8244f7df in dbuf_read_impl (db=0xfffff803b1e8de00, zio=0xfffff80071def398, flags=0xfffffe0677ec0174)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>> ššššššššššdn = (dnode_t *) 0xfffff805c8be5000
>> ššššššššššzb = {zb_objset = 44, zb_object = 7962880, zb_level = 0, zb_blkid = 18446744073709551614}
>> ššššššššššaflags = 100
>> š#10 0xffffffff8244ef8b in dbuf_read (db=0xfffff803b1e8de00, zio=0xfffff80071def398, flags=6)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:621
>> ššššššššššspa = (spa_t *) 0xfffff800719d1000
>> ššššššššššsdt_sdt___blocked__read = {{version = 549749576, prov = 0x841f0f, probe_entry = {tqe_next = 0x20ec8348e5894855, tqe_prev = 0xf0758948f87d8948},
>> ššššššargtype_list = {tqh_first = 0xf0758b48f87d8b48, tqh_last = 0x6b8c289c031}, mod = 0x48c689e875894800 <Address 0x48c689e875894800 out of bounds>,
>> ššššššfunc = 0xfffffc60e8e84d8b <Address 0xfffffc60e8e84d8b out of bounds>, name = 0x2e66c35d20c48348 <Address 0x2e66c35d20c48348 out of bounds>,
>> ššššššid = 8658703, n_args = -443987883, sdtp_lf = 0xf0758948f87d8948}}
>> šššššššššš__set_sdt_probes_set_sym_sdt_sdt___blocked__read = (const void * const) 0xd089b44589f0558b
>> ššššššššššsdta_sdt___blocked__read0 = {{ndx = -125990072, type = 0x7b8c289c031 <Address 0x7b8c289c031 out of bounds>,
>> ššššššxtype = 0x48c689e875894800 <Address 0x48c689e875894800 out of bounds>, argtype_entry = {tqe_next = 0xfffffc20e8e84d8b,
>> šššššššštqe_prev = 0x2e66c35d20c48348}, probe = 0x841f0f}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___blocked__read0 = (const void * const) 0x4c28148c28948
>> ššššššššššsdta_sdt___blocked__read1 = {{ndx = -443987883, type = 0xf0758948f87d8948 <Address 0xf0758948f87d8948 out of bounds>,
>> ššššššxtype = 0xf0758b48f87d8b48 <Address 0xf0758b48f87d8b48 out of bounds>, argtype_entry = {tqe_next = 0x8b8c289c031, tqe_prev = 0x48c689e875894800},
>> ššššššprobe = 0xfffffbe0e8e84d8b}}
>> šššššššššš__set_sdt_argtypes_set_sym_sdta_sdt___blocked__read1 = (const void * const) 0x889c85589480000
>> ššššššššššerr = 0
>> ššššššššššhavepzio = 0
>> ššššššššššprefetch = 1
>> ššššššššššdn = (dnode_t *) 0xfffff805c8be5000
>> š#11 0xffffffff8245a777 in dmu_spill_hold_by_dnode (dn=0xfffff805c8be5000, flags=6, tag=0x0, dbp=0xfffff802f48dfa28)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>> ššššššššššdb = (dmu_buf_impl_t *) 0xfffff803b1e8de00
>> ššššššššššerr = -512
>> š#12 0xffffffff8245a91b in dmu_spill_hold_existing (bonus=0xfffff80052609620, tag=0x0, dbp=0xfffff802f48dfa28)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:359
>> ššššššššššdb = (dmu_buf_impl_t *) 0xfffff80052609620
>> ššššššššššdn = (dnode_t *) 0xfffff805c8be5000
>> ššššššššššerr = 1
>> š#13 0xffffffff824b140c in sa_get_spill (hdl=0xfffff802f48dfa00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:310
>> ššššššššššrc = 0
>> š#14 0xffffffff824b1658 in sa_modify_attrs (hdl=0xfffff802f48dfa00, newattr=19, action=SA_REPLACE, locator=0xffffffff82520d50 <zfs_acl_data_locator>,
>> ššššššdatastart=0xfffffe0677ec05c8, buflen=152, tx=0xfffff8034ef07800)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1681
>> š---Type <return> to continue, or q <return> to quit---
>> ššššššššššsa = (sa_os_t *) 0xfffff80071e5da00
>> ššššššššššdb = (dmu_buf_impl_t *) 0xfffff80052609620
>> ššššššššššdn = (dnode_t *) 0xfffff805c8be5000
>> ššššššššššattr_desc = (sa_bulk_attr_t *) 0xfffff803c1d2a000
>> ššššššššššold_data = {0xfffff80122383800, 0xfffff8034ef07800}
>> ššššššššššbonus_attr_count = 14
>> ššššššššššbonus_data_size = 320
>> ššššššššššspill_data_size = 0
>> ššššššššššspill_attr_count = 0
>> ššššššššššerror = -2045
>> ššššššššššlength = 49618
>> šššššššššši = 0
>> ššššššššššj = 0
>> ššššššššššk = 2
>> ššššššššššlength_idx = 16691240
>> ššššššššššhdr = (sa_hdr_phys_t *) 0xfffff80052600200
>> ššššššššššidx_tab = (sa_idx_tab_t *) 0xffffffff825bb2ea
>> ššššššššššattr_count = 0
>> ššššššššššcount = -1367343104
>> š#15 0xffffffff824b12fb in sa_attr_op (hdl=0xfffff802f48dfa00, bulk=0xfffffe0677ec04f0, count=5, data_op=SA_UPDATE, tx=0xfffff8034ef07800)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:392
>> ššššššššššsa = (sa_os_t *) 0xfffff80071e5da00
>> šššššššššši = 3
>> ššššššššššerror = 0
>> ššššššššššbuftypes = SA_BONUS
>> š#16 0xffffffff824b5a04 in sa_bulk_update_impl (hdl=0xfffff802f48dfa00, bulk=0xfffffe0677ec04f0, count=5, tx=0xfffff8034ef07800)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1790
>> ššššššššššerror = 1888
>> ššššššššššsa = (sa_os_t *) 0xfffff80071e5da00
>> ššššššššššbonustype = DMU_OT_SA
>> š#17 0xffffffff824b5ca5 in sa_bulk_update (hdl=0xfffff802f48dfa00, attrs=0xfffffe0677ec04f0, count=5, tx=0xfffff8034ef07800)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1889
>> ššššššššššerror = 1
>> š#18 0xffffffff82521b5f in zfs_aclset_common (zp=0xfffff803b27e85c0, aclp=0xfffff8001ebcfa00, cr=0xfffff804455e6e00, tx=0xfffff8034ef07800)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c:1329
>> ššššššššššerror = -2045
>> ššššššššššzfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> ššššššššššotype = DMU_OT_ACL
>> ššššššššššlocate = {cb_aclp = 0xfffff8001ebcfa00, cb_acl_node = 0x0}
>> ššššššššššmode = 16893
>> ššššššššššbulk = {{sa_data = 0xfffffe0677ec05c0, sa_data_func = 0, sa_length = 8, sa_attr = 5, sa_addr = 0xfffff80052600208, sa_buftype = 1,
>> ššššššsa_size = 8}, {sa_data = 0xfffff803b27e86c8, sa_data_func = 0, sa_length = 8, sa_attr = 11, sa_addr = 0xfffff80052600238, sa_buftype = 1,
>> ššššššsa_size = 8}, {sa_data = 0xfffffe0677ec04e0, sa_data_func = 0, sa_length = 16, sa_attr = 2, sa_addr = 0xfffff80052600260, sa_buftype = 1,
>> ššššššsa_size = 16}, {sa_data = 0xfffffe0677ec05c8, sa_data_func = 0xffffffff82520d50 <zfs_acl_data_locator>, sa_length = 152, sa_attr = 19,
>> ššššššsa_addr = 0xfffff80052600290, sa_buftype = 1, sa_size = 160}, {sa_data = 0xfffff8001ebcfa00, sa_data_func = 0, sa_length = 8, sa_attr = 16,
>> ššššššsa_addr = 0xfffffe0008161250, sa_buftype = 0, sa_size = 48}}
>> ššššššššššctime = {1457872316, 787888506}
>> ššššššššššcount = 5
>> š#19 0xffffffff82524391 in zfs_setacl (zp=0xfffff803b27e85c0, vsecp=0xfffffe0677ec0748, skipaclchk=0, cr=0xfffff804455e6e00)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c:2010
>> ššššššššššzfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> ššššššššššzilog = (zilog_t *) 0xfffff80086ebd000
>> ššššššššššmask = 16
>> š---Type <return> to continue, or q <return> to quit---
>> šššššššššštx = (dmu_tx_t *) 0xfffff8034ef07800
>> ššššššššššerror = 0
>> ššššššššššaclp = (zfs_acl_t *) 0xfffff8001ebcfa00
>> ššššššššššfuidp = (zfs_fuid_info_t *) 0x0
>> ššššššššššfuid_dirtied = 0
>> ššššššššššacl_obj = 0
>> š#20 0xffffffff8254e33d in zfs_setsecattr (vp=0xfffff8034dbfa000, vsecp=0xfffffe0677ec0748, flag=0, cr=0xfffff804455e6e00, ct=0x0)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5332
>> ššššššššššzp = (znode_t *) 0xfffff803b27e85c0
>> ššššššššššzfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> ššššššššššerror = 2
>> ššššššššššskipaclchk = 0
>> ššššššššššzilog = (zilog_t *) 0xfffff80086ebd000
>> š#21 0xffffffff8254ff5c in zfs_freebsd_setacl (ap=0xfffffe0677ec0808)
>> ššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:7076
>> ššššššššššerror = 0
>> ššššššššššvsecattr = {vsa_mask = 16, vsa_aclcnt = 11, vsa_aclentp = 0xfffff804bd725e00, vsa_dfaclcnt = 2011957208, vsa_dfaclentp = 0xfffffe0677ec07c0,
>> ššššvsa_aclentsz = 132, vsa_aclflags = 2}
>> ššššššššššaclbsize = 132
>> ššššššššššaaclp = (aclent_t *) 0xfffff804bd725e00
>> š#22 0xffffffff816db594 in VOP_SETACL_APV (vop=0xffffffff825de1a0, a=0xfffffe0677ec0808) at vnode_if.c:2993
>> ššššššššššmp__ = (struct mount *) 0xfffff80086df5cc0
>> šššššššššš_enable_stops = 0
>> ššššššššššrc = -1
>> š#23 0xffffffff80f0ba5a in VOP_SETACL (vp=0xfffff8034dbfa000, type=4, aclp=0xfffff800612a1000, cred=0xfffff804455e6e00, td=0xfffff800713124a0)
>> ššššššat vnode_if.h:1253
>> šššššššššša = {a_gen = {a_desc = 0xffffffff81d77eb8}, a_vp = 0xfffff8034dbfa000, a_type = 4, a_aclp = 0xfffff800612a1000, a_cred = 0xfffff804455e6e00,
>> šššša_td = 0xfffff800713124a0}
>> š#24 0xffffffff80f0aea1 in vacl_set_acl (td=0xfffff800713124a0, vp=0xfffff8034dbfa000, type=4, aclp=0x80100a000) at /usr/src/sys/kern/vfs_acl.c:232
>> ššššššššššinkernelacl = (struct acl *) 0xfffff800612a1000
>> ššššššššššmp = (struct mount *) 0xfffff80086df5cc0
>> ššššššššššerror = 0
>> š#25 0xffffffff80f0ad81 in sys___acl_set_file (td=0xfffff800713124a0, uap=0xfffffe0677ec0b98) at /usr/src/sys/kern/vfs_acl.c:375
>> ššššššššššnd = {ni_dirp = 0x7fffffffecd2 <Address 0x7fffffffecd2 out of bounds>, ni_segflg = UIO_USERSPACE, ni_rightsneeded = {cr_rights = {
>> šššššššš144115188075855872, 288230376151711744}}, ni_startdir = 0x0, ni_rootdir = 0xfffff800714cfb10, ni_topdir = 0x0, ni_dirfd = -100,
>> ššššni_strictrelative = 0, ni_filecaps = {fc_rights = {cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, fc_fcntls = 0}, ni_vp = 0xfffff8034dbfa000,
>> ššššni_dvp = 0xfffff805c678d000, ni_pathlen = 1, ni_next = 0xfffff80086ddcc65 "", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 0, cn_flags = 49216,
>> ššššššcn_thread = 0xfffff800713124a0, cn_cred = 0xfffff804455e6e00, cn_lkflags = 2097152,
>> ššššššcn_pnbuf = 0xfffff80086ddcc00 "/hdd/usr/wf/_๓-\202-\203ฤษ-\217_-\223-\237/-\234-\236_-\236-\202-\200มฤฮลฮ-\201หฯล -\223-\237/04--\223-\237-\234-\236_-\237-\230-02/07--\224-\200-\203วฯล", cn_nameptr = 0xfffff80086ddcc56 "07--\224-\200-\203วฯล", cn_namelen = 15, cn_consume = 0}}
>> ššššššššššerror = 0
>> š#26 0xffffffff814e32f4 in syscallenter (td=0xfffff800713124a0, sa=0xfffffe0677ec0b88) at subr_syscall.c:134
>> ššššššššššp = (struct proc *) 0xfffff80071317000
>> ššššššššššerror = 0
>> šššššššššštraced = 0
>> š#27 0xffffffff814e2a8f in amd64_syscall (td=0xfffff800713124a0, traced=0) at /usr/src/sys/amd64/amd64/trap.c:952
>> ššššššššššsa = {code = 348, callp = 0xffffffff81d026d0, args = {140737488350418, 4, 34376556544, 2, 140737488345344, 34376515680, 0, 34376896576},
>> ššššnarg = 3}
>> ššššššššššerror = 0
>> ššššššššššksi = {ksi_link = {tqe_next = 0x202, tqe_prev = 0x202}, ksi_info = {si_signo = 2011958080, si_errno = -506, si_code = -2125728811, si_pid = -1,
>> ššššššsi_uid = 4294960432, si_status = 32767, si_addr = 0x202, si_value = {sival_int = 1392864896, sival_ptr = 0xfffffe0653056e80,
>> ššššššššsigval_int = 1392864896, sigval_ptr = 0xfffffe0653056e80}, _reason = {_fault = {_trapno = 1392864896}, _timer = {_timerid = 1392864896,
>> šššššššššš_overrun = -506}, _mesgq = {_mqd = 1392864896}, _poll = {_band = -2171860586880}, __spare__ = {__spare1__ = -2171860586880, __spare2__ = {
>> š---Type <return> to continue, or q <return> to quit---
>> šššššššššššš1392864896, -506, 0, -2048, -723702400, -507, 1899046048}}}}, ksi_flags = 2011958160, ksi_sigq = 0xffffffff80e3028d}
>> š#28 0xffffffff814b9ddb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>> šNo locals.
>> š#29 0x000000080092a17a in ?? ()
>> šNo symbol table info available.
>> šPrevious frame inner to this frame (corrupt stack?)
>> šCurrent language: auto; currently minimal
>> š(kgdb)q
>>
>> š16.02.2016, 12:10, "Steven Hartland" <killing@multiplay.co.uk>:
>>> šI need the values of the vars specified, so you'll need to:
>>> šprint bp
>>> šIf it reports just an address try:
>>> šprint *bp
>>> šetc.
>>>
>>> šOn 16/02/2016 08:08, DemIS wrote:
>>>> šššframe 8
>>>> ššš#8 0xffffffff81c3669c in dbuf_read (db=0xfffff8013ca46380, zio=0x0, flags=6)
>>>> ššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>>>> ššš573 (void) arc_read(zio, db->db_objset->os_spa, db->db_blkptr,
>>>> šššCurrent language: auto; currently minimal
>>>>
>>>> šššframe 6
>>>> ššš#6 0xffffffff81c2b9f8 in arc_get_data_buf (buf=<value optimized out>)
>>>> ššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>>>> ššš2898 buf->b_data = zio_buf_alloc(size);
>>>>
>>>> ššš16.02.2016, 04:15, "Steven Hartland" <killing@multiplay.co.uk>:
>>>>> šššYou don't need the system live just the kernel and the crash dump to get
>>>>> šššthose values.
>>>>>
>>>>> šššOn 16/02/2016 00:46, DemIS wrote:
>>>>>> šššššToday I fell back to version 10.2 . Users work on the server . To reinstall 10.3 I need 4:00 .
>>>>>> šššššTherefore, it is possible only on weekends . But this has the same effect on the version 10.2
>>>>>>
>>>>>> ššššš16.02.2016, 02:55, "Steven Hartland" <killing@multiplay.co.uk>:
>>>>>>> šššššThat sounds like you have some pool data corruption, from the 10.3
>>>>>>> šššššversion dump can you print out the following:
>>>>>>> ššššš1. frame 8: bp and size
>>>>>>> ššššš2. frame 6: buf->b_hdr
>>>>>>>
>>>>>>> šššššOn 15/02/2016 23:26, DemIS wrote:
>>>>>>>> šššššššAny one knows about problem?
>>>>>>>> šššššššServer: SuperMicro Model:SYS-6026T-6RF+, MB:X8DTU-6F+, RAM 24็ย DDR3, two XEON
>>>>>>>> šššššššRAM: KVR1333D3E9S/4G - DDR3, 1333MHz, ECC, CL9, X8, 1.5V, Unbuffered, DIMM
>>>>>>>> šššššššVersion:uname -a
>>>>>>>>
>>>>>>>> šššššššFreeBSD teo.some.loc 10.2-RELEASE-p12 FreeBSD 10.2-RELEASE-p12 #0: Sat Feb 13 18:04:04 MSK 2016 demis@teo.some.loc:/usr/obj/usr/src/sys/TEO amd64
>>>>>>>> ššššššš(on GENERIC or custom kernel config persist too !!!)
>>>>>>>>
>>>>>>>> šššššššMemtest86+ v.4.40 (ECC mode) test - OK.
>>>>>>>> šššššššEvery disk checked too (physically - mhdd, logically - zpool scrub, and additional checkit in external company recovery disk). No errors.
>>>>>>>>
>>>>>>>> šššššššPart of df -H
>>>>>>>> šššššššFilesystem Size Used Avail Capacity Mounted on
>>>>>>>> šššššššhdd/usr/wf 6,6T 4,1T 2,5T 62% /hdd/usr/wf
>>>>>>>>
>>>>>>>> šššššššzpool status hdd
>>>>>>>> ššššššššššpool: hdd
>>>>>>>> šššššššššstate: ONLINE
>>>>>>>> šššššššstatus: Some supported features are not enabled on the pool. The pool can
>>>>>>>> ššššššššššstill be used, but some features are unavailable.
>>>>>>>> šššššššaction: Enable all features using 'zpool upgrade'. Once this is done,
>>>>>>>> ššššššššššthe pool may no longer be accessible by software that does not support
>>>>>>>> ššššššššššthe features. See zpool-features(7) for details.
>>>>>>>> ššššššššššscan: scrub repaired 0 in 14h57m with 0 errors on Thu Feb 11 03:35:43 2016
>>>>>>>> šššššššconfig:
>>>>>>>>
>>>>>>>> ššššššššššNAME STATE READ WRITE CKSUM
>>>>>>>> ššššššššššhdd ONLINE 0 0 0
>>>>>>>> ššššššššššraidz2-0 ONLINE 0 0 0
>>>>>>>> ššššššššššmfid1p1 ONLINE 0 0 0
>>>>>>>> ššššššššššmfid2p1 ONLINE 0 0 0
>>>>>>>> ššššššššššmfid3p1 ONLINE 0 0 0
>>>>>>>> ššššššššššmfid4p1 ONLINE 0 0 0
>>>>>>>> ššššššššššmfid5p1 ONLINE 0 0 0
>>>>>>>>
>>>>>>>> šššššššerrors: No known data errors
>>>>>>>>
>>>>>>>> šššššššhdd - is My zfs volume.
>>>>>>>> šššššššWhen I run command like:
>>>>>>>> šššššššrm /hdd/usr/some/path/to/file
>>>>>>>> šššššššššor
>>>>>>>> šššššššrm /hdd/usr/some/path/to/folder
>>>>>>>> šššššššššor
>>>>>>>> šššššššchown root:wheel /hdd/usr/some/path/to/file
>>>>>>>> šššššššššor
>>>>>>>> šššššššchown root:wheel /hdd/usr/some/path/to/folder
>>>>>>>> šššššššššor
>>>>>>>> šššššššsetfacl ... to /hdd/usr/some/path/to/file
>>>>>>>>
>>>>>>>> šššššššššI'm get kernel panic:
>>>>>>>> šššššššGNU gdb 6.1.1 [FreeBSD]
>>>>>>>> šššššššCopyright 2004 Free Software Foundation, Inc.
>>>>>>>> šššššššGDB is free software, covered by the GNU General Public License, and you are
>>>>>>>> šššššššwelcome to change it and/or distribute copies of it under certain conditions.
>>>>>>>> šššššššType "show copying" to see the conditions.
>>>>>>>> šššššššThere is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>>>> šššššššThis GDB was configured as "amd64-marcel-freebsd"...
>>>>>>>>
>>>>>>>> šššššššUnread portion of the kernel message buffer:
>>>>>>>> šššššššpanic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270
>>>>>>>> šššššššcpuid = 9
>>>>>>>> šššššššKDB: stack backtrace:
>>>>>>>> ššššššš#0 0xffffffff80984ef0 at kdb_backtrace+0x60
>>>>>>>> ššššššš#1 0xffffffff80948aa6 at vpanic+0x126
>>>>>>>> ššššššš#2 0xffffffff80948973 at panic+0x43
>>>>>>>> ššššššš#3 0xffffffff81c0222f at assfail3+0x2f
>>>>>>>> ššššššš#4 0xffffffff81aa9d40 at zio_buf_alloc+0x50
>>>>>>>> ššššššš#5 0xffffffff81a2b9f8 at arc_get_data_buf+0x358
>>>>>>>> ššššššš#6 0xffffffff81a2e20a at arc_read+0x1ea
>>>>>>>> ššššššš#7 0xffffffff81a3669c at dbuf_read+0x6ac
>>>>>>>> ššššššš#8 0xffffffff81a3d8bf at dmu_spill_hold_existing+0xbf
>>>>>>>> ššššššš#9 0xffffffff81a70dd7 at sa_attr_op+0x167
>>>>>>>> ššššššš#10 0xffffffff81a72ffb at sa_lookup+0x4b
>>>>>>>> ššššššš#11 0xffffffff81abc82a at zfs_rmnode+0x2ba
>>>>>>>> ššššššš#12 0xffffffff81ada58e at zfs_freebsd_reclaim+0x4e
>>>>>>>> ššššššš#13 0xffffffff80e73537 at VOP_RECLAIM_APV+0xa7
>>>>>>>> ššššššš#14 0xffffffff809ec5b4 at vgonel+0x1b4
>>>>>>>> ššššššš#15 0xffffffff809eca49 at vrecycle+0x59
>>>>>>>> ššššššš#16 0xffffffff81ada52d at zfs_freebsd_inactive+0xd
>>>>>>>> ššššššš#17 0xffffffff80e73427 at VOP_INACTIVE_APV+0xa7
>>>>>>>> šššššššUptime: 9m31s
>>>>>>>> šššššššDumping 1286 out of 24543 MB:..2%..12%..22%..32%..42%..51%..61%..71% (CTRL-C to abort) ..81%..91%
>>>>>>>>
>>>>>>>> šššššššReading symbols from /boot/kernel/zfs.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/zfs.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/opensolaris.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/if_lagg.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ums.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ums.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ipfw.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ipfw.ko.symbols
>>>>>>>> ššššššš#0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> ššššššš219 pcpu.h: No such file or directory.
>>>>>>>> ššššššššššin pcpu.h
>>>>>>>> ššššššš(kgdb) bt
>>>>>>>> ššššššš#0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> ššššššš#1 0xffffffff80948702 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
>>>>>>>> ššššššš#2 0xffffffff80948ae5 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
>>>>>>>> ššššššš#3 0xffffffff80948973 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687
>>>>>>>> ššššššš#4 0xffffffff81c0222f in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>,
>>>>>>>> ššššššššššf=<value optimized out>, l=<value optimized out>) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>>>>>>>> ššššššš#5 0xffffffff81aa9d40 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:270
>>>>>>>> ššššššš#6 0xffffffff81a2b9f8 in arc_get_data_buf (buf=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>>>>>>>> ššššššš#7 0xffffffff81a2e20a in arc_read (pio=0xfffff80011791730, spa=0xfffff80011579000, bp=0xfffffe000aee7980, done=0xffffffff81a3a2d0 <dbuf_read_done>,
>>>>>>>> ššššššššššprivate=0xfffff8002244b000, priority=ZIO_PRIORITY_SYNC_READ, zio_flags=-528866606, arc_flags=0xfffffe06727fb3c4, zb=0xffffffff81a3a2d0)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1551
>>>>>>>> ššššššš#8 0xffffffff81a3669c in dbuf_read (db=0xfffff8002244b000, zio=0x0, flags=6)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>>>>>>>> ššššššš#9 0xffffffff81a3d8bf in dmu_spill_hold_existing (bonus=0xfffff800223bed20, tag=0x0, dbp=0xfffff800919966b8)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>>>>>>>> ššššššš#10 0xffffffff81a70dd7 in sa_attr_op (hdl=0xfffff80091996690, bulk=0xfffffe06727fb528, count=1, data_op=SA_LOOKUP, tx=0x0)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:310
>>>>>>>> ššššššš#11 0xffffffff81a72ffb in sa_lookup (hdl=0xfffff80091996690, attr=<value optimized out>, buf=<value optimized out>, buflen=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1441
>>>>>>>> ššššššš#12 0xffffffff81abc82a in zfs_rmnode (zp=0xfffff80091993730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:633
>>>>>>>> ššššššš#13 0xffffffff81ada58e in zfs_freebsd_reclaim (ap=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6569
>>>>>>>> ššššššš#14 0xffffffff80e73537 in VOP_RECLAIM_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2019
>>>>>>>> ššššššš#15 0xffffffff809ec5b4 in vgonel (vp=0xfffff800111733b0) at vnode_if.h:830
>>>>>>>> ššššššš#16 0xffffffff809eca49 in vrecycle (vp=0xfffff800111733b0) at /usr/src/sys/kern/vfs_subr.c:2703
>>>>>>>> ššššššš#17 0xffffffff81ada52d in zfs_freebsd_inactive (ap=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6540
>>>>>>>> ššššššš#18 0xffffffff80e73427 in VOP_INACTIVE_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:1953
>>>>>>>> ššššššš#19 0xffffffff809eb382 in vinactive (vp=0xfffff800111733b0, td=0xfffff800113d8000) at vnode_if.h:807
>>>>>>>> ššššššš#20 0xffffffff809eb772 in vputx (vp=0xfffff800111733b0, func=2) at /usr/src/sys/kern/vfs_subr.c:2306
>>>>>>>> ššššššš#21 0xffffffff809f401e in kern_rmdirat (td=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/kern/vfs_syscalls.c:3842
>>>>>>>> ššššššš#22 0xffffffff80d4b3e7 in amd64_syscall (td=0xfffff800113d8000, traced=0) at subr_syscall.c:134
>>>>>>>> ššššššš#23 0xffffffff80d30acb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>>>>>>>> ššššššš#24 0x00000008008914ea in ?? ()
>>>>>>>> šššššššPrevious frame inner to this frame (corrupt stack?)
>>>>>>>> šššššššCurrent language: auto; currently minimal
>>>>>>>>
>>>>>>>> šššššššIf setup FreeBSD 10.3 BETA (on GENERIC or custom kernel config):
>>>>>>>> šššššššCopyright 2004 Free Software Foundation, Inc.
>>>>>>>> šššššššGDB is free software, covered by the GNU General Public License, and you are
>>>>>>>> šššššššwelcome to change it and/or distribute copies of it under certain conditions.
>>>>>>>> šššššššType "show copying" to see the conditions.
>>>>>>>> šššššššThere is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>>>> šššššššThis GDB was configured as "amd64-marcel-freebsd"...
>>>>>>>>
>>>>>>>> šššššššUnread portion of the kernel message buffer:
>>>>>>>> šššššššpanic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 273
>>>>>>>> šššššššcpuid = 13
>>>>>>>> šššššššKDB: stack backtrace:
>>>>>>>> ššššššš#0 0xffffffff8098f000 at kdb_backtrace+0x60
>>>>>>>> ššššššš#1 0xffffffff80951d06 at vpanic+0x126
>>>>>>>> ššššššš#2 0xffffffff80951bd3 at panic+0x43
>>>>>>>> ššššššš#3 0xffffffff81e0022f at assfail3+0x2f
>>>>>>>> ššššššš#4 0xffffffff81cacc70 at zio_buf_alloc+0x50
>>>>>>>> ššššššš#5 0xffffffff81c2b8f2 at arc_get_data_buf+0x262
>>>>>>>> ššššššš#6 0xffffffff81c2b657 at arc_buf_alloc+0xc7
>>>>>>>> ššššššš#7 0xffffffff81c2d601 at arc_read+0x1c1
>>>>>>>> ššššššš#8 0xffffffff81c36ce9 at dbuf_read+0x6b9
>>>>>>>> ššššššš#9 0xffffffff81c3e415 at dmu_spill_hold_existing+0xc5
>>>>>>>> ššššššš#10 0xffffffff81c73707 at sa_attr_op+0x167
>>>>>>>> ššššššš#11 0xffffffff81c75972 at sa_lookup+0x52
>>>>>>>> ššššššš#12 0xffffffff81cbf8da at zfs_rmnode+0x2ba
>>>>>>>> ššššššš#13 0xffffffff81cdd75e at zfs_freebsd_reclaim+0x4e
>>>>>>>> ššššššš#14 0xffffffff80e81c27 at VOP_RECLAIM_APV+0xa7
>>>>>>>> ššššššš#15 0xffffffff809f9581 at vgonel+0x221
>>>>>>>> ššššššš#16 0xffffffff809f9a19 at vrecycle+0x59
>>>>>>>> ššššššš#17 0xffffffff81cdd6fd at zfs_freebsd_inactive+0xd
>>>>>>>> šššššššUptime: 11m11s
>>>>>>>> šššššššDumping 1368 out of 24542 MB:..2%..11%..22%..31%..41%..51%..61%..71%..81%..91%
>>>>>>>>
>>>>>>>> šššššššReading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/if_lagg.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/aio.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/aio.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ichsmb.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/smbus.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/smbus.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ipmi.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ipmi.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/zfs.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/zfs.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/opensolaris.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ums.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ums.ko.symbols
>>>>>>>> šššššššReading symbols from /boot/kernel/ipfw.ko.symbols...done.
>>>>>>>> šššššššLoaded symbols for /boot/kernel/ipfw.ko.symbols
>>>>>>>> ššššššš#0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> ššššššš219 pcpu.h: No such file or directory.
>>>>>>>> ššššššššššin pcpu.h
>>>>>>>> ššššššš(kgdb) backtrace
>>>>>>>> ššššššš#0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> ššššššš#1 0xffffffff80951962 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
>>>>>>>> ššššššš#2 0xffffffff80951d45 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889
>>>>>>>> ššššššš#3 0xffffffff80951bd3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818
>>>>>>>> ššššššš#4 0xffffffff81e0022f in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>, f=<value optimized out>,
>>>>>>>> ššššššššššl=<value optimized out>) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>>>>>>>> ššššššš#5 0xffffffff81cacc70 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:273
>>>>>>>> ššššššš#6 0xffffffff81c2b8f2 in arc_get_data_buf (buf=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3880
>>>>>>>> ššššššš#7 0xffffffff81c2b657 in arc_buf_alloc (spa=<value optimized out>, size=<value optimized out>, tag=0x0, type=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2057
>>>>>>>> ššššššš#8 0xffffffff81c2d601 in arc_read (pio=0xfffff8000fad03b0, spa=0xfffff8000f63d000, bp=0xfffffe000e509980, done=0xffffffff81c3aed0 <dbuf_read_done>, private=0xfffff8000fdd6360,
>>>>>>>> ššššššššššpriority=ZIO_PRIORITY_SYNC_READ, zio_flags=-2117882160, arc_flags=0xfffffe02925483c4, zb=0xfffff8000fdd6360)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4397
>>>>>>>> ššššššš#9 0xffffffff81c36ce9 in dbuf_read (db=0xfffff8000fdd6360, zio=0x0, flags=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:682
>>>>>>>> ššššššš#10 0xffffffff81c3e415 in dmu_spill_hold_existing (bonus=0xfffff8001f312438, tag=0x0, dbp=0xfffff80062d4e7d0)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>>>>>>>> ššššššš#11 0xffffffff81c73707 in sa_attr_op (hdl=0xfffff80062d4e770, bulk=0xfffffe0292548528, count=1, data_op=SA_LOOKUP, tx=0x0)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:305
>>>>>>>> ššššššš#12 0xffffffff81c75972 in sa_lookup (hdl=0xfffff80062d4e770, attr=<value optimized out>, buf=<value optimized out>, buflen=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1443
>>>>>>>> ššššššš#13 0xffffffff81cbf8da in zfs_rmnode (zp=0xfffff80062d4c8a0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:633
>>>>>>>> ššššššš#14 0xffffffff81cdd75e in zfs_freebsd_reclaim (ap=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6619
>>>>>>>> ššššššš#15 0xffffffff80e81c27 in VOP_RECLAIM_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2019
>>>>>>>> ššššššš#16 0xffffffff809f9581 in vgonel (vp=0xfffff8000f1beb10) at vnode_if.h:830
>>>>>>>> ššššššš#17 0xffffffff809f9a19 in vrecycle (vp=0xfffff8000f1beb10) at /usr/src/sys/kern/vfs_subr.c:2951
>>>>>>>> ššššššš#18 0xffffffff81cdd6fd in zfs_freebsd_inactive (ap=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6590
>>>>>>>> ššššššš#19 0xffffffff80e81b17 in VOP_INACTIVE_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:1953
>>>>>>>> ššššššš#20 0xffffffff809f8322 in vinactive (vp=0xfffff8000f1beb10, td=0xfffff8000f9f34b0) at vnode_if.h:807
>>>>>>>> ššššššš#21 0xffffffff809f8712 in vputx (vp=0xfffff8000f1beb10, func=2) at /usr/src/sys/kern/vfs_subr.c:2547
>>>>>>>> ššššššš#22 0xffffffff80a0137e in kern_rmdirat (td=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>)
>>>>>>>> ššššššššššat /usr/src/sys/kern/vfs_syscalls.c:3964
>>>>>>>> ššššššš#23 0xffffffff80d574bf in amd64_syscall (td=0xfffff8000f9f34b0, traced=0) at subr_syscall.c:141
>>>>>>>> ššššššš#24 0xffffffff80d3c72b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>>>>>>>> ššššššš#25 0x000000080089458a in ?? ()
>>>>>>>> šššššššPrevious frame inner to this frame (corrupt stack?)
>>>>>>>> šššššššCurrent language: auto; currently minimal
>>>>>>>>
>>>>>>>> šššššššCrash folder (or file) have strange rights:
>>>>>>>> šššššššd---------+ 3 anna domain users 3 10 ฤลห 10:32 01-Projcts
>>>>>>>> šššššššd---------+ 2 anna domain users 2 8 ฦลื 21:46 02-Text
>>>>>>>>
>>>>>>>> šššššššHow correct kernel panic?
>>>>>>>>
>>>>>>>> ššššššš_______________________________________________
>>>>>>>> šššššššfreebsd-fs@freebsd.org mailing list
>>>>>>>> šššššššhttps://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>> šššššššTo unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>>>>>>> ššššš_______________________________________________
>>>>>>> šššššfreebsd-fs@freebsd.org mailing list
>>>>>>> šššššhttps://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> šššššTo unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7814671457953923>