STATE     READ WRITE CKSUM
         tank            ONLINE       0     0     0
           raidz1-0      ONLINE       0     0     0
             gpt/data0   ONLINE       0     0     0
             gpt/data1   ONLINE       0     0     0
             gpt/data2   ONLINE       0     0     0
             gpt/data3   ONLINE       0     0     0
             gpt/data4   ONLINE       0     0     0
           raidz1-1      ONLINE       0     0     0
             gpt/data5   ONLINE       0     0    30
             gpt/data6   ONLINE       0     0    30
             gpt/data7   ONLINE       0     0    30
             gpt/data8   ONLINE       0     0    30
             gpt/data9   ONLINE       0     0    30
           raidz1-2      ONLINE       0     0     0
             gpt/data10  ONLINE       0     0     0
             gpt/data11  ONLINE       0     0     0
             gpt/data12  ONLINE       0     0     0
             gpt/data13  ONLINE       0     0     0
             gpt/data14  ONLINE       0     0     0

errors: 229 data errors, use '-v' for a list

===Cut===

(the file unique list is pretty short for now, as these 229 errors 
mostly reference the snapshots, it contains 11 unique filenames which 
mostly are .mp3 files, and one <metadata>:<0x0> entry).

The zfs dataset configuration looks like (the deduplication is on only 
for tank 0, troubled one is the tank3):

===Cut===

NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                        112T  7.12T   141K  /tank
tank/data                   111T  7.12T  3.72G  /usr/local/public
tank/data/tank0            19.9T  7.12T  19.9T /usr/local/public/tank0
tank/data/tank1            16.1T  7.12T  16.1T /usr/local/public/tank1
tank/data/tank2            38.8T  7.12T  38.8T /usr/local/public/tank2
tank/data/tank3            36.6T  7.12T  28.9T /usr/local/public/tank3
tank/obj                   17.7G  7.12T  17.7G  /usr/obj
tank/www                    197M  7.12T   197M  /var/www

===Cut===

As you can probably see it conttains errors. The thing is - I'm 
desperately fighting these errors for almost a year now. They started to 
appear when a drive in this very vdev died, and was successfully 
replaced. But scrub found out that some of the files were damaged beyond 
repair with self-healing; since this entire pool is replicated to a 
sibling SAN (and it's copy of these were intact), they all were deleted 
and copied from it, with no errors (except metadata error that 
persisted). Since that moment, this very vdev keeps on creating these 
errors, and each time new files are affacted. Each time I copy them and 
then new files got corrupted, each time different one. The errors 
persist on a vdev where the initial errors were discovered, the same 
vdev where the drive died. Since then I've done and extended memtest86+ 
memory check (several full cycles, for 14 hours, no errors logged), I've 
also done a firmware upgrade for an Adaptec contoller and of course a 
FreeBSD upgrade (not sure now what was an initial OS versio when the 
pool got struck, but it was probably a 13.0).


The questions obvioulsy are,

- can these be fixed ?

- do they appear because pool metadata is corrupted ?

- can they be fixed without pool destruction and rereceiving ?

and probably what's causing them.


Thanks.

Eugene.