Date: Fri, 11 Nov 2022 15:51:34 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync Message-ID: <cbe8535a-bae7-c6f9-2ba4-ef3ed5ae64c6@denninger.net> In-Reply-To: <alpine.BSF.2.22.395.2211112008220.30520@mail0.time-domain.net> References: <alpine.BSF.2.22.395.2211111709230.29479@mail0.time-domain.net> <CAOgwaMuoLQ9Er67Y%2B=q%2Bf9724v2WT3L5v5TZaRVXq%2B=1vEyJ%2BA@mail.gmail.com> <alpine.BSF.2.22.395.2211112008220.30520@mail0.time-domain.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms090109010601050609040103 Content-Type: multipart/alternative; boundary="------------z7dqCKJCEbxiqibeBbpS56ey" --------------z7dqCKJCEbxiqibeBbpS56ey Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Are you sure there are no snapshots that are holding blocks? If so those become copy-on write and will not be released even if the alleged file is deleted, as the snapshot copy is still there and valid. On 11/11/2022 15:20, andy thomas wrote: > Yes, I can confirm the rsync --delete option is being used and in > fact, 'du' reports some of the mirrored folders as having identical > sizes on both servers, mainly those containing only small amounts of > data. > > It seems almost as if ZFS is not freeing up blocks when rsync has > deleted or shrank files, leaving unwanted blocks lurking around in the > folder that 'du' then discovers and adds to its tally when it works > out the space usage of that folder! > > I suppose I could always destroy the zfs dataset on the mirror server > & resync the whole thing but will take days to complete even over a 10 > Gbit/s network link (the servers ought to be upgraded to FBSD 13.1 as > well). > > Andy > > On Fri, 11 Nov 2022, Mehmet Erol Sanliturk wrote: > >> >> , Nov 11, 2022 at 8:42 PM andy thomas <andy@time-domain.co.uk> wrote: >> I have two identical servers, called clustor2 and >> clustor-backup, each >> with a ZFS RAIDZ-1 pool containing 9 SAS hard disks plus one >> spare and two >> SSDs for the ZIL and ARC functions. clustor2 stores user data >> from a >> HPC while clustor2-backup uses rsync to mirrors all the data >> from clustor2 >> every 24 hours. >> >> However, the disk usage on the mirror server is considerably >> more than on >> the other server - attached is a screenshot showing the two >> servers side >> by side, with the mirror server on the right, and displaying the >> contents >> of the same subdirectory choen at random (named 'ratio_10.0' in >> this >> instance); as you can see, the sizes of the files within each of >> the >> folders are identical but 'du' reports very different >> space usages for each folder and 'zpool list' also reports a >> significant >> difference in ZFS pool size. >> >> I'm not sure if this is relevant but both servers have ZFS pools >> with no >> compression although lz4 compression is enabled on the ZFS >> filesystems & >> both run FreeBSD 11.3 with ZFS version 5. >> >> Perhaps using zfs send/receive instead of rsync for mirroring >> might solve >> this disparity? >> >> Thanks in advance for any suggestions, >> >> Andy >> >> >> >> >> Your question I am understanding the following points . >> >> >> >> I am using rsync in Fedora Linux . >> >> There are parameters of rsync such as >> >> --delete >> >> to delete files from the destination drive when they do not exist in the >> source drive . >> >> >> Please carefully scan rsync parameters and use suitable ones for your >> application . >> >> >> If a parameter like --delete is not used , rsync copies new files >> from >> the source drive and >> it does not delete any files from the destination drive . >> >> >> With my best wishes for all . >> >> >> Mehmet Erol Sanliturk >> >> >> >> >> >> >> >> >> > > > ---------------------------- > Andy Thomas, > Time Domain Systems > > Tel: +44 (0)7866 556626 > http://www.time-domain.co.uk -- Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------z7dqCKJCEbxiqibeBbpS56ey Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">Are you sure there are no snapshots that are holding blocks? If so those become copy-on write and will not be released even if the alleged file is deleted, as the snapshot copy is still there and valid.<br> </div> <div class="moz-cite-prefix"><br> </div> <div class="moz-cite-prefix">On 11/11/2022 15:20, andy thomas wrote:<br> </div> <blockquote type="cite" cite="mid:alpine.BSF.2.22.395.2211112008220.30520@mail0.time-domain.net">Yes, I can confirm the rsync --delete option is being used and in fact, 'du' reports some of the mirrored folders as having identical sizes on both servers, mainly those containing only small amounts of data. <br> <br> It seems almost as if ZFS is not freeing up blocks when rsync has deleted or shrank files, leaving unwanted blocks lurking around in the folder that 'du' then discovers and adds to its tally when it works out the space usage of that folder! <br> <br> I suppose I could always destroy the zfs dataset on the mirror server & resync the whole thing but will take days to complete even over a 10 Gbit/s network link (the servers ought to be upgraded to FBSD 13.1 as well). <br> <br> Andy <br> <br> On Fri, 11 Nov 2022, Mehmet Erol Sanliturk wrote: <br> <br> <blockquote type="cite"> <br> , Nov 11, 2022 at 8:42 PM andy thomas <a class="moz-txt-link-rfc2396E" href="mailto:andy@time-domain.co.uk"><andy@time-domain.co.uk></a> wrote: <br> I have two identical servers, called clustor2 and <br> clustor-backup, each <br> with a ZFS RAIDZ-1 pool containing 9 SAS hard disks plus one <br> spare and two <br> SSDs for the ZIL and ARC functions. clustor2 stores user data <br> from a <br> HPC while clustor2-backup uses rsync to mirrors all the data <br> from clustor2 <br> every 24 hours. <br> <br> However, the disk usage on the mirror server is considerably <br> more than on <br> the other server - attached is a screenshot showing the two <br> servers side <br> by side, with the mirror server on the right, and displaying the <br> contents <br> of the same subdirectory choen at random (named 'ratio_10.0' in <br> this <br> instance); as you can see, the sizes of the files within each of <br> the <br> folders are identical but 'du' reports very different <br> space usages for each folder and 'zpool list' also reports a <br> significant <br> difference in ZFS pool size. <br> <br> I'm not sure if this is relevant but both servers have ZFS pools <br> with no <br> compression although lz4 compression is enabled on the ZFS <br> filesystems & <br> both run FreeBSD 11.3 with ZFS version 5. <br> <br> Perhaps using zfs send/receive instead of rsync for mirroring <br> might solve <br> this disparity? <br> <br> Thanks in advance for any suggestions, <br> <br> Andy <br> <br> <br> <br> <br> Your question I am understanding the following points . <br> <br> <br> <br> I am using rsync in Fedora Linux . <br> <br> There are parameters of rsync such as <br> <br> --delete <br> <br> to delete files from the destination drive when they do not exist in the <br> source drive . <br> <br> <br> Please carefully scan rsync parameters and use suitable ones for your <br> application . <br> <br> <br> If a parameter like --delete is not used , rsync copies new files from <br> the source drive and <br> it does not delete any files from the destination drive . <br> <br> <br> With my best wishes for all . <br> <br> <br> Mehmet Erol Sanliturk <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> </blockquote> <br> <br> ---------------------------- <br> Andy Thomas, <br> Time Domain Systems <br> <br> Tel: +44 (0)7866 556626 <br> <a class="moz-txt-link-freetext" href="http://www.time-domain.co.uk">http://www.time-domain.co.uk</a><br>; </blockquote> <div class="moz-signature">-- <br> Karl Denninger<br> <a href="mailto:karl@denninger.net" class="moz-txt-link-freetext">karl@denninger.net</a><br> <i>The Market Ticker</i><br> <font size="-2"><i>[S/MIME encrypted email preferred]</i></font></div> </body> </html> --------------z7dqCKJCEbxiqibeBbpS56ey-- --------------ms090109010601050609040103 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DbowggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBxIwggT6oAMCAQICEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlk YTEZMBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENB MSUwIwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBMB4XDTIyMDYyOTE2MTYz NloXDTI3MDYyODE2MTYzNlowOjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEX MBUGA1UEAwwOS2FybCBEZW5uaW5nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC AQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvW ZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B 3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTgy+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYI XgVVPgfZZrbJJb5HWOQpvvhILpPCD3xsYJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMi WapsatKm8mxuOOGOEBhAoTVTwUHlMNTg6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMb NQm1mWREQhw3axgGLSntjjnznJr5vsvXSYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZM qa20JLAF1YagutDiMRURU23iWS7bA9tMcXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5 CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1l y+5ZOZbxBAZZMod4y4b4FiRUhRI97r9lCxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY 2BlA7ExM8XShMd9bRPZrNTokPQPUCWCgCdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEE MDAuMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYI KwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBD bGllbnQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNV HSMEgcIwgb+AFF3AXsKnjdPND5+bxVECGKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQ MA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5 c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lz dGVtcyBMTEMgMjAxNyBDQYITAORIioIQzl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJs QGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAKquc7cu0xc8FNtAQauZvocDzWQj 7HG9YvMbWnMi+ckhiA3rdW5NwWg0HBhBho1YlnqV+ntCVE2L8ezohHWm+KAdfXgpraL86Vsn 3ywNlZu/3COMpo2ALuHln8YQtH3Y8ebvzKMdlf2b5WB+7mOFIxXIr4AnNOLKCkq5ZhzC6JW6 Jvw3P0csiGa3UrfatYID5NvPgkaQvEgimEjG3psZqwQTL2Wxohvw783PrDt3wS0XeNhvQ61g 3QJFZKuv+bmGH3YBSPo1t6NUGAr+JozX5lDihB8JGkBt/NwdYec49a08uL0BbPaAJ7NjuIPG 7Y0Ak7PXZT37yx/Zla9PzLMJFgbelOkaatdzbblMZPDEVZ27l4lGMmV83Lm3YP17sdAyS/Wp mav7WmJUkQ9iuIKzSpdc82i9Mfujl1vbBtwtkHNPPtKuulIFM4ZwrPKjlVdLqTSqD8m9yHEi Y0PuAooq63OpJWF6hvMaiIPBWEAVIaDW9uG0MshLl9DnHnMyrJTfuC33Z9mOGMz7dRBjJd5Y W02xAzYnUuEBOpj+LQv5R8XIFMHFXktqEKvQrXeM2RU+PcZqKOBkTktxBLn3NI5VfA15Jk0c 5V5XcOqo3p2hvrwvXrinrb2pEREnoqmfrkXT3zOq5Y6ryRH8u734lGEF0dILXzoV4PM7XFit oTePoEjmMYIFBDCCBQACAQEwgZEwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGEx GTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTEl MCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQQISAsbzIfg9AV1t33FSCanO XWNfMA0GCWCGSAFlAwQCAwUAoIICQzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw0yMjExMTEyMDUxMzZaME8GCSqGSIb3DQEJBDFCBEA/aEwGb1WkRbxbkwED 4PCY20yiklgu5Qn+ldj0I+B3pL1/3TQ7Y65C0DNT+lFnTlJDvy5ywbyA81DUFuFl8/iEMGwG CSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAO BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw gaIGCSsGAQQBgjcQBDGBlDCBkTB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTEZ MBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENBMSUw IwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBAhICxvMh+D0BXW3fcVIJqc5d Y18wgaQGCyqGSIb3DQEJEAILMYGUoIGRMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9y aWRhMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMg Q0ExJTAjBgNVBAMMHEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0ECEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQEFAASCAgCwXofq7rIRqEafdvtgPTU3DtGLsqZBnDDMaLv3 3xbaYhNZV2PJ2/XQAykMSC9OS6jTkrtNP882Z2SENU8paLyBmAjOzxLKIhPjxfb3kIHpMUED 3VmM2iZHQZhdHXhdkRjeSTdE1CizYdtzu404KLk116p18TnMAZm3gdSR+W9vGPfc8YhgsJZG MWx2g8iy5j6K43Um6J7Cj+JLBaTnSYssKqj144B5sWtjyxGTDOX/+lFAIrMm0/i0JPNIHuCf n9IUqpJVKT1FVjBEYnP7nRl0OwX0bxP2vHqFx77LBl57/rsRhW6Mr7syzvdI8KxzXv+U+Pyd 5/VovPSAYopYYu2IXHVrXq8ltLuRFVThM23rILOHJWVfLMWJRwcVOyxlbNHTqeMdPeHOWLBM Gx1/tfZORn+8RTkBozyy0AbqAOlb8o4R12fzGze9JclXnBkYLLADXsiPp5lZN33EwtmiI8ci Jhz+FftV0ZwhF2mSCW0E1H7/uww+aNR0I1EF7EWQUhvj7ertQ5FNKUBNYNvhp7Yywj1T5d5X I/cJ83HSkw/GTA/YFaP1KUdP49HGj3AoucR8egNItfw8Z0GxgC6ZaCJ5Tag27L+IIWSd4UGH EJjZaCruX18qUNajEgEqNOWHzmHxzDFEBux8z4p31gWN85LQ63OQsinfPH9Ba5MRFuaW1QAA AAAAAA== --------------ms090109010601050609040103--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cbe8535a-bae7-c6f9-2ba4-ef3ed5ae64c6>