Date: Wed, 02 Feb 2005 17:00:22 +0300 From: Deomid Ryabkov <myself@rojer.pp.ru> To: Milan Obuch <milan@dino.sk> Cc: freebsd-hackers@freebsd.org Subject: Re: Question: tracking filesystem changes? Message-ID: <4200DCF6.1010002@rojer.pp.ru> In-Reply-To: <200502011746.52908.milan@dino.sk> References: <41FF6869.5060709@rojer.pp.ru> <41FF8B00.2010208@fer.hr> <41FFA801.8070307@rojer.pp.ru> <200502011746.52908.milan@dino.sk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms080404050701030600070001 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Milan Obuch wrote: >>>>>>This could be a custom filesystem wrapper for UFS that would report >>>>>>name of the file/directory being changed. >>>>>> >>>>>> >>>>>Couldn't you use kqueue system to monitor the directory-file? >>>>> >>>>> >>>>I could, if I hadn't near 10 millions of them. >>>> >>>> >>>Hm. I meant monitoring the directory itself, as a file, then parsing >>>the directory list to determine what has changed. But with 10M files, >>>probably nothing would work... >>> >>> >>these are 10M of static documents, the daily change is minmal. >>the question is: where it is the appropriate place to collect those >>changes? the right way seems to implement a customs filesystem, but would >>it possible to obtain a full path at that level? >> >Did you consider using fam (file alteration monitor) from ports? > > No, won't do the trick either. I cannot afford setting up watchdogs for every file or even every directory. And I'm essentially "interested" in every one of them (for mirroring purposes). A more general approach is needed. E.g., if an unlink call is issued and an inode is within a particular filesystem (luckily, most of our data already lives on or can be easily moved to a separate filesystem), a notice is sent to some userland daemon: "file /www/xxx/yyy.shtml is unlinked". Or opened for writing, or renamed... etc. The file is then scheduled for distribution to mirrors. The idea seems simple and straightforward, yet I don't know if it is achievable. The essential part is obtaining the full pathname of the file (won't bother with hardlinks at first, they aren't used here). Could that be done with the FreeBSD's filesystem (vnode/vfs?) code? (which I'm not familiar with) -- Deomid Ryabkov aka Rojer myself@rojer.pp.ru rojer@sysadmins.ru ICQ: 8025844 --------------ms080404050701030600070001 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJIzCC AuwwggJVoAMCAQICAwwKdjANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDQwMzMxMjIxODA5WhcNMDUwMzMxMjIxODA5 WjBfMRAwDgYDVQQEEwdSeWFia292MQ8wDQYDVQQqEwZEZW9taWQxFzAVBgNVBAMTDkRlb21p ZCBSeWFia292MSEwHwYJKoZIhvcNAQkBFhJteXNlbGZAcm9qZXIucHAucnUwggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBxXgFP/1lZDqp0dzUDzR5IBb7aKki6TD+HMMkRjtP IOcaNHsfoDer9RFrFICoxNQZF86iopYFVYr7msgB9y2dKZTRQQoOA72lFrOyH3sgrztx/3LL axEsihA2cxcrglrIgPEm6FF2aabbKVpSdeslMCDPBr0auAm0QLo8ch9c5j0vuQUBrs8TKU6f 6YZLNO2Sk/fPZP2kfJEkXyZhkU6wq3ER1CHq2qgfNpW2Ni7Kuv/eYI/CV1BGgm37ZPubOyxI LNiRUGT0pv0wocrCIehKqoI1uFPZgGS0ANYTqPJQSdjlSzMGQJjT510PNDJnDfKOvLhcadD+ 6gSL/ovNM/LPAgMBAAGjLzAtMB0GA1UdEQQWMBSBEm15c2VsZkByb2plci5wcC5ydTAMBgNV HRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBACunC6DhFX4I6Nvdy/UevjSd3VmKWPRmqwoR l0RXvuI/JVyPO9KHGqxCMpRu7ArJz7d8ShPVs5kynysrB+Nm6/fwWjeaW21+gViojeO9gGP6 Np/LeMIqkqSYMoElq7Feqh/3qp7a/UxuofFtAW9V/2tRunxaPo4/WOxcdcmdcC86MIIC7DCC AlWgAwIBAgIDDAp2MA0GCSqGSIb3DQEBBAUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg RnJlZW1haWwgSXNzdWluZyBDQTAeFw0wNDAzMzEyMjE4MDlaFw0wNTAzMzEyMjE4MDlaMF8x EDAOBgNVBAQTB1J5YWJrb3YxDzANBgNVBCoTBkRlb21pZDEXMBUGA1UEAxMORGVvbWlkIFJ5 YWJrb3YxITAfBgkqhkiG9w0BCQEWEm15c2VsZkByb2plci5wcC5ydTCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAMHFeAU//WVkOqnR3NQPNHkgFvtoqSLpMP4cwyRGO08g5xo0 ex+gN6v1EWsUgKjE1BkXzqKilgVVivuayAH3LZ0plNFBCg4DvaUWs7IfeyCvO3H/cstrESyK EDZzFyuCWsiA8SboUXZpptspWlJ16yUwIM8GvRq4CbRAujxyH1zmPS+5BQGuzxMpTp/phks0 7ZKT989k/aR8kSRfJmGRTrCrcRHUIeraqB82lbY2Lsq6/95gj8JXUEaCbftk+5s7LEgs2JFQ ZPSm/TChysIh6EqqgjW4U9mAZLQA1hOo8lBJ2OVLMwZAmNPnXQ80MmcN8o68uFxp0P7qBIv+ i80z8s8CAwEAAaMvMC0wHQYDVR0RBBYwFIESbXlzZWxmQHJvamVyLnBwLnJ1MAwGA1UdEwEB /wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAK6cLoOEVfgjo293L9R6+NJ3dWYpY9GarChGXRFe+ 4j8lXI870ocarEIylG7sCsnPt3xKE9WzmTKfKysH42br9/BaN5pbbX6BWKiN472AY/o2n8t4 wiqSpJgygSWrsV6qH/eqntr9TG6h8W0Bb1X/a1G6fFo+jj9Y7Fx1yZ1wLzowggM/MIICqKAD AgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVy biBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25zdWx0aW5n MSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYDVQQDExtU aGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZy ZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5WjBiMQsw CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoG A1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJKoZIhvcN AQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2vX8MOmHy v1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9A74r/rsY Pge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEwEgYDVR0T AQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0ZS5jb20v VGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0RBCIwIKQe MBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GBAEiM0VCD 6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZOhl+hLGZ GwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC 3CEZNd4ksdMdRv9dX2VPMYIDOzCCAzcCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMc VGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFs IEZyZWVtYWlsIElzc3VpbmcgQ0ECAwwKdjAJBgUrDgMCGgUAoIIBpzAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNTAyMDIxNDAwMjNaMCMGCSqGSIb3DQEJ BDEWBBSBcGQoDfUUul9tmAEBlNOEccwVfjBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMH MA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIB KDB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29u c3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwg SXNzdWluZyBDQQIDDAp2MHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJaQTElMCMG A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAwwKdjANBgkqhkiG9w0BAQEFAASCAQAcf51x BOTlIPXL3uLeduuLkuWdPHpuF0B0sEmIS6+HFkIIoKnleBgypHGgZo3/bhZ3ouvys59r57ky gDOmByxeeOjTlnAT6nQHNG/L0C2cO/pg4pRaH+tbL4m5Tr7jklJvZDpqQPlbHojWrCG+0eGf p+Rb2DaTZegOuv4PO4wMdoXk25cEshXCUFqwDZ+MWzNIGwW/Iu6Dk/p7cUxBki6xBYUgn/tq 1emqbUyQ64OIStWDgfzt4K9MPTLXcZZsNrZjoB3Y2f074OU143KgMCQEAUdaesCnGieU+XTp l1uDkXMoZ2AWd7Wp9xMBGR/MRC9GCnXgJG38uIZNysbEJc3WAAAAAAAA --------------ms080404050701030600070001--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4200DCF6.1010002>