Date: Wed, 15 Feb 2012 12:21:31 +0200 From: George Kontostanos <gkontos.mail@gmail.com> To: Pavlo <devgs@ukr.net> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS and mem management Message-ID: <CA%2BdUSyqKoqrfD_cgsfupsuZRE0O6dH-4F1roLp_GFaeBKJkN-w@mail.gmail.com> In-Reply-To: <15861.1329298812.1414986334451204096@ffe12.ukr.net> References: <15861.1329298812.1414986334451204096@ffe12.ukr.net>
index | next in thread | previous in thread | raw e-mail
2012/2/15 Pavlo <devgs@ukr.net>: > > > > Hello. > > We have an issue with memory management on FreeBSD and i suspect it is > related to FS. > We are using ZFS, here quick stats: > > > zpool status > pool: disk1 > state: ONLINE > scan: resilvered 657G in 8h30m with 0 errors on Tue Feb 14 21:17:37 2012 > config: > > NAME STATE READ WRITE CKSUM > disk1 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/disk0 ONLINE 0 0 0 > gpt/disk1 ONLINE 0 0 0 > gpt/disk2 ONLINE 0 0 0 > gpt/disk4 ONLINE 0 0 0 > gpt/disk6 ONLINE 0 0 0 > gpt/disk8 ONLINE 0 0 0 > gpt/disk10 ONLINE 0 0 0 > gpt/disk12 ONLINE 0 0 0 > mirror-7 ONLINE 0 0 0 > gpt/disk14 ONLINE 0 0 0 > gpt/disk15 ONLINE 0 0 0 > > errors: No known data errors > > pool: zroot > state: ONLINE > scan: resilvered 34.9G in 0h11m with 0 errors on Tue Feb 14 12:57:52 2012 > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/sys0 ONLINE 0 0 0 > gpt/sys1 ONLINE 0 0 0 > > errors: No known data errors > > ------------------------------------------------------------------------ > > System Memory: > > 0.95% 75.61 MiB Active, 0.24% 19.02 MiB Inact > 18.25% 1.41 GiB Wired, 0.01% 480.00 KiB Cache > 80.54% 6.24 GiB Free, 0.01% 604.00 KiB Gap > > Real Installed: 8.00 GiB > Real Available: 99.84% 7.99 GiB > Real Managed: 96.96% 7.74 GiB > > Logical Total: 8.00 GiB > Logical Used: 21.79% 1.74 GiB > Logical Free: 78.21% 6.26 GiB > > Kernel Memory: 1.18 GiB > Data: 99.05% 1.17 GiB > Text: 0.95% 11.50 MiB > > Kernel Memory Map: 4.39 GiB > Size: 23.32% 1.02 GiB > Free: 76.68% 3.37 GiB > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------ > ZFS Subsystem Report Wed Feb 15 10:53:03 2012 > ------------------------------------------------------------------------ > > System Information: > > Kernel Version: 802516 (osreldate) > Hardware Platform: amd64 > Processor Architecture: amd64 > > ZFS Storage pool Version: 28 > ZFS Filesystem Version: 5 > > FreeBSD 8.2-STABLE #12: Thu Feb 9 11:35:23 EET 2012 root > 10:53AM up 56 mins, 6 users, load averages: 0.00, 0.00, 0.00 > > ------------------------------------------------------------------------ > > > > > Background: > we are using some tool that does indexing of some data and then pushes it > into database (currently bdb-5.2). Instances of indexer are running > continuously one after another. Time of indexing for one instance of > indexer may vary between 2 seconds and 30 minutes. But mostly it is > below one minute. There is nothing else running on the machine except > system stuff and daemons. After several hours of indexing i can see a lot > of active memory, it's ok. Then i check the number of vnodes. and it's > really huge: 300k+ even tho nobody has so many opened files. Reading docs > and googling i figured that's because of cached pages that reside in > memory (unmounting of disk causes whole memory to be freed). Also I > figured that happens only when I am accessing files via mmap(). > > Looks like pretty legit behaviour but the issue is: > This spectacle continues (approximately for 12 hours) unlit indexers > began to be killed out of swap. As I wrote above I observe a lot of used > vnodes and like 7GB of active memory. I made a tool that allocates memory > using malloc() to check what's the limit of available memory that can be > allocated. It is several megabytes, sometimes more. Unless that tool gets > killed out of swap as well. So how i can see the issue: for some reason > after some process had exited normally all mapped pages don't get freed. > I red about and I agree that this is reasonable behaviour if we have > spare memory. But following this logic these pages can be flushed back to > file at any time when system is under stress conditions. So when I ask > for a piece of RAM, OS should do that trick and give me what I ask. But > that's never happens. Those pages are like frozen. Until I unmount disk. > Even after there is not a single instance of indexer running. > > I believe all this is caused by mmap() for sure : BDB uses mmap() for > accessing databases and we tested indexing with out pushing data to DB. > Worked shiny. You may suggest that that's something wrong with BDB. But > we have some more tools of ours that using mmap() as well and the > behaviour is exact. > > Thank you. Paul, Ukraine. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Hi Paul, Are you using dedup anywhere on that pool? Also, could you please post the full zfs-stats -a -- George Kontostanos Aicom telecoms ltd http://www.aisecure.nethelp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BdUSyqKoqrfD_cgsfupsuZRE0O6dH-4F1roLp_GFaeBKJkN-w>