Date: Wed, 15 Aug 2012 08:45:38 +0200 From: =?iso-8859-1?Q?Karli_Sj=F6berg?= <Karli.Sjoberg@slu.se> To: Freddie Cash <fjwcash@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Hang when importing pool Message-ID: <49C9D08A-85EF-4D23-B07F-F3980CBA5A97@slu.se> In-Reply-To: <CAOjFWZ52XvMO%2BA7cwa3fnkJcXMCbGgWD91gvZsmW8Navh0AZ9A@mail.gmail.com> References: <D13A3EA7-B229-4B78-915E-A3CC3162DB8A@slu.se> <CAOjFWZ52XvMO%2BA7cwa3fnkJcXMCbGgWD91gvZsmW8Navh0AZ9A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
31 jul 2012 kl. 17.31 skrev Freddie Cash: On Mon, Jul 30, 2012 at 11:31 PM, Karli Sjöberg <Karli.Sjoberg@slu.se<mailto:Karli.Sjoberg@slu.se>> wrote: I´m really struggling with this. I have had a pool with imported filesystems from a Solaris system that had dedup activated. Then, when the time came to erase them, it just stalled. When rebooting, it stalled again at mounting filesystems, and since then, I´ve installed two USB drives to act as root pool with FreeBSD-9.0-RELEASE so that I could import the original pool in recovery, but it always stalls after a couple of hours. Looking at top, I could see that the 16GB RAM was maxed out, so I have heavily tuned down kmem, arc, etc: You're running out of RAM during the import, as it loads the DDT. Stuff a bunch more RAM into the machine (32 GB, 48 GB, even 64 GB). Then you will be able to load the full DDT into RAM, finish the aborted destroy process, and import the pool. We've run into this three or four times now on systems with dedupe enabled and only 16 GB of RAM. We've since upgraded all our boxes to a minimum of 32 GB, with one having 64 GB. ZFS dataset destruction with dedupe enabled takes *a lot* of time and RAM, as the DDT needs to be updated for every block freed. And rebooting in the middle of a "zfs destroy" operation means that the operation needs to finish at pool import time. -- Freddie Cash fjwcash@gmail.com<mailto:fjwcash@gmail.com> I took your advice. I replaced my Core i5 with a Xeon X3470 and ramped up the RAM to 32GB, maxing out the HW. Sadly enough, it still stalls in the exact same manner:( This has to be the most frustrating thing ever, since there´s tons of data there that I really need and if it wasn´t for that stupid destroy operation, it would still be accessible. I feel that FreeBSD is partly to blame since it was completely possible in the originating SUN machine with Solaris that only has 16GB RAM to do the same destroy to the same dataset without any problem. Sure, it took forever and then some (about two weeks) but it stayed afloat during the whole time. The FreeBSD machine starts to accumulate more and more RAM, but quite steadily until it comes up to 9-9.5GB of RAM Wired and then it just SHOOTS off to swallow it all and cause a stall. That is the same behavior as when there was only 16GB RAM. During the last attempt I had while true; do zfs-stats -A | grep "ARC Size:" zfs-stats -L | egrep '(L2 ARC Size|Bytes Scanned)' sleep 10 done running, so I could monitor the usage just before the crash. The ARC stayed at 6GB, while the last top sample shows 28GB Wired. See for yourselves: top: http://i45.tinypic.com/21do5ra.png gstat: http://i49.tinypic.com/e197ax.png zfs-stats: http://i46.tinypic.com/250uxhz.png Import CTRL+T after it stalled: http://i46.tinypic.com/2uxvb4h.png I´m willing to try anything at this point. Any longshots you have are most welcome, since it couldn´t get any worse:( Med Vänliga Hälsningar ------------------------------------------------------------------------------- Karli Sjöberg Swedish University of Agricultural Sciences Box 7079 (Visiting Address Kronåsvägen 8) S-750 07 Uppsala, Sweden Phone: +46-(0)18-67 15 66 karli.sjoberg@slu.se<mailto:karli.sjoberg@adm.slu.se>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49C9D08A-85EF-4D23-B07F-F3980CBA5A97>