Date: Wed, 15 Aug 2012 08:45:38 +0200 From: =?iso-8859-1?Q?Karli_Sj=F6berg?= <Karli.Sjoberg@slu.se> To: Freddie Cash <fjwcash@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Hang when importing pool Message-ID: <49C9D08A-85EF-4D23-B07F-F3980CBA5A97@slu.se> In-Reply-To: <CAOjFWZ52XvMO%2BA7cwa3fnkJcXMCbGgWD91gvZsmW8Navh0AZ9A@mail.gmail.com> References: <D13A3EA7-B229-4B78-915E-A3CC3162DB8A@slu.se> <CAOjFWZ52XvMO%2BA7cwa3fnkJcXMCbGgWD91gvZsmW8Navh0AZ9A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
31 jul 2012 kl. 17.31 skrev Freddie Cash: On Mon, Jul 30, 2012 at 11:31 PM, Karli Sj=F6berg <Karli.Sjoberg@slu.se<mai= lto:Karli.Sjoberg@slu.se>> wrote: I=B4m really struggling with this. I have had a pool with imported filesyst= ems from a Solaris system that had dedup activated. Then, when the time cam= e to erase them, it just stalled. When rebooting, it stalled again at mount= ing filesystems, and since then, I=B4ve installed two USB drives to act as = root pool with FreeBSD-9.0-RELEASE so that I could import the original pool= in recovery, but it always stalls after a couple of hours. Looking at top,= I could see that the 16GB RAM was maxed out, so I have heavily tuned down = kmem, arc, etc: You're running out of RAM during the import, as it loads the DDT. Stuff a bunch more RAM into the machine (32 GB, 48 GB, even 64 GB). Then you will be able to load the full DDT into RAM, finish the aborted destroy process, and import the pool. We've run into this three or four times now on systems with dedupe enabled and only 16 GB of RAM. We've since upgraded all our boxes to a minimum of 32 GB, with one having 64 GB. ZFS dataset destruction with dedupe enabled takes *a lot* of time and RAM, as the DDT needs to be updated for every block freed. And rebooting in the middle of a "zfs destroy" operation means that the operation needs to finish at pool import time. -- Freddie Cash fjwcash@gmail.com<mailto:fjwcash@gmail.com> I took your advice. I replaced my Core i5 with a Xeon X3470 and ramped up t= he RAM to 32GB, maxing out the HW. Sadly enough, it still stalls in the exa= ct same manner:( This has to be the most frustrating thing ever, since ther= e=B4s tons of data there that I really need and if it wasn=B4t for that stu= pid destroy operation, it would still be accessible. I feel that FreeBSD is partly to blame since it was completely possible in = the originating SUN machine with Solaris that only has 16GB RAM to do the s= ame destroy to the same dataset without any problem. Sure, it took forever = and then some (about two weeks) but it stayed afloat during the whole time. The FreeBSD machine starts to accumulate more and more RAM, but quite stead= ily until it comes up to 9-9.5GB of RAM Wired and then it just SHOOTS off t= o swallow it all and cause a stall. That is the same behavior as when there= was only 16GB RAM. During the last attempt I had while true; do zfs-stats -A | grep "ARC Size:" zfs-stats -L | egrep '(L2 ARC Size|Bytes Scanned)' sleep 10 done running, so I could monitor the usage just before the crash. The ARC stayed= at 6GB, while the last top sample shows 28GB Wired. See for yourselves: top: http://i45.tinypic.com/21do5ra.png gstat: http://i49.tinypic.com/e197ax.png zfs-stats: http://i46.tinypic.com/250uxhz.png Import CTRL+T after it stalled: http://i46.tinypic.com/2uxvb4h.png I=B4m willing to try anything at this point. Any longshots you have are mos= t welcome, since it couldn=B4t get any worse:( Med V=E4nliga H=E4lsningar ---------------------------------------------------------------------------= ---- Karli Sj=F6berg Swedish University of Agricultural Sciences Box 7079 (Visiting Address Kron=E5sv=E4gen 8) S-750 07 Uppsala, Sweden Phone: +46-(0)18-67 15 66 karli.sjoberg@slu.se<mailto:karli.sjoberg@adm.slu.se>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49C9D08A-85EF-4D23-B07F-F3980CBA5A97>