Date: Thu, 3 Apr 2014 20:30:44 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Bryan Drewery <bdrewery@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org Subject: Re: Poudriere: rm -rf: Directory not empty Message-ID: <20140403173044.GY21331@kib.kiev.ua> In-Reply-To: <dc09beae8394e6f23567846bab277ac3@shatow.net> References: <dc09beae8394e6f23567846bab277ac3@shatow.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Thu, Apr 03, 2014 at 09:35:27AM -0500, Bryan Drewery wrote: > Hi, > > While using Poudriere to build packages on segregated tmpfs jails > we sometimes get the following errors: > > ====>> [08] Starting build of devel/qt4-qt3support > ====>> [08] Starting build of graphics/qt4-opengl > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr: > Directory not empty > ====>> [08] Starting build of math/py-numpy > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports: > Directory not empty > rm: > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr: > Directory not empty > > What is happening here is that the devel/qt4-qt3support finishes, > fails to cleanup itself, then the next build tries to cleanup the > previous tempdir and fails. The next build then fails, and so on. > Eventually crashing the whole build. > > This is the result of just "rm -rf > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work". > > devel/qt4-qt3support runs rm -rf, fails. kill -9 -1 is ran in jail. > graphics/qt4-opengl starts, runs jail -r [kills processes], tries rm > -rf, fails > math/py-numpy starts, runs jail -r [kills processes], tries rm -rf, > fails > > Another example is at the bottom of > http://beefy1.isc.freebsd.org/bulk/83i386-default/2014-02-12_03h42m23s/logs/eclipse-3.7.1_4.log > The eclipse one involved a process crashing and a coredump as well. > I thought perhaps there was a race between writing core and > removing the directory, but I found no evidence of that either > by code inspection or testing. > > As shown above, no processes should be running in the jail at this > point. Poudriere itself is not touching these directories outside > of the jail either. There's no nullfs mounts of these > files to elsewhere either that may be getting touched. > > What might cause this? It's very difficult to reproduce and is > reported about once every 2 months or less. Note well this is > not due to flags. A rerun of these same ports won't hit the > issue. > > So far the workaround is to umount the tmpfs and remount it, but this > is not a solution as tmpfs is optional for Poudriere. From past research > it was found to not be tmpfs-specific, but my confidence level is not > 100% > on that. > > This has been seen on at least 9.2-R, and 10.0-R. > > I can't recreate this with simple tests though on ZFS or TMPFS. > > cd /tmp > ( rm -rf test; mkdir test; cat /dev/random > test/foo & sleep 1; rm > -rf test; kill $! ) > ( rm -rf test; mkdir test; mkfifo test/foo; cat test/foo & sleep 1; rm > -rf test; kill $! ) > ( rm -rf test; mkdir test; cd test; rm -rf ../test ) > > In the other cases it's not clear if looping on rm -rf would work or > if it would spin forever. We have not tried it since it's so difficult > to reproduce. > When the situation occured and you notes it, do you still have an access to the tmpfs directory which failed rm -rf ? If yes, try to do ls -la there, and ktrace the "rm -rf". Another approach is to patch tmpfs_rmdir() in tmpfs_vnops.c and dump some information when ENOTEMPTY error is returned; e.g. you could print the directory content and tn_size. [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTPZrDAAoJEJDCuSvBvK1BPVQP/jehDsdsYAgdnbzjKFe+Iiaw RNW+ZPN/1jHNxxB9o2OWs9cVZH9N0TT/yLSkTVmmPSbtIaYD9DS1ele0urgy2tNz +J85tBhjzacP0utYNZlUCJ0wN3cq9hfOkLU9MVY9XNy3NhJ8xjm34I0emBXk9mDU gDJhBkPUwMFsl+KVAD4gwaIGFDX3LQLivoiUCQG0OX2hQB0ixwKGa800QJVkW4BI uU4nLkNzvfuDCgM+sUxBiJT8fkIZAQymomwoa4aaK/6t+RN8Wd2zNr8jZvfeVY+F KmZ+XnckLWLF1jLaXPvglGhpFSpQJmAXXK3V8lA+LfPvhYH3tQnh5VS2H2Tt82c1 UZVQhbBKjsbS1PjRmLxCmxFVCIZQ+PWuPVTJjYOg+f6bzJokaN/dKsJ0TIl5O0BP TorZjTLGRVtfAQ50Vgbh4uWGcgCTQlvxf8tVdiTQkrRx9Nr7MqIpjfLYQwS5ETmL RvXlXLNYqeLzzCtQPc08OYqOq8xp3WQEFcrKxocUs2iEmXNVQyuAvXF24j8aQXA6 BnvDNR/jUqVD3/Km3DCyVBj4Ls53tKBUyABMOFujiFZASXc1AbHADv1lp7cWn4my QzsBJEJb2A5bBJwC/dr+K7MtlLInV6dM35FkMnDG3TmqteKzpnJtZ4VkMz+RvBNX OTIqTSqE+ShvJDYkHaGG =C245 -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140403173044.GY21331>