Date: Fri, 17 Jun 2022 16:52:41 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 246886] [sendfile] Nginx + NFS or FUSE causes VM stall Message-ID: <bug-246886-227-oHjdwED8Z4@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-246886-227@https.bugs.freebsd.org/bugzilla/> References: <bug-246886-227@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246886 --- Comment #70 from firk@cantconnect.ru --- So, the source of the problem seems was base r337165 . I didn't tested revisions around, but I did tested rollback of this specific comit, setting f_iosize back to PAGE_SIZE=3D4096 and the problem is gone. But this is not = the bug itself, it is just a trigger for another problem(s). As already noted, backtrace for the deadlock is: vm_page_grab_pages+0x3f2 allocbuf+0x371 (vfs_vmio_extend inlined inside) getblkx+0x5be breadn_flags+0x3d vfs_bio_getpages+0x403 fuse_vnop_getpages+0x46 VOP_GETPAGES_APV+0x7b vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7b vnode_pager_getpages_async+0x7d vn_sendfile+0xdf2 (sendfile_swapin inlined inside) sendfile+0x12b amd64_syscall+0x387 fast_syscall_common+0xf8 What happens: 1) sendfile_swapin() grabs and exclusively-busies bunch of pages via vm_page_grab_pages(); 2) it then scans them sequentially, unbusies already loaded ones, and calls vm_pages_get_pages_async() for not yet loaded ones, which should load them = and call sendfile_iodone() callback, and that how it was in 11.x; 3) vm_pages_get_pages_async() calls some other nested functions, and now we= are in vfs_bio_getpages(). Note: despite the "async" name all this done synchronously; 4) vfs_bio_getpages() still have vm_page[] array and its size as arguments, passed unchanged straightly from sendfile_swapin(); it downgrades exclusive-busy state to shared-busy for the given pages range; 5) the next step (bread_gb -> breadn_flags) is done using block index and s= ize obtained from fusefs driver via get_lblkno() and get_blksize() callbacks, a= nd the new block size is 65536 by default. And, going through getblkx() -> allocbuf() -> vfs_vmio_extend(), the last one calls vm_page_grab_pages() ag= ain, but the range is not the requested one, but the one matches fusefs block si= ze, effectively aligned to 16-block boundary (65536 =3D 16*4096). This leads to deadlock because the pages after currently requested are still exclusively-= busy (see p.2) What could be fixed: 1) easiest: rollback iosize to PAGE_SIZE, but this will reduce i/o speed ba= ck 2) rework sendfile_swapin() to first scan entire range for being loaded or = not and only then calling queued vm_pager_get_pages_async(); don't think it is = good because everythink already works when fusefs/nfs not used. 3) make "async" functions really async (see p.3) for fusefs; i don't know i= f it easy or not - this will resolve deadlock too because vfs_bio_getpages() will not block the sequential scan of requested pages by sendfile_swapin() 4) prevent partially loaded filesystem f_iosize blocks from happening; agai= n, I don't know is it easy or even desirable or not. PS: I don't know how all this works or not in 13.x --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-246886-227-oHjdwED8Z4>