From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 1 21:15:52 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3500616A417 for ; Thu, 1 Nov 2007 21:15:52 +0000 (UTC) (envelope-from arno@heho.snv.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 951F313C48A for ; Thu, 1 Nov 2007 21:15:51 +0000 (UTC) (envelope-from arno@heho.snv.jussieu.fr) Received: from heho.snv.jussieu.fr (heho.snv.jussieu.fr [134.157.184.22]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id lA1KKiQD000817 for ; Thu, 1 Nov 2007 21:20:44 +0100 (CET) X-Ids: 164 Received: from heho.snv.jussieu.fr (localhost [127.0.0.1]) by heho.snv.jussieu.fr (8.13.3/jtpda-5.2) with ESMTP id lA1KKgWn014715 for ; Thu, 1 Nov 2007 21:20:42 +0100 (MET) Received: (from arno@localhost) by heho.snv.jussieu.fr (8.13.3/8.13.1/Submit) id lA1KKgc7014712; Thu, 1 Nov 2007 21:20:42 +0100 (MET) (envelope-from arno) To: freebsd-hackers@freebsd.org From: "Arno J. Klaassen" Date: 01 Nov 2007 21:20:42 +0100 Message-ID: Lines: 90 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.164]); Thu, 01 Nov 2007 21:20:44 +0100 (CET) X-Virus-Scanned: ClamAV 0.88.7/4659/Thu Nov 1 17:24:40 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 472A351C.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! Subject: "indefinite" wait buffer patch X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2007 21:15:52 -0000 --=-=-= Hello, while slowly testing releng_7, I remembered I have since about two years the attached diff in my releng_6 sources (patch recreated against releng_7 with low timeouts for debugging) : it addresses the situation when one creates a huge swap-space on a (relatively) slow disk-subsystem : e.g. for scientific computing it sometimes makes sense to have, e.g. 8G swap for 2G main memory if you know you're treating N less then 2G matrices and process is CPU-bound for quite a while for 1 matrix before switching to the other. But then, when switching from one matrix to another, dmesg gets flooded by : "indefinite wait buffer" messages. The attached patch shows in fact that the wait buffer is never "indefinite" (unless a real HW-problem probably) and linearly increases timeout to match with reality. The last chunk is just to prevent for a panic at reboot when there is so much data swapped out that is doesn't get treated before 'reboot-finish-time-out'. With this patch, dmesg lookes like following (with low timeout values, on production systems I use TIMO_CHUNK 20 TIMO_START 1O ) : Nov 1 20:09:52 install kernel: swap_pager: wait buffer timeout 1 (1 secs): bufobj: 0, blkno: 1649, size: 28672 Nov 1 20:09:52 install kernel: swap_pager: wait buffer timeout 2 (2 secs): bufobj: 0, blkno: 1649, size: 28672 Nov 1 20:09:52 install kernel: swap_pager: wait buffer completed (2 retry): bufobj: 0, blkno: 1649, size: 28672 Nov 1 20:37:09 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 329073, size: 32768 Nov 1 20:37:10 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 329073, size: 32768 Nov 1 20:39:06 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 381137, size: 32768 Nov 1 20:39:07 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 381137, size: 32768 Nov 1 20:39:11 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 381161, size: 32768 Nov 1 20:39:14 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 381161, size: 32768 Nov 1 20:39:19 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 381209, size: 32768 Nov 1 20:39:20 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 381209, size: 32768 Nov 1 20:43:18 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 300329, size: 32768 Nov 1 20:43:18 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 300329, size: 32768 Nov 1 20:44:23 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 330617, size: 32768 Nov 1 20:44:24 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 330617, size: 32768 Nov 1 20:44:28 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 330649, size: 32768 Nov 1 20:44:28 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 330649, size: 32768 Nov 1 20:44:33 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 330665, size: 32768 Nov 1 20:44:36 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 330665, size: 32768 Nov 1 20:45:18 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 356481, size: 32768 Nov 1 20:45:18 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 356481, size: 32768 Nov 1 20:45:23 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 356521, size: 32768 Nov 1 20:45:27 install kernel: swap_pager: wait buffer timeout 2 (4 secs): bufobj: 0, blkno: 356521, size: 32768 Nov 1 20:45:31 install kernel: swap_pager: wait buffer completed (2 retry): bufobj: 0, blkno: 356521, size: 32768 Nov 1 20:46:37 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 391113, size: 32768 Nov 1 20:46:37 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 391113, size: 32768 Nov 1 20:46:42 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 391129, size: 32768 Nov 1 20:46:45 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 391129, size: 32768 Nov 1 20:48:18 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 432249, size: 32768 Nov 1 20:48:18 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 432249, size: 32768 Nov 1 20:48:25 install kernel: swap_pager: wait buffer timeout 1 (4 secs): bufobj: 0, blkno: 432273, size: 32768 Nov 1 20:48:25 install kernel: swap_pager: wait buffer completed (1 retry): bufobj: 0, blkno: 432273, size: 32768 (And then the timeout of 8 secs apperently is enough for this test setup). Any thoughts? Regards, Arno --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=swap.patch Index: sys/vm/swap_pager.c =================================================================== RCS file: /home/ncvs/src/sys/vm/swap_pager.c,v retrieving revision 1.295 diff -u -r1.295 swap_pager.c --- sys/vm/swap_pager.c 5 Aug 2007 21:04:32 -0000 1.295 +++ sys/vm/swap_pager.c 1 Nov 2007 18:59:18 -0000 @@ -941,6 +941,10 @@ vm_page_t mreq; int i; int j; + int retry = 0; +#define TIMO_CHUNK 2 +#define TIMO_START 1 /* set low to force quick first timeout */ + static int timo_secs = TIMO_START; daddr_t blk; mreq = m[reqpage]; @@ -1066,16 +1070,28 @@ */ VM_OBJECT_LOCK(object); while ((mreq->oflags & VPO_SWAPINPROG) != 0) { + if (retry == 0) { mreq->oflags |= VPO_WANTED; vm_page_lock_queues(); vm_page_flag_set(mreq, PG_REFERENCED); vm_page_unlock_queues(); PCPU_INC(cnt.v_intrans); - if (msleep(mreq, VM_OBJECT_MTX(object), PSWP, "swread", hz*20)) { + } + if (msleep(mreq, VM_OBJECT_MTX(object), PSWP, "swread", hz*timo_secs)) { printf( -"swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n", - bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount); +"swap_pager: wait buffer timeout %d (%d secs): bufobj: %p, blkno: %jd, size: %ld\n", + ++retry, timo_secs, bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount); + if (retry*TIMO_CHUNK > timo_secs) { + timo_secs = retry*TIMO_CHUNK; + } + } else { + if (retry > 0) { + printf( +"swap_pager: wait buffer completed (%d retry): bufobj: %p, blkno: %jd, size: %ld\n", + retry, bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount); + } } + } /* @@ -1553,6 +1569,7 @@ swp_pager_force_pagein(vm_object_t object, vm_pindex_t pindex) { vm_page_t m; + int ret; vm_object_pip_add(object, 1); m = vm_page_grab(object, pindex, VM_ALLOC_NORMAL|VM_ALLOC_RETRY); @@ -1567,8 +1584,18 @@ return; } - if (swap_pager_getpages(object, &m, 1, 0) != VM_PAGER_OK) - panic("swap_pager_force_pagein: read from swap failed");/*XXX*/ + if ((ret=swap_pager_getpages(object, &m, 1, 0)) != VM_PAGER_OK) { + if (ret == VM_PAGER_FAIL) { + printf("swp_pager_force_pagein: VM_PAGER_FAIL\n"); + } + else { + if (ret == VM_PAGER_ERROR) { + printf("swp_pager_force_pagein: VM_PAGER_ERROR\n"); + } + else + panic("swap_pager_force_pagein: read from swap failed");/*XXX*/ + } + } vm_object_pip_subtract(object, 1); vm_page_lock_queues(); vm_page_dirty(m); --=-=-= -- Arno J. Klaassen SCITO S.A. 8 rue des Haies F-75020 Paris, France http://scito.com --=-=-=--