From owner-freebsd-current@FreeBSD.ORG Sat Jan 31 22:04:37 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C959416A4CE for ; Sat, 31 Jan 2004 22:04:37 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id F253B43D1F for ; Sat, 31 Jan 2004 22:04:35 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i11625Ud029182; Sun, 1 Feb 2004 01:02:05 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i11625be029179; Sun, 1 Feb 2004 01:02:05 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Sun, 1 Feb 2004 01:02:05 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: current@FreeBSD.org Subject: Re: Coalescing pipe allocation X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2004 06:04:38 -0000 On Sun, 1 Feb 2004, Dag-Erling Sm=F8rgrav wrote: > Robert Watson writes: > > FYI, the vast majority of the cost of pipe creation appears to be in > > setting up those mappings. I ran some mutex profiling and the amount o= f > > work getting done is pretty high -- we should be able to reap a lot of > > benefit by doing even a little less work there. >=20 > Like I said, setting up a VM mapping is very inefficient in FreeBSD.=20 > This, and the fd allocation issue which I fixed earlier this year, were > the two areas where fefe's scalability benchmark really trounced us last > October.=20 FYI, on that topic, I ran into the following WITNESS warning today while pounding on the pipe subsystem: login: lock order reversal 1st 0xc6585138 filedesc structure (filedesc structure) @ kern/kern_descrip.c:11 75 2nd 0xc0937960 Giant (Giant) @ vm/uma_core.c:2048 Stack backtrace: backtrace(c087d47f,c0937960,c08794df,c08794df,c0892fb4) at backtrace+0x17 witness_checkorder(c0937960,9,c0892fb4,800,c066e432) at witness_checkorder+0x6f8 _mtx_lock_flags(c0937960,0,c0892fab,800,c08d0e00) at _mtx_lock_flags+0x9d uma_large_free(c64ed7e0,c063890a,c6585138,8,c087714b) at uma_large_free+0xae free(c66d9000,c08d0e00,400,497,c6585300) at free+0x11e fdgrowtable(c6585100,800,400,4be,2dc0) at fdgrowtable+0x1b4 fdalloc(c6487bd0,0,e5bd5c9c,52a,c658fa50) at fdalloc+0x93 falloc(c6487bd0,e5bd5ccc,e5bd5cd4,164,0) at falloc+0x1f3 pipe(c6487bd0,e5bd5d14,c0898495,43a,0) at pipe+0x151 syscall(2f,2f,2f,bfbfec50,bfbfec58) at syscall+0x2a0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (42), eip =3D 0x28097e3f, esp =3D 0xbfbfec0c, ebp =3D 0xbfbfec2= 8 --- kern.ipc.maxpipekva exceeded; see tuning(7) The test scenario consisted of forcing paging while running lots of pipe allocation code in parallel on SMP: robert@none:~> more page.c #include #include int main(int argc, char *argv[]) { char *p; while (1) { p =3D malloc(getpagesize()); if (p =3D=3D NULL) { perror("malloc"); while (1); } p[0] =3D '\0'; } } robert@none:~> more test.c=20 #include #include #include int main(int argc, char *argv[]) { int fd[2]; while (1) { if (pipe(fd) =3D=3D -1) { perror("pipe"); exit(-1); } } } I'm not quite sure when it transpired, but somewhere during the following: % ./page & ./page & ./page & ./page & % while (1) ./test & ./test & ./test & end So there was probably pretty strong Giant contention while the paging processes were initially allocation memory. I was somewhat surprised to see UMA hit Giant there.=20 Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research