Date: Sun, 1 Feb 2004 01:02:05 -0500 (EST) From: Robert Watson <rwatson@FreeBSD.org> To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= <des@des.no> Cc: current@FreeBSD.org Subject: Re: Coalescing pipe allocation Message-ID: <Pine.NEB.3.96L.1040201005853.29091A-100000@fledge.watson.org> In-Reply-To: <xzpwu77gz7n.fsf@dwp.des.no>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 1 Feb 2004, Dag-Erling Sm=F8rgrav wrote:
> Robert Watson <rwatson@FreeBSD.org> writes:
> > FYI, the vast majority of the cost of pipe creation appears to be in
> > setting up those mappings. I ran some mutex profiling and the amount o=
f
> > work getting done is pretty high -- we should be able to reap a lot of
> > benefit by doing even a little less work there.
>=20
> Like I said, setting up a VM mapping is very inefficient in FreeBSD.=20
> This, and the fd allocation issue which I fixed earlier this year, were
> the two areas where fefe's scalability benchmark really trounced us last
> October.=20
FYI, on that topic, I ran into the following WITNESS warning today while
pounding on the pipe subsystem:
login: lock order reversal
1st 0xc6585138 filedesc structure (filedesc structure) @
kern/kern_descrip.c:11
75
2nd 0xc0937960 Giant (Giant) @ vm/uma_core.c:2048
Stack backtrace:
backtrace(c087d47f,c0937960,c08794df,c08794df,c0892fb4) at backtrace+0x17
witness_checkorder(c0937960,9,c0892fb4,800,c066e432) at
witness_checkorder+0x6f8
_mtx_lock_flags(c0937960,0,c0892fab,800,c08d0e00) at _mtx_lock_flags+0x9d
uma_large_free(c64ed7e0,c063890a,c6585138,8,c087714b) at
uma_large_free+0xae
free(c66d9000,c08d0e00,400,497,c6585300) at free+0x11e
fdgrowtable(c6585100,800,400,4be,2dc0) at fdgrowtable+0x1b4
fdalloc(c6487bd0,0,e5bd5c9c,52a,c658fa50) at fdalloc+0x93
falloc(c6487bd0,e5bd5ccc,e5bd5cd4,164,0) at falloc+0x1f3
pipe(c6487bd0,e5bd5d14,c0898495,43a,0) at pipe+0x151
syscall(2f,2f,2f,bfbfec50,bfbfec58) at syscall+0x2a0
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (42), eip =3D 0x28097e3f, esp =3D 0xbfbfec0c, ebp =3D 0xbfbfec2=
8 ---
kern.ipc.maxpipekva exceeded; see tuning(7)
The test scenario consisted of forcing paging while running lots of pipe
allocation code in parallel on SMP:
robert@none:~> more page.c
#include <stdio.h>
#include <stdlib.h>
int
main(int argc, char *argv[])
{
char *p;
while (1) {
p =3D malloc(getpagesize());
if (p =3D=3D NULL) {
perror("malloc");
while (1);
}
p[0] =3D '\0';
}
}
robert@none:~> more test.c=20
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int
main(int argc, char *argv[])
{
int fd[2];
while (1) {
if (pipe(fd) =3D=3D -1) {
perror("pipe");
exit(-1);
}
}
}
I'm not quite sure when it transpired, but somewhere during the following:
% ./page & ./page & ./page & ./page &
% while (1)
./test & ./test & ./test &
end
So there was probably pretty strong Giant contention while the paging
processes were initially allocation memory. I was somewhat surprised to
see UMA hit Giant there.=20
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
robert@fledge.watson.org Senior Research Scientist, McAfee Research
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1040201005853.29091A-100000>