From owner-freebsd-hackers  Tue Jan 25  8:48:21 2000
Delivered-To: freebsd-hackers@freebsd.org
Received: from bomber.avantgo.com (ws1.avantgo.com [207.214.200.194])
	by hub.freebsd.org (Postfix) with ESMTP id EF77B14EB8
	for <freebsd-hackers@FreeBSD.ORG>; Tue, 25 Jan 2000 08:48:12 -0800 (PST)
	(envelope-from scott@avantgo.com)
Received: from river ([10.0.128.30]) by bomber.avantgo.com
          (Netscape Messaging Server 3.5)  with SMTP id 214;
          Tue, 25 Jan 2000 08:44:12 -0800
Message-ID: <0c2101bf6753$cf37f280$1e80000a@avantgo.com>
From: "Scott Hess" <scott@avantgo.com>
To: "Scott Hess" <scott@avantgo.com>,
	"Matthew Dillon" <dillon@apollo.backplane.com>
Cc: <freebsd-hackers@FreeBSD.ORG>
References:  <01b601bf6696$60701930$1e80000a@avantgo.com> <200001241939.LAA91219@apollo.backplane.com> <0be801bf6715$601423d0$1e80000a@avantgo.com>
Subject: Re: Performance issue with rfork() and single socketpairs versus multiple socketpairs.
Date: Tue, 25 Jan 2000 08:47:03 -0800
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_0C1E_01BF6710.C075B360"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.00.2919.6600
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600
Sender: owner-freebsd-hackers@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.ORG

This is a multi-part message in MIME format.

------=_NextPart_000_0C1E_01BF6710.C075B360
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

"Scott Hess" <scott@avantgo.com> wrote:
> "Matthew Dillon" <dillon@apollo.backplane.com> wrote:
> > :Unfortunately, I've found that having a group of processes reading
> > :from  a group of socketpairs has better performance than having
> > :them all read from a single socketpair.  I've been unable to
> > :determine why.
> >
> >  The problem is that when you have N processes waiting on a single
> >  socket and you write to the socket, all N processes will wake up even
> >  though only one will get the data you wrote.
> <snip>
> >  As an alternative to socket pairs, I would consider using SysV shared
> >  memory and SysV semaphores.
>
> OK, so let's say I did spend some time implementing it in terms of
semget()
> and semop().  Would you be totally apalled if the performance turned out
to
> be about the same as using a single socketpair?
<snip>
>Unfortunately, I'll have to wait until tomorrow morning
>to rip things out and make a suitable example program for posting.

Find attached a new version of commtest.c which uses semaphores.  I've also
placed a copy at http://www.doubleu.com/~scott/commtest.c.  The performance
is identical enough that I'm guessing that semaphores must suffer from the
exact same problems as the single socketpair version.

Thanks,
scott


------=_NextPart_000_0C1E_01BF6710.C075B360
Content-Type: application/octet-stream;
	name="commtest.c"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="commtest.c"

// commtest.c=0A=
// gcc -Wall -g -o commtest commtest.c=0A=
//=0A=
// Test performance differences for multiple socketpairs versus a=0A=
// single shared socketpair versus SYSV semaphores.=0A=
#include <unistd.h>=0A=
#include <stdio.h>=0A=
#include <stdlib.h>=0A=
#include <fcntl.h>=0A=
#include <sys/types.h>=0A=
#include <sys/socket.h>=0A=
#include <string.h>=0A=
#include <errno.h>=0A=
#include <sys/sem.h>=0A=
#include <sys/ipc.h>=0A=
=0A=
typedef unsigned char request_t;=0A=
=0A=
#define CLIENT_EXIT ((request_t)(~0))=0A=
#define CLIENT_COUNT 32=0A=
#define REQUEST_TARGET 10000=0A=
=0A=
int client_fd_count=3D0;=0A=
int client_fds[ CLIENT_COUNT];=0A=
int server_fds[ CLIENT_COUNT];=0A=
=0A=
/* Reflect requests. */=0A=
void socket_client( int fd)=0A=
{=0A=
    request_t request;=0A=
    int rc;=0A=
    =0A=
    while( 1) {=0A=
        if( (rc=3Dread( fd, &request, sizeof( request)))=3D=3D-1) {=0A=
            perror( "client read");=0A=
            _exit( 1);=0A=
        } else if( rc<sizeof( request)) {=0A=
            _exit( 0);=0A=
        } else if( (rc=3Dwrite( fd, &request, sizeof( =
request)))=3D=3D-1) {=0A=
            perror( "client write");=0A=
            _exit( 1);=0A=
        } else if( rc<sizeof( request)) {=0A=
            _exit( 0);=0A=
        }=0A=
    }=0A=
}=0A=
void sem_client( int sem, int sem_id, int fd, request_t rid)=0A=
{=0A=
    struct sembuf sem_req=3D{ sem_id, -1, 0};=0A=
    int rc;=0A=
    =0A=
    while( 1) {=0A=
	if( (rc=3Dsemop( sem, &sem_req, 1))=3D=3D-1) {=0A=
            perror( "client semop");=0A=
            _exit( 1);=0A=
        } else if( (rc=3Dwrite( fd, &rid, sizeof( rid)))=3D=3D-1) {=0A=
            perror( "client write");=0A=
            _exit( 1);=0A=
        } else if( rc<sizeof( rid)) {=0A=
            _exit( 0);=0A=
        }=0A=
    }=0A=
}=0A=
=0A=
void main( int argc, char **argv)=0A=
{=0A=
    int ii, jj, flags, maxfd, rc;=0A=
    int pair[ 2];=0A=
    fd_set default_fdset, fdset;=0A=
    pid_t cpid;=0A=
    request_t request[32];=0A=
    int request_count=3D0;=0A=
    int sem=3D-1;=0A=
    int sem_count=3D-1;=0A=
    =0A=
    client_fd_count=3D1;=0A=
    if( argc=3D=3D2 && !strcmp( argv[ 1], "--multi")) {=0A=
        client_fd_count=3DCLIENT_COUNT;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--single")) {=0A=
        client_fd_count=3D1;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--handful")) {=0A=
        client_fd_count=3D4;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--sem")) {=0A=
	sem_count=3D1;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--semulti")) {=0A=
	sem_count=3DCLIENT_COUNT;=0A=
    }=0A=
    =0A=
    if( sem_count>-1) {=0A=
	sem=3Dsemget( IPC_PRIVATE, sem_count, SEM_R|SEM_A);=0A=
	if( sem=3D=3D-1) {=0A=
	    perror( "semget");=0A=
	    exit( 1);=0A=
	}=0A=
    }=0A=
    =0A=
    maxfd=3D0;=0A=
    FD_ZERO( &default_fdset);=0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        if( socketpair(AF_UNIX,SOCK_STREAM,0,pair)=3D=3D-1) {=0A=
            perror( "creating socketpair");=0A=
            goto err_exit;=0A=
        } else if( (flags=3Dfcntl( pair[ 0], F_GETFL, NULL))=3D=3D-1) {=0A=
            perror( "getting flags");=0A=
            goto err_exit;=0A=
        } else if( fcntl( pair[ 0], F_SETFL, flags|O_NONBLOCK)=3D=3D-1) {=0A=
            perror( "setting flags");=0A=
            goto err_exit;=0A=
        }=0A=
        =0A=
        client_fds[ ii]=3Dpair[ 0];=0A=
        server_fds[ ii]=3Dpair[ 1];=0A=
        =0A=
        FD_SET( client_fds[ ii], &default_fdset);=0A=
        if( client_fds[ ii]>maxfd) {=0A=
            maxfd=3Dclient_fds[ ii];=0A=
        }=0A=
    }=0A=
    =0A=
    /* Spin off children to process requests. */=0A=
    for( ii=3D0; ii<CLIENT_COUNT; ii++) {=0A=
        if( (cpid=3Drfork(RFPROC))=3D=3D-1) {=0A=
            perror( "rfork");=0A=
            goto err_exit;=0A=
        } else if( cpid=3D=3D0) {=0A=
	    if( sem!=3D-1) {=0A=
		sem_client( sem, ii%sem_count, server_fds[ ii%client_fd_count], ii);=0A=
	    } else {=0A=
		socket_client( server_fds[ ii%client_fd_count]);=0A=
	    }=0A=
        } else {=0A=
            request[0]=3Dii;=0A=
	    if( sem!=3D-1) {=0A=
		struct sembuf sem_req=3D{ ii%sem_count, 1, 0};=0A=
		if( semop( sem, &sem_req, 1)=3D=3D-1) {=0A=
		    perror( "semop");=0A=
		    goto err_exit;=0A=
		}=0A=
            } else if( write( client_fds[ ii%client_fd_count], =
&(request[0]), sizeof( request[0]))=3D=3D-1) {=0A=
                perror( "writing initial request");=0A=
                goto err_exit;=0A=
            }=0A=
        }=0A=
    }=0A=
=0A=
    while( request_count<REQUEST_TARGET) {=0A=
        fdset=3Ddefault_fdset;=0A=
        select( maxfd+1, &fdset, NULL, &fdset, NULL);=0A=
        =0A=
        for( ii=3D0; ii<client_fd_count && request_count<REQUEST_TARGET; =
ii++) {=0A=
            if( (rc=3Dread( client_fds[ ii], request, sizeof( =
request)))=3D=3D-1) {=0A=
		if( rc=3D=3D-1 && errno!=3DEAGAIN && errno!=3DEWOULDBLOCK) {=0A=
		    perror( "reading response");=0A=
		    goto err_exit;=0A=
		}=0A=
	    } else {=0A=
		jj=3Drc/sizeof( request[ 0]);=0A=
		while( jj--) {=0A=
		    if( sem!=3D-1) {=0A=
			struct sembuf sem_req=3D{ request[jj]%sem_count, 1, 0};=0A=
			if( semop( sem, &sem_req, 1)=3D=3D-1) {=0A=
			    perror( "semop");=0A=
			    goto err_exit;=0A=
			}=0A=
		    } else if( write( client_fds[ ii], &(request[jj]), sizeof( =
request[jj]))=3D=3D-1) {=0A=
			perror( "writing request");=0A=
			goto err_exit;=0A=
		    }=0A=
		    request_count++;=0A=
		}=0A=
            }=0A=
        }=0A=
    }=0A=
    =0A=
    if( sem!=3D-1) {=0A=
	semctl( sem, 0, IPC_RMID);=0A=
    }=0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        close( client_fds[ ii]);=0A=
    }=0A=
    exit( 0);=0A=
=0A=
err_exit:=0A=
    if( sem!=3D-1) {=0A=
	semctl( sem, 0, IPC_RMID);=0A=
    }=0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        close( client_fds[ ii]);=0A=
    }=0A=
    exit( 1);=0A=
}=0A=

------=_NextPart_000_0C1E_01BF6710.C075B360--



To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message