From owner-freebsd-net@FreeBSD.ORG Sat Nov 24 19:14:14 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9866B16A417; Sat, 24 Nov 2007 19:14:14 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id ECA9C13C45D; Sat, 24 Nov 2007 19:14:13 +0000 (UTC) (envelope-from max@love2party.net) Received: from amd64.laiers.local (dslb-088-066-037-134.pools.arcor-ip.net [88.66.37.134]) by mrelayeu.kundenserver.de (node=mrelayeu1) with ESMTP (Nemesis) id 0MKwpI-1Iw0SC1jKY-0004bI; Sat, 24 Nov 2007 20:14:12 +0100 From: Max Laier Organization: FreeBSD To: Robert Watson Date: Sat, 24 Nov 2007 20:15:11 +0100 User-Agent: KMail/1.9.7 References: <200711231232.04447.max@love2party.net> <20071123132453.W98338@fledge.watson.org> <200711242006.04753.max@love2party.net> In-Reply-To: <200711242006.04753.max@love2party.net> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1594088.YNyaglIa3o"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200711242015.13417.max@love2party.net> X-Provags-ID: V01U2FsdGVkX1/FKaJA+HWeDOz87JrHeHnZ3d1kKLInu+Ey/Xi pAm322Lhxofb+FjaFB4H4MsfIRTQ8tAHXtukDANXSG+PRbWHjr yfCZ3icoool1Xny6w834johtUi0DpNN2Wq6uNLuAHE= Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org Subject: Re: Switch pfil(9) to rmlocks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 19:14:14 -0000 --nextPart1594088.YNyaglIa3o Content-Type: multipart/mixed; boundary="Boundary-01=_BhHSHgZik2wVaF1" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_BhHSHgZik2wVaF1 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Saturday 24 November 2007, Max Laier wrote: > On Friday 23 November 2007, Robert Watson wrote: > > On Fri, 23 Nov 2007, Max Laier wrote: > > > attached is a diff to switch the pfil(9) subsystem to rmlocks, > > > which are more suited for the task. I'd like some exposure before > > > doing the switch, but I don't expect any fallout. This email is > > > going through the patched pfil already - twice. > > > > Max, > > > > Have you done performance measurements that show rmlocks to be a win > > in this scenario? I did some patchs for UNIX domain sockets to > > replace the rwlock there but it appeared not to have a measurable > > impact on SQL benchmarks, presumbaly because the read/write blend > > wasn't right and/or that wasnt a significant source of overhead in > > the benchmark. I'd anticipate a much more measurable improvement for > > pfil, but would be interested in learning how much is seen? > > I had to roll an artificial benchmark in order to see a significant > change (attached - it's a hack!). attached again text/x-csrc attachment got stripped. > Using 3 threads on a 4 CPU machine I get the following results: > null hook: ~13% +/- 2 > mtx hook: up to 40% [*] > rw hook: ~5% +/- 1 > rm hook: ~35% +/- 5 > > [*] The mtx hook is inconclusive as my measurements vary a lot. If one > thread gets lucky and keeps running the overall time obviously goes > down by a magnitude. It seems however, that rmlocks greatly increase > the chance of that happening - not sure if that's a good thing, though. > If all threads receive approximately equal runtime (which is almost > always the case for rwlocks) the difference is somewhere around 10%. =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --Boundary-01=_BhHSHgZik2wVaF1 Content-Type: text/plain; charset="iso-8859-6"; name="pfilbench.c" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pfilbench.c" /*- * Copyright (c) 2000 Max Laier * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPO= SE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTI= AL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRI= CT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include static struct mtx pb_mtx; MTX_SYSINIT(pb_mtx, &pb_mtx, "pfilbench thread mtx", MTX_DEF); static struct mtx hl_mtx; MTX_SYSINIT(hl_mtx, &hl_mtx, "pfilbench mtx", MTX_DEF); static struct rwlock hl_rw; RW_SYSINIT(hl_rw, &hl_rw, "pfilbench rwlock"); static struct rmlock hl_rm; RM_SYSINIT(hl_rm, &hl_rm, "pfilbench rmlock", 0); static struct pfil_head pb_hook; static struct ifnet pb_dummy; SYSCTL_NODE(_debug, OID_AUTO, pfilbench, CTLFLAG_RW, 0, "pfilbench"); static unsigned long count =3D 1000*1000*10; SYSCTL_ULONG(_debug_pfilbench, OID_AUTO, count, CTLFLAG_RW, &count, 0, "count"); static int die =3D 0; SYSCTL_INT(_debug_pfilbench, OID_AUTO, die, CTLFLAG_RW, &die, 0, "die"); static int threads =3D 1; SYSCTL_INT(_debug_pfilbench, OID_AUTO, threads, CTLFLAG_RW, &threads, 0, "threads"); static int mode =3D 0; /* 0 =3D NULL, 1 =3D mtx, 2 =3D rw, 3 =3D rm */ SYSCTL_INT(_debug_pfilbench, OID_AUTO, mode, CTLFLAG_RW, &mode, 0, "mode"); static int start =3D 0; static int init =3D 0; static int done; static void testloop(void *arg) { struct mbuf *m; unsigned long i; struct timeval stime, etime; m =3D m_gethdr(M_TRYWAIT, MT_HEADER); if (m =3D=3D NULL) return; m->m_pkthdr.rcvif =3D &pb_dummy; while (init && !die && !*(int *)arg) tsleep(arg, 0, "ready", hz); microtime(&stime); for (i =3D 0; i < count && !die; i++) { if (pfil_run_hooks(&pb_hook, &m, &pb_dummy, PFIL_IN, NULL) !=3D 0) return; if (m =3D=3D NULL) return; } microtime(&etime); if (etime.tv_usec - stime.tv_usec < 0) etime.tv_sec--; etime.tv_usec =3D abs(etime.tv_usec - stime.tv_usec); etime.tv_sec =3D etime.tv_sec - stime.tv_sec; =09 mtx_lock(&pb_mtx); printf("RESULT: %jd/%jd.%jd\n", i + 1, etime.tv_sec, etime.tv_usec); done++; mtx_unlock(&pb_mtx); wakeup(&done); kproc_exit(0); } static int null_hook(void *arg, struct mbuf **mp, struct ifnet *ifp, int dir, struct inpcb *arg1) { DELAY(1); return 0; } static int mtx_hook(void *arg, struct mbuf **mp, struct ifnet *ifp, int dir, struct inpcb *arg1) { mtx_lock(&hl_mtx); DELAY(1); mtx_unlock(&hl_mtx); return 0; } static int rw_hook(void *arg, struct mbuf **mp, struct ifnet *ifp, int dir, struct inpcb *arg1) { rw_rlock(&hl_rw); DELAY(1); rw_runlock(&hl_rw); return 0; } static int rm_hook(void *arg, struct mbuf **mp, struct ifnet *ifp, int dir, struct inpcb *arg1) { struct rm_priotracker rmpt; rm_rlock(&hl_rm, &rmpt); DELAY(1); rm_runlock(&hl_rm, &rmpt); return 0; } static int sysctl_pb_init(SYSCTL_HANDLER_ARGS) { int _init =3D init; int error, i; error =3D sysctl_handle_int(oidp, &_init, 0, req); _init =3D _init ? 1 : 0; if (init =3D=3D _init) return (error); if (init =3D=3D 1) { switch (mode) { case 0: pfil_remove_hook(null_hook, NULL, PFIL_ALL, &pb_hook); break; case 1: pfil_remove_hook(mtx_hook, NULL, PFIL_ALL, &pb_hook); break; case 2: pfil_remove_hook(rw_hook, NULL, PFIL_ALL, &pb_hook); break; case 3: pfil_remove_hook(rm_hook, NULL, PFIL_ALL, &pb_hook); break; } mtx_lock(&pb_mtx); die =3D 1; init =3D 0; mtx_unlock(&pb_mtx); wakeup(&start); } if (init =3D=3D 0) { mtx_lock(&pb_mtx); init =3D 1; mtx_unlock(&pb_mtx); for (i =3D 0; i < threads; i++) { kproc_create(testloop, &start, NULL, 0, 0, "pb_loop_%d", i); } switch (mode) { case 0: pfil_add_hook(null_hook, NULL, PFIL_ALL|PFIL_WAITOK, &pb_hook); break; case 1: pfil_add_hook(mtx_hook, NULL, PFIL_ALL|PFIL_WAITOK, &pb_hook); break; case 2: pfil_add_hook(rw_hook, NULL, PFIL_ALL|PFIL_WAITOK, &pb_hook); break; case 3: pfil_add_hook(rm_hook, NULL, PFIL_ALL|PFIL_WAITOK, &pb_hook); break; } done =3D 0; } return (error); } SYSCTL_PROC(_debug_pfilbench, OID_AUTO, init, CTLTYPE_INT|CTLFLAG_RW, &init, 0, &sysctl_pb_init, "I", "Init the benchmark"); static int sysctl_pb_start(SYSCTL_HANDLER_ARGS) { int _start =3D start; int error; error =3D sysctl_handle_int(oidp, &_start, 0, req); _start =3D _start ? 1 : 0; if (init =3D=3D 0) return (EINVAL); if (_start) { start =3D 1; wakeup(&start); } else { start =3D 0; } return (error); } SYSCTL_PROC(_debug_pfilbench, OID_AUTO, start, CTLTYPE_INT|CTLFLAG_RW, &start, 0, &sysctl_pb_start, "I", "Start the benchmark"); static int load (module_t mod, int cmd, void *arg) { int error; error =3D 0; switch (cmd) { case MOD_LOAD: pb_hook.ph_type =3D PFIL_TYPE_IFNET; pb_hook.ph_ifnet =3D &pb_dummy; pfil_head_register(&pb_hook); break; case MOD_UNLOAD: die =3D 1; wakeup(&start); pause("stoping", hz); pfil_head_unregister(&pb_hook); break; default : error =3D EOPNOTSUPP; break; } return error; } static moduledata_t mod_data=3D { "pfilbench", load, 0 }; DECLARE_MODULE(pfilbench, mod_data, SI_SUB_EXEC, SI_ORDER_ANY); --Boundary-01=_BhHSHgZik2wVaF1-- --nextPart1594088.YNyaglIa3o Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHSHhBXyyEoT62BG0RAi5BAJ9/nTNeOVuihizACal3Fk8oJ1ZQlwCfbmlN o1R6WWpEw4UNQwszM9qrLNI= =GM6C -----END PGP SIGNATURE----- --nextPart1594088.YNyaglIa3o--