From owner-freebsd-net@freebsd.org Tue Nov 3 13:14:49 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A71BA2416A for ; Tue, 3 Nov 2015 13:14:49 +0000 (UTC) (envelope-from ralsaadi@swin.edu.au) Received: from iport1.cc.swin.edu.au (iport1.cc.swin.edu.au [136.186.0.49]) by mx1.freebsd.org (Postfix) with ESMTP id D42FD120D for ; Tue, 3 Nov 2015 13:14:47 +0000 (UTC) (envelope-from ralsaadi@swin.edu.au) X-IronPort-AV: E=Sophos;i="5.20,238,1444654800"; d="gif'147?scan'147,208,147";a="14515267" Received: from gsp-ex03.ds.swin.edu.au (HELO outlook.swin.edu.au) ([136.186.126.19]) by iport1.cc.swin.edu.au with ESMTP; 04 Nov 2015 00:14:45 +1100 Received: from GSP-EX02.ds.swin.edu.au ([169.254.2.2]) by gsp-ex03.ds.swin.edu.au ([169.254.3.127]) with mapi id 14.03.0248.002; Wed, 4 Nov 2015 00:14:45 +1100 From: Rasool Al-Saadi To: "freebsd-net@freebsd.org" Subject: Timing issue with Dummynet on high kernel timer interrupt Thread-Topic: Timing issue with Dummynet on high kernel timer interrupt Thread-Index: AdEWOVE5DoAlkzqPQWaBZ8ln4esVdw== Date: Tue, 3 Nov 2015 13:14:45 +0000 Message-ID: <6545444AE21C2749939E637E56594CEA3C0DCCC4@gsp-ex02.ds.swin.edu.au> Accept-Language: en-AU, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [136.186.126.11] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 13:14:49 -0000 Hello all, While we were implementing new AQMs (Codel, PIE, FQ-Codel and FQ-PIE) in Fr= eeBSD targeting ipfw+Dummynet, we noticed timing issue with Dummynet when w= e increase kernel timer interrupt Hz (changing the value of kern.hz in /boo= t/loader.conf) to 3000 or higher. The issue is that spikes (upward and down= ward) appear in RTT when we use FreeBSD+ipfw+Dummynet as a router with high= Hz value. We did simple experiment (configuration listed below) with six runs and Hz= =3D{1000, 2000, 3000, 3250, 3500, 4000} to demonstrate the issue. I have at= tached two RTT vs Time graphs for our experiments. You can see in these gra= phs everything is smooth when Hz ~ <=3D3000, and the issue starts to appear= when Hz ~>=3D3250. As part of figuring out why this happen, I found that removing C_DIRECT_EXE= C and C_HARDCLOCK flags from callout_reset_sbt() call(in dn_reschedule() fu= nction) causes the issue with high kernel interrupt hz (>3000 in my case) t= o disappear even when kern.hz=3D10000. I did another two experiment (kern.hz=3D10000, with and without the flags m= entioned above) and I logged dn_cfg.curr_time from dummynet_task() (after t= he adjustments code). Then I calculated the difference (delta) between curr= _time[i] and curr_time[i-1] of the log file (the logged dn_cfg.curr_time). = I found that when kern.hz=3D10000 and C_DIRECT_EXEC and C_HARDCLOCK flags a= re set, the delta values vary from 0 to ~600, while when C_DIRECT_EXEC and= C_HARDCLOCK flags are unset, the delta values vary from 1 to ~80. It seems= like the high jumps of delta values when the flags are set causes the issu= e. I have attached a graph that compares delta values for the two experimen= ts. Does anyone have thoughts on what we can test next to narrow down the root-= cause of these unusual timing jumps? Our configurations: Router PC: Intel(R) Core(TM)2 Duo @ 2.33GHz processer, 4GiB RAM. FreeBSD 10.1 (also I tried FreeBSD11-CURRENT-r289420). =20 ipfw+Dummynet: 10mbit traffic shaping , emulated delay 30ms (RTT=3D60ms), q= ueue size is 100 slices, droptail Hosts: Linux 3.17 Traffic: One TCP flow, iperf, TCP NewReno NICs: Intel PRO/1000 GT Testbed experiments under the control of TEACUP (http://caia.swin.edu.au/to= ols/teacup/) Regards, Rasool Al-Saadi