Date: Tue, 3 Nov 2015 13:14:45 +0000 From: Rasool Al-Saadi <ralsaadi@swin.edu.au> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Timing issue with Dummynet on high kernel timer interrupt Message-ID: <6545444AE21C2749939E637E56594CEA3C0DCCC4@gsp-ex02.ds.swin.edu.au>
next in thread | raw e-mail | index | archive | help
Hello all,
While we were implementing new AQMs (Codel, PIE, FQ-Codel and FQ-PIE) in Fr=
eeBSD targeting ipfw+Dummynet, we noticed timing issue with Dummynet when w=
e increase kernel timer interrupt Hz (changing the value of kern.hz in /boo=
t/loader.conf) to 3000 or higher. The issue is that spikes (upward and down=
ward) appear in RTT when we use FreeBSD+ipfw+Dummynet as a router with high=
Hz value.
We did simple experiment (configuration listed below) with six runs and Hz=
=3D{1000, 2000, 3000, 3250, 3500, 4000} to demonstrate the issue. I have at=
tached two RTT vs Time graphs for our experiments. You can see in these gra=
phs everything is smooth when Hz ~ <=3D3000, and the issue starts to appear=
when Hz ~>=3D3250.
As part of figuring out why this happen, I found that removing C_DIRECT_EXE=
C and C_HARDCLOCK flags from callout_reset_sbt() call(in dn_reschedule() fu=
nction) causes the issue with high kernel interrupt hz (>3000 in my case) t=
o disappear even when kern.hz=3D10000.
I did another two experiment (kern.hz=3D10000, with and without the flags m=
entioned above) and I logged dn_cfg.curr_time from dummynet_task() (after t=
he adjustments code). Then I calculated the difference (delta) between curr=
_time[i] and curr_time[i-1] of the log file (the logged dn_cfg.curr_time). =
I found that when kern.hz=3D10000 and C_DIRECT_EXEC and C_HARDCLOCK flags a=
re set, the delta values vary from 0 to ~600, while when C_DIRECT_EXEC and=
C_HARDCLOCK flags are unset, the delta values vary from 1 to ~80. It seems=
like the high jumps of delta values when the flags are set causes the issu=
e. I have attached a graph that compares delta values for the two experimen=
ts.
Does anyone have thoughts on what we can test next to narrow down the root-=
cause of these unusual timing jumps?
Our configurations:
Router PC: Intel(R) Core(TM)2 Duo @ 2.33GHz processer, 4GiB RAM.
FreeBSD 10.1 (also I tried FreeBSD11-CURRENT-r289420). =20
ipfw+Dummynet: 10mbit traffic shaping , emulated delay 30ms (RTT=3D60ms), q=
ueue size is 100 slices, droptail
Hosts: Linux 3.17
Traffic: One TCP flow, iperf, TCP NewReno
NICs: Intel PRO/1000 GT
Testbed experiments under the control of TEACUP (http://caia.swin.edu.au/to=
ols/teacup/)
Regards,
Rasool Al-Saadi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6545444AE21C2749939E637E56594CEA3C0DCCC4>