Date: Mon, 6 Feb 2012 23:32:02 +0100 From: "K. Macy" <kmacy@freebsd.org> To: Julian Wissmann <julianwissmann@gmail.com> Cc: freebsd-performance@freebsd.org Subject: Re: Tor on FreeBSD Performance issues Message-ID: <CAHM0Q_P_dB7P9ZuyNymh_LuSf6-1qhSVxECFh3V3qrgZWjBfNA@mail.gmail.com> In-Reply-To: <7FE9C466-2459-4C0E-949A-7C0843B04BB4@gmail.com> References: <7FE9C466-2459-4C0E-949A-7C0843B04BB4@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 6, 2012 at 9:38 PM, Julian Wissmann <julianwissmann@gmail.com> wrote: > Hi, > > I'm an admin for a non-profit that runs Tor exit nodes, most of them on L= inux currently, but due to problems on our high bandwidth nodes, we decided= to migrate one of them to FreeBSD to do some testing. > I've been using FreeBSD for quite some years now, longer than Linux, so I= figured this would probably be a no-brainer, but turns out, that it isn't. > > On FreeBSD I currently manage to push 150-200Mbits with some heavy tuning= applied already, on Linux it is roughly 500Mbits. Tor tends to keep open a lot of connections but that is really very little bandwidth and you really shouldn't need to have polling on or have hz set that high. What does CPU utilization look like on these systems? I don't know if it is part of the problem, but TSO really isn't very useful with large numbers of connections, it is better suited to helping a single connection saturate an interface - could please you turn that off. Thanks > Therefor I'm wondering if I'm really running into some limitations here o= r if I'm actually doing something wrong. > > ifconfig bge0 > bge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 > =A0 =A0 =A0 =A0options=3Dc01db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLL= ING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> > > As you can see I've compiled polling and currently I'm running on kern.hz= =3D16000 as that's given me the best performance, so far. > > This is my netstat output for tcp on ipv4: > > netstat -s > tcp: > =A0 =A0 =A0 =A0265086752 packets sent > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A084255022 data packets (155791599035 bytes) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A02244601 data packets (2698410010 bytes) re= transmitted > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A073011 data packets unnecessarily retransmi= tted > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0684 resends initiated by MTU discovery > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0153151729 ack-only packets (0 delayed) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 URG only packets > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A019864 window probe packets > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A016854982 window update packets > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08551532 control packets > =A0 =A0 =A0 =A0236720260 packets received > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A086600847 acks (for 155842386836 bytes) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A05062568 duplicate acks > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A076267 acks for unsent data > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0138258588 packets (150041170335 bytes) rec= eived in-sequence > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01502804 completely duplicate packets (5622= 43206 bytes) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A044193 old duplicate packets > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A091821 packets with some dup. data (2369339= 1 bytes duped) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A06598536 out-of-order packets (7168950882 b= ytes) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A032074 packets (20848106 bytes) of data aft= er window > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A07806 window probes > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01457705 window update packets > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0159860 packets received after close > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A05219 discarded for bad checksums > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03 discarded for bad header offset fields > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 discarded because packet too short > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01468 discarded due to memory problems > =A0 =A0 =A0 =A05665849 connection requests > =A0 =A0 =A0 =A0694088 connection accepts > =A0 =A0 =A0 =A00 bad connection attempts > =A0 =A0 =A0 =A0129 listen queue overflows > =A0 =A0 =A0 =A09308 ignored RSTs in the windows > =A0 =A0 =A0 =A03289250 connections established (including accepts) > =A0 =A0 =A0 =A06334698 connections closed (including 449721 drops) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0916420 connections updated cached RTT on c= lose > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0923786 connections updated cached RTT vari= ance on close > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0273103 connections updated cached ssthresh= on close > =A0 =A0 =A0 =A0354989 embryonic connections dropped > =A0 =A0 =A0 =A081015541 segments updated rtt (of 56772442 attempts) > =A0 =A0 =A0 =A09127304 retransmit timeouts > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A019875 connections dropped by rexmit timeou= t > =A0 =A0 =A0 =A021274 persist timeouts > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0215 connections dropped by persist timeout > =A0 =A0 =A0 =A04541 Connections (fin_wait_2) dropped because of timeout > =A0 =A0 =A0 =A010657 keepalive timeouts > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 keepalive probes sent > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A010657 connections dropped by keepalive > =A0 =A0 =A0 =A039113689 correct ACK header predictions > =A0 =A0 =A0 =A0121244352 correct data packet header predictions > =A0 =A0 =A0 =A0698461 syncache entries added > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A021576 retransmitted > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A014546 dupsyn > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 dropped > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0694088 completed > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 bucket overflow > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 cache overflow > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01046 reset > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03338 stale > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0135 aborted > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 badack > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A06 unreach > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 zone failures > =A0 =A0 =A0 =A0698461 cookies sent > =A0 =A0 =A0 =A0232 cookies received > =A0 =A0 =A0 =A0173007 hostcache entries added > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 bucket overflow > =A0 =A0 =A0 =A0285122 SACK recovery episodes > =A0 =A0 =A0 =A0584154 segment rexmits in SACK recovery episodes > =A0 =A0 =A0 =A0730380132 byte rexmits in SACK recovery episodes > =A0 =A0 =A0 =A03053612 SACK options (SACK blocks) received > =A0 =A0 =A0 =A06689960 SACK options (SACK blocks) sent > =A0 =A0 =A0 =A00 SACK scoreboard overflow > =A0 =A0 =A0 =A08236 packets with ECN CE bit set > =A0 =A0 =A0 =A026367032 packets with ECN ECT(0) bit set > =A0 =A0 =A0 =A056 packets with ECN ECT(1) bit set > =A0 =A0 =A0 =A0312220 successful ECN handshakes > =A0 =A0 =A0 =A034178 times ECN reduced the congestion window > > My sysctls are roughly equivalent to these: http://serverfault.com/questi= ons/64356/freebsd-performance-tuning-sysctls-loader-conf-kernel > > Any hints? > Do I need to provide more info? > > Julian --=20 =A0 =A0=93The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don=92t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won=92t take measure of their own strength, for fear of antagonizing their own weakness. Those who don=92t like to make waves=97or enemies. =A0 =A0Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It=92s the reductionist approach to life: if you keep it small, you=92ll keep it under control. If you don=92t make any noise, the bogeyman won=92t find you. =A0 =A0But it=92s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! >From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. =A0 =A0I choose my own way to burn.=94 =A0 =A0Sophie Scholl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_P_dB7P9ZuyNymh_LuSf6-1qhSVxECFh3V3qrgZWjBfNA>