Date: Wed, 25 Jul 2007 13:49:05 +0200 From: Gergely CZUCZY <phoemix@harmless.hu> To: freebsd-pf@freebsd.org Subject: Re: connection refused on heavy usage Message-ID: <20070725114905.GA27660@harmless.hu> In-Reply-To: <20070725113824.GB26977@harmless.hu> References: <20070725113824.GB26977@harmless.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
--mYCpIKhGyMATD0i+
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Just catched something in the syslog:
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53051 10.0.0.251=
:53051 10.0.0.1:80 [lo=3D2626749835 high=3D2626816443 win=3D33304 modulator=
=3D0 wscale=3D1] [lo=3D2986152604 high=3D2986219211 win=3D33304 modulator=
=3D0 wscale=3D1] 9:9 S seq=3D2736349746 ack=3D2986152604 len=3D0 ackskew=3D=
0 pkts=3D23:26 dir=3Dout,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1 | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not perm=
itted
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53052 10.0.0.251=
:53052 10.0.0.1:80 [lo=3D368073977 high=3D368140585 win=3D33304 modulator=
=3D0 wscale=3D1] [lo=3D530543602 high=3D530610209 win=3D33304 modulator=3D0=
wscale=3D1] 9:9 S seq=3D477665814 ack=3D530543602 len=3D0 ackskew=3D0 pkts=
=3D24:26 dir=3Dout,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1 | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not perm=
itted
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53053 10.0.0.251=
:53053 10.0.0.1:80 [lo=3D3069306042 high=3D3069372650 win=3D33304 modulator=
=3D0 wscale=3D1] [lo=3D1682247531 high=3D1682314138 win=3D33304 modulator=
=3D0 wscale=3D1] 9:9 S seq=3D3178900053 ack=3D1682247531 len=3D0 ackskew=3D=
0 pkts=3D23:26 dir=3Dout,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1 | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not perm=
itted
Jul 25 13:46:14 lvs1 last message repeated 40 times
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" =
=66rom 192.168.4.21
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" =
=66rom 192.168.4.21
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not perm=
itted
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" =
=66rom 192.168.4.21
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not perm=
itted
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" =
=66rom 192.168.4.21
Jul 25 13:46:17 lvs1 last message repeated 681 times
Jul 25 13:46:17 lvs1 pound: BackEnd 10.0.0.1:80 resurrect
As i see these tend to happen from time to time.
On Wed, Jul 25, 2007 at 01:38:24PM +0200, Gergely CZUCZY wrote:
> Good morning,
>=20
> I've got a problem that disappeared by disabling pf.
>=20
> From the beginning. I'm testing an http reverse proxy[pound], at
> the moment. I've got two gateways in a pfsync+carp+pound configuration
> and two web backends. I'm doing performance testing on the proxy with
> apache benchmarks, this involves hordes of simultaneous connections in
> and out. connections are recieved by pound on the gateway and it
> connects to a given web-backend to make the actual request.
>=20
> The problem is that, periodically it's unable to connect to some
> backends, or just to one of them and renders it DEAD.
> When this happens there's a "connect: operation not permitted" message
> in the syslog. Nor I'm able to connect to the backends directly with
> elinks from the gateway, it also says "operation not permitted". After
> waiting a few seconds it works again.
>=20
> So, the proxy can accept client's connections but it's unable to
> connect forward to the actual web-backends. When I disabled pf with
> pfctl -d these symptons stopped immedietly.
>=20
> I tried playing around with different tcp timeout values, but that
> failed to help.
>=20
> My pf.conf is the following:
> --- chop with axe here ---
> if_ext=3D"em0"
> if_vvv=3D"fxp0"
> if_sync=3D"em1"
>=20
> ip_pub=3D"192.168.4.55"
> ip_vvv=3D"10.0.0.254"
>=20
> ip_vvv1=3D"10.0.0.1"
> ip_vvv2=3D"10.0.0.2"
> ip_vvv3=3D"10.0.0.3"
>=20
> table <vvv> {$ip_vvv1, $ip_vvv2, $ip_vvv3}
>=20
> # Options: tune the behavior of pf, default values are given.
> #set timeout { interval 5, frag 30 }
> #set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
> set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
> #set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
> #set timeout { icmp.first 20, icmp.error 10 }
> #set timeout { other.first 60, other.single 30, other.multiple 60 }
> set timeout { adaptive.start 70000, adaptive.end 120000 }
> set limit { states 100000, frags 2000 }
> #set loginterface none
> set block-policy return
> set require-order yes
> set fingerprints "/etc/pf.os"
> set debug misc
>=20
> set skip on lo0
>=20
> #scrub in all
>=20
> rdr on $if_ext proto tcp from any to $ip_pub port 10001 -> $ip_vvv1 port =
22
> rdr on $if_ext proto tcp from any to $ip_pub port 10002 -> $ip_vvv2 port =
22
> rdr on $if_ext proto tcp from any to $ip_pub port 10003 -> $ip_vvv3 port =
22
>=20
> block in log on $if_ext all
>=20
> pass in quick on {$if_ext,$if_vvv} proto vrrp
> pass out quick on {$if_ext,$if_vvv} proto vrrp
>=20
> pass out quick on $if_ext proto udp from any to 192.168.4.200 port 123 ke=
ep state
>=20
> pass in quick on $if_ext proto tcp from any to $if_ext:0 port 22 flags S/=
SA synproxy state (no-sync)
> pass in quick on $if_ext proto tcp from any to $ip_pub port 80 flags S/SA=
modulate state (no-sync) label "2"
>=20
> pass out quick on $if_ext proto udp from $if_ext:0 to port 53 keep state =
(no-sync)
> pass out quick on $if_ext proto udp from any to port 53 keep state
>=20
> pass out quick on $if_ext proto tcp from $if_ext:0 to port 80 flags S/SA =
keep state (no-sync)
> pass out quick on $if_ext proto tcp from any to port 80 flags S/SA keep s=
tate
>=20
> pass in quick on $if_ext proto tcp from any to <vvv> port 22 flags S/SA s=
ynproxy state
>=20
> pass out quick on $if_vvv proto tcp from ($if_vvv) to <vvv> port 80 flags=
S/SA keep state (no-sync)
> --- chop with axe here ---
>=20
> FreeBSD lvs1.in.publishing.hu 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #1: T=
ue Jul 24 08:07:07 UTC 2007 toor@pointyhat.office:/usr/obj/usr/src/sys/=
LVS i386
>=20
> I'm played with the followings without any success:
> set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
> set timeout { adaptive.start 70000, adaptive.end 120000 }
>=20
> What can cause this issue?
> How could this be fixed?
>=20
> [pound] http://www.apsis.ch/pound/
>=20
> Sincerely,
>=20
> Gergely Czuczy
> mailto: gergely.czuczy@harmless.hu
>=20
> --=20
> Weenies test. Geniuses solve problems that arise.
Sincerely,
Gergely Czuczy
mailto: gergely.czuczy@harmless.hu
--=20
Weenies test. Geniuses solve problems that arise.
--mYCpIKhGyMATD0i+
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)
owHtWc2PHEcVN4m4tEAoFwQX9LReyQmej/6Yrx0YO2vv2jFKSMRuYjlWFNV0V890
tqerXdU944njIxIHDpArQhw5IUBwRwKUG1wQFy78AdxyQVyA3+uP2ZndteO1LSEB
611P96uq937vs97b/cEXX7zwwkt//MWv717+/sc/+tzPvvqX8ddneZYlk+ZM6HmU
NB3bdppuv9ezm14z8IXXs0NHDDqhPfBu/HTwy+sqyWSSNQ+XqRxSJu9n7TQWUfIN
8qdCG5mN8ixsDqx6315kUmWiLFLJkKIkjhK5WjvUIjGh1M39xFdBlEyGdC9XmQya
qY6STIxjaVnfyk1Gvsj8qQzIqJnMptgJVpRNJZmlidVkiF0xuV1yvGGnN3Q6FM+N
Q0dSJzIeUhoO6druHplMZAB9eP0tcuwW/3O7zrDr2V1njTCyNknOcGDT3ViNvD23
5/b6nZ2B16VpNJlWlIHT63Q8WkQJCJ7n2R2aqSCPRab0yALNpoXxRSzx6LxXs9oZ
9Jyu28PmmhUorrPjOs4Ts9oZ7tABGXmPj/e9ntfZ6Xd6JPyjTQmxTIqzWDBHcoHn
kWVTepQZ3ucN3R4FkcazyrNGuAg+25wHbEoKRRTnWhL71qHy6yPqnn08VXkSDGkM
EDIJNqzrqySRfjakN1OpBccKJSojvMxGVpQhIp7Vwe5pB7tnONjrDey+t9Pv104B
wenY3UH3nO5lAR3kjlszAqHn2K698whGYDOyHuncTh8Z2QWUyrfH7E+7tnIsI3M7
/wOu9U671jvLtXZvBz92Z+WSgtJ3e137nM51egPX7fS7nlOzYorndBxv8FS56zn9
wY5tM/DSv2sS/ltzNxao6zNpjJhI0jKVABVQx6YsAvWxMBJVIGkylK2b+4fUTqdp
lISqGS99ZVrTbBbTa4eHb7Wdlr1FMH6vp9WMUFxbsGyr03Kd/6yA52nI/2NeF9B/
XHShmJ8ZXv0NVNcAaf+EolqaXGuoalm7hiLkreT+w+B/Bp8pmoo0lQmFjI9lMI0/
W5b1ZkK3ZdCgUmiDXNvuk8jIdobeYOh23nqDLuNqsBt0U+qJjJd0/d23r797hxYa
3dDQukI3lQpQTXSC1qdhXRm5Noi3Ls0lTWBuQalWaJYgeAq2QWQYi9DQeLwsXscx
90xp2KrP3ihgooEay0mUMNsW+IEkTcZbRULTLEuh91yiq2MB95d3CwO91wB2sODT
M/RjSdY6hpItFE1g64VYGu7RAC00y8S/7AudXi7Oc9CE0SQv4wWMBBsQ5xZyXAeZ
KdEEqoAtdaj0TCS+XOFTZftXwELJzabMJxV+oVHiT9HNHpkG9kQMY67iuTQ0VTrA
hwrJRLM8zkQiVW7qIAYY3lsBQjVtbazAnLCGH8EghVlLXSoclcrFyYiNU500HAWC
JtEcoQH9mnUSgTwTR0UMobJnuYjB/F4O7VYuOiz1K/wKLdi1DbZFhFYZ1wiiJMou
GcoT7pOZYSWTH7lPBovamg1Smj7gRhprKpFsAkieFXg1dsDH4EZ7+7t7kE+3pzIp
bVdGNUuXWkKaoK1VzqtTOV9k/Fadd2C00aW36NuAwX49A3ERixVcvtBAjleelQjf
I1Pl1rG5G4xZxAb6crhtPQpQi3bDTGowWoioDG8K5QJJDABBoflCIV5ITDDJrBxw
oBprQeYjJYTvyxTTSBwh6i9tRs44z0445DgMCPG7EDqoFa08vhYQCPjC6LeqdEWM
pWGtfhrCGNQMqopjlrM0Y5EmU/AOIm42k0EkYbAV9luU6YiZxGJZKKyLcGWGkBBi
6IIGlPlpUaMQ7DQXMcKvUejBsQYm3CLIsrjJOF3xfmPJlYSzuIxL9BIqjtWCBzgs
N5tNzIIqLYWJ+5I4dpjMARG+j5ER3cqWnNlbJWE+nzMhvJ/WFK4Y5R5nq5Yape+n
+Zipx9W/290qVyoWq0aws34Mi87aqnN8xl0/dEz21sjeilMxj9I3sX6FHmxXfBtU
PbmrJ+9hfeIirsUiODAo54ms6u1UzCPkAVIwDRsUyFCgFFXmL6pMUS04DS9inF75
5wGSCUGMfYQrJNRiQp5ND09vg1NbYaSR7I6LK4VfkRhc4nGgfEeZ4SvB8Eg96HXs
ks9pNn6M0R3nduzqYBglnEIrRryB79UzgeRBDaSH7fzGzGBDr3rlGhylIJx9PvJn
NQNWpHiVWsN2zpn7FRepNYnl+5rMknBS6iYTEQj4bC5bMJHOqI+uHCdX1KIBcpm4
OhtHKDM4WUwoBtjKI+whw1f9GlYUwcKJocBVlqAQVyzGsUIXlKo48peox1muk2qF
LwUUwybfXJqW6FxKOhwxkbr4NQkKX1tmfhs5qcxWtR7IcT6hWWT8OhiZao6ilK+s
WNmrGDW+zsfFVR3HNVEHmrdtl9nKBZALF4pFUYFFsuSasF1mJC5C2Im1dqh5pU4C
pySPLNd9an7uGj/3OfDz1vh56/xKpQsnsCHgpXXua3ZJhSm6GvgEW7HnQbWpsV3W
sYcVlrnWab2f4+oJD6xJ2Ti1qShSZ0PRtW4YwVaq63oY+UaWTMuwfKQCn2XCYnVY
cXVdCmOO6oP2yDrYxUWUlHdjIYNeTlSTK/crTyNozVdot2sxB7sjq56i5UkxaPLH
MqYtd+vcljtWDNRCaJcNVpsLcXFKmXM45DTDJwW4sstpgBtWAcA1tOfBetLyZ/Au
OY+sZwic8pY8GTQcMiNrI2gebxhwOSng5Yr+ygkxtQqIlzUlTpnncc1JPRlJee1g
r5gIW+gGEZZ8U/KENM2p13Kb39l/fX/3YL+Z9labT5AvOkM6HFm5LEe+DtmDod3H
N719eL0c//grU0q/mmLGyZZouFoqDCNfDtu50W01/qD4NNpvo3tGvr3+zgFR5A16
x+PfrGjvZNXWbTRipqCxQdkjJkfjaszweVzyz3Bdlrhv84zK3bQvcp6fixHNmFxe
xeJraoGmOY+Dkj6GStF9GVytD1cjaDGcDtvtxWLREqmJTMuftoul9qp7jzAvagzT
q3l5NVx/mPsfLkGYob/N1JAm5ULLLxZenQo9i2Eu+Ls+2mxW2CUaKVzyPIa2wDCJ
oAIacZ4v62GtHNXQyUWGh3/LWkNincDwBAisUvb5JX/v6oufv8B/sqn/3PPSC79r
X/jxV/Y/Xvzm009+fuedj36l//y3L/z1t3+fXfjJnX9e3f7HjS/98E9/+G7ufm37
X7//8oNP/g0=
=yJXf
-----END PGP SIGNATURE-----
--mYCpIKhGyMATD0i+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070725114905.GA27660>