Date: Wed, 15 Jun 2022 12:03:48 -0700 From: Aleksandr Miroslav <alexmiroslav@gmail.com> To: freebsd-questions@freebsd.org Subject: Dokuwiki/Apache/spiped networking problems (sockets stuck in CLOSED/FIN_WAIT_2) Message-ID: <CACcSE1wW%2BSAb=RSexemMzLvniTxSYges2gz_xLyoG827QxDhEA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[All hosts and jails involved below are running FreeBSD 13.0. The Apache
and spiped versions are the latest ones from the ports tree. PHP was
downgraded as mentioned below.]
First some background:
- I have a Dokuwiki site (dokuwiki.example.org) running on my FreeBSD
box (aka foo1.example.org).
- it's been running fine for many years
- earlier in the year, the Dokuwiki port was marked as broken, because
they haven't upgraded the code to PHP 8
- I had some other code on foo1 dependent on PHP 8, so I can't use
Dokuwiki with PHP 7 on machine "foo1"
- while the port remains broken, I thought to spin up a jail, and
install an older version (using portdowngrade) of Dokuwiki and PHP 7
to get my Dokuwiki site back up
- foo1 is kinda overloaded, but I have another machine, foo2 that is
beefy, so I could use it for this purpose
- aside: Now, while I *could* update the DNS and SSL for
dokuwiki.example.org to point to foo2, I figured since this is a
temporary thing (and also because I try not to put public services
on foo2), I did the following...
Steps:
- I setup a jail on foo2, and in it I run the downgraded version of
PHP 7, Dokuwiki, and my site
- I leave the DNS and TLS certs for my site on foo1, as it has always
been
- on foo1 I setup Apache to reverse proxy dokuwiki.example.org to
127.0.0.1:8080
- on foo1 I set up spiped to forward anything from 127.0.0.1:8080 to
foo2.example.org:123 (completely arbitrary, I could have used any
port number)
- on foo2, I setup spiped to listen on port 123 and forward
connections to the jail web host
- also, in foo2's pf, I allow traffic ONLY from foo1 into foo2:123, no
one from the public internet can hit foo2 in any way
- so when someone hits dokuwiki.example.org with an http request, it
goes to Apache on foo1 and gets sent to 127.0.0.1:8080
- spiped on foo1 forwards all traffic on 127.0.0.1:8080 through the
public internet to foo2:123
- spiped on foo2 decrypts the traffic on port 123 and sends it to the
Apache running in the jail on port 80
And everything works fine. I can hit https://dokuwiki.example.org and
from a browser/bookmark point of view, everything works as before with
no changes.
Now, the above is kinda over-engineering and rube Goldberg-ish, I
honestly set it up this way to try out the Bastille jail manager and
spiped and Apache's reverse proxy capabilities, but with the exception of
one problem (described below), everything does work.
(I could point https://dokuwiki.example.org to foo2 directly, but I would
like to avoid any public facing services on that machine, as far as the
world is concerned, foo1 is my only machine talking to them.)
But here's the problem:
- for whatever reason, the sockets opened up by spiped on both foo1
and foo2 seem to stick around forever. Either in CLOSED or
FIN_WAIT_2 mode.
- when spiped reaches the max connections allowed (the default is 100)
after a few hours, I start getting reverse proxy errors from Apache
when I hit the site because spiped can't spawn any more connections
- this happens slowly, over the course of a few hours
- when this limit is reached, I have to "kill -9 spiped" on both boxes,
and restart them again
Things I have tried:
- I extended spiped's max connections to 200, 300 and larger; this
simply makes things run longer until spiped's max connection is
reached, but the problem persists
- some web searching reveals this might be a problem with timeouts
between Apache's reverse proxy (aka ProxyPass) and spiped, I
currently have it setup like this:
ProxyPass "/" "<http://127.0.0.1:8080>/" connectiontimeout=300
timeout=300
but the problem persists and those timeout values are random guesses
on my part.
- I have no idea why this is happening. I suspect this is not a bug in
the OS or Apache, or spiped, but rather some sysctl or Apache
configuration that I need to tweak.
I know I can setup Apache on foo1 to speak directly to foo2, and forego
spiped altogether, but I would like to see if I can fix the above setup.
Any suggestions appreciated.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACcSE1wW%2BSAb=RSexemMzLvniTxSYges2gz_xLyoG827QxDhEA>