Date: Wed, 10 Nov 2004 15:18:50 -0600 From: "Nolan J. Darilek" <nolan@thewordnerd.info> To: freebsd-questions@freebsd.org Subject: Writes failing from jails under pf Message-ID: <E1CRzre-0000oU-00@thewordnerd.thewordnerd.info>
next in thread | raw e-mail | index | archive | help
I posted something similar to the list a few days ago, but it doesn't
appear to have made it into the archives, so I'm trying again after
joining.
My network consists of one real server, acting as a firewall/router,
hosting two separate jails, each of which is a complete system. One
runs various network services, while the other is a shell/application
server for users. This system worked well under ipfw/natd, but I
couldn't accomplish some things under that setup, so I switched to
pf. Now, while almost everything else seems to work fine, writes of
greater than 46 bytes from jailed systems fail. I first noticed this
when esdplay processes on the shell box, which used to play audio
clips on my desktop system quite nicely, failed (even though I could
successfully telnet from the jail to the esd process on the
desktop. Running strace on the esdplay process revealed the following
edited log:
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16)
= 0
. . .
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
write(4, "\0\0\0\0", 4) = 4
read(4, "\0\0\0\0", 4) = 4
. . .
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) =
128
setsockopt(4, SOL_SOCKET, SO_SNDBUF, [0], 4) = 0
setsockopt(4, SOL_SOCKET, SO_RCVBUF, [0], 4) = 0
. . .
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) =
-1 EPERM (Operation not permitted)
close(4) = 0
. . .
Again, this worked fine under ipfw/natd, and the switch to pf is the
only change I recall making. Would a firewall trigger an EPERM,
though? Seems unlikely . . . but what else might? I tried running the
esdplay test as root, but with the same results.
I next wrote a simple client and server. The server sits on the
specified IP and waits for a single string, then terminates. The
client connects to the given IP, sends a user-specified amount of
data, then dies. Running the server on either jailed system and
writing from the non-jailed workstation works fine. Running the server
on the workstation and having the client write up to 46 bytes works
fine . . . but the EPERM errors begin surfacing at 47 bytes, and I
have no clue why.
Again, nothing appears in the firewall logs. I'm running 5.3 (though
this problem existed under 5.2.1 as well) and when I enable raw socket
access in the jail, packets seem to ignore my router entirely;
outbound packets hit my DSL provider's router, and packets bound to
any network boxen travel directly, as if the connection were
bridged. The firewall seems blameless . . . but, again, it's the only
change I recall making.
My ruleset is at http://www.thewordnerd.info/pf.conf if that helps at
all. Does anyone have any suggestions as to what I might try next? Is
there any additional information I can offer?
Thanks.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1CRzre-0000oU-00>