From owner-freebsd-questions@FreeBSD.ORG Thu Nov 11 05:08:02 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE02116A4CE for ; Thu, 11 Nov 2004 05:08:02 +0000 (GMT) Received: from thewordnerd.thewordnerd.info (dsl093-216-223.aus1.dsl.speakeasy.net [66.93.216.223]) by mx1.FreeBSD.org (Postfix) with ESMTP id 58FED43D7C for ; Thu, 11 Nov 2004 05:08:02 +0000 (GMT) (envelope-from nolan@thewordnerd.info) Received: from nolan by thewordnerd.thewordnerd.info with local (Exim 3.36 #1 (Debian)) id 1CRzre-0000oU-00 for ; Wed, 10 Nov 2004 15:18:50 -0600 To: freebsd-questions@freebsd.org User-Agent: SEMI/1.14.6 (Maruoka) FLIM/1.14.6 (Marutamachi) APEL/10.6 Emacs/21.3 (i386-pc-linux-gnu) (with unibyte mode) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Message-Id: From: "Nolan J. Darilek" Date: Wed, 10 Nov 2004 15:18:50 -0600 Subject: Writes failing from jails under pf X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2004 05:08:02 -0000 I posted something similar to the list a few days ago, but it doesn't appear to have made it into the archives, so I'm trying again after joining. My network consists of one real server, acting as a firewall/router, hosting two separate jails, each of which is a complete system. One runs various network services, while the other is a shell/application server for users. This system worked well under ipfw/natd, but I couldn't accomplish some things under that setup, so I switched to pf. Now, while almost everything else seems to work fine, writes of greater than 46 bytes from jailed systems fail. I first noticed this when esdplay processes on the shell box, which used to play audio clips on my desktop system quite nicely, failed (even though I could successfully telnet from the jail to the esd process on the desktop. Running strace on the esdplay process revealed the following edited log: socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0 connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 . . . write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16 write(4, "\0\0\0\0", 4) = 4 read(4, "\0\0\0\0", 4) = 4 . . . write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [0], 4) = 0 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [0], 4) = 0 . . . write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = -1 EPERM (Operation not permitted) close(4) = 0 . . . Again, this worked fine under ipfw/natd, and the switch to pf is the only change I recall making. Would a firewall trigger an EPERM, though? Seems unlikely . . . but what else might? I tried running the esdplay test as root, but with the same results. I next wrote a simple client and server. The server sits on the specified IP and waits for a single string, then terminates. The client connects to the given IP, sends a user-specified amount of data, then dies. Running the server on either jailed system and writing from the non-jailed workstation works fine. Running the server on the workstation and having the client write up to 46 bytes works fine . . . but the EPERM errors begin surfacing at 47 bytes, and I have no clue why. Again, nothing appears in the firewall logs. I'm running 5.3 (though this problem existed under 5.2.1 as well) and when I enable raw socket access in the jail, packets seem to ignore my router entirely; outbound packets hit my DSL provider's router, and packets bound to any network boxen travel directly, as if the connection were bridged. The firewall seems blameless . . . but, again, it's the only change I recall making. My ruleset is at http://www.thewordnerd.info/pf.conf if that helps at all. Does anyone have any suggestions as to what I might try next? Is there any additional information I can offer? Thanks.