From owner-freebsd-pf@FreeBSD.ORG Thu Jul 24 11:01:02 2008 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08A361065671 for ; Thu, 24 Jul 2008 11:01:02 +0000 (UTC) (envelope-from rj@shadowbots.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.190]) by mx1.freebsd.org (Postfix) with ESMTP id DB1498FC14 for ; Thu, 24 Jul 2008 11:01:00 +0000 (UTC) (envelope-from rj@shadowbots.com) Received: by fk-out-0910.google.com with SMTP id k31so2070220fkk.11 for ; Thu, 24 Jul 2008 04:00:59 -0700 (PDT) Received: by 10.180.208.5 with SMTP id f5mr38263bkg.42.1216895696605; Thu, 24 Jul 2008 03:34:56 -0700 (PDT) Received: by 10.103.192.5 with HTTP; Thu, 24 Jul 2008 03:34:56 -0700 (PDT) Message-ID: <9072a4470807240334l7829fddbudbeea941fe1b77ad@mail.gmail.com> Date: Thu, 24 Jul 2008 06:34:56 -0400 From: "Robert Jameson" Sender: rj@shadowbots.com To: freebsd-pf@freebsd.org In-Reply-To: <9072a4470807240321y59f827fdn287011c0336ae866@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_2909_22637454.1216895696594" References: <9072a4470807232259x603f46k49474f5eb309d0fa@mail.gmail.com> <20080724074919.GA36163@eos.sc1.parodius.com> <9072a4470807240255v4d3f8e72gf8bfb39999b2dcbd@mail.gmail.com> <9072a4470807240321y59f827fdn287011c0336ae866@mail.gmail.com> X-Google-Sender-Auth: 0966e4f9ed8c691f X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: network problems 7.0-p3: sendto: Operation not permitted X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2008 11:01:02 -0000 ------=_Part_2909_22637454.1216895696594 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello everyone, I'm not sure about how this works, I was told to share this information with this group because of a possible issue with PF My rules are in place @ http://rj.dawnshosting.com/fbsd_ml/pf.conf If anyone has a chance and can tell me what is wrong about them, it would mean alot to me. My configuration worked fine before the update to 7.0-P3 like i said, but if we fix the rules then we can begin the process of elimanation. Thank's so much guys From: *Robert Jameson* Date: Thu, Jul 24, 2008 at 1:59 AM To: freebsd-stable Hello Everyone, Recently I upgraded to freebsd 7.0-p3 from 7.0-p2, once i upgraded i began to have problems with my network, nothing has changed configuration wise, let me show you guy's an example. (12:46 AM):(root@cube)/$ ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=0 ttl=240 time=64.713 ms ^C --- google.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 64.713/64.713/64.713/0.000 ms (12:46 AM):(root@cube)/$ ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=0 ttl=240 time=73.814 ms 64 bytes from 72.14.207.99: icmp_seq=1 ttl=240 time=64.943 ms ^C --- google.com ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 64.943/69.379/73.814/4.435 ms (12:46 AM):(root@cube)/$ ping google.com PING google.com (72.14.207.99): 56 data bytes ping: sendto: Operation not permitted ^C --- google.com ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss (12:46 AM):(root@cube)/$ As you can see above, I issued the ping command (4) times waiting for output and then doing CTRL+C to interrupt the commands quickly and send them again on the 4th try i did not intterupt it and received the operation not permitted. hitting ctrl+c on this error I can type ping again and it will work correctly. I have the same problem with almost every network command, wget, curl, fetch, lynx, ssh, nslookup, host etc. This appears to be an issue with the network. I have attached my rc.conf and sysctl.conf and pf.conf please let me know if any other information is required. Errors from /var/log/console.log: Jul 18 21:10:02 cube kernel: Jul 18 21:10:02 cube named[908]: socket: too many open file descriptors Jul 19 00:30:13 cube kernel: Jul 19 00:30:13 cube named[9748]: socket: too many open file descriptors Jul 19 00:30:54 cube kernel: Jul 19 00:30:14 cube last message repeated 28 times Initially I figured this problem was bind related and since it has been a planned project for the past few months to switch to djbdns, I took the time to switch to djbdns, so bind is no longer running. I was also receiving this in /var/log/messages: Jul 20 22:15:39 cube kernel: Limiting open port RST response from 318 to 200 packets/sec Jul 20 22:15:40 cube kernel: Limiting open port RST response from 624 to 200 packets/sec Jul 20 22:15:42 cube kernel: Limiting open port RST response from 213 to 200 packets/sec Jul 20 22:15:50 cube kernel: Limiting open port RST response from 439 to 200 packets/sec Jul 20 22:15:51 cube kernel: Limiting open port RST response from 673 to 200 packets/sec Jul 20 22:15:52 cube kernel: Limiting open port RST response from 730 to 200 packets/sec Jul 20 22:15:53 cube kernel: Limiting open port RST response from 307 to 200 packets/sec Jul 20 22:16:02 cube kernel: Limiting open port RST response from 435 to 200 packets/sec Jul 20 22:16:03 cube kernel: Limiting open port RST response from 730 to 200 packets/sec Jul 20 22:16:04 cube kernel: Limiting open port RST response from 287 to 200 packets/sec Jul 20 22:16:13 cube kernel: Limiting open port RST response from 519 to 200 packets/sec Jul 20 22:16:14 cube kernel: Limiting open port RST response from 740 to 200 packets/sec Jul 20 22:16:15 cube kernel: Limiting open port RST response from 258 to 200 packets/sec Jul 20 22:16:24 cube kernel: Limiting open port RST response from 407 to 200 packets/sec Jul 20 22:16:25 cube kernel: Limiting open port RST response from 660 to 200 packets/sec After spending some time on Google i came up with: /etc/sysctl.conf net.inet.icmp.icmplim=2000 I know it seems abit high, but i kept adjusting until the error went away. (not really fixing the problem?) If your mail client or the mailing list prevents you from seeing the attached You can view them here: http://rj.dawnshosting.com/fbsd_ml/ PS: While running tcpdump I see this tcpdump -i fxp0 Neither one of these ip's exist on my system is my cable company doing something wrong? 01:47:12.135929 arp who-has 64.253.3.161.dyn-cm-pool73.pool.hargray.net tell 64.253.3.1.dyn-cm-pool73.pool.hargray.net 01:47:12.155931 arp who-has 216.16.218.141.dyn-cm-pool46.pool.hargray.nettell 216.16.218.1.dyn-cm-pool46.pool.hargray.net 01:47:12.196000 arp who-has 181.131.216.67.181.static.hargray.net tell 1.131.216.67.1.static.hargray.net tcpdump -i fxp0 | grep ICMP: Is this an attack? 01:55:41.231722 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37084, length 64 01:55:42.232794 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37085, length 64 01:55:43.285913 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37086, length 64 01:55:44.286340 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37087, length 64 01:55:45.287380 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37088, length 64 01:55:46.345843 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37089, length 64 01:55:47.346685 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37090, length 64 01:55:48.347366 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37091, length 64 01:55:49.348370 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37092, length 64 01:55:50.360130 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37093, length 64 01:55:51.596916 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37094, length 64 01:55:52.597659 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37095, length 64 01:55:53.640120 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37096, length 64 01:55:54.735275 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37097, length 64 01:55:55.735568 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37098, length 64 01:55:56.745012 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37099, length 64 01:55:57.835442 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37100, length 64 01:55:58.920583 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37101, length 64 01:56:00.022747 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP echo request, id 22055, seq 37102, length 64 ---------- From: *Alex Trull* Date: Thu, Jul 24, 2008 at 3:29 AM To: Robert Jameson Robert, The config files you attached were a series of 403 forbidden htmls. The icmp pings (1 per second) do not constitute an attack. It looks like you are genuinely running out of free states or file descriptors. Had you applied any tuning that may have been lost in the upgrade ? How many packets and sessions is this host meant to be handling - and what sort of traffic ? -- Alex > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" ---------- From: *Alex Trull* Date: Thu, Jul 24, 2008 at 3:31 AM To: Robert Jameson Cc: freebsd-stable ---------- From: *Jeremy Chadwick* Date: Thu, Jul 24, 2008 at 3:49 AM To: Robert Jameson Cc: freebsd-stable Let's see if I can figure out the multitude of things you've posted about, since a bunch are unrelated and you appear to be flailing around with your arms in the air. :-) This usually indicates firewall rules on the local machine, although I believe there are some other operations where EPERM can be returned. Can you provide uname -a output? There was a "cable modem compatibility fix" applied to FreeBSD a while ago (a user informed me of such), although I do not know if it applies to you, as I do not know the original symptoms. I believe that fix was also just for TCP. This indicates a completely different/unrelated problem. This indicates a high number of ICMP packets being received. Keep in mind this can also be seen due to TCP connections which are being reset and other such things -- ICMP is at a higher layer than TCP. I don't think there's necessarily anything "wrong" with that number (you show up to 740), but it would be worthwhile investigating what's soliciting that amount of ICMP traffic. Are you seeing this 24x7x365? It's not a big high; FreeBSD's 200 default is too low for any production server, if you ask me. Setting it to 2000 is probably fine. You should discuss your firewalling rules on freebsd-pf, and not here. I believe you may have some mistakes which are inducing said problem. Nope. This is normal behaviour for a cable modem network; they constantly spam layer 2 ARP for *everyone* on the entire cable network segment. Yes, you read that right. At this rate (1 ICMP packet a second), absolutely not. You also don't mention which FQDN/IP is yours; I assume "cube.dawnshosting.com", based on your local hostname in the above. Your machine is sending out an ICMP ping packet to purple.haze.bluntroll.in every 1 second. If you don't know why, you need to investigate why. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ---------- From: *Robert Jameson* Date: Thu, Jul 24, 2008 at 5:55 AM To: Jeremy Chadwick Sorry about that, bit of a information overload, i really am flailing my arms around! Tried running with my firewall disabled/wide problem still occurs FreeBSD cube.dawnshosting.com 7.0-RELEASE-p3 FreeBSD 7.0-RELEASE-p3 #5: Wed Jul 16 21:55:02 EDT 2008 root@cube.dawnshosting.com:/usr/obj/usr/src/sys/CUBE i386 Was the patch applied upstream? if not and its not too much trouble can you point me in the direction of it. Ah, thought they were related, what's causing this :)! Yes its constant. let it me known i also have a 2 network cards in the machne, 1 into my cable modem and nother into a linksys 16port vpn router. the defaultrouter is set to a WAN IP (not 10.192.240.1), not that any of that matters, i dont think? I read a bit about it from the handbook, i think it's a non issue. Might be worth mentioning the only real service change to this machine was an ircd daemon w/ about 500 users. I will send them an e-mail shortly, thanks. ah, ok, nothing to see here, keep moving. Correct, cube.dawnshosting.com is the actual FreeBSD machinr. sorry for the newbish question, off the top of your head how can i see who/what is using this process? > > -- ---------- From: *Robert Jameson* Date: Thu, Jul 24, 2008 at 6:21 AM To: freebsd-stable Still don't know whats going on, im currently sitting here with no firewall between me and the internet (very nervous) seeing if it fixes the problems, as of right this moment, still seeing permission denied errors. I have fixed the 403 errors now. http://rj.dawnshosting.com/fbsd_ml/ now contains sysctl.conf rc.conf pf.conf ------=_Part_2909_22637454.1216895696594 Content-Type: application/octet-stream; name=rc.conf Content-Transfer-Encoding: base64 X-Attachment-Id: f_fj0xi4u90 Content-Disposition: attachment; filename=rc.conf PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pgo8IURPQ1RZUEUgaHRt bCBQVUJMSUMgIi0vL1czQy8vRFREIFhIVE1MIDEuMCBUcmFuc2l0aW9uYWwvL0VOIgogICAgICAg ICAiaHR0cDovL3d3dy53My5vcmcvVFIveGh0bWwxL0RURC94aHRtbDEtdHJhbnNpdGlvbmFsLmR0 ZCI+CjxodG1sIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiB4bWw6bGFuZz0i ZW4iIGxhbmc9ImVuIj4KIDxoZWFkPgogIDx0aXRsZT40MDMgLSBGb3JiaWRkZW48L3RpdGxlPgog PC9oZWFkPgogPGJvZHk+CiAgPGgxPjQwMyAtIEZvcmJpZGRlbjwvaDE+CiA8L2JvZHk+CjwvaHRt bD4K ------=_Part_2909_22637454.1216895696594 Content-Type: application/octet-stream; name=sysctl.conf Content-Transfer-Encoding: base64 X-Attachment-Id: f_fj0xic0h1 Content-Disposition: attachment; filename=sysctl.conf PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pgo8IURPQ1RZUEUgaHRt bCBQVUJMSUMgIi0vL1czQy8vRFREIFhIVE1MIDEuMCBUcmFuc2l0aW9uYWwvL0VOIgogICAgICAg ICAiaHR0cDovL3d3dy53My5vcmcvVFIveGh0bWwxL0RURC94aHRtbDEtdHJhbnNpdGlvbmFsLmR0 ZCI+CjxodG1sIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiB4bWw6bGFuZz0i ZW4iIGxhbmc9ImVuIj4KIDxoZWFkPgogIDx0aXRsZT40MDMgLSBGb3JiaWRkZW48L3RpdGxlPgog PC9oZWFkPgogPGJvZHk+CiAgPGgxPjQwMyAtIEZvcmJpZGRlbjwvaDE+CiA8L2JvZHk+CjwvaHRt bD4K ------=_Part_2909_22637454.1216895696594 Content-Type: application/octet-stream; name=pf.conf Content-Transfer-Encoding: base64 X-Attachment-Id: f_fj0xir4a2 Content-Disposition: attachment; filename=pf.conf PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pgo8IURPQ1RZUEUgaHRt bCBQVUJMSUMgIi0vL1czQy8vRFREIFhIVE1MIDEuMCBUcmFuc2l0aW9uYWwvL0VOIgogICAgICAg ICAiaHR0cDovL3d3dy53My5vcmcvVFIveGh0bWwxL0RURC94aHRtbDEtdHJhbnNpdGlvbmFsLmR0 ZCI+CjxodG1sIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiB4bWw6bGFuZz0i ZW4iIGxhbmc9ImVuIj4KIDxoZWFkPgogIDx0aXRsZT40MDMgLSBGb3JiaWRkZW48L3RpdGxlPgog PC9oZWFkPgogPGJvZHk+CiAgPGgxPjQwMyAtIEZvcmJpZGRlbjwvaDE+CiA8L2JvZHk+CjwvaHRt bD4K ------=_Part_2909_22637454.1216895696594 Content-Type: text/plain; name=signature.asc Content-Transfer-Encoding: base64 X-Attachment-Id: 0.1 Content-Disposition: attachment; filename=signature.asc LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KVmVyc2lvbjogR251UEcgdjEuNC42IChHTlUv TGludXgpCgppRDhEQlFCSWlDL21leTRtNi9lV3hUUVJBdm5RQUo5SDJFZWVPWWNwTnF4dDFETHdH NjlzdFRETEJBQ2dpYnZyCkZJTVA3YWh4VUhqUDE5ZjJMalROYzdrPQo9VVBvQgotLS0tLUVORCBQ R1AgU0lHTkFUVVJFLS0tLS0K ------=_Part_2909_22637454.1216895696594--