Date: Wed, 22 Oct 2014 16:41:52 -0700 From: javocado <javocado@gmail.com> To: freebsd-net@freebsd.org Subject: ipfw command freezes system Message-ID: <CAP1HOmSBcvTcJs4FHGk%2BO2dVmyn=UN6URnre74bQOcyJx8hWog@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
I'm seeing an occasional, recurring problem on my 8.3-RELEASE amd64 systems
where when I enter an ipfw rule, the system becomes locked up.
For example, when entering a command like this:
ipfw add 1 allow ip from x.x.x.x to me
or other times with a command like:
ipfw add xxx skipto ........
the server becomes unreachable via the network. I am however still able to
get to a shell via console where I ran top:
last pid: 25518; load averages: 0.75, 1.12, 0.93 up 4+00:13:02
13:55:34
221 processes: 1 running, 215 sleeping, 5 lock
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 486M Active, 40G Inact, 174G Wired, 24M Cache, 24G Buf, 18G Free
Swap: 32G Total, 32G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
25518 root 1 44 0 9372K 2768K CPU12 12 0:00 0.10% top
79561 xxxx 1 46 0 35284K 13424K *IPFW 12 271:00 0.00% sshd
79564 xxxx 1 45 0 156M 16808K select 0 65:08 0.00% rsync
22311 xxxx 1 44 0 27092K 4676K select 4 13:23 0.00% sshd
13056 xxxx 1 44 0 27092K 4720K select 0 12:13 0.00% sshd
43625 xxxx 1 45 0 29140K 6924K select 5 10:25 0.00% sshd
18925 xxxx 1 44 0 27092K 4640K select 8 10:08 0.00% sshd
14423 xxxx 1 44 0 29140K 6328K select 3 7:59 0.00% sshd
11356 xxxx 1 44 0 12992K 2544K select 2 7:52 0.00%
sftp-server
20619 xxxx 1 44 0 116M 101M select 5 6:25 0.00% rsync
98406 xxxx 1 44 0 29140K 7136K select 2 6:09 0.00% sshd
20617 xxxx 1 44 0 35284K 12992K *IPFW 15 6:03 0.00% sshd
54146 xxxx 1 44 0 27092K 4556K select 3 5:04 0.00% sshd
63688 xxxx 1 44 0 27092K 5728K select 16 4:07 0.00% sshd
20624 xxxx 1 44 0 156M 102M select 0 3:45 0.00% rsync
43629 xxxx 1 44 0 5832K 2084K select 0 3:41 0.00% rsync
Note those "*IPFW" state processes are long running child sshd processes,
not master sshd daemon itself.
I've tried to do an ipfw flush in these situations before but those
commands never return and the system just stays locked up and unreachable.
I was able to issue a reboot from the console, but even that did not
complete:
Oct 21 13:56:53 xxxx reboot: rebooted by root
Oct 21 13:56:54 xxxx syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...
and I had to reset.
Here's the ipfw ruleset in place:
00110 count ip from any to any via igb0 in
00111 count ip from any to any via igb0 out
00210 skipto 410 ip from x.x.x.x/27,x.x.x.x/29,x.x.x.x/24,x.x.x.x/28 to me
00210 skipto 410 ip from me to
x.x.x.x/27,x.x.x.x/29,x.x.x.x/24,x.x.x.x/28
00211 skipto 410 ip from x.x.x.x/24 to me
00211 skipto 410 ip from me to x.x.x.x/24
00212 skipto 410 ip from x.x.x.x/24 to me
00212 skipto 410 ip from me to x.x.x.x/24
00214 skipto 410 ip from x.x.x.x,x.x.x.x to me
00214 skipto 410 ip from me to x.x.x.x,x.x.x.x
00218 skipto 410 ip from x.x.x.x to me
00218 skipto 410 ip from me to x.x.x.x
00219 skipto 410 ip from x.x.x.x to me
00219 skipto 410 ip from me to x.x.x.x
00225 skipto 410 ip from x.x.x.x to me
00225 skipto 410 ip from me to x.x.x.x
00226 skipto 410 ip from x.x.x.x to me
00226 skipto 410 ip from me to x.x.x.x
00227 skipto 410 ip from x.x.x.x to me
00227 skipto 410 ip from me to x.x.x.x
00310 pipe 5 ip from me to x.x.x.x
00310 pipe 5 ip from x.x.x.x to me
00311 skipto 410 ip from me to x.x.x.x
00311 skipto 410 ip from x.x.x.x to me
00312 pipe 1 ip from any to me
00313 pipe 2 ip from any to me
00314 pipe 3 ip from me to any
00315 pipe 4 ip from me to any
00410 allow tcp from any to any established
00411 allow icmp from any to any icmptypes 0,3,8,11
00420 deny icmp from any to any
00430 allow ip from any to any via lo0
00510 deny ip from any to 127.0.0.0/8
00511 deny ip from 127.0.0.0/8 to any
00512 deny ip from 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 to any
00513 deny ip from any to 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
00514 deny ip from
0.0.0.0/8,169.254.0.0/16,192.0.2.0/24,224.0.0.0/4,240.0.0.0/4 to
any
00515 deny ip from any to
0.0.0.0/8,169.254.0.0/16,192.0.2.0/24,224.0.0.0/4,240.0.0.0/4
00610 deny tcp from any to any tcpflags syn tcpoptions !mss
00611 deny tcp from any to any tcpflags syn,fin
00612 deny tcp from any to any tcpflags fin,psh,rst,urg
00613 deny tcp from any to any tcpflags fin,psh,urg
00614 deny tcp from any to any tcpflags syn,fin,ack,rst
00615 deny tcp from any to any tcpflags !syn,!fin,!ack
01010 allow udp from me to any dst-port 53
01011 allow udp from x.x.x.x,x.x.x.x 53 to me
01012 allow udp from any to me dst-port 33433-33499
01020 allow tcp from any to x.x.x.x dst-port 21,22,62000-64000 setup
65000 deny log logamount 1000 ip from any to me
65001 deny log logamount 1000 ip from any to me6
65535 allow ip from any to any
# ipfw pipe list
00001: 200.000 Mbit/s 0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0
droptail
sched 65537 type FIFO flags 0x0 0 buckets 1 active
0 ip 0.0.0.0/0 0.0.0.0/0 10 15000 0 0 0
00002: 32.000 Mbit/s 0 ms burst 0
q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0
droptail
sched 65538 type FIFO flags 0x1 64 buckets 10 active
mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
0
00003: 100.000 Mbit/s 0 ms burst 0
q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0
droptail
sched 65539 type FIFO flags 0x0 0 buckets 1 active
0 ip 0.0.0.0/0 0.0.0.0/0 6 312 1 52 0
00004: 24.000 Mbit/s 0 ms burst 0
q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0
droptail
sched 65540 type FIFO flags 0x1 64 buckets 9 active
mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
0
00005: 40.000 Mbit/s 0 ms burst 0
q131077 50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0
droptail
sched 65541 type FIFO flags 0x0 0 buckets 0 active
What other troubleshooting or remedy should I do via console when this
happens, or perhaps is there a problem with the way we've setup our ruleset?
Thanks for your help!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAP1HOmSBcvTcJs4FHGk%2BO2dVmyn=UN6URnre74bQOcyJx8hWog>