Date: Fri, 02 Jan 2015 15:14:43 +0000 From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-net@freebsd.org Subject: Re: CARP Problem/Bug? on 10.1-RELEASE Message-ID: <54A6B5E3.5090200@multiplay.co.uk> In-Reply-To: <54A69F72.6060405@gmail.com> References: <54A69F72.6060405@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Since 10 we're had to use the following settings in sysctl for carp to
work as expected:
net.inet.carp.preempt=1
net.inet.carp.senderr_demotion_factor=0
See if these help for your setup too?
Regards
Steve
On 02/01/2015 13:38, Sascha wrote:
> Hi and a happy new year to everyone!
>
> I have problems with my carp setup between two Routers/Firewalls after
> upgrade from 10.0 RELEASE to 10.1 RELEASE.
> Before the upgrade my setup worked without any problems!
>
> After the upgrade I checked the carp status. Machine 2 (Backup
> machine) is in Master state for all Interfaces. Machine 1 remains in
> Backup state. Machine 1 is my primary Machine and should be the master!
>
> I restarted both machines several times and checked the rc.conf for
> errors. But Machine 2 gets every time the master. Machine 1 gets only
> the master when I turn off Machine 2.
> All interfaces remain in Master state even when I manually take a
> Interface for example igb4 with ifconfig on Machine 2 down. Then igb4
> on Machine 1 goes into Master state.But the other interfaces remain in
> Backup state on Machine 1.
>
> preemt Option is set to 1
>
> The machines are connected with Link Aggregation to the switch and
> they do VLAN Tagging. The CARP Interface is the Gateway for the clients.
> The address on the physical interface is used to bind demons for
> network services.
> PF Sync is done over a cross cable direct attached to both machines.
> PF is not blocking any carp traffic.
>
>
> I Think the problem has something to do with the net.inet.carp.demotion
>
> You can see in my output that machine 1 had this value
> net.inet.carp.demotion: 3840
>
> Is this a bug or does someone see a configuration error in the config?
>
>
>
> Detailed config(Some output is cleared or omitted):
>
> rc.conf Machine 1:
>
> ################################################################################################################################################################################
>
> ifconfig_igb0="up"
> ifconfig_igb1="up"
> ifconfig_igb2="up"
> ifconfig_igb3="up"
> ifconfig_igb4="up"
> ifconfig_igb5="up"
>
> ifconfig_igb4="inet xxx.xxx.0.253 netmask 255.255.255.0"
> ifconfig_igb4_alias0="vhid 1 pass secret xxx.xxx.0.1/32"
>
> # PF Sync Interface
> ifconfig_igb5="inet xxx.xxx.255.253/30"
>
> # Create Virutal Interfaces
> cloned_interfaces="lagg0 vlan10 vlan25 vlan35 vlan90 vlan95 vlan97
> vlan98 vlan99 vlan100 vlan101 vlan102 vlan103 vlan106 vlan107 vlan108
> vlan109"
> # lagg Interface
> ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport
> igb2 laggport igb3"
>
> # VLAN Interfaces
> ifconfig_vlanxx0="inet xxx.xxxx.xxx.197/27"
> ifconfig_vlanxx0alias0="inet vhid 2 pass secret xxx.xxx.xxx.199/32
> vlan xx0 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.25.253/24"
> ifconfig_vlanxx5_alias0="inet vhid 3 pass secret xxx.xxx.25.1/32 vlan
> xx5 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.35.253/24"
> ifconfig_vlanxx5_alias0="inet vhid 4 pass secret xxx.35.1/32 vlan xx5
> vlandev lagg0"
> ifconfig_vlanxx0="inet xxx.xxx.7.253/21"
> ifconfig_vlanxx0_alias0="inet vhid 5 pass secret xxx.xxx.7.1/32 vlan
> xx0 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.95.253/24"
> ifconfig_vlanxx5_alias0="inet vhid 6 pass secret xxx.xxx.95.1/32 vlan
> xx5 vlandev lagg0"
> ifconfig_vlanxx7="inet xxx.xxx.90.253/16"
> ifconfig_vlanxx7_alias0="inet vhid 7 pass secret xxx.xxx.90.6/32 vlan
> xx7 vlandev lagg0"
> ifconfig_vlanxx8="inet xxx.xxx.239.253/21"
> ifconfig_vlanxx8_alias0="inet vhid 8 pass secret xxx.xxx.232.1/32 vlan
> xx8 vlandev lagg0"
> ifconfig_vlanxx9="inet xxx.xxx.0.29/27"
> ifconfig_vlanxx9_alias0="inet vhid 9 pass secret xxx.xxx.0.1/32 vlan
> xx9 vlandev lagg0"
> ifconfig_vlanxx0="inet xxx.xxx.100.253/24"
> ifconfig_vlanxx0_alias0="inet vhid 10 pass secret xxx.xxx.100.1/32
> vlan xx0 vlandev lagg0"
> ifconfig_vlanxx1="inet xxx.xxx.1.253/24"
> ifconfig_vlanxx1_alias0="inet vhid 11 pass secret xxx.xxx.1.1/32 vlan
> xx1 vlandev lagg0"
> ifconfig_vlanxx2="inet xxx.xxx.116.253/24"
> ifconfig_vlanxx2_alias0="inet vhid 12 pass secret xxx.xxx.116.1/32
> vlan xx2 vlandev lagg0"
> ifconfig_vlanxx3="inet xxx.xxx.10.253/24"
> ifconfig_vlanxx3_alias0="inet vhid 13 pass secret xxx.xxx.10.1/32 vlan
> xx3 vlandev lagg0"
> ifconfig_vlanxx6="inet xxx.xxx.255.252/16"
> ifconfig_vlanxx6_alias0="inet vhid 14 pass secret xxx.xxx.255.254/32
> vlan xx6 vlandev lagg0"
> ifconfig_vlanxx7="inet xxx.xxx.90.253/16"
> ifconfig_vlanxx7_alias0="inet vhid 15 pass secret xxx.xxx.90.140/32
> vlan xx7 vlandev lagg0"
> ifconfig_vlanxx8="inet xxx.xxx.89.12/29"
> ifconfig_vlanxx8_alias0="inet vhid 16 pass secret xxx.xxx.89.14/32
> vlan xx8 vlandev lagg0"
> ifconfig_vlanxx9="inet xxx.xxx.110.253/24"
> ifconfig_vlanxx9_alias0="inet vhid 17 pass secret xxx.xxx.110.1/32
> vlan xx9 vlandev lagg0"
>
> igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
>
> ether 00:1b:21:96:65:78
> inet xxx.xxx.0.253 netmask 0xffffff00 broadcast xxx.xxx.0.255
> inet xxx.xxx.0.1 netmask 0xffffffff broadcast xxx.xxxx.0.1 vhid 1
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> carp: BACKUP vhid 1 advbase 1 advskew 0
>
> vlanxx0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=303<RXCSUM,TXCSUM,TSO4,TSO6>
> ether 00:1b:21:96:c0:38
> inet xxx.xxx.xxx.197 netmask 0xffffffe0 broadcast xxx.xxx.xxx.223
> inet xxx.xxx.xxx.199 netmask 0xffffffff broadcast
> xxx.xxx.xxx.199 vhid 2
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 10 parent interface: lagg0
> carp: BACKUP vhid 2 advbase 1 advskew 0
>
>
>
> root@xxx1:/# sysctl net.inet.carp
> net.inet.carp.allow: 1
> net.inet.carp.preempt: 1
> net.inet.carp.log: 2
> net.inet.carp.demotion: 3840
> net.inet.carp.senderr_demotion_factor: 240
> net.inet.carp.ifdown_demotion_factor: 240
>
>
> Log File when restarting machine 2
>
> root@xxx1:/# less /var/log/messages
>
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 15@vlanxx7: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 16@vlanxx8: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 17@vlanxx9: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 1@igb4: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 2@vlanxx0: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 3@vlanxx5: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 4@vlanxx5: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 5@vlanxx0: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 6@vlanxx5: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 7@vlanxx7: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 8@vlanxx8: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 10@vlanxx0: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 9@vlanxx9: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 11@vlanxx1: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 12@vlanxx2: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 13@vlanxx3: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:11 xxx1 kernel: carp: VHID 14@vlanxx6: BACKUP -> MASTER
> (master down)
> Jan 2 12:29:12 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:29:14 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:29:14 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:29:20 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:29:22 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:29:22 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:29:23 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:29:24 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:29:24 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:29:35 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:29:37 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:29:37 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:31:06 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:31:08 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:31:08 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:31:17 xxx1 kernel: igb5: link state changed to DOWN
> Jan 2 12:31:19 xxx1 kernel: igb5: link state changed to UP
> Jan 2 12:31:19 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5'
> Jan 2 12:31:21 xxx1 kernel: carp: VHID 1@igb4: MASTER -> BACKUP (more
> frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 2@vlanxx0: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 3@vlanxx5: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 5@vlanxx0: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 6@vlanxx5: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 9@vlanxx9: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 8@vlanxx8: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 7@vlanxx7: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 14@vlanxx6: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 10@vlanxx0: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 16@vlanxx8: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 11@vlanxx1: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 12@vlanxx2: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 15@vlanxx7: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 13@vlanxx3: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 17@vlanxx9: MASTER -> BACKUP
> (more frequent advertisement received)
> Jan 2 12:31:52 xxx1 kernel: carp: VHID 4@vlanxx5: MASTER -> BACKUP
> (more frequent advertisement received)
>
>
> TCP Dump on machine 1 during restarting machine 2
>
> root@xxx1:/# sudo tcpdump -npi igb4 -T carp dst 224.0.0.18
>
> 13:47:55.455580 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167172
> 13:47:56.881428 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167173
> 13:47:59.883366 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167174
> 13:48:00.894534 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167175
> .
> .
> 13:50:07.842383 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167240
> 13:50:09.782055 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167241
> 13:50:09.782914 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167242
> 13:50:11.206767 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167243
> 13:50:11.510203 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167244
> 13:50:12.450678 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36:
> vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167245
> ################################################################################################################################################################################
>
>
>
> rc.conf Machine 2:
>
> ################################################################################################################################################################################
>
> ifconfig_igb0="up"
> ifconfig_igb1="up"
> ifconfig_igb2="up"
> ifconfig_igb3="up"
> ifconfig_igb4="up"
> ifconfig_igb5="up"
>
> ifconfig_igb4="inet xxx.xxx.0.254 netmask 255.255.255.0"
> ifconfig_igb4_alias0="vhid 1 advskew 100 pass secret xxx.xxx.0.1/32"
>
> # PF Sync Interface
> ifconfig_igb5="inet 192.168.255.254/30"
>
> # Create Virutal Interfaces
> cloned_interfaces="lagg0 vlan10 vlan25 vlan35 vlan90 vlan95 vlan97
> vlan98 vlan99 vlan100 vlan101 vlan102 vlan103 vlan106 vlan107 vlan108
> vlan109"
> # lagg Interface
> ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport
> igb2 laggport igb3"
>
> # VLAN Interfaces
> ifconfig_vlanxx0="inet xxx.xxx.22.198/27"
> ifconfig_vlanxx0_alias0="inet vhid 2 advskew 100 pass secret
> xxx.xxx.22.199/32 vlan 10 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.25.254/24"
> ifconfig_vlanxx5_alias0="inet vhid 3 advskew 100 pass secret
> xxx.xxx.25.1/32 vlan 25 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.35.254/24"
> ifconfig_vlanxx5_alias0="inet vhid 4 advskew 100 pass secret
> xxx.xxx.35.1/32 vlan 35 vlandev lagg0"
> ifconfig_vlanxx0="inet xxx.xxx.7.254/21"
> ifconfig_vlanxx0_alias0="inet vhid 5 advskew 100 pass secret
> xxx.xxx.0.1/32 vlan 90 vlandev lagg0"
> ifconfig_vlanxx5="inet xxx.xxx.95.254/24"
> ifconfig_vlanxx5_alias0="inet vhid 6 advskew 100 pass secret
> xxx.xxx.95.1/32 vlan 95 vlandev lagg0"
> ifconfig_vlanxx7="inet xxx.xxx.90.254/16"
> ifconfig_vlanxx7_alias0="inet vhid 7 advskew 100 pass secret
> xxx.xxx.90.6/32 vlan 97 vlandev lagg0"
> ifconfig_vlanxx8="inet xxx.xxx.239.254/21"
> ifconfig_vlanxx8_alias0="inet vhid 8 advskew 100 pass secret
> xxx.xxx.232.1/32 vlan 98 vlandev lagg0"
> ifconfig_vlanxx9="inet xxx.xxx.0.30/27"
> ifconfig_vlanxx9_alias0="inet vhid 9 advskew 100 pass secret
> xxx.xxx.0.1/32 vlan 99 vlandev lagg0"
> ifconfig_vlanxx0="inet xxx.xxx.100.254/24"
> ifconfig_vlanxx0_alias0="inet vhid 10 advskew 100 pass secret
> xxx.xxx.100.1/32 vlan 100 vlandev lagg0"
> ifconfig_vlanxx1="inet xxx.xxx.1.254/24"
> ifconfig_vlanxx1_alias0="inet vhid 11 advskew 100 pass secret
> xxx.xxx.1.1/32 vlan 101 vlandev lagg0"
> ifconfig_vlanxx2="inet xxx.xxx.116.254/24"
> ifconfig_vlanxx2_alias0="inet vhid 12 advskew 100 pass secret
> xxx.xxx.116.1/32 vlan 102 vlandev lagg0"
> ifconfig_vlanxx3="inet xxx.xxx.10.254/24"
> ifconfig_vlanxx3_alias0="inet vhid 13 advskew 100 pass secret
> xxx.xxx.10.1/32 vlan 103 vlandev lagg0"
> ifconfig_vlanxx6="inet xxx.xxx.255.253/16"
> ifconfig_vlanxx6_alias0="inet vhid 14 advskew 100 pass secret
> xxx.xxx.255.254/32 vlan 106 vlandev lagg0"
> ifconfig_vlanxx7="inet xxx.xxx.90.254/16"
> ifconfig_vlanxx7_alias0="inet vhid 15 advskew 100 pass secret
> xxx.xxx.90.140/32 vlan 107 vlandev lagg0"
> ifconfig_vlanxx8="inet xxx.xxx.89.13/29"
> ifconfig_vlanxx8_alias0="inet vhid 16 advskew 100 pass secret
> xxx.xxx.89.14/32 vlan 108 vlandev lagg0"
> ifconfig_vlanxx9="inet xxx.xxx.110.254/24"
> ifconfig_vlanxx9_alias0="inet vhid 17 advskew 100 pass secret
> xxx.xxx.110.1/32 vlan 109 vlandev lagg0"
>
>
>
> igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
>
> ether 00:1b:21:96:66:70
> inet xxx.xxx.0.254 netmask 0xffffff00 broadcast xxx.xxx.0.255
> inet xxx.xxx.0.1 netmask 0xffffffff broadcast xxx.xxx.0.1 vhid 1
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> carp: MASTER vhid 1 advbase 1 advskew 100
>
> vlanxx0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=303<RXCSUM,TXCSUM,TSO4,TSO6>
> ether 00:1b:21:96:65:68
> inet xxx.xxx.xxx.198 netmask 0xffffffe0 broadcast xxx.xxx.xxx.223
> inet xxx.xxx.xxx.199 netmask 0xffffffff broadcast
> xxx.xxx.xxx.199 vhid 2
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 10 parent interface: lagg0
> carp: MASTER vhid 2 advbase 1 advskew 100
>
>
>
>
> root@xxx2:/# sysctl net.inet.carp
> net.inet.carp.allow: 1
> net.inet.carp.preempt: 1
> net.inet.carp.log: 2
> net.inet.carp.demotion: 720
> net.inet.carp.senderr_demotion_factor: 240
> net.inet.carp.ifdown_demotion_factor: 240
> root@xxx2:/#
> ################################################################################################################################################################################
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54A6B5E3.5090200>