From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 15:48:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 44098BCE for ; Wed, 21 Nov 2012 15:48:44 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id A6BC58FC14 for ; Wed, 21 Nov 2012 15:48:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 4929113200F for ; Wed, 21 Nov 2012 16:41:45 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RlO6qfZYLOm0 for ; Wed, 21 Nov 2012 16:41:33 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id F19C913200E for ; Wed, 21 Nov 2012 16:41:32 +0100 (CET) Message-ID: <50ACF62C.8000408@mpeters.org> Date: Wed, 21 Nov 2012 16:41:32 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Low Bandwidth on intercontinental connections X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 15:48:44 -0000 Hi list, we are experiencing low throughput on interncontinental connections with our FreeBSD Servers. We made several tests and are wondering, why this would be. The first tests were on an IPSEC VPN between our datacenter in DE and Santa Clara, CA. We are connected with two gigabit uplinks in each DC. Pushing data by scp between our FreeBSD servers takes ages. Starting with several MB/s it drops to 60-70KB/s: [root@freebsd ~]# ls -alh test.tgz -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz [root@freebsd ~]# scp test.tgz 172.16.3.10:. Password: test.tgz 28% 17MB 75.3KB/s 09:32 ETA For comparision, we did a similiar test with Linux, which didn't show this behaviour: root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: root@172.16.4.50's password: jdk-6u33-linux-x64.bin 100% 69MB 3.4MB/s 00:20 root@linux:~# Otherwise, the servers are really fast, when copying data to a machine nearby: [root@freebsd ~]# ls -alh test -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test [root@freebsd ~]# scp test 172.16.3.11: Password: test 100% 1000MB 38.5MB/s 00:26 Intercontinental ftp downloads are the same: [root@freebsd ~]# fetch ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps [root@freebsd ~]# fetch ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s [root@freebsd ~]# fetch ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C Linux: root@linux:~# wget ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso --2012-11-21 15:07:57-- ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso => `FreeBSD-9.1-RC3-amd64-disc1.iso' Resolving ftp1.de.freebsd.org... 137.226.34.42 Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. Length: 718800896 (686M) (unauthoritative) 100%[=====================================================================>] 718,800,896 19.1M/s in 61s 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' saved [718800896] Doing some googling brought up a lot of tuning hints, but nothing worked for us. We tweaked some sysctls: kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.hostcache.expire=1 but to no good. Disabling MSI and TSO4 for the card didn't change anything, too. The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and pciconf -lvc at the end of this mail). Can someone hit me with a cluestick to get the BSDs on speed? marc PS: The version is FreeBSD-RC2 amd64, because we need the patch for process migration on the CPUs which didn't make it 9.0 or an errata, as we were the only ones, hitting this bug (so kib@ said). ifconfig: [root@freebsd ~]# ifconfig bce0: flags=8843 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:f4 inet 172.16.3.10 netmask 0xffffff00 broadcast 172.16.3.255 inet6 fe80::ae16:2dff:feb7:f4%bce0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bce1: flags=8843 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:f6 inet 172.17.3.10 netmask 0xfffff800 broadcast 172.17.7.255 inet6 fe80::ae16:2dff:feb7:f6%bce1 prefixlen 64 scopeid 0x2 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bce2: flags=8802 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:fc nd6 options=29 media: Ethernet autoselect bce3: flags=8802 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:fe nd6 options=29 media: Ethernet autoselect lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb inet 127.0.0.1 netmask 0xff000000 nd6 options=21 pciconf -lvc: [root@freebsd ~]# pciconf -lvc hostb0@pci0:0:0:0: class=0x060000 card=0x330b103c chip=0x34068086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520 I/O Hub to ESI Port' class = bridge subclass = HOST-PCI cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 128(128) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib1@pci0:0:1:0: class=0x060400 card=0x330b103c chip=0x34088086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 1' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib2@pci0:0:2:0: class=0x060400 card=0x330b103c chip=0x34098086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 2' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib3@pci0:0:3:0: class=0x060400 card=0x330b103c chip=0x340a8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 3' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x16) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib4@pci0:0:4:0: class=0x060400 card=0x330b103c chip=0x340b8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 4' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib5@pci0:0:5:0: class=0x060400 card=0x330b103c chip=0x340c8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 5' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib6@pci0:0:6:0: class=0x060400 card=0x330b103c chip=0x340d8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 6' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib7@pci0:0:7:0: class=0x060400 card=0x330b103c chip=0x340e8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 7' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib8@pci0:0:8:0: class=0x060400 card=0x330b103c chip=0x340f8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 8' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib9@pci0:0:9:0: class=0x060400 card=0x330b103c chip=0x34108086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 9' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x8) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib10@pci0:0:10:0: class=0x060400 card=0x330b103c chip=0x34118086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 10' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 hostb1@pci0:0:13:0: class=0x060000 card=0x00000000 chip=0x343a8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 ecap 000b[800] = unknown 0 hostb2@pci0:0:13:1: class=0x060000 card=0x00000000 chip=0x343b8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 ecap 000b[800] = unknown 0 hostb3@pci0:0:13:2: class=0x060000 card=0x00000000 chip=0x343c8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb4@pci0:0:13:3: class=0x060000 card=0x00000000 chip=0x343d8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 hostb5@pci0:0:13:4: class=0x060000 card=0x00000000 chip=0x34188086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 Physical Layer Port 0' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb6@pci0:0:13:5: class=0x060000 card=0x00000000 chip=0x34198086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500 Physical Layer Port 1' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb7@pci0:0:13:6: class=0x060000 card=0x00000000 chip=0x341a8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb8@pci0:0:14:0: class=0x060000 card=0x00000000 chip=0x341c8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 hostb9@pci0:0:14:1: class=0x060000 card=0x00000000 chip=0x341d8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb10@pci0:0:14:2: class=0x060000 card=0x00000000 chip=0x341e8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb11@pci0:0:14:3: class=0x060000 card=0x00000000 chip=0x341f8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb12@pci0:0:14:4: class=0x060000 card=0x00000000 chip=0x34398086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none0@pci0:0:20:0: class=0x080000 card=0x000b003c chip=0x342e8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub System Management Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none1@pci0:0:20:1: class=0x080000 card=0x000b003c chip=0x34228086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none2@pci0:0:20:2: class=0x080000 card=0x000b003c chip=0x34238086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub Control Status and RAS Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) pcib11@pci0:0:28:0: class=0x060400 card=0x330d103c chip=0x3a408086 rev=0x00 hdr=0x01 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) PCI Express Root Port 1' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 1 root port max data 128(128) link x0(x4) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x330d103c cap 01[a0] = powerspec 2 supports D0 D3 current D0 ecap 0002[100] = VC 1 max VC0 ecap 0005[180] = unknown 1 pcib12@pci0:0:28:4: class=0x060400 card=0x330d103c chip=0x3a488086 rev=0x00 hdr=0x01 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) PCI Express Root Port 5' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x330d103c cap 01[a0] = powerspec 2 supports D0 D3 current D0 ecap 0002[100] = VC 1 max VC0 ecap 0005[180] = unknown 1 uhci1@pci0:0:29:0: class=0x0c0300 card=0x330d103c chip=0x3a348086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci2@pci0:0:29:1: class=0x0c0300 card=0x330d103c chip=0x3a358086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci3@pci0:0:29:2: class=0x0c0300 card=0x330d103c chip=0x3a368086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci4@pci0:0:29:3: class=0x0c0300 card=0x330d103c chip=0x3a398086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP ehci0@pci0:0:29:7: class=0x0c0320 card=0x330d103c chip=0x3a3a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB2 EHCI Controller' class = serial bus subclass = USB cap 01[50] = powerspec 2 supports D0 D3 current D0 cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14 cap 13[98] = PCI Advanced Features: FLR TP pcib13@pci0:0:30:0: class=0x060401 card=0x330d103c chip=0x244e8086 rev=0x90 hdr=0x01 vendor = 'Intel Corporation' device = '82801 PCI Bridge' class = bridge subclass = PCI-PCI cap 0d[50] = PCI Bridge card=0x330d103c isab0@pci0:0:31:0: class=0x060100 card=0x00000000 chip=0x3a188086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JIB (ICH10) LPC Interface Controller' class = bridge subclass = PCI-ISA cap 09[e0] = vendor (length 12) Intel cap 1 version 0 features: Quick Resume, SATA RAID-5, 4 PCI-e x1 slots, SATA RAID-0/1/10 atapci0@pci0:0:31:2: class=0x01018f card=0x330d103c chip=0x3a208086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) 4 port SATA IDE Controller' class = mass storage subclass = ATA cap 01[70] = powerspec 3 supports D0 D3 current D0 cap 13[b0] = PCI Advanced Features: FLR TP ciss0@pci0:5:0:0: class=0x010400 card=0x3245103c chip=0x323a103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Smart Array G6 controllers' class = mass storage subclass = RAID cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 10[70] = PCI-Express 2 endpoint max data 256(256) link x4(x8) cap 11[ac] = MSI-X supports 16 messages in map 0x10 enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected bce0@pci0:3:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700f4 ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce1@pci0:3:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700f6 ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce2@pci0:4:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700fc ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce3@pci0:4:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700fe ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 none3@pci0:2:0:0: class=0x088000 card=0x3309103c chip=0x3306103c rev=0x04 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Slave Instrumentation & System Support' class = base peripheral cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 05[b0] = MSI supports 1 message, 64 bit cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) none4@pci0:2:0:2: class=0x088000 card=0x3309103c chip=0x3307103c rev=0x04 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Management Processor Support and Messaging' class = base peripheral cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 05[b0] = MSI supports 1 message, 64 bit cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) uhci0@pci0:2:0:4: class=0x0c0300 card=0x3309103c chip=0x3300103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Virtual USB Controller' class = serial bus subclass = USB cap 05[70] = MSI supports 1 message, 64 bit cap 10[80] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) cap 01[f0] = powerspec 3 supports D0 D3 current D0 vgapci0@pci0:1:3:0: class=0x030000 card=0x31fb103c chip=0x515e1002 rev=0x02 hdr=0x00 vendor = 'ATI Technologies Inc' device = 'ES1000' class = display subclass = VGA cap 01[50] = powerspec 2 supports D0 D1 D2 D3 current D0 dmesg: [root@freebsd ~]# dmesg Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-RC2 #0 r241106: Mon Oct 1 18:26:44 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU X5660 @ 2.80GHz (2800.16-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0xbfebfbff Features2=0x29ee3ff AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33095315456 (31562 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 16 cpu7 (AP): APIC ID: 17 cpu8 (AP): APIC ID: 18 cpu9 (AP): APIC ID: 19 cpu10 (AP): APIC ID: 20 cpu11 (AP): APIC ID: 21 cpu12 (AP): APIC ID: 32 cpu13 (AP): APIC ID: 33 cpu14 (AP): APIC ID: 34 cpu15 (AP): APIC ID: 35 cpu16 (AP): APIC ID: 36 cpu17 (AP): APIC ID: 37 cpu18 (AP): APIC ID: 48 cpu19 (AP): APIC ID: 49 cpu20 (AP): APIC ID: 50 cpu21 (AP): APIC ID: 51 cpu22 (AP): APIC ID: 52 cpu23 (AP): APIC ID: 53 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20110527/tbfadt-638) ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 (20110527/tbfadt-638) ioapic1 irqs 24-47 on motherboard ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 atrtc0: port 0x70-0x71 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci5: on pcib1 ciss0: port 0x4000-0x40ff mem 0xfbe00000-0xfbffffff,0xfbdf0000-0xfbdf0fff irq 28 at device 0.0 on pci5 ciss0: PERFORMANT Transport pcib2: at device 2.0 on pci0 pci12: on pcib2 pcib3: at device 3.0 on pci0 pci9: on pcib3 pcib4: at device 4.0 on pci0 pci13: on pcib4 pcib5: at device 5.0 on pci0 pci14: on pcib5 pcib6: at device 6.0 on pci0 pci15: on pcib6 pcib7: at device 7.0 on pci0 pci3: on pcib7 bce0: mem 0xf4000000-0xf5ffffff irq 30 at device 0.0 on pci3 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: ac:16:2d:b7:00:f4 bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) bce1: mem 0xf2000000-0xf3ffffff irq 37 at device 0.1 on pci3 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce1: Ethernet address: ac:16:2d:b7:00:f6 bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) pcib8: at device 8.0 on pci0 pci4: on pcib8 bce2: mem 0xf8000000-0xf9ffffff irq 31 at device 0.0 on pci4 miibus2: on bce2 brgphy2: PHY 1 on miibus2 brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce2: Ethernet address: ac:16:2d:b7:00:fc bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) bce3: mem 0xf6000000-0xf7ffffff irq 39 at device 0.1 on pci4 miibus3: on bce3 brgphy3: PHY 1 on miibus3 brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce3: Ethernet address: ac:16:2d:b7:00:fe bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) pcib9: at device 9.0 on pci0 pci6: on pcib9 pcib10: at device 10.0 on pci0 pci16: on pcib10 pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pcib11: at device 28.0 on pci0 pci17: on pcib11 pcib12: at device 28.4 on pci0 pci2: on pcib12 pci2: at device 0.0 (no driver attached) pci2: at device 0.2 (no driver attached) uhci0: port 0x3c00-0x3c1f irq 17 at device 0.4 on pci2 usbus0 on uhci0 uhci1: port 0x1000-0x101f irq 20 at device 29.0 on pci0 usbus1 on uhci1 uhci2: port 0x1020-0x103f irq 23 at device 29.1 on pci0 usbus2 on uhci2 uhci3: port 0x1040-0x105f irq 22 at device 29.2 on pci0 usbus3 on uhci3 uhci4: port 0x1060-0x107f irq 23 at device 29.3 on pci0 usbus4 on uhci4 ehci0: mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci0 pcib13: at device 30.0 on pci0 pci1: on pcib13 vgapci0: port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci1 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1080-0x1087,0x1088-0x108b,0x1090-0x1097,0x1098-0x109b,0x10a0-0x10af,0x10b0-0x10bf irq 17 at device 31.2 on pci0 ata2: at channel 0 on atapci0 ata3: at channel 1 on atapci0 acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 qpi0: on motherboard orm0: at iomem 0xc0000-0xcafff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range uart1: at port 0x2f8-0x2ff irq 3 on isa0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est1 attach returned 6 p4tcc1: on cpu1 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est2 attach returned 6 p4tcc2: on cpu2 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est3 attach returned 6 p4tcc3: on cpu3 est4: on cpu4 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est4 attach returned 6 p4tcc4: on cpu4 est5: on cpu5 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est5 attach returned 6 p4tcc5: on cpu5 est6: on cpu6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est6 attach returned 6 p4tcc6: on cpu6 est7: on cpu7 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est7 attach returned 6 p4tcc7: on cpu7 est8: on cpu8 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est8 attach returned 6 p4tcc8: on cpu8 est9: on cpu9 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est9 attach returned 6 p4tcc9: on cpu9 est10: on cpu10 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est10 attach returned 6 p4tcc10: on cpu10 est11: on cpu11 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est11 attach returned 6 p4tcc11: on cpu11 est12: on cpu12 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est12 attach returned 6 p4tcc12: on cpu12 est13: on cpu13 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est13 attach returned 6 p4tcc13: on cpu13 est14: on cpu14 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est14 attach returned 6 p4tcc14: on cpu14 est15: on cpu15 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est15 attach returned 6 p4tcc15: on cpu15 est16: on cpu16 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est16 attach returned 6 p4tcc16: on cpu16 est17: on cpu17 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est17 attach returned 6 p4tcc17: on cpu17 est18: on cpu18 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est18 attach returned 6 p4tcc18: on cpu18 est19: on cpu19 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est19 attach returned 6 p4tcc19: on cpu19 est20: on cpu20 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est20 attach returned 6 p4tcc20: on cpu20 est21: on cpu21 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est21 attach returned 6 p4tcc21: on cpu21 est22: on cpu22 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est22 attach returned 6 p4tcc22: on cpu22 est23: on cpu23 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est23 attach returned 6 p4tcc23: on cpu23 ZFS filesystem version 5 ZFS storage pool version 28 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen0.1: <0x103c> at usbus0 uhub0: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered ugen0.2: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 ums0: on usbus0 (probe0:ciss0:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 (probe0:ciss0:0:0:0): CAM status: SCSI Status Error (probe0:ciss0:0:0:0): SCSI status: Check Condition (probe0:ciss0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe0:ciss0:0:0:0): Error 22, Unretryable error da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C) SMP: AP CPU #1 Launched! cd0 at ata2 bus 0 scbus2 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device SMP: AP CPU #12 Launched! cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #11 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #15 Launched! Timecounter "TSC-low" frequency 10938126 Hz quality 1000 uhub5: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/da0p2 [rw]... bce0: link state changed to UP bce0: Gigabit link up! bce0: Gigabit link up! bce0: Gigabit link up! bce1: link state changed to UP bce1: Gigabit link up! bce1: Gigabit link up! bce1: Gigabit link up!