Date: Tue, 28 Apr 2015 14:19:34 +0200 From: Mark Schouten <mark@tuxis.nl> To: Chris Forgeron <cforgeron@acsi.ca>, "freebsd-net@FreeBSD.org" <freebsd-net@FreeBSD.org> Subject: Re: Frequent hickups on the networking layer Message-ID: <4294293883-7795@kerio.tuxis.nl> In-Reply-To: <46D80686C389884BB0C047851038EC4501C98541EB@AA-EX0.acsi.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-0J1G8JZsM31y1lJuUDfE Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable I've disabled TSO a while ago, after my networking stopped working complete= ly, with `ifconfig em0 -tso`. em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=3D4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGI= C,VLAN_HWTSO> ether 00:25:90:dc:0f:a2 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active nfsv212: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 1500 options=3D3<RXCSUM,TXCSUM> ether 00:25:90:dc:0f:a2 inet6 fe80::225:90ff:fedc:fa2%nfsv212 prefixlen 64 scopeid 0x6=C2=A0 inet6 fd::212:31:3:111:fffe prefixlen 64=C2=A0 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 212 parent interface: em0 nfsv308: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 9000 options=3D3<RXCSUM,TXCSUM> ether 00:25:90:dc:0f:a2 inet 10.38.0.253 netmask 0xffffff00 broadcast 10.38.0.255=C2=A0 inet6 fe80::225:90ff:fedc:fa2%nfsv308 prefixlen 64 scopeid 0x8=C2=A0 inet6 fd:308::30 prefixlen 64=C2=A0 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 308 parent interface: em0 Or should I disable VLAN_HWTSO as well. If so, how? Met vriendelijke groeten, --=C2=A0 Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/ T: 0318 200208 | info@tuxis.nl Van: Chris Forgeron <cforgeron@acsi.ca>=20 Aan: Mark Schouten <mark@tuxis.nl>, "freebsd-net@FreeBSD.org" <freebsd-n= et@FreeBSD.org>=20 Verzonden: 28-4-2015 13:58=20 Onderwerp: RE: Frequent hickups on the networking layer=20 What network care are you using?=20 =20 There have been a few reports of issues with TSO, if you check the list. I = had some myself a while ago, but are now resolved thanks to a few helpful f= olk here.=20 =20 You could increase your mbufs, but if the problem is a leak/error in the st= ack, then you're just delaying the behaviour. =20 =20 -----Original Message-----=20 From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Mark Schouten=20 Sent: Tuesday, April 28, 2015 5:48 AM=20 To: freebsd-net@FreeBSD.org=20 Subject: Frequent hickups on the networking layer=20 =20 Hi,=20 =20 =20 I've got a FreeBSD 10.1-RELEASE box running with iscsi on top of ZFS. I've = had some major issues with it where it would stop processing traffic for a = minute or two, but that's 'fixed' by disabling TSO. I do have frequent iscs= i errors, which are luckily fixed on the iscsi layer, but they do cause an = occasional errormessage on both the iscsi client and server. Also, I see in= put errors on the FreeBSD server, but I'm unable to find out what those are= . I do see a relation between iscsi-errormessages and the number of etherne= t inputerrors on the server.=20 =20 =20 I saw this message [1] which made me have a look at `vmstat -z`, and that s= hows me the following:=20 =20 =20 vmstat -z | head -n 1; vmstat -z | sort -k 6 -t , | tail -10 ITEM =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SIZE =C2=A0LIMIT = =C2=A0 =C2=A0 USED =C2=A0 =C2=A0 FREE =C2=A0 =C2=A0 =C2=A0REQ FAIL SLEEP zi= o_data_buf_94208: =C2=A0 94208, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 162, = =C2=A0 =C2=A0 =C2=A0 5, =C2=A0135632, =C2=A0 0, =C2=A0 0 zio_data_buf_98304= : =C2=A0 98304, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 118, =C2=A0 =C2=A0 =C2= =A0 9, =C2=A0101606, =C2=A0 0, =C2=A0 0 zio_link_cache: =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A048, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 =C2=A0 6, =C2=A0 = 30870,24853549414, =C2=A0 0, =C2=A0 0 8 Bucket: =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A064, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 145, = =C2=A0 =C2=A02831,148672720, =C2=A011, =C2=A0 0 32 Bucket: =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0256, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0= 859, =C2=A0 =C2=A0 731,231513474, =C2=A052, =C2=A0 0 mbuf_jumbo_9k: =C2=A0= =C2=A0 =C2=A0 =C2=A0 9216, 604528, =C2=A0 =C2=A07230, =C2=A0 =C2=A02002,11= 764806459,108298123, =C2=A0 0 64 Bucket: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0512, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 808, =C2=A0 =C2=A0 = 352,147120342,16375582, =C2=A0 0 256 Bucket: =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A02048, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 500, =C2=A0 =C2=A0 = =C2=A050,307051808,189685088, =C2=A0 0 vmem btag: =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 56, =C2=A0 =C2=A0 =C2=A00, 1671605, 1291509,198933= 250,36431, =C2=A0 0 128 Bucket: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A010= 24, =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 410, =C2=A0 =C2=A0 106,65267164,77= 2374, =C2=A0 0 =20 =20 =20 I am using jumboframes. Could it be that the inputerrors AND my frequent hi= ckups come from all those failures to allocate 9k jumbo mbufs? And can I in= crease the in [1] mentioned sysctls at will?=20 =20 =20 Thanks=20 =20 =20 =20 =20 =20 =20 [1]: https://lists.freebsd.org/pipermail/freebsd-questions/2013-August/2528= 27.html=20 =20 =20 Met vriendelijke groeten,=20 =20 --=C2=A0=20 Kerio Operator in de Cloud? https://www.kerioindecloud.nl/=20 Mark Schouten =C2=A0| Tuxis Internet Engineering=20 KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/=20 T: 0318 200208 | info@tuxis.nl=20 = --=-0J1G8JZsM31y1lJuUDfE Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Description: Electronic Signature S/MIME Content-Transfer-Encoding: base64 MIIRwQYJKoZIhvcNAQcCoIIRsjCCEa4CAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCCDt4w ggUbMIIEA6ADAgECAhAsv+VdGX6YsSHI/WRu2j2JMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYD VQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNh dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBMB4XDTE0MDcwMTAwMDAwMFoXDTE1MDcwMTIzNTk1OVow HjEcMBoGCSqGSIb3DQEJARYNbWFya0B0dXhpcy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBANHu3SxlMZOG5GA0/mqtRXR1QmWwhUXzmCIprI0IPtSBWSA31YBJ5qcmXRhLzaiTB3Fr UpIGkW5aAZnDms9DD64kasF3oZE00Fvfnj/BDGbw098px1PukKfg4hasbTaELAjQTSUj8xRSHzKk VVynvLA/YmyRT/+u3ueK4wdaxcej241xH6mNfZeiKMAvbkv6Tm9vdup0BtqqbRSKcnc01KKrspun Eh73jLIUhP21uJv8vuOTxS1I9zJSlhIcMCEapjBQ+26cQl+s+qBuAs/LP3UPVytSbxvicdhDxtqH npN2h3jJ/+J86zGfQi8bG7EamsULTGHkbgIJL9AdKZRgAKECAwEAAaOCAd0wggHZMB8GA1UdIwQY MBaAFHoTTgB0W8Z4Y2QnwS/ioFu8ecV7MB0GA1UdDgQWBBR+dMZE22X/SMCyTxYzIP+NMZBNXTAO BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAgBgNVHSUEGTAXBggrBgEFBQcDBAYLKwYBBAGy MQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUgMEYGA1UdIAQ/MD0wOwYMKwYBBAGyMQECAQEBMCswKQYI KwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQvQ1BTMFcGA1UdHwRQME4wTKBKoEiG Rmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET0NsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2Vj dXJlRW1haWxDQS5jcmwwgYgGCCsGAQUFBwEBBHwwejBSBggrBgEFBQcwAoZGaHR0cDovL2NydC5j b21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNy dDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMBgGA1UdEQQRMA+BDW1hcmtA dHV4aXMubmwwDQYJKoZIhvcNAQEFBQADggEBAIB8FhqaML1EzfvgNwwHDC3k0ICeMerOncgee6uJ KLxwU2mstttX5jtAmgK9RuDOu+TrMkkpF2yxYMTPpSM8nL7r+N/kdogu5Bustol8WTsW1e5vs+Nh hJYFORk113ouur1kSjXuHF8TWy+/PjFJBS/xm/H+/fkghppRU+4Dj2IReUBvlexAPYr4VDxjV7AD xPOXqTQkP15LWGvhTz2YVbJ3IAVOyUNkRhr9QwzToUxXa9k/QAOpXMuvS74AT2RBV/YCEEx7ebRD MAR6lZcbYiV8sXv1ASbnMdO3Fh2F98g+5rJn5PfFH8qLpapsZx0I2/axtSG09QMDJqXd3Ab6NpEw ggUaMIIEAqADAgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQG EwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUg VVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQG A1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTEx MDQyODAwMDAwMFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVh dGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1p dGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1h aWwgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hf ZmXxMk73nzJ9VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrh o/+43x9IbWVDjCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0 mZVZEjH/CaLSTNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MP bXohV+Y0sNsyfuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRc WzW6FvOnm//BAgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAd BgNVHQ4EFgQUehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQI MAYBAf8CAQAwEQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwu dXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwu Y3JsMHQGCCsGAQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29t L1VUTkFkZFRydXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRy dXN0LmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ 9RtaxdKtG3NuPukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1 ei+eupN5yv7ikR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B 080zX5vQvWBqszv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG 1l1XBxunML5LSUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCC BJ0wggOFoAMCAQICEDQ96SusJzT/j8s0lPvMcFQwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMC U0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAg TmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0wNTA2MDcwODA5 MTBaFw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcT DlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsT GGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQg QXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA sjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIxB8dOtINknS4p 1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K 2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bU MSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjK aJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4H0MIHxMB8GA1UdIwQYMBaAFK29mHo0 tCb3+sQmVO8DveAky1QaMB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8E BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDeg NYYzaHR0cDovL2NybC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDUG CCsGAQUFBwEBBCkwJzAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkq hkiG9w0BAQUFAAOCAQEAAbyc42MosPMxAcLfe91ioAGdIzEPnJJzU1HqH0z61p/Eyi9nfngzD3QW uZGHkfWKJvpkcADYHvkLBGJQh5OB1Nr1I9s0u4VWtHA0bniDNx6FHMURFZJfhxe9rGr98cLRzIlf sXzwPlHyNfN87GCYazor4O/fs32G67Ub9VvsonyYE9cAULnRLXPeA3h04QWFMV7LmrmdlMa5lDd1 ctxE+2fo8PolHlKn2iXpR+CgxzygTrEKNvt3SJ/vl4r7tP7jlBSog7xcLT/SYHFg7sJxggzpiDbj 2iC0o6BsqpZLuICOdcpJB/Y7FLrf3AXZn9vgsuZNoHgm5+ctbn9fxh6IFTGCAqswggKnAgEBMIGo MIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdT YWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVu dCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhAsv+VdGX6YsSHI/WRu2j2JMAkG BSsOAwIaBQCggdgwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTUw NDI4MTIxOTM0WjAjBgkqhkiG9w0BCQQxFgQUf6rnm6ptwBbG18Tr2mA2o9HBOBgweQYJKoZIhvcN AQkPMWwwajALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0D BzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw DQYJKoZIhvcNAQEBBQAEggEAk/ungwBOWatVqg79H0PSLGfgCNDk5Lh2FzRpMpqJlfyq+MBBCywd X8uN6CUWWa6D8qMXDVjA8O1ta3D2OlX2LqTVa2XGl9vRJNos1Vf4tNSUMZ4coEKrMfl00fproSVq 2sp2LqeDgfR4GOyqc1ViYPM5MdFc/xXfQcldIfF/g+1DYHQS4ba2oeB9r0R56cpAmWxOGzOUX3Kd JzJ45K8E+hmItVWYHj6g1CyVvoYup2tXr1XJw38g/nKyBp+jqTkmRDmGztz0wJfoopVF8wryBele HQsWE0XhXRFkaRH1lDkVpMuJvojvZVj/37OymdZY9Qr2V4wV/hJV1fQEmezlyQ== --=-0J1G8JZsM31y1lJuUDfE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4294293883-7795>