From owner-freebsd-net@freebsd.org Mon Jun 13 01:39:43 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAE7DAF0553 for ; Mon, 13 Jun 2016 01:39:43 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 8F326296F for ; Mon, 13 Jun 2016 01:39:43 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 2538418275; Mon, 13 Jun 2016 01:39:43 +0000 (UTC) Date: Mon, 13 Jun 2016 01:39:43 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D6689+325+6c89ed8b7a9bc66d@reviews.freebsd.org Subject: [Differential] D6689: tcp/lro: Implement hash table for LRO entries. Message-ID: <1b595f1cb3b0b27d165e6cd6ca9b2bec@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D6689: tcp/lro: Implement hash table for LRO entries. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ZGRkYTdkZDNmZDVlODIxOWE3MGU3NDg3NmVjIFdeDt8= MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2016 01:39:43 -0000 From owner-freebsd-net@freebsd.org Mon Jun 13 01:41:10 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98636AF0648 for ; Mon, 13 Jun 2016 01:41:10 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 5A1392ACB for ; Mon, 13 Jun 2016 01:41:10 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id DAA4A18401; Mon, 13 Jun 2016 01:41:09 +0000 (UTC) Date: Mon, 13 Jun 2016 01:41:09 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D6689+325+6c89ed8b7a9bc66d@reviews.freebsd.org Subject: [Differential] D6689: tcp/lro: Implement hash table for LRO entries. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D6689: tcp/lro: Implement hash table for LRO entries. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ZGRkYTdkZDNmZDVlODIxOWE3MGU3NDg3NmVjIFdeDzU= MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2016 01:41:10 -0000 From owner-freebsd-net@freebsd.org Mon Jun 13 03:04:01 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C503AAF07F1 for ; Mon, 13 Jun 2016 03:04:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B60942CFA for ; Mon, 13 Jun 2016 03:04:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5D3411D021783 for ; Mon, 13 Jun 2016 03:04:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 209758] Broadcom 5717 C not working Date: Mon, 13 Jun 2016 03:04:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2016 03:04:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209758 --- Comment #15 from commit-hook@freebsd.org --- A commit references this bug: Author: sephe Date: Mon Jun 13 03:03:09 UTC 2016 New revision: 301848 URL: https://svnweb.freebsd.org/changeset/base/301848 Log: MFC 300985, 301103 r300985 bge: Support 5717 C0, which is almost same as 5720 A0 PR: 209758 Obtained from: DragonFlyBSD d79f5d8f5fe94cd6769207b2901422977d502bc0 MFC after: 1 week =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D r301103 bge: Force chipid to 5720 A0 for 5717 C0 in an early place Discussed with: yongari MFC after: 1 week Sponsored by: Microsoft OSTC Changes: _U stable/10/ stable/10/sys/dev/bge/if_bge.c stable/10/sys/dev/bge/if_bgereg.h --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Jun 14 02:10:26 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F5DFAF27D8 for ; Tue, 14 Jun 2016 02:10:26 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id C724B24CB for ; Tue, 14 Jun 2016 02:10:25 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 426FE1873C; Tue, 14 Jun 2016 02:10:25 +0000 (UTC) Date: Tue, 14 Jun 2016 02:10:25 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D6689+325+6c89ed8b7a9bc66d@reviews.freebsd.org Subject: [Differential] D6689: tcp/lro: Implement hash table for LRO entries. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , , Thread-Topic: D6689: tcp/lro: Implement hash table for LRO entries. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ZGRkYTdkZDNmZDVlODIxOWE3MGU3NDg3NmVjIFdfZ5E= MIME-Version: 1.0 Content-Type: text/x-patch; charset=utf-8; name="D6689.17569.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="D6689.17569.patch" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2016 02:10:26 -0000 ZGlmZiAtLWdpdCBhL3N5cy9uZXRpbmV0L3RjcF9scm8uaCBiL3N5cy9uZXRpbmV0L3RjcF9scm8u aAotLS0gYS9zeXMvbmV0aW5ldC90Y3BfbHJvLmgKKysrIGIvc3lzL25ldGluZXQvdGNwX2xyby5o CkBAIC00MCw2ICs0MCw3IEBACiAKIHN0cnVjdCBscm9fZW50cnkgewogCUxJU1RfRU5UUlkobHJv X2VudHJ5KQluZXh0OworCUxJU1RfRU5UUlkobHJvX2VudHJ5KQloYXNoX25leHQ7CiAJc3RydWN0 IG1idWYJCSptX2hlYWQ7CiAJc3RydWN0IG1idWYJCSptX3RhaWw7CiAJdW5pb24gewpAQCAtOTUs NiArOTYsOCBAQAogCXVuc2lnbmVkIHNob3J0CWxyb19hY2tjbnRfbGltOwkJLyogbWF4ICMgb2Yg YWdncmVnYXRlZCBBQ0tzICovCiAJdW5zaWduZWQgCWxyb19sZW5ndGhfbGltOwkJLyogbWF4IGxl biBvZiBhZ2dyZWdhdGVkIGRhdGEgKi8KIAorCXVfbG9uZwkJbHJvX2hhc2hzejsKKwlzdHJ1Y3Qg bHJvX2hlYWQJKmxyb19oYXNoOwogCXN0cnVjdCBscm9faGVhZAlscm9fYWN0aXZlOwogCXN0cnVj dCBscm9faGVhZAlscm9fZnJlZTsKIH07CmRpZmYgLS1naXQgYS9zeXMvbmV0aW5ldC90Y3BfbHJv LmMgYi9zeXMvbmV0aW5ldC90Y3BfbHJvLmMKLS0tIGEvc3lzL25ldGluZXQvdGNwX2xyby5jCisr KyBiL3N5cy9uZXRpbmV0L3RjcF9scm8uYwpAQCAtNjgsMTkgKzY4LDI0IEBACiAjZW5kaWYKIAog c3RhdGljIHZvaWQJdGNwX2xyb19yeF9kb25lKHN0cnVjdCBscm9fY3RybCAqbGMpOworc3RhdGlj IGludAl0Y3BfbHJvX3J4MihzdHJ1Y3QgbHJvX2N0cmwgKmxjLCBzdHJ1Y3QgbWJ1ZiAqbSwKKwkJ ICAgIHVpbnQzMl90IGNzdW0sIGludCB1c2VfaGFzaCk7CiAKIHN0YXRpYyBfX2lubGluZSB2b2lk Ci10Y3BfbHJvX2FjdGl2ZV9pbnNlcnQoc3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IGxyb19l bnRyeSAqbGUpCit0Y3BfbHJvX2FjdGl2ZV9pbnNlcnQoc3RydWN0IGxyb19jdHJsICpsYywgc3Ry dWN0IGxyb19oZWFkICpidWNrZXQsCisgICAgc3RydWN0IGxyb19lbnRyeSAqbGUpCiB7CiAKIAlM SVNUX0lOU0VSVF9IRUFEKCZsYy0+bHJvX2FjdGl2ZSwgbGUsIG5leHQpOworCUxJU1RfSU5TRVJU X0hFQUQoYnVja2V0LCBsZSwgaGFzaF9uZXh0KTsKIH0KIAogc3RhdGljIF9faW5saW5lIHZvaWQK IHRjcF9scm9fYWN0aXZlX3JlbW92ZShzdHJ1Y3QgbHJvX2VudHJ5ICpsZSkKIHsKIAotCUxJU1Rf UkVNT1ZFKGxlLCBuZXh0KTsKKwlMSVNUX1JFTU9WRShsZSwgbmV4dCk7CQkvKiBhY3RpdmUgbGlz dCAqLworCUxJU1RfUkVNT1ZFKGxlLCBoYXNoX25leHQpOwkvKiBoYXNoIGJ1Y2tldCAqLwogfQog CiBpbnQKQEAgLTk1LDcgKzEwMCw3IEBACiB7CiAJc3RydWN0IGxyb19lbnRyeSAqbGU7CiAJc2l6 ZV90IHNpemU7Ci0JdW5zaWduZWQgaTsKKwl1bnNpZ25lZCBpLCBlbGVtZW50czsKIAogCWxjLT5s cm9fYmFkX2NzdW0gPSAwOwogCWxjLT5scm9fcXVldWVkID0gMDsKQEAgLTExMCw2ICsxMTUsMTgg QEAKIAlMSVNUX0lOSVQoJmxjLT5scm9fZnJlZSk7CiAJTElTVF9JTklUKCZsYy0+bHJvX2FjdGl2 ZSk7CiAKKwkvKiBjcmVhdGUgaGFzaCB0YWJsZSB0byBhY2NlbGVyYXRlIGVudHJ5IGxvb2t1cCAq LworCWlmIChscm9fZW50cmllcyA+IGxyb19tYnVmcykKKwkJZWxlbWVudHMgPSBscm9fZW50cmll czsKKwllbHNlCisJCWVsZW1lbnRzID0gbHJvX21idWZzOworCWxjLT5scm9faGFzaCA9IHBoYXNo aW5pdF9mbGFncyhlbGVtZW50cywgTV9MUk8sICZsYy0+bHJvX2hhc2hzeiwKKwkgICAgSEFTSF9O T1dBSVQpOworCWlmIChsYy0+bHJvX2hhc2ggPT0gTlVMTCkgeworCQltZW1zZXQobGMsIDAsIHNp emVvZigqbGMpKTsKKwkJcmV0dXJuIChFTk9NRU0pOworCX0KKwogCS8qIGNvbXB1dGUgc2l6ZSB0 byBhbGxvY2F0ZSAqLwogCXNpemUgPSAobHJvX21idWZzICogc2l6ZW9mKHN0cnVjdCBscm9fbWJ1 Zl9zb3J0KSkgKwogCSAgICAobHJvX2VudHJpZXMgKiBzaXplb2YoKmxlKSk7CkBAIC0xNDcsNiAr MTY0LDEzIEBACiAJCW1fZnJlZW0obGUtPm1faGVhZCk7CiAJfQogCisJLyogZnJlZSBoYXNoIHRh YmxlICovCisJaWYgKGxjLT5scm9faGFzaCAhPSBOVUxMKSB7CisJCWZyZWUobGMtPmxyb19oYXNo LCBNX0xSTyk7CisJCWxjLT5scm9faGFzaCA9IE5VTEw7CisJfQorCWxjLT5scm9faGFzaHN6ID0g MDsKKwogCS8qIGZyZWUgbWJ1ZiBhcnJheSwgaWYgYW55ICovCiAJZm9yICh4ID0gMDsgeCAhPSBs Yy0+bHJvX21idWZfY291bnQ7IHgrKykKIAkJbV9mcmVlbShsYy0+bHJvX21idWZfZGF0YVt4XS5t Yik7CkBAIC00ODcsNyArNTExLDcgQEAKIAkJfQogCiAJCS8qIGFkZCBwYWNrZXQgdG8gTFJPIGVu Z2luZSAqLwotCQlpZiAodGNwX2xyb19yeChsYywgbWIsIDApICE9IDApIHsKKwkJaWYgKHRjcF9s cm9fcngyKGxjLCBtYiwgMCwgMCkgIT0gMCkgewogCQkJLyogaW5wdXQgcGFja2V0IHRvIG5ldHdv cmsgbGF5ZXIgKi8KIAkJCSgqbGMtPmlmcC0+aWZfaW5wdXQpKGxjLT5pZnAsIG1iKTsKIAkJCWxj LT5scm9fcXVldWVkKys7CkBAIC01NjEsOCArNTg1LDggQEAKIH0KICNlbmRpZgogCi1pbnQKLXRj cF9scm9fcngoc3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IG1idWYgKm0sIHVpbnQzMl90IGNz dW0pCitzdGF0aWMgaW50Cit0Y3BfbHJvX3J4MihzdHJ1Y3QgbHJvX2N0cmwgKmxjLCBzdHJ1Y3Qg bWJ1ZiAqbSwgdWludDMyX3QgY3N1bSwgaW50IHVzZV9oYXNoKQogewogCXN0cnVjdCBscm9fZW50 cnkgKmxlOwogCXN0cnVjdCBldGhlcl9oZWFkZXIgKmVoOwpAQCAtNTc4LDYgKzYwMiw3IEBACiAJ dGNwX3NlcSBzZXE7CiAJaW50IGVycm9yLCBpcF9sZW4sIGw7CiAJdWludDE2X3QgZWhfdHlwZSwg dGNwX2RhdGFfbGVuOworCXN0cnVjdCBscm9faGVhZCAqYnVja2V0OwogCiAJLyogV2UgZXhwZWN0 IGEgY29udGlndW91cyBoZWFkZXIgW2VoLCBpcCwgdGNwXS4gKi8KIApAQCAtNjcwLDggKzY5NSw0 MSBAQAogCiAJc2VxID0gbnRvaGwodGgtPnRoX3NlcSk7CiAKKwlpZiAoIXVzZV9oYXNoKSB7CisJ CWJ1Y2tldCA9ICZsYy0+bHJvX2hhc2hbMF07CisJfSBlbHNlIGlmIChNX0hBU0hUWVBFX0lTSEFT SChtKSkgeworCQlidWNrZXQgPSAmbGMtPmxyb19oYXNoW20tPm1fcGt0aGRyLmZsb3dpZCAlIGxj LT5scm9faGFzaHN6XTsKKwl9IGVsc2UgeworCQl1aW50MzJfdCBoYXNoOworCisJCXN3aXRjaCAo ZWhfdHlwZSkgeworI2lmZGVmIElORVQKKwkJY2FzZSBFVEhFUlRZUEVfSVA6CisJCQloYXNoID0g aXA0LT5pcF9zcmMuc19hZGRyICsgaXA0LT5pcF9kc3Quc19hZGRyOworCQkJYnJlYWs7CisjZW5k aWYKKyNpZmRlZiBJTkVUNgorCQljYXNlIEVUSEVSVFlQRV9JUFY2OgorCQkJaGFzaCA9IGlwNi0+ aXA2X3NyYy5zNl9hZGRyMzJbMF0gKworCQkJICAgIGlwNi0+aXA2X2RzdC5zNl9hZGRyMzJbMF07 CisJCQloYXNoICs9IGlwNi0+aXA2X3NyYy5zNl9hZGRyMzJbMV0gKworCQkJICAgIGlwNi0+aXA2 X2RzdC5zNl9hZGRyMzJbMV07CisJCQloYXNoICs9IGlwNi0+aXA2X3NyYy5zNl9hZGRyMzJbMl0g KworCQkJICAgIGlwNi0+aXA2X2RzdC5zNl9hZGRyMzJbMl07CisJCQloYXNoICs9IGlwNi0+aXA2 X3NyYy5zNl9hZGRyMzJbM10gKworCQkJICAgIGlwNi0+aXA2X2RzdC5zNl9hZGRyMzJbM107CisJ CQlicmVhazsKKyNlbmRpZgorCQlkZWZhdWx0OgorCQkJaGFzaCA9IDA7CisJCQlicmVhazsKKwkJ fQorCQloYXNoICs9IHRoLT50aF9zcG9ydCArIHRoLT50aF9kcG9ydDsKKwkJYnVja2V0ID0gJmxj LT5scm9faGFzaFtoYXNoICUgbGMtPmxyb19oYXNoc3pdOworCX0KKwogCS8qIFRyeSB0byBmaW5k IGEgbWF0Y2hpbmcgcHJldmlvdXMgc2VnbWVudC4gKi8KLQlMSVNUX0ZPUkVBQ0gobGUsICZsYy0+ bHJvX2FjdGl2ZSwgbmV4dCkgeworCUxJU1RfRk9SRUFDSChsZSwgYnVja2V0LCBoYXNoX25leHQp IHsKIAkJaWYgKGxlLT5laF90eXBlICE9IGVoX3R5cGUpCiAJCQljb250aW51ZTsKIAkJaWYgKGxl LT5zb3VyY2VfcG9ydCAhPSB0aC0+dGhfc3BvcnQgfHwKQEAgLTc3OSw3ICs4MzcsNyBAQAogCS8q IFN0YXJ0IGEgbmV3IHNlZ21lbnQgY2hhaW4uICovCiAJbGUgPSBMSVNUX0ZJUlNUKCZsYy0+bHJv X2ZyZWUpOwogCUxJU1RfUkVNT1ZFKGxlLCBuZXh0KTsKLQl0Y3BfbHJvX2FjdGl2ZV9pbnNlcnQo bGMsIGxlKTsKKwl0Y3BfbHJvX2FjdGl2ZV9pbnNlcnQobGMsIGJ1Y2tldCwgbGUpOwogCWdldG1p Y3JvdGltZSgmbGUtPm10aW1lKTsKIAogCS8qIFN0YXJ0IGZpbGxpbmcgaW4gZGV0YWlscy4gKi8K QEAgLTgzNyw2ICs4OTUsMTMgQEAKIAlyZXR1cm4gKDApOwogfQogCitpbnQKK3RjcF9scm9fcngo c3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IG1idWYgKm0sIHVpbnQzMl90IGNzdW0pCit7CisK KwlyZXR1cm4gdGNwX2xyb19yeDIobGMsIG0sIGNzdW0sIDEpOworfQorCiB2b2lkCiB0Y3BfbHJv X3F1ZXVlX21idWYoc3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IG1idWYgKm1iKQogewoK From owner-freebsd-net@freebsd.org Tue Jun 14 15:13:44 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DAD8B69E97 for ; Tue, 14 Jun 2016 15:13:44 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 939352633; Tue, 14 Jun 2016 15:13:42 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.200.206] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 3D1911928BE; Tue, 14 Jun 2016 15:13:39 +0000 (UTC) From: Sean Bruno Subject: lagg(4): LOR, deadlock and panic To: "freebsd-net@freebsd.org" Message-ID: <459d2639-b490-beee-9cd4-05f38983eaed@freebsd.org> Date: Tue, 14 Jun 2016 08:13:34 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Is90jhGO75kilTfiCxPNF5lta6GvJPfeR" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2016 15:13:44 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Is90jhGO75kilTfiCxPNF5lta6GvJPfeR Content-Type: multipart/mixed; boundary="kUdht4diC0KWBgcQ6tLskKqTDJCRT2ow6" From: Sean Bruno To: "freebsd-net@freebsd.org" Message-ID: <459d2639-b490-beee-9cd4-05f38983eaed@freebsd.org> Subject: lagg(4): LOR, deadlock and panic --kUdht4diC0KWBgcQ6tLskKqTDJCRT2ow6 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable tl;dr --> https://reviews.freebsd.org/D6845 Navdeep and I have been poking at an LOR that seems to be popping up in -current that is related to lagg(4) and lagg_get_counter(). root@sysdev07:~ # ifconfig lagg0 create laggport ix0 laggproto lacp 192.168.100.11/24 lagg0: link state changed to DOWN root@sysdev07:~ # ifconfig ix0 up lock order reversal: 1st 0xfffff8002d7c9190 if_addr_lock (if_addr_lock) @ /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1717 2nd 0xfffff800271a5808 if_lagg rmlock (if_lagg rmlock) @ /usr/home/sbruno/fbsd_head/sys/modules/if_lagg/../../net/if_lagg.c:1057 stack backtrace: #0 0xffffffff80aa5ab0 at witness_debugger+0x70 #1 0xffffffff80aa59a4 at witness_checkorder+0xe54 #2 0xffffffff80a42521 at _rm_rlock_debug+0x111 #3 0xffffffff82222b2c at lagg_get_counter+0x4c #4 0xffffffff80b2ebd1 at if_data_copy+0xa1 #5 0xffffffff80b533bc at sysctl_rtsock+0x56c #6 0xffffffff80a53f0a at sysctl_root_handler_locked+0x8a #7 0xffffffff80a536c8 at sysctl_root+0x188 #8 0xffffffff80a53cbe at userland_sysctl+0x16e #9 0xffffffff80a53b14 at sys___sysctl+0x74 #10 0xffffffff80eb5b3b at amd64_syscall+0x2db #11 0xffffffff80e95c4b at Xfast_syscall+0xfb Running a netstat -w 1 in the backgrouund while repeatedly creating destroying the interface lagg0 will lead to either a panic or a deadlock:= e.g. netstat -w 1 > /dev/null & while [ 1 ]; do ifconfig lagg0 destroy ifconfig lagg0 create laggport ix0 laggproto lacp 192.168.100.11/24 done When the system deadlocks on the console, kdb sees the locks held like this: KDB: enter: Break to debugger [ thread pid 11 tid 100007 ] Stopped at kdb_alt_break_internal+0x18e: movq $0,kdb_why db> show allocks No such command db> show alllocks Process 2173 (ifconfig) thread 0xfffff8002d125a00 (100186) exclusive rm if_lagg rmlock (if_lagg rmlock) r =3D 0 (0xfffff8002717e408) locked @ /usr/home/sbruno/fbsd_head/sys/modules/if_lagg/../../net/if_lagg.c:1530 exclusive sleep mutex in6_multi_mtx (in6_multi_mtx) r =3D 0 (0xffffffff81d7e288) locked @ /usr/home/sbruno/fbsd_head/sys/netinet6/in6_mcast.c:1142 Process 792 (netstat) thread 0xfffff80027e67a00 (100167) shared rw if_addr_lock (if_addr_lock) r =3D 0 (0xfffff80103e95190) locked @ /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1717 shared rw ifnet_rw (ifnet_rw) r =3D 0 (0xffffffff81d7b760) locked @ /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1713 exclusive sleep mutex Giant (Giant) r =3D 0 (0xffffffff81d55e08) locked @ /usr/home/sbruno/fbsd_head/sys/kern/kern_sysctl.c:164 This looks like the netstat is causing a call into the counter function while the destruction or creation is ongoing. Removing the LAGG_RLOCK() calls from lagg_get_counter() makes the deadlock, LOR and panic go away, however this can't be that easy. I'm unsure what the RLOCK is for in lagg_get_counter(). It appears that there is a higher lock in the ifnet access that is protecting simultaneous access already, but I'm very ignorant of what's going on here. I don't see any other driver with locks in its get_counter() functions, so I'm not sure what the best course of action here is. Sean --kUdht4diC0KWBgcQ6tLskKqTDJCRT2ow6-- --Is90jhGO75kilTfiCxPNF5lta6GvJPfeR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJXYB8iXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kxXAIAMct0GyKd0fgQfCpxzwCuOHE Wr2sH1wjaVIhj3tRBYFvpd9OcAb5UKTUX1qyiOJrn6LJDzetKmZbiTblGDcteJx/ bCp+Zq+/dxD5FoxJEWqLDLXFipdo2i6xX+rJ9zvIOt1gmzhLuesU40lM0cVFTSZA BMO+a6362ECT7OCNyPUK8Bo5WrLBp0rwbdbsybNFl9anB0A9CXy1Kk9hMcueuGdd QjRJ5e3kmIzEkjbX97v52+s2inLSXSNuIBmzxYk5nYuTgWwf2jyef+rel/dLKr6e LwZYoK1SlMSnpG3dHxNwCkfEndVSkU2XNQvxxUxwTPzmQb0cdzBaNgb1RN3/ewk= =pc1V -----END PGP SIGNATURE----- --Is90jhGO75kilTfiCxPNF5lta6GvJPfeR-- From owner-freebsd-net@freebsd.org Tue Jun 14 15:14:57 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A10BB69F15 for ; Tue, 14 Jun 2016 15:14:57 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 74B0E2718 for ; Tue, 14 Jun 2016 15:14:57 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: by mailman.ysv.freebsd.org (Postfix) id 7010DB69F14; Tue, 14 Jun 2016 15:14:57 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FA8CB69F13 for ; Tue, 14 Jun 2016 15:14:57 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: from mail-out1.informatik.tu-muenchen.de (mail-out1.informatik.tu-muenchen.de [131.159.0.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14BD62717 for ; Tue, 14 Jun 2016 15:14:55 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: (Authenticated sender: schoeffm) by mail.in.tum.de (Postfix) with ESMTPSA id D9B771C1131 for ; Tue, 14 Jun 2016 17:08:54 +0200 (CEST) From: Dominik Schoeffmann Subject: Netmap Checksum Offloading To: net@freebsd.org Message-ID: <57601DFA.2040008@in.tum.de> Date: Tue, 14 Jun 2016 17:08:42 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="c2ilmUOL4VmDEpGuPjGWmDn46sh7vDw5d" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2016 15:14:57 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --c2ilmUOL4VmDEpGuPjGWmDn46sh7vDw5d Content-Type: multipart/mixed; boundary="ojWM2RfT27D5rxalIH6m7sQf9oPQ8T9jk" From: Dominik Schoeffmann To: net@freebsd.org Message-ID: <57601DFA.2040008@in.tum.de> Subject: Netmap Checksum Offloading --ojWM2RfT27D5rxalIH6m7sQf9oPQ8T9jk Content-Type: multipart/mixed; boundary="------------070703060403080105060100" This is a multi-part message in MIME format. --------------070703060403080105060100 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear Netmap Developers, during the course of my bachelor's thesis, I modified a packet generator called MoonGen [1] in order to utilize netmap. One key component was to flexibly offload checksums for different kinds of packets (IPv4, UDP, TCP). The ixgbe netmap patch was modified [2] in order to construct context descriptors and suitable data descriptors. This is implemented in less than 250 LoC (including pseudo-header calculations). The man page states, that checksum offloading is available via ethtool, although a solution inside the netmap API might be a cleaner way for applications to actually use these features. Attached is a graph showing the performance implication of using offloading in the current implementation. As can be seen, offloading has only a minor impact. When regarding this data (and comparing it to other frameworks), please keep in mind, that internally a lot of per-packet effort is needed due to the software architecture of the packet generator. The question being: Would it not make sense to include checksum offloading inside of netmap in order to accomodate applications operating on layer 3 and above? As these programs need to calculate the checksums in software, it would be just as fast to move these calculations to the kernel for NICs without checksum offloading support (and the kernel would act as a librar= y). The problem which currently is imposed by the fact, that netmap exports the complete ring, is that context descriptors disrupt the data descriptors, which is unpleasant for the application. But you may find the data interesting nevertheless. Best Regards, Dominik Schoeffmann [1] https://github.com/dschoeffm/MoonGen/tree/netmap [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading --------------070703060403080105060100 Content-Type: application/pdf; name="netmap-offloading.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="netmap-offloading.pdf" JVBERi0xLjUKJbXtrvsKMyAwIG9iago8PCAvTGVuZ3RoIDQgMCBSCiAgIC9GaWx0ZXIgL0Zs YXRlRGVjb2RlCj4+CnN0cmVhbQp4nJWbyY4ktxGG73yKOlbB6FRyTfI6kGFYgAHLHsOHgQ5C abM8I1mSN/jpHcFYWV2t6YIh90R08CeDZJIfmdk/hX07ytjT6fbnn353+uTL/fTtLxCRR6+n /4T99Bn893149wVE7aevQtxPfzi1tB0tnVLfWoqnD2rXuNXeTu9P49hG7hYgtgbEnLcYc7AQ 9viY1rcErbAQcWhI2vNWWuSQADHqsZgMv8yHyajDQo689X1oSDCPxuS9b6MXk1GHhZSyxeYi ph1cxJ9DLluLrjH5SFsu2lUoMgNq23JuLoAdEhBOvWx79RLsUIkY43YcLoAdFpDHVnoJPoRc LqYdG/SpD2GPhYyyjdix2zSGXRqTUoI+9zLisZAKs64mCwnqwo7bt7rHASP7/B8wb/ctjpFw xr4wN3Pa+oGdczNXbfKWAzo46eQNz2avStzMZpveKqGTWeeleFTkdr67J8Bkbp6AoB6TuXkm 3FOiMutTYg9JMJnbx8Y9SCbz0oNEKsGHmIx4TObmYVOHNOb502cxJvPi86g5sSM4GS1kMnef WSfCvyeJYBEmoQ91xl+UOYG3UobY73XOW0AbW8VqNIIfanj8jv1wEmQ7CQ0QCY2gRpS6pRpM YdpOgGwrr7/ndQc2g5G1eBCHCVgEK1gESbQIfTesBWQHJ2ERLMEOk4CSKTkJst/b0mARIqEh JAGBo1WTYNsaQY7gJCyEJTqsMN1JkO0kNIAUggshiQ7+4STYNgkL4EaQI5jEqFss0STYNgkL YAmL4FkV931r3c1McZiIC2EVF8Pr/N62lI7gZaZnkSGPl9EY2ZbgwYFlbm4XJMMeJyMxJmMx LAP7R6/NzVTxOBmLERmLEZkOrONmKzvCIqMxKkMet5XmbRQ3Y8Xx3m2KFiMyEqQy8DTuyT3/ 4nAz38WIjAWxDDyS1auQ7TKaDj/3LUQk2tyMnAY5vIiEBFPRIMGHuDU3SupwMhYiT5ELYhmI PYqXYYeTsRBpDHmCl8Gt0Q82O7yMhqgMxwSV6bAZZT/Y7HAyFiIyFiNPJTzq+ahehhxeRkNU RmMYm+AxnXu5yrDHyYjHZFyMyLRt34t7KsXjZTjGyWgMywBi1nK4p1I8TsZiRMZiRObYdjdO ZIdFRCJUYzoccfYtHm7KiMM9Si5GVCyIZeARPaJvCjtcU8jjnyYXJDJjSy15GXJ4GQ4JTkaD WAbYoy8dww4nYyHyOFnQnfNH33o73PFjPY+gCvTInm4PKMGdUEjCDijricUkbo4wfGAJJmEn mPVIYxIvn3GmQnBnnPXQYxIvnIKsFXosciIWIyq/clBiGTknuR3AYkTmxcOUythRSncAixGZ Xzlv8QDbcWs9gAU/xr9yJmMZPZGpjIQEk/HHthP+b57Q+EpBpQnKM8zkI5+esG6kr5+/1lI/ f3t68zbEfRv7iNMj/247MEXNucB6uO3HHkHo7YfwyTdP+xNEnd5+E96d0+UptZbPhX92+Fny fo6XcZwbGHH0c76MBpFfvP0sxLr1PvY0oKK3X4FAw0AoDYfMRqUSRoNOPHIio7IYeDIaGoiK UBKmaxpAx8BSre5AZaB83i9vv4cc01ZzK7FOX1Jf2VumuGI+mGt9+tqduI6+J1zR6w4w73/3 jtqzz/ZwlTll96tkv7otVdyvpAHvpP/wV799Gz4//fSKIf3r6YfwU+BBhP+gsTnB6f10hVGb d06nT/7bTp/+GD7Hwd/a0UvGI/6AeVYrTgW5mrrzW5hecYuj1vzylVXZjgrTBU61DZ8cteFn hwkFm3PLYFYQT3InIOYVDyitFXEEnIDHYaXZmtpXNaHbjm5F4WfvJj3NgOFSNwdIy7j40vDr 6btwm8qbl3rsha4aA6AzwgrWYDLM+w1xHLDEDzT3gssgnGVSmZcbpap5PY224VwIGlC3AT2v xcUk+as56laylI4wZ2oerB7Uvlr1HCCNo9Jr27E7nqXzaH+gdJwrOED9TneR4oFFfi72ZZ5h 4gHrzsh0UwMPmthX8CRE8OBCYEbCUmoSYlMlV+ehLYAlwIZtEmFHayHHLMLtYI+1UyTWTLB7 nmf3cP/MlHAnagA1CKPmObbu+2n0eRlBdVW1r9LC4EIghZichNhUydV56rLB4l53pBxcLdMx i0g7yGPtZImbTGb/PMvu0f5JsCgUWO8S7k6D7s7EA0M6Ctgwl3ua9uid7ta6mldwwGba8Tpc IyJEdKcgNtVxNc8+tj6qSEwbkDxoJWTPEtwK9mgrVWHNA3vneW4P904Bd0W22Oda+cF56NYH KXfewMJ2cRyFrgybmtBwWFlqS8FFNIjoTkFsquNqHtiB8QcpgAmzr+OphOsgexbgRrBHG8kC N1nMvnmW2cN9cwBy5DZHuUxINU+eq386ylx9Uyywuw+6By1FbWj4POVi52gIdMpsuUiITZVc nSfOsyRLzEGe4Gu1TMcsIu0gj7VTJNZMZv88y+7R/smgvkfqjZ4m8KoH2pPB3OcSmZDd0inj zE5ZzCs4YHtJs3M4AA6JUIkUV5P0r84DXQIrLQvMPCGX4KqYDiyibSCPNZElbpLArnme2MNd UxoSD40y3UGqB2gMr9NLncsjpg7kOe+tcf0UG1peMvJ6cCFpgzSchNhUydV5YEyjKNABHo+a Vsl0zBLSjMhX3tJMVljzmL3zLLc3wb/3wW5ZbOgPhaJI1wKOoiJClTBW3vbu+I5Mx3fTERzC zdJsTW3hO+SOrniH1ALrNAuzdQ1SsfyaWiVFlzYvaMdZfDxvpZ+OLyUcy40t1a4wBUtVhCde WI5NZTl2BGM1Li7mVL961NvrMJZr8zWGkuI0g2c5DjDQnMWXti8sx9l8PH/FG1hs6KbMPG1u XQZvx5bb4eCN7AXeJITRTCXEpkocvMG6CGujwRusnHMjt1qmw8Mbe6ydIrFmssCbZveKHlGg GXx5Ih689MNhF0xChNhLVpAS2+GahRCMmYTAGVVyXYBuzItewbU52B7XaPQ9rpHHY+WUuMlk xTXJ7uM9YhBTZ+M9oDWY5x7Qji3iNieARuYCaBIhQCYKYld+tMxDd1YKaLCqzVs5qSPT7Lc2 TIc1UYqvSax0Jom9ojOUWuq8QfQ8BicorEl5DO8Yk/EYmQuPSYTwlyiITXU4HkuFLzsZyBJ0 Ar6J0Eqm7YGMPdZKVrjJYyUyye0V/SGUArnkVDyDpT5fKSr7pAEz/DA6YtszmIYwYamE2FSJ Y7CEr8eyMRg0PefkGIwcnsHYY+0UiTWThcE0u4/3iMIJ6MJS7akLVBJ+CSHIg7flcORXKGLb g5eGMFiJhNpUiUMvSG/Pw9ALBjziFYHVUviptHaQx9rJEjeZLOil2b2iRwRI4PzBbzfVA4t5 6kY50BI4MhgHse1hS0MYpVRCbKrEwRYcczBNpS04cKRcHG2Rw9MWe6ydIrFmsuCWZocwyl1B //CAVXndVUeba48S1jE3DCWsg7cU5qSgAcxUXJqtygu93aghb3g6g32CtIPa1wXvMMBuyGbx tekLaFXdRZ6nLDQC3V5HcXAF3d6gwxSu6rz2VLgi012USQDTkxQXk+SNrmDYy0jGZpD9nuym jG2jKwmQ1nHxtfUerzShe1kbc4xt4am+z2qUYzrsb6MZ6bDteUpDmJZUQugJq/A01fmyTGjq wANAWJiNTgme6tBjrRYJn8XKUpTX3dyVlPrW95WdoO92x06RvnpRdmLbs5OGCDuJhNhUydV5 DpgZxk74erXvjp3Y4dhJPNZOlrjJxLOTZXevDxQqcOONHpZg4aqzlQwquMvOJhDIsOlhSSMY hVRB7FmFYyVYgvArD2UlWLZK7AZLZHtaYo81UhSWLBZaksTuZu8QAh/SFY9wOD0e4X29w6NG l3kejyjC8IgUDI+wjhWP8Mn0eNTGWPAIR3fFI/R4PJoKN3nc4hHldrcHHDLstd4AUaxtASJc gDwQoX0DRBRiQEQSBkRYiQciWgU9EO0tr0AEjhsgQo+1UyTWTG6BiLK71wceeFpbLp4ggaM1 hx4DB8bYZJoLAHGA0A2XdzzU2nL1hBDd3NUTjtbhr57IsfAPeTzbTImbNFb+kdTudoAyAX3a 6olnEFspaYx5laYoMs2FdzhAYEbKi001eNzBjc/RTqWNz6qYjoV2yGNtZIU1iRV2JDFIH/q5 ljGv2ur8Eh674pnTQ1Cf5xwHQWM+WcIhcZ/fswgEsekgSAIYc7g0WyRuEASHLA9BeMbyEES2 gyAOkKZx8bXpCwRJNo90hbBE3fnrQ3XECfgCJzXNrV3giE0HRxLA8CPFxSR5B0c0fRSOYD0p 7jUi2w6OOEDQh4uvrfdwpAk90hsKFyPTl3fmKPMVsvIK5FGbe4XItqcmDWEqUgmxZx0Om0aa N/+KTSPizb/DJnJ4bGKPNVMkljwWbpLUHuoXQY44n0jHU/GYnz4aT/X5eZzxFNkLT0kI05JK iD3rcDgV63Zk++wGr1L6/IBGK5kOj1PssWayxJrHQlOS2iP9YizStr0v7wxhYYrdXUkl/MLF 3hmyuVCWRAhEiYLYVMd14bD5GbxiVsHraYdZhe6vXSsKv4c0FiSFNY+VsyS3h3pGIQVpuXkA g+m+793QB+yO6MNsxKYHMI1gvFIFwS2qwwMYonF1AAZzLJbgKQ/sBcDIY+1mhZs8VgCT3B7q GUGXLJ9qmyduB/7JhwARLATIA4pMbHsw0xDGLpUQmyrxYDbmKdbADHpkLDdV07GAGXkMukRi zWQBM83ukb5RqsElkT7UU0+iv5gRVsI18XDExrZHNg1hIhMJtakSx2zQaJzqxmwDKCOFhQvj 7BsPjuixdrLETSYLs2l2D/WNIA+UxstJx3IZ38EV4yhIPO/u9optT3MawrCmEmJHvhQ1D6yT 0d9eDbyKDAsy0n2lZ0r0WDtFYs1kATrN7s38Jk0/IeTvxl74khCvKPbW8VPChPv/3S8J31xy PX956cf5n5e4n6/4md93lydYE86/XGJq57+B0c//uxzx/DV/MycVPFmtCH1jn18tJvxCrnFl 785/uORy/selJPq/X+TjOmgf/I/eAt98Rnn/M7cE+/t83whQhbcCuBXAceuptnnx8vPXL7xo Xr6gw7WuwOPb8/wCC9aZWidDsoO/63zoI5A+4coUxQFoU+I08R6u1/kNV4IxxRdAbMKCAhNj 7twaAAtMPaw4m5m/IVMHTM75h3mz+KnTZ+YsH8S+Wv0SwK2T4kvraSW/SejNR79ZTRXqKAWH v+NEuD/V/nJ5ivn86eWp1PMf4d9tP/8GZ1o+/x5+oG/s53/PIPwytWacjRVmIzi+vhxtmuH8 98tT5bn5r0vazx9sRt28/uCxBpjJ/XBjzQ4c64+/UqLOMA1xRPpYhHsXvxjGr3V4dNl0o6sB NHpanE2St9EFZqM/dKTRRagb0UaXbRtdDaDWafGl9X50NaFXjC5sgrUPHF38S7R8vHZ07wzf y6NnlzQ8bBXfpkQ3bOygR/T55RblZaXE0emNEndUHfRCiQeKTTdQGkADocXZJHkbKPyroDR0 oKCpaT5WPFBs20BpALdOii+t9wOlCb1moAosiREHqsL+0/PdgXrgYfuV0bpLcTxy0DUI7TZy 7KC/dX49LVMHmJo46kQY6VHo6l0XPrZ0PMkOOlxals0iJw5xwE55HFY6TXBR8WkGN5wawE2T 4kvT/XBqNq8Zzr6lNlfVgq+4xt3h/AG37R9hCHuR3TsmHcKGQ9juDuHn4f+AycSMCmVuZHN0 cmVhbQplbmRvYmoKNCAwIG9iagogICA0Mjk0CmVuZG9iagoyIDAgb2JqCjw8CiAgIC9FeHRH U3RhdGUgPDwKICAgICAgL2EwIDw8IC9DQSAxIC9jYSAxID4+CiAgID4+CiAgIC9YT2JqZWN0 IDw8IC94NiA2IDAgUiA+PgogICAvRm9udCA8PAogICAgICAvZi0wLTAgNSAwIFIKICAgPj4K Pj4KZW5kb2JqCjcgMCBvYmoKPDwgL1R5cGUgL1BhZ2UKICAgL1BhcmVudCAxIDAgUgogICAv TWVkaWFCb3ggWyAwIDAgMzc1LjM4MzM2MiAyNTcuNjc5ODEgXQogICAvQ29udGVudHMgMyAw IFIKICAgL0dyb3VwIDw8CiAgICAgIC9UeXBlIC9Hcm91cAogICAgICAvUyAvVHJhbnNwYXJl bmN5CiAgICAgIC9JIHRydWUKICAgICAgL0NTIC9EZXZpY2VSR0IKICAgPj4KICAgL1Jlc291 cmNlcyAyIDAgUgo+PgplbmRvYmoKNiAwIG9iago8PCAvTGVuZ3RoIDkgMCBSCiAgIC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlCiAgIC9UeXBlIC9YT2JqZWN0CiAgIC9TdWJ0eXBlIC9Gb3JtCiAg IC9CQm94IFsgNjIgMTAxIDM0NSAyMzMgXQogICAvUmVzb3VyY2VzIDggMCBSCj4+CnN0cmVh bQp4nF3STUrGQAwG4P2cIid4ze9k5gSC4EJdigtBEEQX4sLrm/aTOnVRGl7C5Jk2n42J0XO4 SRXTUiPo/pqunplev5pAZoTRd/Xd1PPWHp+qj+mlCdMtdUV2JWGnD5qJ6UFiHSOU3qsyiFbS B7rKllRlXEn1skUlygYfjVQEJn1LrLo7qdZw3o7RNAz3aqlpMysxnmC1ShziY0vcUUdRU6lm 3aKHtl0pWGavYrvUoRW4rOCB/ZDDm4E+4+SdsJSTN+BpqzeQuXI10T1WrnZ45MJVVYRn27lM /52xD/2DumGKn6SzbrtAtUaJnqBqODs7+PKhF6jlGVoDxzhBBzTGLxRTwydtu1Pv+kkremCc 0MHwLit61ty5roNK4kI60PWleN0GY5RqVZuA2Vd1JXFJjm2oqFvu6rv2A9OFk9gKZW5kc3Ry ZWFtCmVuZG9iago5IDAgb2JqCiAgIDMxNQplbmRvYmoKOCAwIG9iago8PAogICAvRXh0R1N0 YXRlIDw8CiAgICAgIC9hMCA8PCAvQ0EgMSAvY2EgMSA+PgogICA+Pgo+PgplbmRvYmoKMTAg MCBvYmoKPDwgL0xlbmd0aCAxMSAwIFIKICAgL0ZpbHRlciAvRmxhdGVEZWNvZGUKICAgL0xl bmd0aDEgNjgyMAo+PgpzdHJlYW0KeJzlWIl/U1W+/527ZWna3jRJtxRIGprypJ0W0haoHYQn RZERkUVTpEClCGWolrZCUVbZQihWEcFh9436KC7cXh0hRZ36hrJ05jFVGRVEBlzeKFOfTwcF aXM7v3PSNGnB+Qfezefec353Off7+/7WGyAAYITVwINjbmVZ1Se79q4GsGYBcDPmLql1DB4r fQmQ6MO7fvVQ1fzK+VdmvozyGQDy1vxFyx56/tbbvsdrrwBYVi+YV1Z+bfzLcQDJT+G5ggV4 wtgs/h7lkygPXlBZW+ebEDcf5Q6Uly96ZG7ZPYumLQdIWYDymMqyuiruHu4nlA+g7KiqnldV u78B702hz38CIpQD8ENEBdFKYAATxMNi1SSbzQmjFJOsQDs9SuxoYMfYdlCg2KtwOfYmLm10 CRMABZBHl6gCB/ikKoYGXWjQs6HJaLqimkJnYtmgcHJTnOlK7jCn0+zkzYSYCe8k+cTJDwkW cccKtP/VmonpC47XNMIFg6Jy/QVRF1zBLe00c3XBWdwsHzcLtSBQCiB4xGOQBHeHwMmIR74B XBIxDvU6P7ZfKVG4dpWXkygGWVYNkfOyrJjaVVuSjJcornxSUJCf53alSzZX3mDP8ESblRCn 4OkaT65VTffX+H8TIPzZP3Yg1Me4T9Zzuav2Ta/euqf+1LW/NH2kfaSVID4O8hHfZOTYCHEw IYTQgKAMYYQCCgJFSAwCxURyVC6CCdknshLTruqlGEYeuyl3GPHYXGaXGTF6iNOs4ze0tTUH K7j61uAa0ppIvt6hvUamVPLfdY3i2oYgDh6qujuEOMaTE4ZCWQiJDV9uCyNhsCiSTIONIsnM Uf+cSUrV1F44Kp+aSVHEZ6bSGwyyOigC1Sar6b0SIrRKukSkTHKlu/PzCka46RFJ5PPc6ZI1 0TO8gCCx1sThBXluvqPmwer13e3vB9dUl1V1vNvyzY5d13c8s/aJbdrlyo3rL6z3C3mVB3OH vbX07YuX3lryzrDcg4uOfPxx1/OP7Xzu2pMNQurG2kc2bbpQT/3hIJJ+VjyMHj0qpKOIaok3 OiuIVAUhR+Uj6nG8GLK9yyzmZ3i4s83aZi5hgPDepgMnqC3XIYdpYhNYkcNJEFkwvHosCrEy FWSVRJbVxTKP1+WotghdsbKaFk2XzSUJOiQowWblXOmQny9DBp8u6WyMK/RD/p70j7vBWnPi nc9/OP2e1kXuJ1Pfn71/0H8sW9HwtNi0R7h2aZ125cwl7XsyNngH2UoaxWBV9X3j3jh/5Nnt ARYrs7o7+J/QB9xQBxHAYfQpKKRQbiA2heIFtD+g/fcBWazm0NlsINUUuBStBs0NqK87oq/F zfRNkZW0djxGOQl1XAwqORRTBSymQm4iOAbne8KqZrrIhs1rt3fXNQSCfzh9+fGFdWu7QXtI 627evnLDk7uf2cwP5zZUE9i0+NUvz/3XbDXLraw69j8X3qzx169d5eOYrjvRVtkYdwPCvk5Q PXKDpYisCtGWIj2WinJ6Uce8HTU1t1PtrRHtdbIyoD13mMXitDn5XuO5Mz2JPZq40nW603dx l4OvZ/3ad+LyP869e8V8yPzUkjXP7Fu3rHgYd4778GWt5jbtp4uXtOCHb69Ypeza2pQ/BH2t BvEb0VaxYA97shFBG8Ma8CjwVAOjrJojiHhZTY6mW0Y8HJ8vJyA4SybLZjoz43mEYLzQ8fVF 4cI3f7/AB9Y1PPkEt2nzpvU8V6kd1Y5hKvZcJWPJSO0D7Xjs3z/68IL2ccelM1+wXNKIOc2H 3MZBIgyC6TfhN5J3gTGalKPGRxjVx7P0i14jR5CTvo7iMTsj3iG5CIroOM4MZ4hXZyM5/823 teVLNmlfaSfILzfs0j7TWkj6yh31DdoXonKs5aE9Q52B1ccuco3BK5sfI7qdKxfVVQLLyZUY B5eFSTAQhsAyNenfbmEQZcXRU9+SEH1Scr8E3VcvLokmYjWGDUpMDj4erWFCDNNQ6JMhY2TV FaWh050ZSo6DPZ7eOlNAgyLJbE2yZaCJ8l0OjH9+iGxc8/r+/ybk6zdqF8/d0FzTuuToGcGt xdy/2/W09kqtY8qG321uPDq9rKb8jnt3eI++oMU965W3zLjzwon7H8RYMHR3cPeJY8AGs9W4 RIosymRhPS0oWJJvjA/aB0AONQ/fJ3vpo52OhkYCRgIx08JE/WuEjSlnduV7zGRjW1vBWMfI CcXLV7a2imO06w3BsrFjTdut2/3cvgYiIcaZaJMOtMkAmKDKAwf1YmTlPAxLj4KepajIy/Wy wrfT4m2lx6hCnzuMUuiJo+nUnN/DrMvNiEZWudFzt+kPiUtaH/5U66w9u+3N7/SH9A0VW3bt XFs3o/RAOckkMGjvj75zr1Vs/GOL62gbsLyCRYb/qwRgRpzGBAvDKedE2qIwVFZ2wm4Th0Ic w82JoeZHNUZHaYHHYcFy7g6Fp8vsIcWPt5L7+QCpeGSGzx0I8Id3aCuC+dyfHq2aM6krKIWw zEbOjiJnbihTDZlDejljlTwMJA2FNPpuPs0QqtrYx9HaLffJGlGlKU1WnO14VAdHY8xzDw7X cPRTxqI1UQiXeHd+KOfx3GLtby/ec+nQkS+b1zw4r3ohsb0y9avAEycXB8TN1RWryKCJU4um 1U5ef+Sdbb962HvH7eNG37fs/qcPzXxxTmnldIzNLajYAsx7PPajo1UpNo5ilmSFRPeeyf3C EQlVjO2YIFVJjAowJJLPKxieaJXS3WR6IHByxKKRIxeNEArJoOzRo2cUFSGHXrTn60Ih9r/j Qi/Q4Zq68AvY2/o6HLYOHCsSsipGThpEHWsdMrASYBeb77SRQm5Z14vc9OAbfN6OHT4+5Tdr ae4pRZv9A21mQ0+vUPXM06kPJ/box1y8v35qqp6V1NScN36f+udUrlQ1RczVJ42qltCdellN jLKeSDsKFgIJZpuTWWpEkoQdmM6Z73Zzd5/XOh7/dM0Hl4Mu4XX/gz7PYp92tuq5BG6g3mcl zu/Tnw82aJe14N37Wyf/u/c9vu23z8Rt2cV8EGuBNIXVgpKbdER9vhT6ZNJwKDPBhIKJNcHA 0qouNBgh1O0aOA8GhYW4CI0S8jw5T87/1JKgpT+juSwtotI5U3jh+mRuBFfeOU94LrgveAoB Idf12hxhIHIdD6nwsGqxp1FmLLTNh8hrk6Pb37AgoSAl9Cu1qmjiGSw24AeRavn5wksbgUwa ykmMdFZ6dWZKvDDws3dbFwcaDYtP/eHzwG7fgWlTX16/hzNf1d5fEbwqnqur185p14XDZ54N dm77gOUc7AUOokoJUKSKFmtvnPehus9XRTj7qHHRmMwS6HTpkGn2hFqAfLNwkGjd5z3fal9w 77z2/G9fFZWu9NPadZkj3Gf8xS733kOv7eU/oVzSmn8cMcSg5476FxWfFoq+zhnbr6oLCUnY Jgk63ozEoFdymY1k1ztk8D6yTzt77IO2C1e//lBUDmhtp0pPa20vcWJC52Zi7Z5+jVi4UP2m WFrYN5UF7laNVva5YozYlRksDIyhpFTE8CRUq6OrB5otoQ9FLooQ+1FXT6eR6SQVLVzKtyRe u3pVayQlO194oUHbzRUGkY0rp858uefpTU/s5lksTIKtQp7wJnKUCmgPWjkNfbplnpZIzBDs 081p9nB7yXZtYbO2kGxv5mMPaXnkT4dII1vrMB6Wsm/zAf/iozuU5MjSFkoHe86J+awZ85kE I1XQ6W9e65l7JPd3bxD5UA7Dj0oX7t8R09va8i1a3dud/G2dx3FRXB/zs2DC+mOCRyES1QOj o7p/8mRCDAox7C16mj7H2ArhLpgBC+Ex8MNO/GxrhjYwlqpSz+UCabx0n/SQtETaIG2XXpLe lI5LxlKkj9VH8wjEhyCbawOBueTeS9oD5MOz5Idl2joJumYvJeVaUdDP+PBhvj2P0GWYqhrN Cb18sAjv0+7cFDZrP0IfRdxNKncGrX0snrDjwcHFD1n03qu/I4GTZ8YHlF+vOtXKtQSLf9zL WzqPh+yzGfNlHetzlqpCVJ/DrJB8MxqZrZP70cgEKwrWsGBHwc5yqJ79VSCygRaCqEKfTB2Q ZqSkgp6eIy/cc+TR/ITZtVBc2vb5lIJXl5ByKbBwxXxfTPPf3rw9IBTW1b82aY62MTiUa6ut eXxBcDjX2rGr6zJ1i3BfhHqZkeeYnr7oBr36qMIsEEYfj0I8Q89LFDbGZUx0XCb1b5HIrdLy k1EtklDo2x3dIjFYmCsewNyZx2qAHW5XTWkDev7dsrRHuW9y/zoUldixh0+JQmJhVRSwg5Qk +p2eZO391JMeWPHXhk+Jedmlree1b5v/s37LS431mw5wmfs0v3Zai93bWU+GdxneOPfpCfXT c4w1Hug/hiYQuEk4DkQv5bGSroJuMpWUkTqykmzljnPnHW5HrqPQ8Yozvbub/pcH+8kUMgev r+i5bsHro3qv//xG8B3nyU6ym+zF3/6e33H8nSQnf/appN6ZHgRcQ0JedSCyb8EbNwPutl4p uWdMYMfE3vPWn3mXkfFB+2u6xaLdALMpQBpGzEDM93SLwwxrh5SeJ+Sfxf3/d7sF9sBqTKx1 cAbKsS7NgI0wB+6DPK4I3oJjeIdVmwNWbhc4eCfEYLBYhS0gYxGxSslgIy+BWWpBnnFTIGui YpjsbSLkyZIjpHu9sm5Ak4GfPStbIVkOR3HFOIXMyVa4LIXc4sxW+CzHeIXPGD/F6ypx+B3+ CeV+x3jHgrJyRchgI16Y5y/JcSgw1VuBx2lepzKmxN47nVdSUpitCHQZgS3jL8EFFvYssJAt gM8HsxUxa6JD4d2Tvfd6ldXj7MqYcSV2p9NRrLRM9iot4+zOkpJsRerFiOOKiuQQWl2WIt2S rehDK0z1KmPsCpT4/SHJ5VRW+/12P2oQllv6ykcI9D8xJvoEMlB8hKyezK6sdjnt9ITL6XIi wpJx2Yoha+JUbzFCdCJEY5YyuDhbiclSMnAwZTVlEp/DP9UbGIPRNveIHnzTvAEYzH9VVWJX XLi4w3dEht5zVMvYLGWM74gDZnibMmCcPQAZ/FfjSrJZdsGm6XPt8MnZ8UU/gF3P3OPY/5l/ Qcf2TybtvG4P/sWQq68BGrlcj//gc7pFGsahYcJ1u3aXIZetFL0N4uuhXFgJpbjn416F+0Hc 1+E+C/eduNcIndAozIBKbg8YhFyYKSyAesEBs8l3sIW/Cl7hl1AqdeE9K6BezMX9NmgUTShv gklcNhzmL4JTiIMtQgX4dDxsFhbi80/BA/j+B0GFH8lC8hy5xj3MnebT+OX8FWG8sFO4KN4l PiE2S+lSlXRGl6ibqTugO627rE/VV+ubDW5DtaGFaTMIGjAfTcM8FtItjlLFWTG7EcU0VIGh oJI4bihR4wkeD8cX5mbYZdAPhcPkzlt/4bTi9DDnvXPkEDbjp9yel5FIZ8Ksu4uGptCZWJCV nhJPZ9Li0vHD7XSme3rpzEIXnelXzp88KpXODNOKC9xJdGZcMmdiwQA6i1lbPil0n+ms6p9b RGexVnOsQaKzuKLhmWlmCkaVjToEG/JtNTeB1KiKFQ+TraRWXU0PVVbyqBqfhuceSSOPwj8B oOJEpQplbmRzdHJlYW0KZW5kb2JqCjExIDAgb2JqCiAgIDQxOTUKZW5kb2JqCjEyIDAgb2Jq Cjw8IC9MZW5ndGggMTMgMCBSCiAgIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCj4+CnN0cmVhbQp4 nF2Sy26DMBBF9/4KL9NFhHnZjYSQqnTDog+V9gOMPaRIxViGLPj7ejxRKnUBHMZ37jWMs3P3 3Llp49l7WEwPGx8nZwOsyzUY4ANcJsfygtvJbLe3dDez9iyLzf2+bjB3blxY0/DsIy6uW9j5 4ckuAzwwznn2FiyEyV344evcU6m/ev8DM7iNC9a23MIY7V60f9Uz8Cw1Hzsb16dtP8a2P8Xn 7oEX6T2nLZnFwuq1gaDdBVgjRMubcWwZOPtvrRTUMozmWwfWlEWUChEfkSviCvmR+BE5J86R JbFELolL5Jq4RhbEInJF/hX6S/KR6KMoS2GWJB+JPpJyJeYqqqtUP1H9hHVNdY11ypWYW1nK sqihPSjcQ02aOmkot8LcmjQ1aoohcXxEDWVVKYu+V+H3StJI1CjyVOgpKVdirgRiQB6JaRC3 P44jwbNzn7W5hhDHnA5Ymi9OdnJwP4N+8diVrl8qGrS8CmVuZHN0cmVhbQplbmRvYmoKMTMg MCBvYmoKICAgMzU4CmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IK ICAgL0ZvbnROYW1lIC9XU1JDQkwrUm9ib3RvLVJlZ3VsYXIKICAgL0ZvbnRGYW1pbHkgKFJv Ym90bykKICAgL0ZsYWdzIDMyCiAgIC9Gb250QkJveCBbIC03MzYgLTI3MCAxMTQ4IDEwNTYg XQogICAvSXRhbGljQW5nbGUgMAogICAvQXNjZW50IDkyNwogICAvRGVzY2VudCAtMjQ0CiAg IC9DYXBIZWlnaHQgMTA1NgogICAvU3RlbVYgODAKICAgL1N0ZW1IIDgwCiAgIC9Gb250Rmls ZTIgMTAgMCBSCj4+CmVuZG9iago1IDAgb2JqCjw8IC9UeXBlIC9Gb250CiAgIC9TdWJ0eXBl IC9UcnVlVHlwZQogICAvQmFzZUZvbnQgL1dTUkNCTCtSb2JvdG8tUmVndWxhcgogICAvRmly c3RDaGFyIDMyCiAgIC9MYXN0Q2hhciAxMjIKICAgL0ZvbnREZXNjcmlwdG9yIDE0IDAgUgog ICAvRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZwogICAvV2lkdGhzIFsgMCAwIDAgMCAwIDAg MCAwIDAgMCAwIDU2NiAwIDAgMCAwIDU2MSA1NjEgNTYxIDU2MSA1NjEgNTYxIDU2MSAwIDU2 MSAwIDAgMCAwIDAgMCAwIDAgMCA2MjIgMCA2NTUgMCAwIDAgMCAyNzEgMCAwIDAgODczIDAg MCA2MzAgMCAwIDAgMCA2NDggMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDU0MyAwIDUyMyAwIDUy OSAwIDAgNTUwIDI0MiAwIDUwNiAwIDg3NiA1NTEgNTcwIDU2MSAwIDAgNTE1IDMyNiA1NTEg NDg0IDAgMCAwIDQ5NSBdCiAgICAvVG9Vbmljb2RlIDEyIDAgUgo+PgplbmRvYmoKMSAwIG9i ago8PCAvVHlwZSAvUGFnZXMKICAgL0tpZHMgWyA3IDAgUiBdCiAgIC9Db3VudCAxCj4+CmVu ZG9iagoxNSAwIG9iago8PCAvQ3JlYXRvciAoY2Fpcm8gMS4xNC4yIChodHRwOi8vY2Fpcm9n cmFwaGljcy5vcmcpKQogICAvUHJvZHVjZXIgKGNhaXJvIDEuMTQuMiAoaHR0cDovL2NhaXJv Z3JhcGhpY3Mub3JnKSkKPj4KZW5kb2JqCjE2IDAgb2JqCjw8IC9UeXBlIC9DYXRhbG9nCiAg IC9QYWdlcyAxIDAgUgo+PgplbmRvYmoKeHJlZgowIDE3CjAwMDAwMDAwMDAgNjU1MzUgZiAK MDAwMDAxMDg0OCAwMDAwMCBuIAowMDAwMDA0NDA5IDAwMDAwIG4gCjAwMDAwMDAwMTUgMDAw MDAgbiAKMDAwMDAwNDM4NiAwMDAwMCBuIAowMDAwMDEwMzg4IDAwMDAwIG4gCjAwMDAwMDQ3 NzMgMDAwMDAgbiAKMDAwMDAwNDU0NiAwMDAwMCBuIAowMDAwMDA1MjcxIDAwMDAwIG4gCjAw MDAwMDUyNDkgMDAwMDAgbiAKMDAwMDAwNTM0MyAwMDAwMCBuIAowMDAwMDA5NjM0IDAwMDAw IG4gCjAwMDAwMDk2NTggMDAwMDAgbiAKMDAwMDAxMDA5NSAwMDAwMCBuIAowMDAwMDEwMTE4 IDAwMDAwIG4gCjAwMDAwMTA5MTMgMDAwMDAgbiAKMDAwMDAxMTA0MSAwMDAwMCBuIAp0cmFp bGVyCjw8IC9TaXplIDE3CiAgIC9Sb290IDE2IDAgUgogICAvSW5mbyAxNSAwIFIKPj4Kc3Rh cnR4cmVmCjExMDk0CiUlRU9GCg== --------------070703060403080105060100-- --ojWM2RfT27D5rxalIH6m7sQf9oPQ8T9jk-- --c2ilmUOL4VmDEpGuPjGWmDn46sh7vDw5d Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXYB36AAoJEJiuUT5oYfea1VYP/iX/gu3TiUPUNkvt3evn1d6J v47kDlUdkenQPt0LuJk4/DLw4DBrjdfGP1pA/O2s0Jq8+rQqXn1qMUvY1DgkP3RH G43UtE6OWlMG9x3SlAlNfYo4nCNTfREGcE/iSpvC6GAzKkI9/9nlTvGTwhC+K+lO vcP5i8B62YYpg599TbIUnzLSGoazy+SIOtnEYKFswMZOxS6E1/lgtSDlYobI+cv4 N1tvZrJ+ONXrJHgbP7pqki8qrwNfJoh4QQls+7rwp+U/CQvKmxLdGSCmaBHxT0qN MuF7bITwbQuuguNECXd1EUgbfoAuUs8WQUR5oiFwEW7emzSpJ2AvgTiYkPstfi2O 322MMyZ4XDrB8P8RZbJmBP17b7lNtI4MmruZNu3ePvceX1eUQupLqNY87//LZQW7 aIqebKrkzguYhwR/Oa8Di7EET92P4vDsm9HSwGIIukVXpsNQnyN37/iuJ8dKwZtk 0TqobaOrSmUcwJps5BRRf08oxX88H+c/FDYo+lVtLDuwgbampx8n4mxM8P7GYoJA YUJ3aCwhk2L3iRChCpQN6Q8BD3u8w1RviFiJcqBLrth9O3UczY0P4IJT/jSF0c6V 76qdzXmeFWPQw4RS1kCC255Y21qLqzCaIP/CxDlkGugQoSn8IhpXt2ivzf+rQEIy jZyA5VwqTmn3dac99Eoh =to2t -----END PGP SIGNATURE----- --c2ilmUOL4VmDEpGuPjGWmDn46sh7vDw5d-- From owner-freebsd-net@freebsd.org Tue Jun 14 15:26:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB4BDAF23E0 for ; Tue, 14 Jun 2016 15:26:23 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B41862F02; Tue, 14 Jun 2016 15:26:23 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi0-x22c.google.com with SMTP id d132so157935408oig.1; Tue, 14 Jun 2016 08:26:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=hCw6zfilP471N+yzQ8SaIKMr6Pim059AiwShFGK5SMM=; b=TuiasPTzmHNHtLra5OMXoHSFzeb8mmlJqtC/n2e679CU9dxikh//WMWNzM8w0NGILw ef0ZiSKy//QGhsOE1kwqG03ySkqhNUSmSbGZ20iZcSSQUOHvUE0cbTLc1XJPQds4uZ8i W+tindHKa10ejolveIvIcANH1t0E0d0ykUN+/uLzxEcb3xTwHiyD6z4cME5nd/D3ilZn 50LjjkK7kLBsqrB6L+4suiKUWIQfyizZaCODr+OokuDzYSKjZ93pG1JzAsTpuCeSWlxb oxdg2AteKPxhwiMNwAjb9mwphpzo0svYofdBCCoysRtUP6orEQmF89uT42kdc5UdXEPY z3eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=hCw6zfilP471N+yzQ8SaIKMr6Pim059AiwShFGK5SMM=; b=EgYm2f7I/oFO92PwboeHELSUp70UMjlN+d7Wah1gyGFLoE1X6QdL4QaDlJxKEuf6G2 kgu4g/AmDmDPYTA1AXvi7RKcUgsrfH2cl8M1StN30DcZpNrVKxPB5aNk5KjhKAcs+h2o DfzeF9xqufEGxSzdQWq3t61G0Qf9NXuly5Fx9Rr6Pgn4SuzGpmr2L4QP1KY1AY+FJwAo 0A4XzBJ3mogRjFSy/LTDmVLjcQA5WXiYmAsVRQUph1LWPu0xO0Czloc10Lt0F2qYQVFr XgSXSOEvaOkN2d681rfDuWDLaHphHoHzc+I/KlsqzQjGSu+qJTQyNz3TOk7w/L4sGCSr UuEg== X-Gm-Message-State: ALyK8tKIhUknsW8J4O84upRqpjKL71kqJioZi3QUO9erT0IE31XJ3AmNfasa8dvps8zkJQb6dJsKJPehHUaAOQ== X-Received: by 10.202.114.65 with SMTP id p62mr11431767oic.105.1465917982909; Tue, 14 Jun 2016 08:26:22 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.202.102.206 with HTTP; Tue, 14 Jun 2016 08:26:22 -0700 (PDT) In-Reply-To: <459d2639-b490-beee-9cd4-05f38983eaed@freebsd.org> References: <459d2639-b490-beee-9cd4-05f38983eaed@freebsd.org> From: Alan Somers Date: Tue, 14 Jun 2016 09:26:22 -0600 X-Google-Sender-Auth: xgDcCb3pREXs52cLjG_1eyVNc0I Message-ID: Subject: Re: lagg(4): LOR, deadlock and panic To: Sean Bruno Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2016 15:26:24 -0000 On Tue, Jun 14, 2016 at 9:13 AM, Sean Bruno wrote: > tl;dr --> https://reviews.freebsd.org/D6845 > > Navdeep and I have been poking at an LOR that seems to be popping up > in -current that is related to lagg(4) and lagg_get_counter(). > > root@sysdev07:~ # ifconfig lagg0 create laggport ix0 laggproto lacp > 192.168.100.11/24 > lagg0: link state changed to DOWN > root@sysdev07:~ # ifconfig ix0 up > lock order reversal: > 1st 0xfffff8002d7c9190 if_addr_lock (if_addr_lock) @ > /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1717 > 2nd 0xfffff800271a5808 if_lagg rmlock (if_lagg rmlock) @ > /usr/home/sbruno/fbsd_head/sys/modules/if_lagg/../../net/if_lagg.c:1057 > stack backtrace: > #0 0xffffffff80aa5ab0 at witness_debugger+0x70 > #1 0xffffffff80aa59a4 at witness_checkorder+0xe54 > #2 0xffffffff80a42521 at _rm_rlock_debug+0x111 > #3 0xffffffff82222b2c at lagg_get_counter+0x4c > #4 0xffffffff80b2ebd1 at if_data_copy+0xa1 > #5 0xffffffff80b533bc at sysctl_rtsock+0x56c > #6 0xffffffff80a53f0a at sysctl_root_handler_locked+0x8a > #7 0xffffffff80a536c8 at sysctl_root+0x188 > #8 0xffffffff80a53cbe at userland_sysctl+0x16e > #9 0xffffffff80a53b14 at sys___sysctl+0x74 > #10 0xffffffff80eb5b3b at amd64_syscall+0x2db > #11 0xffffffff80e95c4b at Xfast_syscall+0xfb > > Running a netstat -w 1 in the backgrouund while repeatedly creating > destroying the interface lagg0 will lead to either a panic or a deadlock: > > e.g. netstat -w 1 > /dev/null & > while [ 1 ]; do > ifconfig lagg0 destroy > ifconfig lagg0 create laggport ix0 laggproto lacp 192.168.100.11/24 > done > > When the system deadlocks on the console, kdb sees the locks held like > this: > KDB: enter: Break to debugger > [ thread pid 11 tid 100007 ] > Stopped at kdb_alt_break_internal+0x18e: movq $0,kdb_why > db> show allocks > No such command > db> show alllocks > Process 2173 (ifconfig) thread 0xfffff8002d125a00 (100186) > exclusive rm if_lagg rmlock (if_lagg rmlock) r = 0 > (0xfffff8002717e408) locked @ > /usr/home/sbruno/fbsd_head/sys/modules/if_lagg/../../net/if_lagg.c:1530 > exclusive sleep mutex in6_multi_mtx (in6_multi_mtx) r = 0 > (0xffffffff81d7e288) locked @ > /usr/home/sbruno/fbsd_head/sys/netinet6/in6_mcast.c:1142 > Process 792 (netstat) thread 0xfffff80027e67a00 (100167) > shared rw if_addr_lock (if_addr_lock) r = 0 (0xfffff80103e95190) > locked @ /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1717 > shared rw ifnet_rw (ifnet_rw) r = 0 (0xffffffff81d7b760) locked @ > /usr/home/sbruno/fbsd_head/sys/net/rtsock.c:1713 > exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff81d55e08) locked > @ /usr/home/sbruno/fbsd_head/sys/kern/kern_sysctl.c:164 > > This looks like the netstat is causing a call into the counter > function while the destruction or creation is ongoing. > > Removing the LAGG_RLOCK() calls from lagg_get_counter() makes the > deadlock, LOR and panic go away, however this can't be that easy. I'm > unsure what the RLOCK is for in lagg_get_counter(). It appears that > there is a higher lock in the ifnet access that is protecting > simultaneous access already, but I'm very ignorant of what's going on > here. > > I don't see any other driver with locks in its get_counter() > functions, so I'm not sure what the best course of action here is. > > Sean I don't know the best answer either. But while you're in there, are you interested in fixing any other lagg panics too? I've written some ATF torture tests for lagg, but I haven't checked them into head yet because most of them quickly panic. -Alan From owner-freebsd-net@freebsd.org Wed Jun 15 09:01:33 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2627A13869 for ; Wed, 15 Jun 2016 09:01:33 +0000 (UTC) (envelope-from ibk@labhijau.net) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D02419D1 for ; Wed, 15 Jun 2016 09:01:33 +0000 (UTC) (envelope-from ibk@labhijau.net) Received: by mail-qk0-x234.google.com with SMTP id s186so12507081qkc.1 for ; Wed, 15 Jun 2016 02:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=labhijau-net.20150623.gappssmtp.com; s=20150623; h=mime-version:date:message-id:subject:from:to; bh=dvaqHX4IbdMRWAmYXL0tcjObFp3G21jYy7PR6hapuNo=; b=o/YrRvIeijnpZh034Yvar4xs2kEdRKRpDJJCjpujKAbJkjl6+uwdS2XynqsCKmenw8 yszx2jGeFK04Abiy8SlYcYlUiUmE9Dd5hRTMqjIoNCTaBNj3f6mWX84yRo/2drprRXjw rZqiJpYFDENspfo66+01W+QdPFIQjskDHmeKU9lSXPcUmgej/LxB/qYpLLcQOyFjgU2d Rf/hjukQf5NiaxBRpEY/DoQmmy0CWJK75onjEz5VILJBVASfCs6NLiVjkaL82xFbQnoZ Hthruamag4uXun5BHwVsh67fa8e5H2jPztydNqbNzMnRAjpDUz0+EpZnUR86FZS2aUFA gRaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=dvaqHX4IbdMRWAmYXL0tcjObFp3G21jYy7PR6hapuNo=; b=NeGEmwKXex/i8+QqspuebDdiHnYOsMmmghlVMhGuVpoOjaNiwlnZQPx4cSYfcmY6DS PfZgXw20YAkB1Ksdw8ajSo4htQqVraijaXo19Aj4nyd3m+1jgUuSGnnePVGnnU8fypLt fI2c034b2I4lCK8/UNuOMn92rGpCOZ0n19zsHv8wp8p5c+JRAyhByuC4zlxzoC0kZqaJ RSLcJk3EHlOp13TDpnEqFv/0ns31loWirEZ75uNYShytaWx0lUxIimrQxvbZYsMCTpew kZNx+Mon3YmmpnIwBe2qIcMPleiUX/PTmOmsa32jqc3ICigjhPnvKB7LNRnjDXZZaY3T TSXQ== X-Gm-Message-State: ALyK8tLSWotHALHPHBFFTGIzMCM1/GAoHYaGPwFeFXDKrF8ioW/e0yCHL3Oxaqa1PnjJ9QQzTKbzuywQ+NE0+A== MIME-Version: 1.0 X-Received: by 10.55.162.68 with SMTP id l65mr24815235qke.5.1465981292532; Wed, 15 Jun 2016 02:01:32 -0700 (PDT) Received: by 10.55.78.150 with HTTP; Wed, 15 Jun 2016 02:01:32 -0700 (PDT) X-Originating-IP: [180.253.25.48] Date: Wed, 15 Jun 2016 16:01:32 +0700 Message-ID: Subject: specify destination when using netmap From: Iwan Budi Kusnanto To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 09:01:33 -0000 Hi all, I am just learning netmap these last few days and wrote some simple receiver & sender based on example from https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4. And then i just realized that the sender doesn't specify the destination address, neither mac or IP address. It only specify the interface to use. Same case with pkt-gen example. So, how we specify the destination address? -- Iwan Budi Kusnanto From owner-freebsd-net@freebsd.org Wed Jun 15 13:13:57 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA501A311A8 for ; Wed, 15 Jun 2016 13:13:57 +0000 (UTC) (envelope-from joe@truespeed.com) Received: from mail.karthauser.co.uk (babel.karthauser.co.uk [212.13.197.151]) by mx1.freebsd.org (Postfix) with ESMTP id 9043118D4 for ; Wed, 15 Jun 2016 13:13:56 +0000 (UTC) (envelope-from joe@truespeed.com) Received: from dspam (babel.karthauser.co.uk [212.13.197.151]) by mail.karthauser.co.uk (Postfix) with SMTP id C84D3C4D for ; Wed, 15 Jun 2016 13:04:43 +0000 (UTC) Received: from phoenix.domain_not_set.invalid (unknown [31.210.26.211]) (Authenticated sender: joemail@tao.org.uk) by mail.karthauser.co.uk (Postfix) with ESMTPSA id 7CA56C49; Wed, 15 Jun 2016 13:04:28 +0000 (UTC) From: Dr Josef Karthauser Date: Wed, 15 Jun 2016 14:04:27 +0100 Subject: IPFW: Packet forwarding with bridges and vlans and Vimage? With an IP address. To: freebsd-net@freebsd.org Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) X-Mailer: Apple Mail (2.2104) X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Jun 15 13:04:43 2016 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 5761526b28816613420626 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 13:13:57 -0000 I=E2=80=99m bridging traffic on a FreeBSD-10.3 machine, between a vlan = and a VIMAGE enabled Jail: vlan9: flags=3D8943 = metric 0 mtu 1500 ether 0c:c4:7a:7d:4f:1e nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active vlan: 9 parent interface: igb0 bridge9: flags=3D8943 = metric 0 mtu 1500 ether 02:02:28:ac:d2:09 nd6 options=3D9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0:6 flags=3D143 ifmaxaddr 0 port 12 priority 128 path cost 2000 member: vlan9 flags=3D143 ifmaxaddr 0 port 9 priority 128 path cost 20000 vnet0:6: flags=3D8943 = metric 0 mtu 1500 description: associated with jail: = aec07207-31b9-11e6-8bed-0cc47a7d4f1e options=3D8 ether 02:ff:60:ae:c0:72 inet6 fe80::ff:60ff:feae:c072%vnet0:6 prefixlen 64 scopeid 0xc=20= nd6 options=3D21 media: Ethernet 10Gbase-T (10Gbase-T ) status: active All is good in the world, until I also add an IP address to vlan9. When = that happens IPFW appears to gobble up packages originating from = vnet0:6. They appear on bridge9, but aren=E2=80=99t forwarded in an = egress direction down vlan9. I don=E2=80=99t have any sysctls relating to bridge filtering enabled: net.link.ether.ipfw: 0 net.link.bridge.ipfw: 0 net.link.bridge.ipfw_arp: 0 But, with an IP address assigned to vlan9, packets are getting filtered: # ifconfig vlan9 inet 192.168.9.250/24 # tcpdump -i bridge9 13:58:02.498307 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 13:58:02.498442 IP 192.168.9.3.bootps > 255.255.255.255.bootpc: = BOOTP/DHCP, Reply, length 300 13:58:10.497760 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 13:58:10.497892 IP 192.168.9.3.bootps > 255.255.255.255.bootpc: = BOOTP/DHCP, Reply, length 300 # tcpdump -i vlan9 13:58:02.498273 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 13:58:10.497725 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 # ifconfig vlan9 inet delete # tcpdump -i bridge9 14:00:58.486653 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 14:00:58.486795 IP 192.168.9.3.bootps > 255.255.255.255.bootpc: = BOOTP/DHCP, Reply, length 300 # tcpdump -i vlan9 14:00:58.486634 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, = Request from 00:14:f2:76:46:e6 (oui Unknown), length 320 14:00:58.486792 IP 192.168.9.3.bootps > 255.255.255.255.bootpc: = BOOTP/DHCP, Reply, length 300 I don=E2=80=99t have IP forwarding switched on and so I=E2=80=99d expect = bridged packets to carry on being bridged irrespective of whether vlan9 = has an IP address or not. What=E2=80=99s strange is that ingress packets to the bridge are being = forwarded ok, but egress packets out onto the vlan are being filtered. Is there something obvious that I=E2=80=99ve missed? Cheers, Joe =E2=80=94=20 Dr Josef Karthauser Chief Technical Officer (01225) 300371 / (07703) 596893 www.truespeed.com / theTRUESPEED =20 @theTRUESPEED =20 This email contains TrueSpeed information, which may be privileged or = confidential. It's meant only for the individual(s) or entity named = above. If you're not the intended recipient, note that disclosing, = copying, distributing or using this information is prohibited. If you've = received this email in error, please let me know immediately on the = email address above. Thank you. We monitor our email system, and may record your emails. From owner-freebsd-net@freebsd.org Wed Jun 15 13:25:44 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D87DA317AB for ; Wed, 15 Jun 2016 13:25:44 +0000 (UTC) (envelope-from joe@truespeed.com) Received: from mail.karthauser.co.uk (babel.karthauser.co.uk [212.13.197.151]) by mx1.freebsd.org (Postfix) with ESMTP id 3619B1225 for ; Wed, 15 Jun 2016 13:25:43 +0000 (UTC) (envelope-from joe@truespeed.com) Received: from dspam (babel.karthauser.co.uk [212.13.197.151]) by mail.karthauser.co.uk (Postfix) with SMTP id 011ADCEE for ; Wed, 15 Jun 2016 13:25:42 +0000 (UTC) Received: from phoenix.domain_not_set.invalid (unknown [31.210.26.211]) (Authenticated sender: joemail@tao.org.uk) by mail.karthauser.co.uk (Postfix) with ESMTPSA id 13DE2CEC; Wed, 15 Jun 2016 13:25:36 +0000 (UTC) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: IPFW: Packet forwarding with bridges and vlans and Vimage? With an IP address. From: Dr Josef Karthauser In-Reply-To: Date: Wed, 15 Jun 2016 14:25:35 +0100 Message-Id: <33CB1553-0C61-410A-BB94-9C0CBB51E78C@truespeed.com> References: To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.2104) X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Jun 15 13:25:42 2016 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 5761575628811371313876 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 13:25:44 -0000 > On 15 Jun 2016, at 14:04, Dr Josef Karthauser = wrote: >=20 > I don=E2=80=99t have IP forwarding switched on and so I=E2=80=99d = expect bridged packets to carry on being bridged irrespective of whether = vlan9 has an IP address or not. >=20 > What=E2=80=99s strange is that ingress packets to the bridge are being = forwarded ok, but egress packets out onto the vlan are being filtered. >=20 > Is there something obvious that I=E2=80=99ve missed? >=20 > Cheers, > Joe Ok, I=E2=80=99ve narrowed the problem down. It=E2=80=99s related to the = anti spoofing ruleset. I=E2=80=99ve also got this in my ruleset: deny log ip from any to any not antispoof in What=E2=80=99s strange is that when vlan9 has an ip address this rule = starts triggering for interfaces that it didn=E2=80=99t before: Jun 15 14:19:39 kernel: ipfw: 10000 Deny UDP 192.168.9.3:67 = 255.255.255.255:68 in via vnet0:13 Jun 15 14:19:39 kernel: ipfw: 10000 Deny UDP 192.168.9.3:67 = 255.255.255.255:68 in via bridge9 Jun 15 14:19:39 kernel: ipfw: 10000 Deny UDP 192.168.9.3:67 = 255.255.255.255:68 in via vnet0:13 Without the IP address I don=E2=80=99t get any of these logged and no = packets are filtered. Why would anti-spoof filtering filter traffic on interfaces without IP = addresses assigned when vlan9 is given an interface? I might expect that = behaviour on the vlan, but but the other bridged interfaces. Is this a =E2=80=9Cfeature=E2=80=9D? Joe =E2=80=94=20 Dr Josef Karthauser Chief Technical Officer (01225) 300371 / (07703) 596893 www.truespeed.com / theTRUESPEED =20 @theTRUESPEED =20 This email contains TrueSpeed information, which may be privileged or = confidential. It's meant only for the individual(s) or entity named = above. If you're not the intended recipient, note that disclosing, = copying, distributing or using this information is prohibited. If you've = received this email in error, please let me know immediately on the = email address above. Thank you. We monitor our email system, and may record your emails. From owner-freebsd-net@freebsd.org Wed Jun 15 23:15:25 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00275A478F8 for ; Wed, 15 Jun 2016 23:15:24 +0000 (UTC) (envelope-from andy.yakov@ya.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id DB4781CA7 for ; Wed, 15 Jun 2016 23:15:24 +0000 (UTC) (envelope-from andy.yakov@ya.ru) Received: by mailman.ysv.freebsd.org (Postfix) id D6B28A478F6; Wed, 15 Jun 2016 23:15:24 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D65DEA478F5 for ; Wed, 15 Jun 2016 23:15:24 +0000 (UTC) (envelope-from andy.yakov@ya.ru) Received: from forward5o.cmail.yandex.net (forward5o.cmail.yandex.net [IPv6:2a02:6b8:0:1a72::28a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9026C1CA6 for ; Wed, 15 Jun 2016 23:15:24 +0000 (UTC) (envelope-from andy.yakov@ya.ru) Received: from mxback6o.mail.yandex.net (mxback6o.mail.yandex.net [37.140.190.20]) by forward5o.cmail.yandex.net (Yandex) with ESMTP id 14F6C20BEF; Thu, 16 Jun 2016 02:15:10 +0300 (MSK) Received: from web23o.yandex.ru (web23o.yandex.ru [95.108.205.123]) by mxback6o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id YULR402WVY-FAWONwaV; Thu, 16 Jun 2016 02:15:10 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ya.ru; s=mail; t=1466032510; bh=AlmwrRVuDxa+bYucSWNdzgn484MxjBfcY2Hovd5sxFc=; h=X-Yandex-Sender-Uid:From:Envelope-From:To:In-Reply-To:References: Subject:MIME-Version:Message-Id:X-Mailer:Date: Content-Transfer-Encoding:Content-Type; b=te6zkuPoXkbRvF2LNSEIGQkA6w7gRK9ZvQtiyQkP6xsS6T5wLSaJSRr5cDJGIxoPG zrmZeDBnJqq8rvlI3+M0KLF72U8eTylurNnGEEKx8B2s5w4wUZf1jQ4bmUU7lcv+Nz 1cl0GLeOUdF7KQcXqZnFJ/b23HnbZ/TjvEO2mS9M= Authentication-Results: mxback6o.mail.yandex.net; dkim=pass header.i=@ya.ru X-Yandex-Suid-Status: 1 0,1 0,1 896244277 X-Yandex-Sender-Uid: 359195858 Received: by web23o.yandex.ru with HTTP; Thu, 16 Jun 2016 02:15:10 +0300 From: Andrey Yakovlev Envelope-From: andy-yakov@yandex.ru To: Dominik Schoeffmann , "net@freebsd.org" In-Reply-To: <57601DFA.2040008@in.tum.de> References: <57601DFA.2040008@in.tum.de> Subject: Re: Netmap Checksum Offloading MIME-Version: 1.0 Message-Id: <470941466032510@web23o.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Wed, 15 Jun 2016 20:15:10 -0300 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 23:15:25 -0000 ive heard on bsdcan this year that some patches exist to add hwcsum offloading to netmap, hope to see it chelsio at least --  ./andy 14.06.2016, 12:15, "Dominik Schoeffmann" : > Dear Netmap Developers, > > during the course of my bachelor's thesis, I modified a packet generator > called MoonGen [1] in order to utilize netmap. > One key component was to flexibly offload checksums for different kinds > of packets (IPv4, UDP, TCP). > The ixgbe netmap patch was modified [2] in order to construct context > descriptors and suitable data descriptors. This is implemented in less > than 250 LoC (including pseudo-header calculations). > The man page states, that checksum offloading is available via ethtool, > although a solution inside the netmap API might be a cleaner way for > applications to actually use these features. > Attached is a graph showing the performance implication of using > offloading in the current implementation. > As can be seen, offloading has only a minor impact. > When regarding this data (and comparing it to other frameworks), please > keep in mind, that internally a lot of per-packet effort is needed due > to the software architecture of the packet generator. > > The question being: > Would it not make sense to include checksum offloading inside of netmap > in order to accomodate applications operating on layer 3 and above? > As these programs need to calculate the checksums in software, it would > be just as fast to move these calculations to the kernel for NICs > without checksum offloading support (and the kernel would act as a library). > The problem which currently is imposed by the fact, that netmap exports > the complete ring, is that context descriptors disrupt the data > descriptors, which is unpleasant for the application. > But you may find the data interesting nevertheless. > > Best Regards, > Dominik Schoeffmann > > [1] https://github.com/dschoeffm/MoonGen/tree/netmap > [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading From owner-freebsd-net@freebsd.org Wed Jun 15 23:50:34 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D053A47071 for ; Wed, 15 Jun 2016 23:50:34 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 5B3371A61 for ; Wed, 15 Jun 2016 23:50:34 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5752AA47070; Wed, 15 Jun 2016 23:50:34 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56ED7A4706F for ; Wed, 15 Jun 2016 23:50:34 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pf0-x22d.google.com (mail-pf0-x22d.google.com [IPv6:2607:f8b0:400e:c00::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20BAE1A5F for ; Wed, 15 Jun 2016 23:50:34 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pf0-x22d.google.com with SMTP id i123so9710853pfg.0 for ; Wed, 15 Jun 2016 16:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=+ghvbP1sfHgID43no6vmNMWBtsw3qh/ah1K3m3nAzkQ=; b=DcmoHsTLwO2mBP6sGs1jS7ankPaEXADjv9fzUw2/bjJ63gU2IXTHJcDIIUvlHZGiwf nucPF8aKQREiMMsxPcDQ4FBaJViSPg0LbGvgL2ov9sHpgl3Rj61gsbSR0gQUZHoY53RO 7hjM0j6AfoiTHUejjhgKuupYp758B2iHHVEGEc9G3K9lUhiEvvjJDNi4JwbVqlmeuFDy Mi/pjjYXJimjdT+QIlaVmmqb/TBxNRxQCq/QtEk5SE5e14xkEmEZPERCExuNYe232MGa flkLENzUFuR4JdKChhpfOmcEm40mU1fC8SJ0R8qY2+wyqIdvMyHFqoCVrvQ+shCZc1or FF+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=+ghvbP1sfHgID43no6vmNMWBtsw3qh/ah1K3m3nAzkQ=; b=Wcq7wiabuGv0v/jEtXErEl9ht6j5Z2pwS2TYGu1yNSLnzFlvlsQlArdiuguPsWpDZg B+9OBXF6qi+sRaYQ7/3i3iF1fL1XHlC5C2tLT7aDSpswyO7mQKhwZbGwBuc33tb5ztA0 zT4Uc4BwZq2ocbdaiRzYBi1B9pQnvDSUr5CaFS6boTCa6GpcZHfquckqo6Irh0yJZ1Pm 9USApHaD3KzC2NKMMnOv2+WIfVLNfj+7gQx+v32qqXH6sLRnrNfG5+wQCFbbiLGfwaZN tQIUxL4/XIxAx5UPvMCW/ct3/ZqUoLZi5LfJRT7Xsdmbkx1suMVE2v1uEepZ7dsBV/L9 euEA== X-Gm-Message-State: ALyK8tIOQY/XLRwatlI7dx+JCHPsapkx3YvpbBzxGeaWKYrbxYop5hZYuvDB05kEV5I/FA== X-Received: by 10.98.11.4 with SMTP id t4mr1471947pfi.159.1466034632918; Wed, 15 Jun 2016 16:50:32 -0700 (PDT) Received: from [10.192.166.0] ([12.32.117.8]) by smtp.googlemail.com with ESMTPSA id z88sm27912198pfa.59.2016.06.15.16.50.31 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 15 Jun 2016 16:50:31 -0700 (PDT) Sender: Navdeep Parhar Subject: Re: Netmap Checksum Offloading To: Andrey Yakovlev , Dominik Schoeffmann , "net@freebsd.org" References: <57601DFA.2040008@in.tum.de> <470941466032510@web23o.yandex.ru> From: Navdeep Parhar Message-ID: Date: Wed, 15 Jun 2016 16:50:30 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <470941466032510@web23o.yandex.ru> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 23:50:34 -0000 On 06/15/2016 16:15, Andrey Yakovlev wrote: > ive heard on bsdcan this year that some patches exist to add hwcsum offloading to netmap, hope to see it chelsio at least cxgbe/cxl is a bit sneaky and will let you override netmap (on tx only). The ncxl interfaces declare themselves capable of checksumming but all such capabilities are disabled by default. Just enable txcsum on the interface and the hardware will do checksum insertion on tx. No way to solve the rx part entirely within the driver -- netmap has to be willing to accept checksum related flags from the driver. Regards, Navdeep > > -- > ./andy > > > 14.06.2016, 12:15, "Dominik Schoeffmann" : >> Dear Netmap Developers, >> >> during the course of my bachelor's thesis, I modified a packet generator >> called MoonGen [1] in order to utilize netmap. >> One key component was to flexibly offload checksums for different kinds >> of packets (IPv4, UDP, TCP). >> The ixgbe netmap patch was modified [2] in order to construct context >> descriptors and suitable data descriptors. This is implemented in less >> than 250 LoC (including pseudo-header calculations). >> The man page states, that checksum offloading is available via ethtool, >> although a solution inside the netmap API might be a cleaner way for >> applications to actually use these features. >> Attached is a graph showing the performance implication of using >> offloading in the current implementation. >> As can be seen, offloading has only a minor impact. >> When regarding this data (and comparing it to other frameworks), please >> keep in mind, that internally a lot of per-packet effort is needed due >> to the software architecture of the packet generator. >> >> The question being: >> Would it not make sense to include checksum offloading inside of netmap >> in order to accomodate applications operating on layer 3 and above? >> As these programs need to calculate the checksums in software, it would >> be just as fast to move these calculations to the kernel for NICs >> without checksum offloading support (and the kernel would act as a library). >> The problem which currently is imposed by the fact, that netmap exports >> the complete ring, is that context descriptors disrupt the data >> descriptors, which is unpleasant for the application. >> But you may find the data interesting nevertheless. >> >> Best Regards, >> Dominik Schoeffmann >> >> [1] https://github.com/dschoeffm/MoonGen/tree/netmap >> [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Jun 16 00:04:56 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40833A47960 for ; Thu, 16 Jun 2016 00:04:56 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1F13912CC for ; Thu, 16 Jun 2016 00:04:56 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1AC7DA4795E; Thu, 16 Jun 2016 00:04:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 183FDA4795D for ; Thu, 16 Jun 2016 00:04:56 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C28EF12CA for ; Thu, 16 Jun 2016 00:04:55 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mail-oi0-x22d.google.com with SMTP id d132so45009943oig.1 for ; Wed, 15 Jun 2016 17:04:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=74EQhOAhkDSWoiPhNNAgzA4HKMBvlXe5RrPZsmt/jX8=; b=B7gDtG6WpCBFWX0bsmKKRl/KZfYHeh2QGGj82KTqXutoy8QwsqRN4QcAE1c5/udigg a/9pt2iMJiB48ofhnPv/W+ATRTZGsdYfO3YvW/HJIK95icbwkMNozRzTUr5MVlSS4pnX ZE0kIVfZ0z0iQLP6P1PsTib0NBidU7We+zIUY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=74EQhOAhkDSWoiPhNNAgzA4HKMBvlXe5RrPZsmt/jX8=; b=K9SevwCpmvTBnsxpWLnDF+5ZVliWAw1lEdCT+9E9AJRTs+Qa00lZDFAd00BgJF1xtI 7aVojydd6Xf1MPMFxQCk/DQ9eWBQT5cEsk4DMJeqrzZ7kCCIXgZHC5WQWhgFy9K35NU6 ntszaaDj7ivLjqXp6CpJeXkdKQFcs+jGFt7M1JkotQa18VyFo2ofQkGV6k/sfogEVHDP oY4E0hgKysVyhMpQq8tV00Q/LU34yHPg6A7GPzkiS5abjpyezBXhWvllyoummN42TOuK lmuyjQPY1XTbOGDIn9HsV/7t6T6vciESBqCsVB49h5yPLKH/fDRJLu/zNbpR+T1TlKMG RnKw== X-Gm-Message-State: ALyK8tLcWTXfICvOEa0OwJpQExESYNwpSYYEAtT7fqc6Y0B/KDrsk9+df3wd1LzeWWXIg0On X-Received: by 10.157.20.101 with SMTP id h92mr899811oth.114.1466035494996; Wed, 15 Jun 2016 17:04:54 -0700 (PDT) Received: from [172.21.0.108] (65-36-116-65.dyn.grandenetworks.net. [65.36.116.65]) by smtp.gmail.com with ESMTPSA id 3sm13643338otc.24.2016.06.15.17.04.54 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 15 Jun 2016 17:04:54 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Netmap Checksum Offloading From: Jim Thompson In-Reply-To: Date: Wed, 15 Jun 2016 19:04:53 -0500 Cc: Andrey Yakovlev , Dominik Schoeffmann , "net@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <57601DFA.2040008@in.tum.de> <470941466032510@web23o.yandex.ru> To: Navdeep Parhar X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 00:04:56 -0000 Luiz Otavio O Souza (loos@) developed these for igb(4) and, by = extension, em(4) for use in netmap-fwd. He=E2=80=99s just gone back to Brazil with 82599 ixgb(4) hardware. = I=E2=80=99m sure he=E2=80=99ll develop similar patches for ixgb(4) in = the near future. Chelsio is also =E2=80=9Con the list=E2=80=9D, but I figured I=E2=80=99d = speak to np@ about it first. ;-) =20 We might do ixl(4) as well. =20 Before Luiz retired to Brazil, we discussed upstreaming these to = FreeBSD. We=E2=80=99re committed to make it happen, but I doubt they = make 11. Jim > On Jun 15, 2016, at 6:50 PM, Navdeep Parhar wrote: >=20 > On 06/15/2016 16:15, Andrey Yakovlev wrote: >> ive heard on bsdcan this year that some patches exist to add hwcsum = offloading to netmap, hope to see it chelsio at least >=20 > cxgbe/cxl is a bit sneaky and will let you override netmap (on tx = only). > The ncxl interfaces declare themselves capable of checksumming but all > such capabilities are disabled by default. Just enable txcsum on the > interface and the hardware will do checksum insertion on tx. No way = to > solve the rx part entirely within the driver -- netmap has to be = willing > to accept checksum related flags from the driver. >=20 > Regards, > Navdeep >=20 >>=20 >> --=20 >> ./andy >>=20 >>=20 >> 14.06.2016, 12:15, "Dominik Schoeffmann" : >>> Dear Netmap Developers, >>>=20 >>> during the course of my bachelor's thesis, I modified a packet = generator >>> called MoonGen [1] in order to utilize netmap. >>> One key component was to flexibly offload checksums for different = kinds >>> of packets (IPv4, UDP, TCP). >>> The ixgbe netmap patch was modified [2] in order to construct = context >>> descriptors and suitable data descriptors. This is implemented in = less >>> than 250 LoC (including pseudo-header calculations). >>> The man page states, that checksum offloading is available via = ethtool, >>> although a solution inside the netmap API might be a cleaner way for >>> applications to actually use these features. >>> Attached is a graph showing the performance implication of using >>> offloading in the current implementation. >>> As can be seen, offloading has only a minor impact. >>> When regarding this data (and comparing it to other frameworks), = please >>> keep in mind, that internally a lot of per-packet effort is needed = due >>> to the software architecture of the packet generator. >>>=20 >>> The question being: >>> Would it not make sense to include checksum offloading inside of = netmap >>> in order to accomodate applications operating on layer 3 and above? >>> As these programs need to calculate the checksums in software, it = would >>> be just as fast to move these calculations to the kernel for NICs >>> without checksum offloading support (and the kernel would act as a = library). >>> The problem which currently is imposed by the fact, that netmap = exports >>> the complete ring, is that context descriptors disrupt the data >>> descriptors, which is unpleasant for the application. >>> But you may find the data interesting nevertheless. >>>=20 >>> Best Regards, >>> Dominik Schoeffmann >>>=20 >>> [1] https://github.com/dschoeffm/MoonGen/tree/netmap >>> [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>=20 >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Thu Jun 16 03:51:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A168A47B8D for ; Thu, 16 Jun 2016 03:51:23 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3D96D18B5 for ; Thu, 16 Jun 2016 03:51:23 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mailman.ysv.freebsd.org (Postfix) id 3CE9AA47B8C; Thu, 16 Jun 2016 03:51:23 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C980A47B8B for ; Thu, 16 Jun 2016 03:51:23 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19CE618B3 for ; Thu, 16 Jun 2016 03:51:23 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mail-it0-x236.google.com with SMTP id e5so37463103ith.0 for ; Wed, 15 Jun 2016 20:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mNsyQ47o6jOd63/ioIGUb5k19w/rM5QZd22PNcbPY8E=; b=AVCdmlJrL4RypYU/Ddl6KLzr7mWRDmfpQxDo9BC3gZDlxSswqvREYiwtQWG0agGnsu ctmZN2ubBIOCx+HiGNFYuSNJQa3LqDW2AWHuYzTNlKA5c+gWdiZdQsfY1aD3DsljvuC8 22fEsnUBgi56wxEWn3XNcJ5qbggx5gJ8yiIFs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mNsyQ47o6jOd63/ioIGUb5k19w/rM5QZd22PNcbPY8E=; b=VjMV2XrRy1fkLGKg8o4gWmK2d4HnSxYS459TupIX+dnUT1yRTSs/W08yZZFA+bUbQh B/5gLMYAB1efSmJOX1un7ueTiKQCWnSNHhGIvNKIUcZFUov2tYHu78YALzLLJLxLKpkJ IrexWdnnXYf9enAD9Jtc2nFoz4wG3xSHaXnOw+vSZPtHSGZ2S/J94/muAQRbj1evdkIU 38Gg1n94+4kQ931wTZ4IIlH7gmXvjMgY4vO2tDIiyq3Bn3uUhJPrqjDhNKFQu3x6L/+8 DnJ2sZ/IpcGKiLWknf/i/1YGnvyjKztyhR/EtTLYHDPN62CQFT0xE6p6U/OeT2y6fGaV 0eGQ== X-Gm-Message-State: ALyK8tIXZYWnks+aayCl4yDkDV068lzzqED/P65ZeYLSi1J+33PT7lOflAfdHXmH1GMkDYupQ/9D+UzjbklOnDWW X-Received: by 10.36.2.2 with SMTP id 2mr4552246itu.34.1466049082265; Wed, 15 Jun 2016 20:51:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.156.167 with HTTP; Wed, 15 Jun 2016 20:51:21 -0700 (PDT) In-Reply-To: <57601DFA.2040008@in.tum.de> References: <57601DFA.2040008@in.tum.de> From: Jim Thompson Date: Wed, 15 Jun 2016 22:51:21 -0500 Message-ID: Subject: Re: Netmap Checksum Offloading To: Dominik Schoeffmann Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 03:51:23 -0000 We've focused on just the IP header checksum, but it's possible to add L4 checksum offload as well. I asked Luigi why he hadn't included checksum offload (with a library in software for devices that don't offer a hw offload), and his answer was that when he wrote netmap, he wanted a fast path to the wire. On Tue, Jun 14, 2016 at 10:08 AM, Dominik Schoeffmann wrote: > Dear Netmap Developers, > > during the course of my bachelor's thesis, I modified a packet generator > called MoonGen [1] in order to utilize netmap. > One key component was to flexibly offload checksums for different kinds > of packets (IPv4, UDP, TCP). > The ixgbe netmap patch was modified [2] in order to construct context > descriptors and suitable data descriptors. This is implemented in less > than 250 LoC (including pseudo-header calculations). > The man page states, that checksum offloading is available via ethtool, > although a solution inside the netmap API might be a cleaner way for > applications to actually use these features. > Attached is a graph showing the performance implication of using > offloading in the current implementation. > As can be seen, offloading has only a minor impact. > When regarding this data (and comparing it to other frameworks), please > keep in mind, that internally a lot of per-packet effort is needed due > to the software architecture of the packet generator. > > The question being: > Would it not make sense to include checksum offloading inside of netmap > in order to accomodate applications operating on layer 3 and above? > As these programs need to calculate the checksums in software, it would > be just as fast to move these calculations to the kernel for NICs > without checksum offloading support (and the kernel would act as a library). > The problem which currently is imposed by the fact, that netmap exports > the complete ring, is that context descriptors disrupt the data > descriptors, which is unpleasant for the application. > But you may find the data interesting nevertheless. > > > Best Regards, > Dominik Schoeffmann > > > > [1] https://github.com/dschoeffm/MoonGen/tree/netmap > [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading > From owner-freebsd-net@freebsd.org Thu Jun 16 06:41:40 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55FA9A69577 for ; Thu, 16 Jun 2016 06:41:40 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 308B71A6F for ; Thu, 16 Jun 2016 06:41:40 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: by mailman.ysv.freebsd.org (Postfix) id 2FE7BA69576; Thu, 16 Jun 2016 06:41:40 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F960A69575 for ; Thu, 16 Jun 2016 06:41:40 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: from mail-out1.informatik.tu-muenchen.de (mail-out1.informatik.tu-muenchen.de [131.159.0.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD9B41A6B; Thu, 16 Jun 2016 06:41:38 +0000 (UTC) (envelope-from schoeffm@in.tum.de) Received: (Authenticated sender: schoeffm) by mail.in.tum.de (Postfix) with ESMTPSA id 20B311C114E; Thu, 16 Jun 2016 08:41:30 +0200 (CEST) Subject: Re: Netmap Checksum Offloading To: Jim Thompson , Navdeep Parhar References: <57601DFA.2040008@in.tum.de> <470941466032510@web23o.yandex.ru> Cc: Andrey Yakovlev , "net@freebsd.org" From: Dominik Schoeffmann Message-ID: <576249FA.2050809@in.tum.de> Date: Thu, 16 Jun 2016 08:40:58 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Rp2hROBL0nIX7gVAa8o2At7qCNNanGtim" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 06:41:40 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Rp2hROBL0nIX7gVAa8o2At7qCNNanGtim Content-Type: multipart/mixed; boundary="o52g7rCliRbO2helNQB4tL0uESVrn2nBu" From: Dominik Schoeffmann To: Jim Thompson , Navdeep Parhar Cc: Andrey Yakovlev , "net@freebsd.org" Message-ID: <576249FA.2050809@in.tum.de> Subject: Re: Netmap Checksum Offloading References: <57601DFA.2040008@in.tum.de> <470941466032510@web23o.yandex.ru> In-Reply-To: --o52g7rCliRbO2helNQB4tL0uESVrn2nBu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Is the checksum offloading patch for the igb(4) driver available online? (I could not find it) I would really like to take a look at it, espacially the context descriptor part of it. Best regards, Dominik On 16.06.2016 02:04, Jim Thompson wrote: >=20 > Luiz Otavio O Souza (loos@) developed these for igb(4) and, by extensio= n, em(4) for use in netmap-fwd. >=20 > He=E2=80=99s just gone back to Brazil with 82599 ixgb(4) hardware. I=E2= =80=99m sure he=E2=80=99ll develop similar patches for ixgb(4) in the nea= r future. >=20 > Chelsio is also =E2=80=9Con the list=E2=80=9D, but I figured I=E2=80=99= d speak to np@ about it first. ;-) =20 > We might do ixl(4) as well. =20 >=20 > Before Luiz retired to Brazil, we discussed upstreaming these to FreeBS= D. We=E2=80=99re committed to make it happen, but I doubt they make 11. >=20 > Jim >=20 >> On Jun 15, 2016, at 6:50 PM, Navdeep Parhar wrote: >> >> On 06/15/2016 16:15, Andrey Yakovlev wrote: >>> ive heard on bsdcan this year that some patches exist to add hwcsum o= ffloading to netmap, hope to see it chelsio at least >> >> cxgbe/cxl is a bit sneaky and will let you override netmap (on tx only= ). >> The ncxl interfaces declare themselves capable of checksumming but all= >> such capabilities are disabled by default. Just enable txcsum on the >> interface and the hardware will do checksum insertion on tx. No way t= o >> solve the rx part entirely within the driver -- netmap has to be willi= ng >> to accept checksum related flags from the driver. >> >> Regards, >> Navdeep >> >>> >>> --=20 >>> ./andy >>> >>> >>> 14.06.2016, 12:15, "Dominik Schoeffmann" : >>>> Dear Netmap Developers, >>>> >>>> during the course of my bachelor's thesis, I modified a packet gener= ator >>>> called MoonGen [1] in order to utilize netmap. >>>> One key component was to flexibly offload checksums for different ki= nds >>>> of packets (IPv4, UDP, TCP). >>>> The ixgbe netmap patch was modified [2] in order to construct contex= t >>>> descriptors and suitable data descriptors. This is implemented in le= ss >>>> than 250 LoC (including pseudo-header calculations). >>>> The man page states, that checksum offloading is available via ethto= ol, >>>> although a solution inside the netmap API might be a cleaner way for= >>>> applications to actually use these features. >>>> Attached is a graph showing the performance implication of using >>>> offloading in the current implementation. >>>> As can be seen, offloading has only a minor impact. >>>> When regarding this data (and comparing it to other frameworks), ple= ase >>>> keep in mind, that internally a lot of per-packet effort is needed d= ue >>>> to the software architecture of the packet generator. >>>> >>>> The question being: >>>> Would it not make sense to include checksum offloading inside of net= map >>>> in order to accomodate applications operating on layer 3 and above? >>>> As these programs need to calculate the checksums in software, it wo= uld >>>> be just as fast to move these calculations to the kernel for NICs >>>> without checksum offloading support (and the kernel would act as a l= ibrary). >>>> The problem which currently is imposed by the fact, that netmap expo= rts >>>> the complete ring, is that context descriptors disrupt the data >>>> descriptors, which is unpleasant for the application. >>>> But you may find the data interesting nevertheless. >>>> >>>> Best Regards, >>>> Dominik Schoeffmann >>>> >>>> [1] https://github.com/dschoeffm/MoonGen/tree/netmap >>>> [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >>> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"= >=20 --o52g7rCliRbO2helNQB4tL0uESVrn2nBu-- --Rp2hROBL0nIX7gVAa8o2At7qCNNanGtim Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXYkoYAAoJEJiuUT5oYfeaANwQAKwExI//FOoLJ2VEsNlSO3E9 PyvdHPZhNjZWuwztiQGGN7KTF2uHbKtgnL5+dlks7XB4DvwOcU50UtbaNlP7bMtK EOG9QqJRjJ1DkRKm+ZgYS0Uu3pqevQFu7QmfeH2j4+NH89Yav86azwtvuGMlUEBh v+v5SJK0D3aZRupZ9nla5cClugIm4eKJdlGzRsypM1OG+FeE//kir1y6YHfSw00R 4La852QXZyFCmIzPy4b1hKKnVKYpYmP6+3J6lTBGE1OVfF3LyHT1RtNWetv+BGGp 7tRZLohNpK9EwwHLV1WkJbzXFkQsPf4o4y/GnVwPaah0ACT2WSMo324cQCSPq31F nBFrxyydD6LP0GGN4kp/T2T4DdmdSUXVYHX+Mo1e/C5hK/IflqdOm8q/Iy7lwmk/ qoJMg733SEAKrpZQsc2ACGn0/EEuxm9tM2q0BHbDvu9KMgbKpVz/ACsNY2iVIPh6 atQXO5rfDFRfGqPMbOamqNaw0x54S2fpyVKyCZVPghOZJZgsl/df3Kdq+EMIcDSZ qVd1h7Dq5UtmWtTgVqpyT+kJK6kOHDCyoY5oPg7glwFuNKLpEERpQl4jgpklUqTG xjkHGTA/TFmBYkgiu+RMlDk2/OhaPVRJ9KxuWEKJYm77lY5R/APuhpRPZ5S/pW3v 4mJy/6YNrWt10aibGLrP =ekw+ -----END PGP SIGNATURE----- --Rp2hROBL0nIX7gVAa8o2At7qCNNanGtim-- From owner-freebsd-net@freebsd.org Thu Jun 16 07:18:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F222A69D01 for ; Thu, 16 Jun 2016 07:18:14 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id EC69417F2 for ; Thu, 16 Jun 2016 07:18:13 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: by mailman.ysv.freebsd.org (Postfix) id EBBE8A69D00; Thu, 16 Jun 2016 07:18:13 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB61FA69CFF for ; Thu, 16 Jun 2016 07:18:13 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 94AF617F0; Thu, 16 Jun 2016 07:18:13 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from [IPv6:2a02:c6a0:3061:50e1:6002:30fd:f135:9450] (unknown [IPv6:2a02:c6a0:3061:50e1:6002:30fd:f135:9450]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 629C9721E280C; Thu, 16 Jun 2016 09:18:04 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Netmap Checksum Offloading From: Michael Tuexen In-Reply-To: <576249FA.2050809@in.tum.de> Date: Thu, 16 Jun 2016 09:18:03 +0200 Cc: Jim Thompson , Navdeep Parhar , "net@freebsd.org" , Andrey Yakovlev Content-Transfer-Encoding: quoted-printable Message-Id: References: <57601DFA.2040008@in.tum.de> <470941466032510@web23o.yandex.ru> <576249FA.2050809@in.tum.de> To: Dominik Schoeffmann X-Mailer: Apple Mail (2.3124) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 07:18:14 -0000 > On 16 Jun 2016, at 08:40, Dominik Schoeffmann = wrote: >=20 > Is the checksum offloading patch for the igb(4) driver available = online? > (I could not find it) > I would really like to take a look at it, espacially the context > descriptor part of it. I would also be interested in it... For looking at SCTP checksum = offloading. Best regards Michael >=20 > Best regards, > Dominik >=20 > On 16.06.2016 02:04, Jim Thompson wrote: >>=20 >> Luiz Otavio O Souza (loos@) developed these for igb(4) and, by = extension, em(4) for use in netmap-fwd. >>=20 >> He=E2=80=99s just gone back to Brazil with 82599 ixgb(4) hardware. = I=E2=80=99m sure he=E2=80=99ll develop similar patches for ixgb(4) in = the near future. >>=20 >> Chelsio is also =E2=80=9Con the list=E2=80=9D, but I figured I=E2=80=99= d speak to np@ about it first. ;-) =20 >> We might do ixl(4) as well. =20 >>=20 >> Before Luiz retired to Brazil, we discussed upstreaming these to = FreeBSD. We=E2=80=99re committed to make it happen, but I doubt they = make 11. >>=20 >> Jim >>=20 >>> On Jun 15, 2016, at 6:50 PM, Navdeep Parhar wrote: >>>=20 >>> On 06/15/2016 16:15, Andrey Yakovlev wrote: >>>> ive heard on bsdcan this year that some patches exist to add hwcsum = offloading to netmap, hope to see it chelsio at least >>>=20 >>> cxgbe/cxl is a bit sneaky and will let you override netmap (on tx = only). >>> The ncxl interfaces declare themselves capable of checksumming but = all >>> such capabilities are disabled by default. Just enable txcsum on = the >>> interface and the hardware will do checksum insertion on tx. No way = to >>> solve the rx part entirely within the driver -- netmap has to be = willing >>> to accept checksum related flags from the driver. >>>=20 >>> Regards, >>> Navdeep >>>=20 >>>>=20 >>>> --=20 >>>> ./andy >>>>=20 >>>>=20 >>>> 14.06.2016, 12:15, "Dominik Schoeffmann" : >>>>> Dear Netmap Developers, >>>>>=20 >>>>> during the course of my bachelor's thesis, I modified a packet = generator >>>>> called MoonGen [1] in order to utilize netmap. >>>>> One key component was to flexibly offload checksums for different = kinds >>>>> of packets (IPv4, UDP, TCP). >>>>> The ixgbe netmap patch was modified [2] in order to construct = context >>>>> descriptors and suitable data descriptors. This is implemented in = less >>>>> than 250 LoC (including pseudo-header calculations). >>>>> The man page states, that checksum offloading is available via = ethtool, >>>>> although a solution inside the netmap API might be a cleaner way = for >>>>> applications to actually use these features. >>>>> Attached is a graph showing the performance implication of using >>>>> offloading in the current implementation. >>>>> As can be seen, offloading has only a minor impact. >>>>> When regarding this data (and comparing it to other frameworks), = please >>>>> keep in mind, that internally a lot of per-packet effort is needed = due >>>>> to the software architecture of the packet generator. >>>>>=20 >>>>> The question being: >>>>> Would it not make sense to include checksum offloading inside of = netmap >>>>> in order to accomodate applications operating on layer 3 and = above? >>>>> As these programs need to calculate the checksums in software, it = would >>>>> be just as fast to move these calculations to the kernel for NICs >>>>> without checksum offloading support (and the kernel would act as a = library). >>>>> The problem which currently is imposed by the fact, that netmap = exports >>>>> the complete ring, is that context descriptors disrupt the data >>>>> descriptors, which is unpleasant for the application. >>>>> But you may find the data interesting nevertheless. >>>>>=20 >>>>> Best Regards, >>>>> Dominik Schoeffmann >>>>>=20 >>>>> [1] https://github.com/dschoeffm/MoonGen/tree/netmap >>>>> [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>=20 >=20 From owner-freebsd-net@freebsd.org Thu Jun 16 17:26:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2650A7686F for ; Thu, 16 Jun 2016 17:26:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D74E91FAA for ; Thu, 16 Jun 2016 17:26:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GHQo21054419 for ; Thu, 16 Jun 2016 17:26:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 17:26:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 17:26:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 arcadiy@ivanovy.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arcadiy@ivanovy.net --- Comment #1 from arcadiy@ivanovy.net --- I can report the same for 10.3-RELEASE-p4 and stock kernel. # cat /boot/loader.conf: kern.geom.label.gptid.enable=3D"0" kern.ipc.nmbclusters=3D"1000000" hw.pci.enable_msi=3D"1" hw.pci.enable_msix=3D"0" zfs_load=3D"YES" aesni_load=3D"YES" ichsmb_load=3D"YES" ipmi_load=3D"YES" beastie_disable=3D"YES" autoboot_delay=3D"3" # pciconf -lcv igb0@pci0:2:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet cap 01[40] =3D powerspec 3 supports D0 D3 current D0 cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks enabled wit= h 1 message cap 11[70] =3D MSI-X supports 5 messages Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) FLR RO NS link = x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] =3D Serial 1 002590ffffxxxxx2 ecap 0017[1a0] =3D TPH Requester 1 igb1@pci0:5:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet cap 01[40] =3D powerspec 3 supports D0 D3 current D0 cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks enabled wit= h 1 message cap 11[70] =3D MSI-X supports 5 messages Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) FLR RO NS link = x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] =3D Serial 1 002590ffffxxxxx3 ecap 0017[1a0] =3D TPH Requester 1 This configuration is fairly stable (1h so far, no watchdog timeouts on igb= 1). Disabling MSI altogether causes igb1 that doesn't pass any traffic and does= n't even come up. Disabling MSI-X seems to help. This is a SuperMicro X11SBA-F board (http://www.supermicro.com/products/motherboard/X11/X11SBA-F.cfm) with BIOS 1.0b (latest). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jun 16 17:37:49 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9C84A76B84 for ; Thu, 16 Jun 2016 17:37:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9F5D177F for ; Thu, 16 Jun 2016 17:37:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GHbnou075685 for ; Thu, 16 Jun 2016 17:37:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 17:37:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 17:37:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 --- Comment #2 from arcadiy@ivanovy.net --- Spoke too soon, disabling MSI-X seems to help only marginally: Jun 16 13:29:49 fw1 kernel: igb1: Watchdog timeout -- resetting Jun 16 13:29:49 fw1 kernel: igb1: Queue(846295657) tdh =3D -1249464976, hw tdt =3D 589458993 Jun 16 13:29:49 fw1 kernel: igb1: TX(846295657) desc avail =3D = 0,Next TX to Clean =3D 0 Jun 16 13:29:49 fw1 kernel: igb1: link state changed to DOWN Jun 16 13:29:53 fw1 kernel: igb1: link state changed to UP Jun 16 13:29:53 fw1 devd: Executing '/etc/rc.d/dhclient quietstart igb1' Jun 16 13:34:26 fw1 kernel: igb1: Watchdog timeout -- resetting Jun 16 13:34:26 fw1 kernel: igb1: Queue(846295657) tdh =3D -1249464976, hw tdt =3D 589458993 Jun 16 13:34:26 fw1 kernel: igb1: TX(846295657) desc avail =3D = 0,Next TX to Clean =3D 0 Jun 16 13:34:26 fw1 kernel: igb1: link state changed to DOWN Jun 16 13:34:31 fw1 kernel: igb1: link state changed to UP Jun 16 13:34:31 fw1 devd: Executing '/etc/rc.d/dhclient quietstart igb1' --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jun 16 17:43:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B29D9A76F05 for ; Thu, 16 Jun 2016 17:43:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2B181D77 for ; Thu, 16 Jun 2016 17:43:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GHhJMg090046 for ; Thu, 16 Jun 2016 17:43:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 17:43:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 17:43:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 --- Comment #3 from arcadiy@ivanovy.net --- Possibly related to #200221 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jun 16 17:44:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DD17A77013 for ; Thu, 16 Jun 2016 17:44:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DD6A1FAD for ; Thu, 16 Jun 2016 17:44:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GHiVU3091532 for ; Thu, 16 Jun 2016 17:44:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 17:44:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 17:44:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 --- Comment #4 from arcadiy@ivanovy.net --- igb0: flags=3D8843 metric 0 mtu 1500 options=3Dbb igb1: flags=3D8843 metric 0 mtu 1500 options=3Dbb I have TSO, VLAN_HWTSO and LRO disabled. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jun 16 18:41:52 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1C2DA77EA5 for ; Thu, 16 Jun 2016 18:41:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D1E60150C for ; Thu, 16 Jun 2016 18:41:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GIfqAT012259 for ; Thu, 16 Jun 2016 18:41:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 18:41:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 18:41:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 --- Comment #5 from arcadiy@ivanovy.net --- some PCI-E errors: pcib4@pci0:3:0:0: class=3D0x060400 card=3D0x00000000 chip=3D0x260812d= 8 rev=3D0x00 hdr=3D0x01 vendor =3D 'Pericom Semiconductor' class =3D bridge subclass =3D PCI-PCI PCI-e errors =3D Correctable Error Detected Unsupported Request Detected Corrected =3D Receiver Error Bad TLP Bad DLLP REPLAY_NUM Rollover Replay Timer Timeout Advisory Non-Fatal Error igb0@pci0:2:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet Corrected =3D Advisory Non-Fatal Error igb1@pci0:5:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet PCI-e errors =3D Correctable Error Detected Corrected =3D Advisory Non-Fatal Error --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jun 16 19:02:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6833A772CC for ; Thu, 16 Jun 2016 19:02:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6C002110 for ; Thu, 16 Jun 2016 19:02:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5GJ2WN2089173 for ; Thu, 16 Jun 2016 19:02:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 200420] [igb] igb0: Watchdog timeout -- resetting Date: Thu, 16 Jun 2016 19:02:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arcadiy@ivanovy.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2016 19:02:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200420 --- Comment #6 from arcadiy@ivanovy.net --- # netstat -m 2048/3772/5820 mbufs in use (current/cache/total) 2046/2514/4560/1000000 mbuf clusters in use (current/cache/total/max) 2046/2508 mbuf+clusters out of packet secondary zone in use (current/cache) 0/4/4/250101 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/74104 9k jumbo clusters in use (current/cache/total/max) 0/0/0/41683 16k jumbo clusters in use (current/cache/total/max) 4604K/5987K/10591K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jun 17 01:40:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18C9CA471E2 for ; Fri, 17 Jun 2016 01:40:28 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-yw0-x22d.google.com (mail-yw0-x22d.google.com [IPv6:2607:f8b0:4002:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C729D196F for ; Fri, 17 Jun 2016 01:40:27 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-yw0-x22d.google.com with SMTP id v137so59514611ywa.3 for ; Thu, 16 Jun 2016 18:40:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ubzQLIjagyBVGGph5oTAmcaCu2XATcM94icQmTtNr3s=; b=EBYgySYFggXPltYZEy1A83S9xhNifzrALFDGdL9u1dTlIRwthEiNVfJnCR9mqtCiR+ gExAun5o/oAfNwUwCH1ckV3eBlybRp5GYy6HUzWmTUxCB5H6gUeBdPSkSF6gZjHkTlUJ UuBv7C4iAeO3/ldFW85dUt2zp81eFBNwgCPtzcBytnu11S5kzwYY71tB6KsHIDJckXY5 XA/6EwVUY+cJmZ81UunyiopCKCzdCs5eIcsrmyjxtkbZMFRzwFRLrnI/NZTpiGRs9O5I dCU4BEe3ucoACMd83iN+rsBMSIa1iVqnyNq1rqkGv7OSEGQ5bD4TWhafTteBrPgaau7Y zl1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ubzQLIjagyBVGGph5oTAmcaCu2XATcM94icQmTtNr3s=; b=BpT+rVRS/g6fi+Bj5H1/H0UWilo+MOOyJs8fhIvYkykYM/s3GHzp5HPdyDQeVupzjj RDiZcSkUGoq0pOYpnUcW0E+x+etoWSL8gHzAD/GNafxouxr/35ancGA+NwlQf9howLSI zsOZd6fSmILUfJ9F7a0a559/aZDoVXdQd4nqWVmcAolWKufQ3N0uo+hIkG7EyYEenOqm HQ3CARBmP4ZrscQ+8Lw4XaWEdZFs88RkDZh1ToGDtujgm8dd9IKVv6tWjHgibw6LwSDV nKoRm9p8c39zDfcmAxaCqMhWNW3eAPhvHREPmgn4fUeOEJb6MxBmh/1gP7C65nNJBIs/ bphQ== X-Gm-Message-State: ALyK8tIUiKuVJMB6l0QczKfDAXM1zm5S3mHyasU51rRXs22D/si3iv+pn6l4fBTEqs+7idPZOepfVL7j/5NL7A== X-Received: by 10.13.204.22 with SMTP id o22mr5019201ywd.203.1466127627051; Thu, 16 Jun 2016 18:40:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.212.195 with HTTP; Thu, 16 Jun 2016 18:40:26 -0700 (PDT) In-Reply-To: <1502708678.10983095.1465385308094.JavaMail.zimbra@ulg.ac.be> References: <5756C17D.1090409@yandex.ru> <1502708678.10983095.1465385308094.JavaMail.zimbra@ulg.ac.be> From: Andrew Vylegzhanin Date: Fri, 17 Jun 2016 04:40:26 +0300 Message-ID: Subject: Re: Is netmap jumbo frames broken in STABLE? To: freebsd-net@freebsd.org Cc: Luigi Rizzo , "Andrey V. Elsukov" , Ryan Stone , tom.barbette@ulg.ac.be Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 01:40:28 -0000 About week ago I've patched if_ix.c, just returned back code fragment of max_frame_size determination from 2.8.3 version of ixgbe driver: /* ** Determine the correct mbuf pool ** for doing jumbo frames */ if (adapter->max_frame_size <= 2048) adapter->rx_mbuf_sz = MCLBYTES; else if (adapter->max_frame_size <= 4096) adapter->rx_mbuf_sz = MJUMPAGESIZE; else if (adapter->max_frame_size <= 9216) adapter->rx_mbuf_sz = MJUM9BYTES; else adapter->rx_mbuf_sz = MJUM16BYTES; After week of heavy testing everything is ok. This solution is acceptable for me for now, because I use eight interfaces in netmap mode only. In future I want to make support something like dev.ix.N.jumbo_mbuf_enable variable, since one of ix interfaces will be used for generic kernel stack with jumbo frame support. Doing with fragmented packet in netmap ring is also possible way, but need more user cpu cycles for processing, I guess. In any case I can test it when this patch will be merged for netmap. -- Andrew 2016-06-08 14:28 GMT+03:00 : > > Support for fragmented packets with ixgbe was recently added on the linux version of Netmap : > > https://github.com/luigirizzo/netmap/commit/fc1e77560a8a8ea93cc3594de5fae94334debcd3 > > I think the change for freebsd would be quite the same looking at https://github.com/freebsd/freebsd/blob/master/sys/dev/netmap/ixgbe_netmap.h#L396 > > After that, your userspace application simply have to check for the NS_MOREFRAG flag in the receive ring, and if it's set he knows the end of the packet will follow in the next buf. > > Tom From owner-freebsd-net@freebsd.org Fri Jun 17 03:05:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 871B8A774AC for ; Fri, 17 Jun 2016 03:05:50 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: from mail-pa0-x22c.google.com (mail-pa0-x22c.google.com [IPv6:2607:f8b0:400e:c03::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 58EC31652; Fri, 17 Jun 2016 03:05:50 +0000 (UTC) (envelope-from matt.joras@gmail.com) Received: by mail-pa0-x22c.google.com with SMTP id b5so23868178pas.3; Thu, 16 Jun 2016 20:05:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=sqDYN8tlLWFlkmE6nUSz772jlbwVAj/Hlq2wxflMt24=; b=B2trNH5Iz5zzVtrI1sa9nzaLOO+qwAOQFJjXT1w3TKwxb2UsMnupXgmRmhlWWAlqo8 8GA+6FQEOz6x0d86tBHuokMkwwkAHFQaeGnmvc4no99MOiMz8mwqOdf9qrPKCkfkdtbR yA5bEqNaYm5fYRtidIe+gmwrXaIg0eNK9PJudefeF2j4Dw3WIZTaSM2DFvbfz0wf5XVp rhQrVlCvaXjH7JPXVqXJQE/amOMRQPV0UkK5aZWS8vNrNvTSTI5E82jvtiOMx2zGU9lo CdKY4bnW7m141nrgDGizNIvxhpk80tBZM42DH0Xdeg67ishJ9BTMYpCBqsveLt9yMAYd 5NSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=sqDYN8tlLWFlkmE6nUSz772jlbwVAj/Hlq2wxflMt24=; b=Js5EWnKfZDkYf/L5NulNdr5gcJZdbqGlWCGdvid2MAGBeTI62hkDRpLtOIPaw7hUY4 rlQvSZwlgk0myPn4f9CvuogmUVQo0xVmeapZQ9hdneTuWEHDXfbHfYiXC+c09TXCqyup WU9tFSe/Tz0xGPLLnvmJu6tZ1eskoEZhpli5NSo+hw1CHf3hVo4B9+3AL1soS55ma+ac xIzd5iFnlsSNhVB4+I0lTyUKMw3K8GvWls+RclasuKjtGDsmt4Y7PtNqdea8g27/q7oz rCKZNkyXBA7O3Gv9QWHFgxWQ0JsBXUK3DeVviRvVGYyoeXiF6yN7M9V2tpYET2saquUn uTlw== X-Gm-Message-State: ALyK8tJljLsh8VMRjtWlloEBt5E7cH5hcNTR27wdCjxXW7hNYcdHXeuMd5p54Lt3/If73w== X-Received: by 10.66.122.196 with SMTP id lu4mr9143310pab.52.1466132749815; Thu, 16 Jun 2016 20:05:49 -0700 (PDT) Received: from buzzell ([23.252.50.43]) by smtp.googlemail.com with ESMTPSA id g86sm63777809pfd.67.2016.06.16.20.05.48 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Jun 2016 20:05:49 -0700 (PDT) Message-ID: <1466132748.8545.5.camel@gmail.com> Subject: Re: lagg(4): LOR, deadlock and panic From: Matt Joras To: Alan Somers , Sean Bruno Cc: "freebsd-net@freebsd.org" Date: Thu, 16 Jun 2016 20:05:48 -0700 In-Reply-To: References: <459d2639-b490-beee-9cd4-05f38983eaed@freebsd.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.20.2 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 03:05:50 -0000 On Tue, 2016-06-14 at 09:26 -0600, Alan Somers wrote: > > I don't know the best answer either.  But while you're in there, are > you interested in fixing any other lagg panics too?  I've written > some > ATF torture tests for lagg, but I haven't checked them into head yet > because most of them quickly panic. > > -Alan We run into if_lagg and if_vlan panics at $WORK all the time in our automation. I've fixed the if_vlan panics and I'm hoping to update this review: https://reviews.freebsd.org/D5825 soon with something that accomodates drivers sleeping in the vlan_*config event handlers (which involves having a global rmlock _and_ sx in if_vlan). I was planning on doing a similar audit/fixing of if_lagg soon when I get the chance. Matt Joras From owner-freebsd-net@freebsd.org Fri Jun 17 04:53:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3028EA7889D for ; Fri, 17 Jun 2016 04:53:27 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 1DD7612A4 for ; Fri, 17 Jun 2016 04:53:27 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 19E3CA7889B; Fri, 17 Jun 2016 04:53:27 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19368A78899; Fri, 17 Jun 2016 04:53:27 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 81A7A129F; Fri, 17 Jun 2016 04:53:26 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id u5H4rJH9075730 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 16 Jun 2016 21:53:19 -0700 (PDT) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id u5H4rJvq075729; Thu, 16 Jun 2016 21:53:19 -0700 (PDT) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 16 Jun 2016 21:53:19 -0700 From: Gleb Smirnoff To: jch@FreeBSD.org, hselasky@FreeBSD.org Cc: rrs@FreeBSD.org, net@FreeBSD.org, current@FreeBSD.org Subject: panic with tcp timers Message-ID: <20160617045319.GE1076@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 04:53:27 -0000 Hi! At Netflix we are observing a race in TCP timers with head. The problem is a regression, that doesn't happen on stable/10. The panic usually happens after several hours at 55 Gbit/s of traffic. What happens is that tcp_timer_keep finds t_tcpcb being NULL. Some coredumps have tcpcb already initialized, with non-NULL t_tcpcb and in TCPS_ESTABLISHED state. Which means that other CPU was working on the tcpcb while the faulted one was working on the panic. So, this all looks like a use after free, which conflicts with new allocation. Comparing stable/10 and head, I see two changes that could affect that: - callout_async_drain - switch to READ lock for inp info in tcp timers That's why you are in To, Julien and Hans :) We continue investigating, and I will keep you updated. However, any help is welcome. I can share cores. -- Totus tuus, Glebius. From owner-freebsd-net@freebsd.org Fri Jun 17 11:07:31 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1F89A78EA0 for ; Fri, 17 Jun 2016 11:07:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A050D1846 for ; Fri, 17 Jun 2016 11:07:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.ysv.freebsd.org (Postfix) id 99596A78E9E; Fri, 17 Jun 2016 11:07:31 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9892CA78E9C; Fri, 17 Jun 2016 11:07:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6169B1844; Fri, 17 Jun 2016 11:07:30 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 6383E1FE023; Fri, 17 Jun 2016 13:07:22 +0200 (CEST) Subject: Re: panic with tcp timers To: Gleb Smirnoff , jch@FreeBSD.org References: <20160617045319.GE1076@FreeBSD.org> Cc: rrs@FreeBSD.org, net@FreeBSD.org, current@FreeBSD.org From: Hans Petter Selasky Message-ID: Date: Fri, 17 Jun 2016 13:10:57 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <20160617045319.GE1076@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 11:07:31 -0000 On 06/17/16 06:53, Gleb Smirnoff wrote: > Hi! > > At Netflix we are observing a race in TCP timers with head. > The problem is a regression, that doesn't happen on stable/10. > The panic usually happens after several hours at 55 Gbit/s of > traffic. > > What happens is that tcp_timer_keep finds t_tcpcb being > NULL. Some coredumps have tcpcb already initialized, > with non-NULL t_tcpcb and in TCPS_ESTABLISHED state. Which > means that other CPU was working on the tcpcb while > the faulted one was working on the panic. So, this all looks > like a use after free, which conflicts with new allocation. > > Comparing stable/10 and head, I see two changes that could > affect that: > > - callout_async_drain > - switch to READ lock for inp info in tcp timers > > That's why you are in To, Julien and Hans :) > > We continue investigating, and I will keep you updated. > However, any help is welcome. I can share cores. > Hi, I do have projects/hps_head around, which is not that much behind 11-current, which has a completely different callout implementation. If you can reproduce the issue separately might we worth a try to rule out the callout stack. --HPS From owner-freebsd-net@freebsd.org Fri Jun 17 14:41:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92A6CA77CD3 for ; Fri, 17 Jun 2016 14:41:19 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 785182642 for ; Fri, 17 Jun 2016 14:41:19 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: by mailman.ysv.freebsd.org (Postfix) id 77AA2A77CD1; Fri, 17 Jun 2016 14:41:19 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 772CDA77CD0; Fri, 17 Jun 2016 14:41:19 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A55A2640; Fri, 17 Jun 2016 14:41:19 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id AAE3925D388E; Fri, 17 Jun 2016 14:41:06 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id D27C9D1F8BC; Fri, 17 Jun 2016 14:41:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id uNrClogwaQYJ; Fri, 17 Jun 2016 14:41:04 +0000 (UTC) Received: from [192.168.124.1] (unknown [IPv6:fde9:577b:c1a9:4410:392c:10a8:d881:6c0f]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id AAC8CD1F7FA; Fri, 17 Jun 2016 14:41:03 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Gleb Smirnoff" Cc: jch@FreeBSD.org, hselasky@FreeBSD.org, rrs@FreeBSD.org, current@FreeBSD.org, net@FreeBSD.org Subject: Re: panic with tcp timers Date: Fri, 17 Jun 2016 14:41:02 +0000 Message-ID: In-Reply-To: <20160617045319.GE1076@FreeBSD.org> References: <20160617045319.GE1076@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailer: MailMate Trial (2.0BETAr6032) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 14:41:19 -0000 On 17 Jun 2016, at 4:53, Gleb Smirnoff wrote: > Hi! > > At Netflix we are observing a race in TCP timers with head. > The problem is a regression, that doesn't happen on stable/10. > The panic usually happens after several hours at 55 Gbit/s of > traffic. > > What happens is that tcp_timer_keep finds t_tcpcb being > NULL. Some coredumps have tcpcb already initialized, > with non-NULL t_tcpcb and in TCPS_ESTABLISHED state. Which > means that other CPU was working on the tcpcb while > the faulted one was working on the panic. So, this all looks > like a use after free, which conflicts with new allocation. > > Comparing stable/10 and head, I see two changes that could > affect that: > > - callout_async_drain > - switch to READ lock for inp info in tcp timers > > That's why you are in To, Julien and Hans :) > > We continue investigating, and I will keep you updated. > However, any help is welcome. I can share cores. There’s also the change to no longer mark the zones NO_FREE. In theory I was convinced at the time that it should not be an issue anymore. If I had overlooked something or follow-up timer changes invalidated assumptions then that could also be trouble. That said, I was not able to get any related panics or log entries anymore lately (but I am currently slightly behind head with my branch). We should get the problem fixed however and not try to “paint over” again. /bz From owner-freebsd-net@freebsd.org Fri Jun 17 14:51:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72599A77F94 for ; Fri, 17 Jun 2016 14:51:22 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 515972BF5 for ; Fri, 17 Jun 2016 14:51:22 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4D900A77F92; Fri, 17 Jun 2016 14:51:22 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A827A77F90; Fri, 17 Jun 2016 14:51:22 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-f51.google.com (mail-wm0-f51.google.com [74.125.82.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D28BF2BF3; Fri, 17 Jun 2016 14:51:21 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-f51.google.com with SMTP id a66so3028224wme.0; Fri, 17 Jun 2016 07:51:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=6OY9gjWbYgOcgcVDQVySHWPAeT0fFPIwXKTx6HtXeBo=; b=MCTi+qXHRsFeL/RpIYwFIa7gvXMb8DDbKMN/EcGXDlaUHnwMkD8g/ZE4PwtH7laQS9 aapzakL7GYUUNC/uV0+rH/7zq037ff1mDoMQpsXhJSDjt/wtx2g8DLd8Nlgz/SDFI/g4 PgmMCZr86KN3dWkHZmBCAJQINXO+4vLbLUG1wct9kCKdLIqd7BlLEHcvJtF7tHboPCD5 aQ92HTqslu3sKVcPU9cfXHdGy1oJQmktZKFCqVatQdNpWeSjbUloPZ+9JkPS0+0FvLgt k94iLFVmtstC4tLhGeDMes5EQSilfPZPBtMvg7r6DfBSO79F800fRFXIed73B0uggtTl dNaw== X-Gm-Message-State: ALyK8tJ1EvphMzuptAYnb+JiXxgHbcnWlE6CJ38Sg0VBf8T1Fqlrx0VX40ZEWX+FQFk3kQ== X-Received: by 10.194.151.73 with SMTP id uo9mr1199856wjb.177.1466155666454; Fri, 17 Jun 2016 02:27:46 -0700 (PDT) Received: from [10.100.64.43] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id x124sm19060810wmg.24.2016.06.17.02.27.44 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 17 Jun 2016 02:27:45 -0700 (PDT) Subject: Re: panic with tcp timers To: Gleb Smirnoff , hselasky@FreeBSD.org References: <20160617045319.GE1076@FreeBSD.org> Cc: rrs@FreeBSD.org, net@FreeBSD.org, current@FreeBSD.org From: Julien Charbon Message-ID: <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> Date: Fri, 17 Jun 2016 11:27:39 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <20160617045319.GE1076@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="FGNRPDI6hUU9D3ssP7mvM3Up4quRr6JfV" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 14:51:22 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --FGNRPDI6hUU9D3ssP7mvM3Up4quRr6JfV Content-Type: multipart/mixed; boundary="iNgJ25Kd6kg6dTtPv0T8RCFla3TDH3fl9" From: Julien Charbon To: Gleb Smirnoff , hselasky@FreeBSD.org Cc: rrs@FreeBSD.org, net@FreeBSD.org, current@FreeBSD.org Message-ID: <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> Subject: Re: panic with tcp timers References: <20160617045319.GE1076@FreeBSD.org> In-Reply-To: <20160617045319.GE1076@FreeBSD.org> --iNgJ25Kd6kg6dTtPv0T8RCFla3TDH3fl9 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Gleb, On 6/17/16 6:53 AM, Gleb Smirnoff wrote: > At Netflix we are observing a race in TCP timers with head. > The problem is a regression, that doesn't happen on stable/10. > The panic usually happens after several hours at 55 Gbit/s of > traffic. >=20 > What happens is that tcp_timer_keep finds t_tcpcb being > NULL. Some coredumps have tcpcb already initialized, > with non-NULL t_tcpcb and in TCPS_ESTABLISHED state. Which > means that other CPU was working on the tcpcb while > the faulted one was working on the panic. So, this all looks > like a use after free, which conflicts with new allocation. >=20 > Comparing stable/10 and head, I see two changes that could > affect that: >=20 > - callout_async_drain > - switch to READ lock for inp info in tcp timers >=20 > That's why you are in To, Julien and Hans :) >=20 > We continue investigating, and I will keep you updated. > However, any help is welcome. I can share cores. Thanks for sharing. Let me run our TCP tests on a recent version of HEAD to see if by chance I can reproduce it. If I am not able to reproduce it I will ask for debug kernel and cores and see if I can help.= Few notes here: - Around 2 months ago I did test HEAD with callout_async_drain() in TCP timers with our TCP QA testsuite but no kernel panic. That said I did not let our test run during several hours. - At Verisign we run 10 with READ lock for inp info in tcp timers change. Again, it does not mean this change has no impact here. My 2 cents. -- Julien --iNgJ25Kd6kg6dTtPv0T8RCFla3TDH3fl9-- --FGNRPDI6hUU9D3ssP7mvM3Up4quRr6JfV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJXY8KPAAoJEKVlQ5Je6dhxY2YH/RMWLRYKV0VwtKNw6YgGhLss JaZhOzuHg6W751fBk1LXGJp1pg3CICVMtRX7jQVtGVjAPiT4en6M0M2DzHlgb8un IFUfnwAfP9DSdIpclzc8vOci4QBI3inziIuQ5vLDayuExS1gswZk8fRSkW9BroVu 4TVIPk7vVLyK5bo/VlWK8e1+d5Ypdd+2rGKPinB28GVmBwejWf0GnTV80O/Qr2JE jBldQM44ZU0nnxUj/yIq8NiswoTGQxdx2h4KPnCLIe+BJ6lygYMwrg8LdGbH/359 s0yiJoiwhPAmhvaS73dPmps7WUtS2e+QPq001r+IdNebWjXW8OwvbExGNHrH8pQ= =PA0C -----END PGP SIGNATURE----- --FGNRPDI6hUU9D3ssP7mvM3Up4quRr6JfV--