From owner-freebsd-net@FreeBSD.ORG Mon Jun 22 12:01:48 2015 Return-Path: Delivered-To: freebsd-net@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 74DB73C1 for ; Mon, 22 Jun 2015 12:01:48 +0000 (UTC) (envelope-from asossi@dotcom.ts.it) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0138.outbound.protection.outlook.com [157.55.234.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D9A8CAFC for ; Mon, 22 Jun 2015 12:01:47 +0000 (UTC) (envelope-from asossi@dotcom.ts.it) Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none; Received: from [192.168.0.2] (79.60.165.11) by DBXPR06MB416.eurprd06.prod.outlook.com (10.141.14.154) with Microsoft SMTP Server (TLS) id 15.1.190.14; Mon, 22 Jun 2015 12:01:38 +0000 Message-ID: <5587F8F7.2000700@dotcom.ts.it> Date: Mon, 22 Jun 2015 14:00:55 +0200 From: Andrej Sossi Organization: DOTCOM S.R.L. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Subject: Strange problem with TCP checksum Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [79.60.165.11] X-ClientProxiedBy: AM3PR05CA0049.eurprd05.prod.outlook.com (25.162.114.17) To DBXPR06MB416.eurprd06.prod.outlook.com (10.141.14.154) X-Microsoft-Exchange-Diagnostics: 1; DBXPR06MB416; 2:0hmtzlpkf0TjjRQtRNFkFkiFM9IOugPfjL3eHl1e/9dMk0EvyW5EnOHzb2qwhoqE; 2:T7lUQEW6FHCNcFUol9gn+gRC78g2QezH3VVxeZaluA3RmWjXv5kfmbx0BzncwM6XNdHKpRwAQBtpxzhIep2kcMZJu5nPZoaczri5QR8w8Gx09NxQ9wO0pJcOYXIj9Pv+Dd2I3E01vDFYeX9khJZuHw==; 3:VKMGFuPif2WmlTPmiL53t78HBiXThU1lKfEIg94+YhQUIxdEY2UgPWAqBIVxoutPEfmPYv/bWaLTeIvc13Q0Ha//IvYv/a4/k0ku40mdPNSlTk1D5507lapOozraUC+CEtcChCY5J2gLEnFSKy5M012J+TTvAMhR7FbHyBvlSPyLe0aEfTubi/vdQdZa07+UdQ/AWM2HBM5dWobrztTfu64eQMnFB4pn2lxhczfjKyd7vvnViBlZdKMx+4M6Gvq6orFbA1zSdYLfv1MNDZkK1nGPw8cXCJvIioGYoChliTUx19N/US9sri8Na1/nzoKg X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DBXPR06MB416; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(520004)(5005006)(3002001); SRVR:DBXPR06MB416; BCL:0; PCL:0; RULEID:; SRVR:DBXPR06MB416; X-Forefront-PRVS: 06157D541C X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(6009001)(6049001)(51164003)(110136002)(107886002)(80316001)(5001920100001)(77096005)(92566002)(19580395003)(19580405001)(5001960100002)(74482002)(46102003)(33656002)(189998001)(50466002)(122386002)(42186005)(117156001)(77156002)(40100003)(62966003)(450100001)(23676002)(50986999)(65816999)(54356999)(65956001)(66066001)(47776003)(36756003)(4001350100001)(86362001)(83506001)(87976001)(229853001)(2351001)(64126003)(554374003); DIR:OUT; SFP:1102; SCL:1; SRVR:DBXPR06MB416; H:[192.168.0.2]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtEQlhQUjA2TUI0MTY7OTo5MGRQYzdCS0lOTmNBb0l1QVliaEQyckQ1RlR2?= =?utf-8?B?VHI4TGExTG9NaHRxZTVGNUgxQnZsWG1xSW9tb3pVTFNTQSsxVm9NclExNVoy?= =?utf-8?B?ZXNqWE45c2JGZUZpSitiWVF1L0thczQyM3lyb3ZGQmtnUTlCbmIvc0NLRjdy?= =?utf-8?B?K3A5akRadEFMeEovWG5lWVdhRm05aEFPMVc3Qkh1QnZ1V0E2b2ppWG8yRG9H?= =?utf-8?B?RzVzb056eTR1ZkNrVDhodEQxVTJBdXZVTkxmb25qRnV1SmU0K0tINzF1VVZO?= =?utf-8?B?bmN3Um5DYndWcVJwaHVWdjU5dUx5OHBaMm5nSEsybDA5aU9Oa1YwWU10dHdw?= =?utf-8?B?WDhqUEZrbWR4dHFtZ2NvNWNUT1ZLelpUYld4TFlxbUlZTjdYNFUvTHVPbVJz?= =?utf-8?B?OU05NUNwQ1EvQ3pRL3ZDT0JBVjlObG1tbExZRXRsZlVkSzJUd2owRHlLdGhi?= =?utf-8?B?enY5ek14enNKOGtHRlg3SHo4NUtkajROOWViRHRQUFZoZ3JjVWorTmhSc3JH?= =?utf-8?B?TjRDYWdOVnI2WS8zeWFqeUhyeWFxalVadE1pd1FUcW94Zisvd0owYzhUbWRL?= =?utf-8?B?MnVQV0QwazIrWE9ZT01xMzV0RVZtM2ZQTngyTnZGYzJ2SFIzZytqWVFHSlBy?= =?utf-8?B?SDA3bmNjOWtOdWdpeXpiZ0xHL09sTTgwenBJdWE4dUhxYlVjK3I4RUp3S1Ew?= =?utf-8?B?ZGhwMVFIc1JoRFgyN2JYck9kckRBRkJWakFqK0Zob1dWYU5Jd2l6NEllVHdY?= =?utf-8?B?U0dGRHczTlhid3dwdU1ldFJzeUhlM1pmbWgzRlhPWDM0Q1ZZU3Evanpvc2dS?= =?utf-8?B?aHNDUlFuVElhWjVLY0pYQ3U4ZnJsYzZjcVB2b2daeGRHN01LSU9BNmxma3Mw?= =?utf-8?B?RjlZZzhnWGFmaDhxMUoycUN3L3JYUjFIVEFXQk9PdFFJOWJoKzdqRmtzWEYz?= =?utf-8?B?WXRjUWZheFhLcnA1N0hBNVhFS2ZpTG1XWjlxd1QzZk1MK1NmcklQbEVjejNq?= =?utf-8?B?MHBjRUl6QjJHeHZyMnJNd2lHR2hLQ1VvSHorWldSdEVKRmVGYVJOdkNIeU02?= =?utf-8?B?MVUzVE5YOWhpcTZVWTQveGkrYmxsaitWanFmZndjcmdKNUpzbDljb3dTTDZj?= =?utf-8?B?UTE3LzJ6WkphWW5IdE9PcEJycGQ5eENNa0VaRVg3YmZnVzg2YzBFanVxR0lo?= =?utf-8?B?THlNVnVPeWs1aWxPNThuUGxlL2dMY01lSG5ZZVFkK0pkRWk5RzVYNHpjYjZm?= =?utf-8?B?aWVuZlFxNzNtWkcrT1RoNWx0OVRrWXd6WVJWV1g0S1lJQ0xGZnJzeU02ZW9T?= =?utf-8?B?Ykx3OHV0dHFVeXhzOSs1Y0l0K2ZxKzQzSUNBRUFpa3NUUnpPOXpsZkpHMHdq?= =?utf-8?B?cVNqdFd0YzhKdWlGRmY5UFNUWDczUTVNR2xSTXBsa1lmTEhYTlhvRnYwZFBo?= =?utf-8?Q?uyUU=3D?= X-Microsoft-Exchange-Diagnostics: 1; DBXPR06MB416; 3:/QtEngxxGvvHobhiWq1Yxwb0RfKm/gqDfLAvyzm0iarPBtL3p6C0F0IOBU5p6FAfoVilGPx+wrbypxMNkJHqDJ4TnMtbe238zM3IPp4bIJjB83t04Fimw2hSjr9SivyOFBZ1Wc83bmnB+rxroPPkdg==; 10:MbTlkEmGCzPJlSltN66y8Ywbnny22j6oXr2UmLxYGxAAjelh+KU0lGe5GTfIQY7MIK4fstt1rnIyOxZE/6gREl2KdKKyn5P1NsuGk66g5oY=; 6:n3UJkf6s4Ihql9gKj53mIA/Ad3rvvXtswYxTxb2aQPHVgTjYU5j3TO+Nl23tv5/N X-OriginatorOrg: dotcom.ts.it X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2015 12:01:38.3384 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBXPR06MB416 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 12:01:48 -0000 Hello, I have a weird network problem which I believe may be caused by the FreeBSD igb driver or perhaps even the network adapter. Let me try to explain the scenario in brief: I have a FreeBSD 10.0-RELEASE-p10 server with a public IP address, in which N virtual machines are installed through JAIL; the machines hold private IP addresses on the loopback1 adapter. The VMs access the internet through NATting on the public IP via ipfw: nat 1 config ip X.Y.Z.W if igb0 unreg_only same_ports add 60000 nat 1 ip from 192.168.250.0/24 to any out xmit igb0 keep-state add 60001 nat 1 ip from any to X.Y.Z.W in recv igb0 In addition, port forwarding is configured on the real machine towards the VMs in order to support public services (Apache httpd, database, etc.) The network adapter is: igb0: flags=8843 metric 0 mtu 1500 options=403bb ether 00:45:80:dd:32:30 inet X.Y.Z.W netmask 0xffffff00 broadcast X.Y.Z.W inet6 XX::YY:ZZ:WWW:VVV%igb0 prefixlen 64 scopeid 0x1 inet6 XX:YY:ZZ:WWW::1 prefixlen 64 nd6 options=23 media: Ethernet autoselect (1000baseT ) status: active The loopback1 adapter, where the VMs' IPs are assigned, too, has MTU 1500. So far so good, in the sense that everything works as expected, almost. Occasionally there are requests originated by the VMs towards internet servers which end in timeout (http, sftp, etc.). The very same requests, if executed by the real machine, end correctly with a response. After countless experiments I have managed to reproduce the problem deterministically. Through a tcpdump executed on the request's recipient I have noticed that all TCP packets with a payload between 101 e 106 (inclusive) bytes in size arrive with a wrong TCP checksum and as such are rejected. Subsequent retransmissions of the same packet continue to bear a wrong checksum and this continues until the connection timeout is reached. The IP checksum, instead, is always correct. Packets smaller than 101 bytes are transmitted and received with the correct checksum, as the same happens to packets with a payload in excess of 116 bytes in size. If TXCSUM is disabled, the problem disappears. The same problem I have on second server with same configuration and hardware bat with FreeBSD 10.0-RELEASE-p1 . I believe the above behavior is something error with the driver, as on a third machine, with identical configuration with jail machines NATting but with an em driver, the checksum problem didn't appear. -- Cordiali saluti Sossi Andrej ------------------------- DOTCOM Information technology Via Machiavelli, 28 34132 - Trieste (TS) Italy tel: +39 040 9828090 fax: +39 040 0641954 E-mail: asossi@dotcom.ts.it ---------------------------- Ai sensi del D.lgs n. 196 del 30.06.03 (Codice Privacy) si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo del destinatario. Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a terzi, dandocene gentilmente comunicazione. Grazie This message, for the D.lgs n. 196 / 30.06.03 (Privacy Code), may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.