Date: Wed, 16 Sep 1998 09:59:09 +0300 From: Ruslan Ermilov <ru@ucb.crimea.ua> To: "Jan B. Koum " <jkb@best.com>, Bengt Gorden <bengan@sunet.se> Cc: beatteam@austasia.net, FreeBSD -Questions <freebsd-questions@FreeBSD.ORG> Subject: Re: MTU Size & Connection reset by.. & Lost Input channel Message-ID: <19980916095909.B22964@relay.ucb.crimea.ua> In-Reply-To: <Pine.BSF.4.02A.9809151213500.17708-100000@shell6.ba.best.com>; from Jan B. Koum on Tue, Sep 15, 1998 at 12:15:52PM -0700 References: <19980915203908.A19511@sunet.se> <Pine.BSF.4.02A.9809151213500.17708-100000@shell6.ba.best.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--K8nIJk4ghYZn606h Content-Type: text/plain; charset=us-ascii On Tue, Sep 15, 1998 at 12:15:52PM -0700, Jan B. Koum wrote: > > And to beat around dead horse: > > % ifconfig -a > xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 > ^^^^^^^^ > inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 > ether 00:60:08:15:bc:65 > media: autoselect (10baseT/UTP <half-duplex>) > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 > ^^^^^^^^^ > ifconfig command will let you know MTU for your interfaces. > Sorry, I didn't catch all this thread, but "Connection resets" usually happen because of broken PMTU discovery. PMTU uses ICMP to adjust a proper MTU size along a specific route. Some sites protect themselves agains DoS ICMP attacks and disable ICMP on their routers, and this breaks PMTU discovery from working. Attached is my recent post to -hackers regarding this issue. Best regards, -- Ruslan Ermilov Sysadmin and DBA of the ru@ucb.crimea.ua United Commercial Bank +380.652.247.647 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --K8nIJk4ghYZn606h Content-Type: message/rfc822 Received: from spider.cris.net (root@spider.cris.net [194.93.176.65]) by relay.ucb.crimea.ua (8.8.8/8.8.8) with ESMTP id QAA21113 for <ru@ucb.crimea.ua>; Mon, 31 Aug 1998 16:45:26 +0300 (EEST) (envelope-from owner-freebsd-hackers@FreeBSD.ORG) Received: from hub.freebsd.org (hub.FreeBSD.ORG [204.216.27.18]) by spider.cris.net (8.8.8/8.8.8) with ESMTP id QAA24934 for <ru@ucb.crimea.ua>; Mon, 31 Aug 1998 16:44:30 +0300 (EET DST) Received: from localhost (daemon@localhost) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id GAA18776; Mon, 31 Aug 1998 06:25:14 -0700 (PDT) (envelope-from owner-freebsd-hackers) Received: by hub.freebsd.org (bulk_mailer v1.6); Mon, 31 Aug 1998 06:25:09 -0700 Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id GAA18758 for freebsd-hackers-outgoing; Mon, 31 Aug 1998 06:25:08 -0700 (PDT) (envelope-from owner-freebsd-hackers@FreeBSD.ORG) Received: from relay.ucb.crimea.ua (relay.ucb.crimea.ua [194.93.177.113]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id GAA18552 for <hackers@FreeBSD.org>; Mon, 31 Aug 1998 06:23:42 -0700 (PDT) (envelope-from ru@ucb.crimea.ua) Received: (from ru@localhost) by relay.ucb.crimea.ua (8.8.8/8.8.8) id QAA20826 for hackers@FreeBSD.org; Mon, 31 Aug 1998 16:22:29 +0300 (EEST) (envelope-from ru) Message-ID: <19980831162228.A20318@ucb.crimea.ua> Date: Mon, 31 Aug 1998 16:22:28 +0300 From: Ruslan Ermilov <ru@ucb.crimea.ua> To: hackers@FreeBSD.ORG Subject: PMTU discovery, Firewalls and Sendmail Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.2i X-Operating-System: FreeBSD 2.2.7-STABLE i386 Sender: owner-freebsd-hackers@FreeBSD.ORG X-Loop: FreeBSD.ORG Hi! Recently I had problems receiving mail from hub.FreeBSD.org. The logs were full of sendmail's timeout errors. Now I finally discovered the problem and want to share my experience with you ;-) As you know (if you don't, please refer to RFC1191) today almost every host uses ``Path MTU Discovery''. The ICMP participates in it too. What happens if some router along the path is blocking ICMP? Let's start from an example. As you know, hub.FreeBSD.org is a mailhub that serves all mailing lists dedicated to FreeBSD. I'll call it just HUB, OK? Somewhere on CRL.NET (the Walnut Creek CDROM's ISP) along the path from my mail machine to the HUB the ICMP blocked. This was made to protect HUB from ICMP DoS attacks. Well, HUB is trying to send mail to me <RU@UCB.CRIMEA.UA>. The host route to RELAY.UCB.CRIMEA.UA is cloned, and its MTU (by default) is equal to 1500. More exactly, it will be equal to MTU of its first hop. In case of Ethernet it is equal to 1500. All following SMTP packets have the DF flag set. While the packet size is small (at EHLO, MAIL and RCPT stages) all goes smooth. After that, the message itself is transferred at DATA stage. The size of TCP packets increases. Somewhere (I don't know exactly where) along the path from HUB.FREEBSD.ORG to RELAY.UCB.CRIMEA.UA a router can't forward the packet due to its lower next-hop MTU (at least, my site is connected to the Internet via SLIP with MTU equal to 552). This router, following RFC1191 rules, constructs and sends ``Datagram Too Big'' ICMP message to the HUB. Router at CRL.NET blocks ICMP and doesn't let it to come through and reach the HUB. The connection hangs at DATA stage. Then, after a certain period of time, the standard timeout-driven retransmit procedure takes place. HUB then re-transmits the same packet (of the same large size), router that can't forward it sends back ICMP ``too big'', and CRL.NET blocks it. This scenario will take place until HUB's sendmail drops the connection by timeout. So, HUB will never adjust the MTU. My conclusion is: ~~~~~~~~~~~~~~~~ Don't block ICMP, because blocking breaks PMTU discovery. At least, don't block ICMP type 3 code 4 messages (fragmentation needed and DF set). P.S. Now, when ICMP to HUB is not anymore blocked, everything is OK. With best Regards, -- Ruslan Ermilov Sysadmin and DBA of the ru@ucb.crimea.ua United Commercial Bank +380.652.247.647 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message --K8nIJk4ghYZn606h-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19980916095909.B22964>