From owner-freebsd-bugs@freebsd.org Tue Jan 8 16:04:47 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5FBA1482112 for ; Tue, 8 Jan 2019 16:04:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4FE986AC46 for ; Tue, 8 Jan 2019 16:04:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 12ECC1482108; Tue, 8 Jan 2019 16:04:47 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E19311482107 for ; Tue, 8 Jan 2019 16:04:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B2AD6AC43 for ; Tue, 8 Jan 2019 16:04:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id AF66B4CFC for ; Tue, 8 Jan 2019 16:04:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x08G4jl8025637 for ; Tue, 8 Jan 2019 16:04:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x08G4j3c025636 for bugs@FreeBSD.org; Tue, 8 Jan 2019 16:04:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234754] FreeBSD 12.0 ena Network Driver on AWS EC2 Packet Loss Date: Tue, 08 Jan 2019 16:04:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mwalker@carbonhouse.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2019 16:04:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234754 Bug ID: 234754 Summary: FreeBSD 12.0 ena Network Driver on AWS EC2 Packet Loss Product: Base System Version: 12.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: mwalker@carbonhouse.com Overview:=20 Given a single connection transferring anywhere a 100MB file, using scp or = just plain netcat, there will eventually be sufficient packet loss on the server side such that the connection will stall and eventually reset. Steps to Reproduce: 1. Boot the standard FreeBSD 12.0 AMI ami-04331586c79df8e01 in us-west-2 2. Attempt to use scp to copy a 100MB file onto the server Additional Information: I've been able to replicate this against a bone-stock boot-up of the FreeBSD 12.0 AMI on the following instance+region combinations (every one I've tried I've been able to replicate it on): * eu-west-1 / t3.medium * us-west-2 / t3.small * us-west-2 / t3.medium * us-west-2 / t3.large * us-west-2 / m5.large * us-west-2 / m5a.large * us-west-2 / m5a.xlarge I was able to replicate this using a residental (~50Mbps) connection as wel= l as another hosted machine at another provider with a 1Gbps connection, though I cannot replicate this using a machine in the same VPC as the server as the client. This really seems like the network driver is just black-holeing packets. I = see no error messages in dmesg, /var/log/, the console, or anything untoward in= the dev.ena.* sysctl values :( Here's `sysctl dev.ena` from a t3.small instance after a number of tcp connections experiencing packet loss: dev.ena.0.admin_stats.no_completion: 0 dev.ena.0.admin_stats.out_of_space: 0 dev.ena.0.admin_stats.completed_cmd: 37 dev.ena.0.admin_stats.sumbitted_cmd: 37 dev.ena.0.admin_stats.aborted_cmd: 0 dev.ena.0.hw_stats.rx_drops: 0 dev.ena.0.hw_stats.tx_bytes: 5681243 dev.ena.0.hw_stats.rx_bytes: 47398849 dev.ena.0.hw_stats.tx_packets: 27863 dev.ena.0.hw_stats.rx_packets: 42686 dev.ena.0.queue1.rx_ring.empty_rx_ring: 0 dev.ena.0.queue1.rx_ring.bad_req_id: 0 dev.ena.0.queue1.rx_ring.bad_desc_num: 0 dev.ena.0.queue1.rx_ring.dma_mapping_err: 0 dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0 dev.ena.0.queue1.rx_ring.mbuf_alloc_fail: 0 dev.ena.0.queue1.rx_ring.bad_csum: 0 dev.ena.0.queue1.rx_ring.refil_partial: 0 dev.ena.0.queue1.rx_ring.bytes: 33640679 dev.ena.0.queue1.rx_ring.count: 28882 dev.ena.0.queue1.tx_ring.mbuf_collapse_err: 0 dev.ena.0.queue1.tx_ring.mbuf_collapses: 0 dev.ena.0.queue1.tx_ring.bad_req_id: 0 dev.ena.0.queue1.tx_ring.missing_tx_comp: 0 dev.ena.0.queue1.tx_ring.doorbells: 18803 dev.ena.0.queue1.tx_ring.dma_mapping_err: 0 dev.ena.0.queue1.tx_ring.prepare_ctx_err: 0 dev.ena.0.queue1.tx_ring.bytes: 4843829 dev.ena.0.queue1.tx_ring.count: 18812 dev.ena.0.queue0.rx_ring.empty_rx_ring: 0 dev.ena.0.queue0.rx_ring.bad_req_id: 0 dev.ena.0.queue0.rx_ring.bad_desc_num: 0 dev.ena.0.queue0.rx_ring.dma_mapping_err: 0 dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0 dev.ena.0.queue0.rx_ring.mbuf_alloc_fail: 0 dev.ena.0.queue0.rx_ring.bad_csum: 0 dev.ena.0.queue0.rx_ring.refil_partial: 0 dev.ena.0.queue0.rx_ring.bytes: 13758170 dev.ena.0.queue0.rx_ring.count: 13804 dev.ena.0.queue0.tx_ring.mbuf_collapse_err: 0 dev.ena.0.queue0.tx_ring.mbuf_collapses: 0 dev.ena.0.queue0.tx_ring.bad_req_id: 0 dev.ena.0.queue0.tx_ring.missing_tx_comp: 0 dev.ena.0.queue0.tx_ring.doorbells: 9057 dev.ena.0.queue0.tx_ring.dma_mapping_err: 0 dev.ena.0.queue0.tx_ring.prepare_ctx_err: 0 dev.ena.0.queue0.tx_ring.bytes: 838748 dev.ena.0.queue0.tx_ring.count: 9057 dev.ena.0.admin_q_pause: 0 dev.ena.0.interface_down: 1 dev.ena.0.interface_up: 2 dev.ena.0.wd_expired: 0 dev.ena.0.missing_tx_threshold: 128 dev.ena.0.missing_tx_max_queues: 4 dev.ena.0.missing_tx_timeout: 21474836480 dev.ena.0.keep_alive_timeout: 25769803776 dev.ena.0.wd_active: 1 dev.ena.0.%parent: pci0 dev.ena.0.%pnpinfo: vendor=3D0x1d0f device=3D0xec20 subvendor=3D0x0000 subdevice=3D0x0000 class=3D0x020000 dev.ena.0.%location: slot=3D5 function=3D0 dbsf=3Dpci0:0:5:0 handle=3D\_SB_= .PCI0.S05_ dev.ena.0.%driver: ena dev.ena.0.%desc: ENA adapter dev.ena.%parent: --=20 You are receiving this mail because: You are the assignee for the bug.=