From owner-freebsd-current@FreeBSD.ORG  Wed Jun  4 13:45:03 2014
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 137B4ADD;
 Wed,  4 Jun 2014 13:45:03 +0000 (UTC)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
 by mx1.freebsd.org (Postfix) with ESMTP id CAEBB2467;
 Wed,  4 Jun 2014 13:45:02 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
 id ED2607300A; Wed,  4 Jun 2014 15:49:45 +0200 (CEST)
Date: Wed, 4 Jun 2014 15:49:45 +0200
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: current@freebsd.org, bryanv@freebsd.org, jfv@freebsd.org
Subject: BUG: some drivers return ENOBUFS when the mbuf is actually queued
Message-ID: <20140604134945.GA64688@onelab2.iet.unipi.it>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.20 (2009-06-14)
Cc: stefanogarzarella@gmail.com
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Jun 2014 13:45:03 -0000

Hi,
if I read correctly the code, there are a few network device drivers
(igb, ixgbe, i40e, vtnet, vmxnet) where ifp->if_transmit(ifp, m)
can return ENOBUFS even when 'm' has _not_ been dropped:

    e1000/if_igb.c :: igb_mq_start()
            can return ENOBUFS from igb_xmit()

    ixgbe/ixgbe_main.c :: ixgbe_mq_start_locked()
            can return ENOBUFS from ixgbe_xmit()
       (similar for i40)

    virtio/network/if_vtnet.c :: vtnet_txq_mq_start
            can return ENOBUFS if virtqueue_full()

In all these cases, the error comes from a later attempt to transfer
mbufs from the buf_ring to the NIC ring.

All drivers using if_transmit() seem correct, as well as a bunch
of others (cxgbe, sfxge, mxge ...) that reassign if_transmit and I
checked for correctness.

I think that when the current buffer has been queued, returning
ENOBUFS is extremely confusing and should not be done.

I would also argue that the return from ifp->if_transmit(ifp, m)
should only tell what happened to 'm', not other things
such as the status of the queue.

Any objections if i fix the above drivers ?

	cheers
	luigi

(For those curious: i found this issue when using emulated
netmap mode on top of a standard driver. The netmap emulation
code assumes that ENOBUFS indicates that the driver has
m_free()'d the mbuf, same as it happens on linux, and the
bug was causing panics in my system).