From owner-freebsd-net@FreeBSD.ORG Wed Jul 16 18:00:50 2014 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90AA3D03 for ; Wed, 16 Jul 2014 18:00:50 +0000 (UTC) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 526DF21E7 for ; Wed, 16 Jul 2014 18:00:50 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id w7so1092124qcr.40 for ; Wed, 16 Jul 2014 11:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=xZCG6GRKAfStMP/0X6FOunvhiQYfMrrKyOwvYA8Ad38=; b=zCDEVosU0Affo/+pzuJP8elRND6uyznsNxBEjwiZ0WTfMnvoVxLPb3RTNj6MaNiTFO po5Zo61GFA8cbmMcQ63HITOE+UD45rtw20s8iV0ugeDD8tO/PxRf9jwT1Vi10rxu8fcd x3E6lyB1qLhR2GL0GgC5M8/VnG9D0VniGQHP7gaGASOIoenY6fPHIImpVF08BRBZrTCh BG8YTa3PoLLGbDeASGPD04yOJcGwclSCu5tX77IFP56zzL8+n3nlzOvMrvrDkaj04OTy 2jpvR+eZB98VKb4DEUWWleH7abDtL59pP2Kw7zureFAUq3v25+tQ+wWMGEwgk65kpe5K Z52w== MIME-Version: 1.0 X-Received: by 10.224.171.197 with SMTP id i5mr48707672qaz.55.1405533648737; Wed, 16 Jul 2014 11:00:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.202.193 with HTTP; Wed, 16 Jul 2014 11:00:48 -0700 (PDT) In-Reply-To: References: Date: Wed, 16 Jul 2014 11:00:48 -0700 X-Google-Sender-Auth: 3MSHRCDzJ0oArz-IxtNMsPsMGpo Message-ID: Subject: Re: UDP sendto() returning ENOBUFS - "No buffer space available" From: Adrian Chadd To: hiren panchasara Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jul 2014 18:00:50 -0000 Hi! So the UDP transmit path is udp_usrreqs->pru_send() == udp_send() -> udp_output() -> ip_output() udp_output() does do a M_PREPEND() which can return ENOBUFS. ip_output can also return ENOBUFS. it doesn't look like the socket code (eg sosend_dgram()) is doing any buffering - it's just copying the frame and stuffing it up to the driver. No queuing involved before the NIC. So a _well behaved_ driver will return ENOBUFS _and_ not queue the frame. However, it's entirely plausible that the driver isn't well behaved - the intel drivers screwed up here and there with transmit queue and failure to queue vs failure to transmit. So yeah, try tweaking the tx ring descriptor for the driver your'e using and see how big a bufring it's allocating. -a On 16 July 2014 01:58, hiren panchasara wrote: > Return values in sendto() manpage says: > > [ENOBUFS] The system was unable to allocate an internal buffer. > The operation may succeed when buffers become avail- > able. > > [ENOBUFS] The output queue for a network interface was full. > This generally indicates that the interface has > stopped sending, but may be caused by transient con- > gestion. > > If I hit the first condition, it should reflect as failures in > "netstat -m". Is that a correct assumption? > > I want to understand what happens when/if we hit the second condition. > And how to prevent that from happening. > Is it just application's job to rate-limit data it sends to the n/w > interface card so that it doesn't saturate? > Does kernel do any sort of queuing in the case of ENOBUFS? OR does the > message just gets dropped? > > For an application sending a lot of UDP data and returning ENOBUFS, > what all udp and other tunables I should tweak? I can only think of: > - number of tx ring descriptors - increasing this will get us more txds. > - kern.ipc.maxsockbuf: Increasing this will increase buffer size > allocated for sockets. > > what else? > > Any comments/suggestions/corrections? > > cheers, > Hiren > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"