From owner-freebsd-net@FreeBSD.ORG Thu Nov 10 23:40:37 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF1AA106566C for ; Thu, 10 Nov 2011 23:40:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 65A898FC17 for ; Thu, 10 Nov 2011 23:40:37 +0000 (UTC) Received: by vws11 with SMTP id 11so4291141vws.13 for ; Thu, 10 Nov 2011 15:40:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=AwKkgCem7GMrqrTGiGH0+zdjgL50s7AxRw9+m2ikAGQ=; b=XGqg2UJDtsqF+iMtFSujUctxCK49e5Xe+mle/FqJ+gfJeuly1SUCnQmcWRu/TS2c/k d7shJpzm2NtF/Z5RK/iPtMvCM7v92SrU6i/+rQTCDG9awu/hzdqDsJlUmpqu3GCwAA4Q kHVfHOt3VQ4vaRa2CS5C4//nbgGywu33OoDuQ= MIME-Version: 1.0 Received: by 10.52.72.104 with SMTP id c8mr16384928vdv.105.1320968436498; Thu, 10 Nov 2011 15:40:36 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.29.198 with HTTP; Thu, 10 Nov 2011 15:40:36 -0800 (PST) In-Reply-To: References: <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> <4EA959EE.2070806@hotplug.ru> <4EAD116A.8090006@gmail.com> <4EAE58A2.9040803@gmail.com> <4EB96511.50701@gmail.com> <4EBA3F22.2060204@gmail.com> <4EBB9CDF.9090300@gmail.com> Date: Thu, 10 Nov 2011 15:40:36 -0800 X-Google-Sender-Auth: HR0rhuVeHXHTk9N-f3eZAAdhegg Message-ID: From: Adrian Chadd To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, Jason Wolfe , Emil Muratov , Hooman Fazaeli Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2011 23:40:37 -0000 In the case of multi-threaded OACTIVE, why not just do this: upon queue completion: * lock; * do queue completion; * if any frames were completed, clear queue busy; * if (queue busy == clear && OACTIVE) clear OACTIVE; * run queue completion; * unlock. upon queue transmit: * lock; * queue frame; * if (fail) - mark queue as busy; * if (all busy) - set OACTIVE. * unlock The stack will stop sending you frames whilst OACTIVE Is set, but will continue sending them when you clear it. If you just clear OACTIVE when _a_ queue is clear then you'll get frames queued via start/transmit when queues are busy, but what you won't do is starve them all until all queues have free slots. Ie, you don't mind clearing it, then getting a frame and immediately finding you're busy. Once you've done that, any further issues are likely either concurrency with accessing the tx/rx descriptor rings (per-queue), or the interrupt handling races. Ie, you disable interrupts, handle the tx queue completion, -then- re-enable interrupts and clear masks. Maybe that logic in igb_enable_intr() is to blame? Ie, que_mask / link_mask, is that before or after the queue has been handled? If it's called after the queue is handled, and que_mask / link_mask is 0, but the queue is full, you'll never pick up the next interrupt as there won't be one? 2c, Adrian