From owner-freebsd-smp@FreeBSD.ORG Tue Nov 11 07:02:20 2008 Return-Path: Delivered-To: smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56A9B1065676 for ; Tue, 11 Nov 2008 07:02:20 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.178]) by mx1.freebsd.org (Postfix) with ESMTP id 1EF0F8FC18 for ; Tue, 11 Nov 2008 07:02:19 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: by wa-out-1112.google.com with SMTP id m34so1484692wag.27 for ; Mon, 10 Nov 2008 23:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=07Kg6z8piMlG3OjzHDr5XMr+H0liA0p7hgcN9JCzcbo=; b=XyV5Ul5xeFVNuI8+O8OLTQRbfU5g6Lx35iNzmctocYv1YOepbF5MbWCSK/BI0qumYz a4m8vs6KcSdqrEymm3fmTK9MBqaRZ1w96Cti8LxNan0hBi5VrqFyoAefSLp6feLlWBXc +9k7M38JkI721LVwKPlQpLQfbHBFuPXXRwRFg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=GUwZeAGZVWIRx2yLMJnNE6ecR9FZ4Co49xa+1HLxAa/cvJq1FJwkuJpGT/0IHyczWi kfxUy8L/3wNqNCNogTCtHrH71CaWcwJBrd9K0d6ByEKf7JdjyeReExLPNp4BOjd34C8P FeSWYBMW27NXZha+HWLxn3ApRmAHVWsGft33U= Received: by 10.115.110.6 with SMTP id n6mr4994440wam.72.1226386939656; Mon, 10 Nov 2008 23:02:19 -0800 (PST) Received: by 10.115.76.12 with HTTP; Mon, 10 Nov 2008 23:02:19 -0800 (PST) Message-ID: <42e3d810811102302h3a0e38bcuf1195cf0a89c29a7@mail.gmail.com> Date: Tue, 11 Nov 2008 15:02:19 +0800 From: "Archimedes Gaviola" To: ivoras@freebsd.org In-Reply-To: <42e3d810811102032w7850a1c0t386d80ce747f37d3@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <42e3d810811100033w172e90dbl209ecbab640cc24f@mail.gmail.com> <200811101733.04547.jhb@freebsd.org> <42e3d810811102032w7850a1c0t386d80ce747f37d3@mail.gmail.com> Cc: smp@freebsd.org, freebsd-smp@freebsd.org Subject: Re: CPU affinity with ULE scheduler X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Nov 2008 07:02:20 -0000 On Tue, Nov 11, 2008 at 12:32 PM, Archimedes Gaviola wrote: > On Tue, Nov 11, 2008 at 6:33 AM, John Baldwin wrote: >> On Monday 10 November 2008 03:33:23 am Archimedes Gaviola wrote: >>> To Whom It May Concerned: >>> >>> Can someone explain or share about ULE scheduler (latest version 2 if >>> I'm not mistaken) dealing with CPU affinity? Is there any existing >>> benchmarks on this with FreeBSD? Because I am currently using 4BSD >>> scheduler and as what I have observed especially on processing high >>> network load traffic on multiple CPU cores, only one CPU were being >>> stressed with network interrupt while the rests are mostly in idle >>> state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom >>> network interface cards (bce0 and bce1). Below is the snapshot of the >>> case. >> >> Interrupts are routed to a single CPU. Since bce0 and bce1 are both on the >> same interrupt (irq 23), the CPU that interrupt is routed to is going to end >> up handling all the interrupts for bce0 and bce1. This not something ULE or >> 4BSD have any control over. >> >> -- >> John Baldwin >> > > Hi John, > > I'm sorry for the wrong snapshot. Here's the right one with my concern. > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 17 root 1 171 52 0K 16K CPU0 0 54:28 95.17% idle: cpu0 > 15 root 1 171 52 0K 16K CPU2 2 55:55 93.65% idle: cpu2 > 14 root 1 171 52 0K 16K CPU3 3 58:53 93.55% idle: cpu3 > 13 root 1 171 52 0K 16K RUN 4 59:14 82.47% idle: cpu4 > 12 root 1 171 52 0K 16K RUN 5 55:42 82.23% idle: cpu5 > 16 root 1 171 52 0K 16K CPU1 1 58:13 77.78% idle: cpu1 > 11 root 1 171 52 0K 16K CPU6 6 54:08 76.17% idle: cpu6 > 36 root 1 -68 -187 0K 16K WAIT 7 8:50 65.53% > irq23: bce0 bce1 > 10 root 1 171 52 0K 16K CPU7 7 48:19 29.79% idle: cpu7 > 43 root 1 171 52 0K 16K pgzero 2 0:35 1.51% pagezero > 1372 root 10 20 0 16716K 5764K kserel 6 58:42 0.00% kmd > 4488 root 1 96 0 30676K 4236K select 2 1:51 0.00% sshd > 18 root 1 -32 -151 0K 16K WAIT 0 1:14 0.00% swi4: clock s > 20 root 1 -44 -163 0K 16K WAIT 0 0:30 0.00% swi1: net > 218 root 1 96 0 3852K 1376K select 0 0:23 0.00% syslogd > 2171 root 1 96 0 30676K 4224K select 6 0:19 0.00% sshd > > Actually I was doing a network performance testing on this system with > FreeBSD-6.2 RELEASE using its default scheduler 4BSD and then I used a > tool to generate big amount of traffic around 600Mbps-700Mbps > traversing the FreeBSD system in bi-direction, meaning both network > interfaces are receiving traffic. What happened was, the CPU (cpu7) > that handles the (irq 23) on both interfaces consumed big amount of > CPU utilization around 65.53% in which it affects other running > applications and services like sshd and httpd. It's no longer > accessible when traffic is bombarded. With the current situation of my > FreeBSD system with only one CPU being stressed, I was thinking of > moving to FreeBSD-7.0 RELEASE with the ULE scheduler because I thought > my concern has something to do with the distributions of load on > multiple CPU cores handled by the scheduler especially at the network > level, processing network load. So, if it is more of interrupt > handling and not on the scheduler, is there a way we can optimize it? > Because if it still routed only to one CPU then for me it's still > inefficient. Who handles interrupt scheduling for bounding CPU in > order to prevent shared IRQ? Is there any improvements with > FreeBSD-7.0 with regards to interrupt handling? > > Thanks, > Archimedes > Hi Ivan, Archimedes Gaviola wrote: > To Whom It May Concerned: >=20 > Can someone explain or share about ULE scheduler (latest version 2 if > I'm not mistaken) dealing with CPU affinity? Is there any existing > benchmarks on this with FreeBSD? Because I am currently using 4BSD Yes but not for network loads. See for example benchmarks in http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf [Archimedes] Ah okay, so based on my understanding with ULE scheduler in FreeBSD-7.0, it only scale well with userland applications scheduling such as database and DNS? > scheduler and as what I have observed especially on processing high > network load traffic on multiple CPU cores, only one CPU were being > stressed with network interrupt while the rests are mostly in idle > state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom > network interface cards (bce0 and bce1). Below is the snapshot of the > case. This is unfortunately so and cannot be changed for now - you are not the first with this particular performance problem. [Archimedes] Meaning, you still have to improve the ULE scheduler processing network load? I have read some papers and articles that FreeBSD is implementing parallelized network stack, what is the status of this development? Is processing high network load can address this? BUT, looking at the data in the snapshot you gave, it's not clear that there is a performance problem in your case - bce is not nearly taking as much CPU time to be bottlenecking. What exactly do you think is wrong in your case? [Archimedes] Oh I'm sorry this is not the right one. Here below, PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 17 root 1 171 52 0K 16K CPU0 0 54:28 95.17% idle: cpu0 15 root 1 171 52 0K 16K CPU2 2 55:55 93.65% idle: cpu2 14 root 1 171 52 0K 16K CPU3 3 58:53 93.55% idle: cpu3 13 root 1 171 52 0K 16K RUN 4 59:14 82.47% idle: cpu4 12 root 1 171 52 0K 16K RUN 5 55:42 82.23% idle: cpu5 16 root 1 171 52 0K 16K CPU1 1 58:13 77.78% idle: cpu1 11 root 1 171 52 0K 16K CPU6 6 54:08 76.17% idle: cpu6 36 root 1 -68 -187 0K 16K WAIT 7 8:50 65.53% irq23: bce0 bce1 10 root 1 171 52 0K 16K CPU7 7 48:19 29.79% idle: cpu7 43 root 1 171 52 0K 16K pgzero 2 0:35 1.51% pagezero 1372 root 10 20 0 16716K 5764K kserel 6 58:42 0.00% kmd 4488 root 1 96 0 30676K 4236K select 2 1:51 0.00% sshd 18 root 1 -32 -151 0K 16K WAIT 0 1:14 0.00% swi4: clock s 20 root 1 -44 -163 0K 16K WAIT 0 0:30 0.00% swi1: net 218 root 1 96 0 3852K 1376K select 0 0:23 0.00% syslogd 2171 root 1 96 0 30676K 4224K select 6 0:19 0.00% sshd I was doing network performance testing with a traffic generator tool bombarding 600Mbps-700Mbps traversing my FreeBSD system in both directions. As you can see cpu7 is bounded to irq23 shared on both network interfaces bce0 and bce1. cpu7 takes up 65.53% CPU utilization which affects some of the applications running on the system like sshd and httpd. These services are no longer accessible when bombarding that amount of traffic. Since there are still more idled CPUs, I'm concern about CPU load distribution so that not only one CPU will be stressed. Thanks, Archimedes