From owner-freebsd-net@freebsd.org Thu Apr 9 17:29:14 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 70C97278270; Thu, 9 Apr 2020 17:29:14 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-vs1-f65.google.com (mail-vs1-f65.google.com [209.85.217.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48yp6K3wqCz3QcV; Thu, 9 Apr 2020 17:29:13 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-vs1-f65.google.com with SMTP id u9so7432535vsp.6; Thu, 09 Apr 2020 10:29:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+FiWlM0JvZZqa9MBjftqXtQ47GZYrye0Dadae5IeD9A=; b=VrcrF67ujkrKA4xJYBYPOUWkdLPdq+Q0vics3EjvNZ48t0Mzom9vMgUT6GF+n54Xsu 8W2bvYqEOZDqzaFvCg+sadFmhVCmHM7zN0jb78N5DpqgimQdnUxx9aC0NbzVEQP83ID7 l8wZFDlybBCCMUZjRwsjToIE6Ah0UHrLwYGtY297v389EJQIH8qdq93qOBfwznviJLEd EbFvd/vZDOClxzg72MVMI1olM+W03qUhfvEPpbgcjbtCKSyLY9Bp2bqfC5M06retFwXt 2oxM4eH6lUc/FKt32b2R0yFKmF8ipYhgaPp37FERUaL7my7lk3DOb8RZ+hWR7FF/mc4F 49KQ== X-Gm-Message-State: AGi0PuaTCJoRZ9Nxif4W1djYb8wA/QrNIC/lvWUtqKjeRy9puEwLSrod SfHBCUKAJZHAY2aRjfxclTivE3kfZScaIFk722A= X-Google-Smtp-Source: APiQypJJcaJARD6KtFR7oFpCExpZLtjvDgbCL89zQbl2u8LzumyvXWN5YuFLqePfsF+J9TZzqzZH0KExcZsP9qfSyjM= X-Received: by 2002:a67:e308:: with SMTP id j8mr922317vsf.74.1586453352440; Thu, 09 Apr 2020 10:29:12 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Luigi Rizzo Date: Thu, 9 Apr 2020 10:28:56 -0700 Message-ID: Subject: Re: dummynet performance To: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Cc: "freebsd-net@freebsd.org" , freebsd-ipfw , "Andrey V. Elsukov" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48yp6K3wqCz3QcV X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rizzounipi@gmail.com designates 209.85.217.65 as permitted sender) smtp.mailfrom=rizzounipi@gmail.com X-Spamd-Result: default: False [-0.49 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; FORGED_SENDER(0.30)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[rizzo@iet.unipi.it,rizzounipi@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.839,0]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-0.99)[-0.986,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[unipi.it]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[65.217.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.16)[ip: (0.05), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; RWL_MAILSPIKE_POSSIBLE(0.00)[65.217.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2020 17:29:14 -0000 On Thu, Apr 9, 2020 at 2:18 AM =C3=96zkan KIRIK wro= te: > > Hello, > > I wonder if it is possible to update dummynet code multi-threading capabl= e > practically? > My idea is below: > - A new sysctl tunable will be defined as > "net.inet.ip.dummynet.thread_count" (default 1) > - To distribute tasks along threads, each sched instance can be assigned = to > different thread. > - By default, all schedulers assigned to thread 0 if no thread_id is set. > With a new option named "thr_id" affinity can be set. as below: > ipfw sched 1 config type QFQ (by defaut thr_id =3D 0) > ipfw sched 2 config type QFQ thr_id 1 > ipfw sched 3 config type QFQ thr_id 2 > ipfw sched 4 config type FIFO thr_id 3 > ipfw sched 5 config type FIFO thr_id 3 ( sched 4 & 5 assigned to same > thread 3 ) > > I think this approach is more easy then completely rewrite or other > solutions. > This change may not be completely trivial given that dummynet was designed = as single threaded and protected by a single lock. If you want to proceed, you should do as follows: - some members of struct dn_parms need to be duplicated for each thread, including prev_t, evheap, curr_time, the locks, and perhaps others. Probably may be useful to create a new struct for them, with a back pointer to the parent dn_parms, and include a refcount. - add another rwlock to protect access to dn_cfg - dummynet_io() does the initial demultiplexing calling dn_ht_find(), currently protected DN_BH_WLOCK(). This could be changed to an RWLOCK and I believe this initial step could be protected by a read lock (maybe there are corner cases where you need to create a new entry so you need to drop the lock, acquire in write mode and retry...) - once the demux has returned a dn_fsk, maybe you can use a refcount to avoid the object being destroyed, release the read lock on dn_cfg.fshash, acquire an exclusive lock on the thread handling the dn_fsk, and proceed from there. - the kernel thread that process the heap should now run each on one instance of the evheap cheers luigi