From owner-freebsd-bugs@FreeBSD.ORG Tue Jul 29 19:35:46 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48AB3C52 for ; Tue, 29 Jul 2014 19:35:46 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AFA429E5 for ; Tue, 29 Jul 2014 19:35:46 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6TJZkup068979 for ; Tue, 29 Jul 2014 19:35:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 192249] New: [PATCH] Load reloc modules (amd64, mips) faster by caching global syms Date: Tue, 29 Jul 2014 19:35:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: conrad.meyer@isilon.com X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 19:35:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192249 Bug ID: 192249 Summary: [PATCH] Load reloc modules (amd64, mips) faster by caching global syms Product: Base System Version: 11.0-CURRENT Hardware: Any OS: Any Status: Needs Triage Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: conrad.meyer@isilon.com Created attachment 145126 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=145126&action=edit (cd src/ && patch -p1 ; diff is against recent-ish CURRENT) The problem: At Isilon, efs.ko is a very large kernel module; we were seeing kldload take 12-13 seconds. Most of that time was spent repeatedly resolving the same global symbols for relocations. link_elf_obj would lookup the same globals many times; each time, it would lookup in the local module first, before missing and falling back to deps, where it would find 'kernel' and get a fast hit (hash table hit). Since ld(1) doesn't generate lookup hashes (SysV or GNU) for relocatable objects (kmods on AMD64), lookups in the local module involved walking the entire symtab and strcmp()ing. efs.ko has 32k symbols. This was very slow. Solution: Add a special ELF SHN_ value (in the SHN_LOOS to SHN_HIOS range) to represent cached SHN_UNDEF globals. Cache the results of global lookup during relocatable object load. Modify Dtrace to ignore these SHN_FBSD_CACHED symbols the same way it ignored SHN_UNDEF symbols before. I didn't find other users of SHN_UNDEF that might be negatively affected by this change. Testing done: Basic Dtrace testing suggests it isn't broken. Performance results: Before # for i in 1 2 3 ; do time kldload efs.ko ; kldunload efs.ko ; done kldload efs.ko 0.00s user 13.00s system 103% cpu 12.607 total kldload efs.ko 0.00s user 13.25s system 102% cpu 12.876 total kldload efs.ko 0.00s user 13.14s system 103% cpu 12.752 total kldload zfs.ko 0.00s user 1.77s system 102% cpu 1.729 total # Smaller, for comparison After kldload efs.ko 0.00s user 0.25s system 99% cpu 0.252 total kldload efs.ko 0.00s user 0.20s system 99% cpu 0.204 total kldload efs.ko 0.00s user 0.21s system 99% cpu 0.211 total kldload zfs.ko 0.00s user 0.04s system 96% cpu 0.040 total # For comparison Sponsored by: EMC / Isilon storage division -- You are receiving this mail because: You are the assignee for the bug.