From owner-freebsd-hackers@freebsd.org Wed Sep 16 20:02:32 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E79343EF9CD for ; Wed, 16 Sep 2020 20:02:32 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bs9xM65J0z4CMj for ; Wed, 16 Sep 2020 20:02:31 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f173.google.com with SMTP id r24so7034663ljm.3 for ; Wed, 16 Sep 2020 13:02:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=FuSFqhnww5g6YakjmCjfYih8fjHm+8t0eibP1A2sz6A=; b=eFGzulkFFaZiQ8INH3jvnBHjSFUoRtRCOOl5cceKiZ9CgcliIg2mmr90MGmvDDbNAn jnp8+YMEND1sYZPPT+sACfJ63W0mZfsTh4xRxZQ+ULgHoUME4HoSUhOlBVBFotcPR/Dm EIN8bIBCD0xHeVRYMvzGCPuPq/V+CH6+DYIA7DUsdEA5RBudOun9NkL3pPU+S+O2opQ+ qJ2skTVgYIoB79Vwx65mWgc8DZnA3UplcPJqM1a7/YL6ZkTrmhXL3rXsKfiZffJVw76n 3qd28mbbLYj6VEIhFSdKZmzfKHkDd5sZA/st7kcsh61np/ffa3TQR5j8jsezejMMLSd4 maxQ== X-Gm-Message-State: AOAM533HErFHyMO8SRuajmBa6ORjas16Ynlh8yO0YW2P3fICDYICyC5f 5BsWoxKYerq/cxeGJ+CY25aTZpql03Q= X-Google-Smtp-Source: ABdhPJxc4DVnEtGQUgxQ+M7m/64qVnOlEZN4FKZD+lYGsmBFZxsHpjTPkofDi9ymX4TSLJYun0ICUw== X-Received: by 2002:a05:651c:1188:: with SMTP id w8mr9834385ljo.344.1600286549510; Wed, 16 Sep 2020 13:02:29 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id j4sm5554880lja.31.2020.09.16.13.02.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 16 Sep 2020 13:02:28 -0700 (PDT) Subject: Re: MSR accesses that slows down the hypervisor/host To: Konstantin Belousov , Wei Hu Cc: "freebsd-hackers@freebsd.org" References: <20200916135727.GO94807@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mDMEX1iFDhYJKwYBBAHaRw8BAQdAiu8JG/oLFkVkOAJqJc7Dx5KI/Q6C3SBI20EQm+DXnAu0 HkFuZHJpeSBHYXBvbiA8YXZnQEZyZWVCU0Qub3JnPoiWBBMWCAA+FiEEyCHHZM09l0OE3Ir/ 1A1+Gq8+L1EFAl9YhQ4CGwMFCQeEzgAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ1A1+ Gq8+L1Fc0wD/ZjmhHfbCJywZU3aOxXIPjcz73FYEGMvqMCCLAWyLbSABALFL+1ZNrjV3BGjq 889cOYFuboA/Yn3eWezS+tfqYBsGuDgEX1iFDhIKKwYBBAGXVQEFAQEHQL6B20Xi600TrkpG P9fWjl7JtHNxqrHKhX6Kg7kgb4ILAwEIB4h+BBgWCAAmFiEEyCHHZM09l0OE3Ir/1A1+Gq8+ L1EFAl9YhQ4CGwwFCQeEzgAACgkQ1A1+Gq8+L1F3cgEAktp4h+IJUJxL1vn6zMOt//znni/J TanKfQuA8wGXcGkBAKpZJhqMkg+pKk7MGvJhgJ6nCpTZ+rMK6vZVZLUWc3QF Message-ID: <6d17f16e-d672-11e1-eabc-e9d64bc13ff4@FreeBSD.org> Date: Wed, 16 Sep 2020 23:02:27 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20200916135727.GO94807@kib.kiev.ua> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Bs9xM65J0z4CMj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.173 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.23 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.25)[-0.254]; FREEMAIL_TO(0.00)[gmail.com,microsoft.com]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.966]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.01)[-1.012]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.173:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.173:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Sep 2020 20:02:33 -0000 On 16/09/2020 16:57, Konstantin Belousov wrote: > On Wed, Sep 16, 2020 at 06:52:42AM +0000, Wei Hu via freebsd-hackers wrote: >> Hello, >> >> There are couple AMD processor related MSRs which are being accessed in FreeBSD. >> >> #define MSR_AMDK8_IPM 0xc0010055 >> #define MSR_LS_CFG 0xc0011020 >> >> We are seeing a lot of CPU time being spent in the host (Hyper-V) in handling traps when accessing these MSRs. Especially the first MRS is frequently being accessed in cpu_idle() so the performance impact to host is significant. >> >> We noted that Linux made some code changes in the 4.10 kernel to access the first MSR much less frequently. So we are wondering if there are similar changes in FreeBSD that might be in the plan. Microsoft Hyper-V team is also planning some work to speed up the accesses to these MSRs. However any suggestions or plan to improve in the FreeBSD guest are welcome. >> > > Where do you see accesses to MSR_LS_CFG ? I can only find manipulations > of that MSR in init_amd(), and then it is all under check that we are not > virtualized. > > For MSR_AMDK8_IPM access in cpu_idle(), it seems that the workaround was > applied too wide. It might be that we do not need to do it on recent CPUs, > but I need to spent more time looking at datasheets to confirm/deny. > > But, do you (hypervisor) indeed allow guest to initiate C1 or deeper idle > state ? If not, perhaps as the first measure, we can avoid manipulating > MSR_AMDK8_IPM under hypervisor at all. I agree with these observations and suggestions. Additionally, I am not sure why we check and clear AMDK8_CMPHALT bit on every call of cpu_idle(). Maybe in the past we had to deal with firmware / SMM code that aggresivekly restored that bit despite the wishes of our OS. I would assume that "firmware" of virtual machines does not do that. Having said that, echoing what Kostik said, I really doubt that any hypervisor faithfully emulates AMDK8_CMPHALT. I do not see any reason to do that. -- Andriy Gapon