From owner-freebsd-hackers@freebsd.org Thu Apr 26 20:19:39 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68EF8FB746E for ; Thu, 26 Apr 2018 20:19:39 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DA64478479 for ; Thu, 26 Apr 2018 20:19:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f47.google.com with SMTP id m18-v6so15301566lfb.0 for ; Thu, 26 Apr 2018 13:19:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:resent-from:resent-to:resent-date :resent-message-id:resent-user-agent:to:from:subject:openpgp :message-id:date:user-agent:mime-version:content-language :content-transfer-encoding; bh=mQjAbz4b/NxuVh1zRWVH0rJbYsupxA3YxLvt4MLHuK8=; b=mCya49pTnyrxtH0qvGWFWGUkuyI/2XKNJwsa/LCKVdEfTY7FzXSmQr86Ho1N7mEvCi x0sbpd1KPOnYmtx2BgF6ebyZK1X6CZA4WqBGrvBB9YpTpg3sVcGL6S6A1TyFjxIybPHp xLTz8FwQQnGDI4IaH0bNdUwTfYY2WPusKP6OBTBaC1/aPrjKfJyX/4K4V+tI6zh7GmcC H5dwOiHFV9vLXyidBq3mU2jMuC5vEmnUzYiYNF9yMALgqZwrh/GBgtTz7qAADZOGsTx4 oJdsdumXwp4JNrFRVHVXu7+lyfRdoHvEBTuz1GybF8oSQkgY/JsXxXUniHebrSi/icWr g3Tw== X-Gm-Message-State: ALQs6tDSilCz819K87TQKTrivAezUncoJibYrhcs+BeZLOlt5z2zNGXb y97khyxoShNZUSWEYqd6jQyZg5CB X-Google-Smtp-Source: AB8JxZptNehyQUUS31SAbupQgX2hnGp+KUrUvkb5GxAWXPiwmI0ZOh0aO4wMh8Z3zOYR2ThvJAV0EQ== X-Received: by 10.46.158.85 with SMTP id g21mr114558ljk.30.1524773976983; Thu, 26 Apr 2018 13:19:36 -0700 (PDT) Received: from [10.141.214.5] ([78.111.187.12]) by smtp.googlemail.com with ESMTPSA id z4sm293534lji.14.2018.04.26.13.19.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Apr 2018 13:19:36 -0700 (PDT) Resent-From: Andriy Gapon Resent-To: "freebsd-hackers@freebsd.org" Resent-Date: Thu, 26 Apr 2018 23:19:27 +0300 Resent-Message-ID: <5a296ca4-ac55-c4c8-186b-cc4a72ade651@FreeBSD.org> Resent-User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 To: "freebsd-hackers@freebsd.org" From: Andriy Gapon Subject: suspend to ram and AP cache Openpgp: preference=signencrypt Message-ID: Date: Thu, 26 Apr 2018 23:07:20 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 20:19:39 -0000 I am playing with some changes to suspend / resume code and I see some behavior that I didn't quite expect. Here is a sketch of the code. AP: - savectx - wbinvd - atomic_add_rel_int(&suspend_count, 1) // lock add $1, addr - while (1) cpu_spinwait() BSP: - while (suspend_count < expect_count) cpu_spinwait() - wbinvd - enter ACPI suspend What I see is that after the resume suspend_count may have a value different from what it was before the suspend (expect_count). In fact, it's just zero in all tests I have done so far. If I move wbinvd to a position just after the suspend_count increment, then the post-resume value is consistently the expected value. It appears that the changes to suspend_count performed by the APs may never make it to the main memory unless the caches of the APs are flushed afterwards. This is the part that I find surprising. I will double-check the code and the test to be more confident that what I described is what actually happens. The hardware is AMD and the architecture is amd64. I understand that on modern hardware even lock-ed stores do not have to go to the main memory immediately as they can be handled by cache coherency protocols. For AMD it seems to be MOESI. I guess that one of the AP caches Owns the cached (dirty) value and the BSP has it only in the Shared mode, so the cache flush on the BSP did nothing to store the cached value and the owning AP didn't get enough time to do the store. Just wanted to share this and get some feedback on whether the theory is plausible. -- Andriy Gapon