From owner-freebsd-hackers@freebsd.org Sun Feb 23 19:48:31 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A8CE246420 for ; Sun, 23 Feb 2020 19:48:31 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound3d.ore.mailhop.org (outbound3d.ore.mailhop.org [54.186.57.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48QbNG1DZSz41Pq for ; Sun, 23 Feb 2020 19:48:29 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1582487308; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=dQ2MueuSAaLfkKomJGG3ZiX1Tz/0x5M3N+nc1FdGAi1+pQZMUw3qyPiB6arQs84AxKzNuTzISCAeV Y+iBhqLwBMwV3NG1CxdRsfw50DUKz2UDP7Vs4lX2Rtuu9/bwBHaYlvnbQWbXtIBjYhjHWnI0wAyND8 PPCECweziZH+r1X6rSoRutIK1AJ+FRtZ7UE4IPjt9zTmQt1QuIsymEtxjHJsvWXq3YYuNuEB8tvD4Q e27BcLc2mQnDyGpfL8NtDwwZpuUDPweOCQp0mudY4mAilzUqeN/F6L48QZ2+Ymc2UFh1MYnaNTCxGK U6EUpuQgF9CG2KimFCvf9Uh5pRWXY9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=sKknGiZFuF7RnYPwxL1LAwKqR8u011HA0fP1FpscuFo=; b=VCzNhmopJBF+WNPLPfpWnl7+yD5iBYKa8oJdkMq7dy1JM8bEIPpmjfcI9pb18VYXO3TKka6hAAppa ITEkxR0NwCSZdNFGIyw6gPYIxLmLfKKD0DZDdST8/GcfKp/un8M5dg8WIAU15Vx4WvlIe3P0xLj9Yx E0NfWdkexQzFh+fQReFOg5xBoqW8QiNCY7aMC/6HDo22mMYMKXXhLOWSBHb05lCwtzd4nWX+HVI6JY RXsmKvqZsvFonbDVyqSF7TiZRXTbbuS6vVLkMDX943h1OOKXwRavPcs6rTXsxpfpjsqg/kc/4oHKsx iXAh/nkyUTHzbCElC68hmzvmVfISKaQ== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=sKknGiZFuF7RnYPwxL1LAwKqR8u011HA0fP1FpscuFo=; b=CDCOAn4oYoMdZ9HDdloZFXNoVGWmV5S7RlkIL1+8qr8oEXVAbwmYk20wOIuj6IwyoDLqDpCKQf8dD zi4MnKzXgRN6Gl3cOTV8Nga9Aq5pRIFIuBcYziRisnDPA24989pr8pW8qvA0XSRmR5cM0izN0rGJMd hoo3SpQe1nBviLOpE90g68CjlWUpOv89oNcK+F77loOg8K+ikCoyYK7BaUp2D9zYVgEyowoB8uhis2 73Nj36cYEbWgd0fpaH6cn3VgYBvcrfRyPIVLXC2A2TXYzVZ4+yzTav3JyZUQrgDEIWE+ssBg7fnVYd zWsoJ/r6D0AHqgbhs+an071hrZskgMg== X-MHO-RoutePath: aGlwcGll X-MHO-User: 7515ba2f-5675-11ea-b80e-052b4a66b6b2 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id 7515ba2f-5675-11ea-b80e-052b4a66b6b2; Sun, 23 Feb 2020 19:48:27 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id 01NJmQQD020048; Sun, 23 Feb 2020 12:48:26 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <08fd4dcf7231237abf749834d6bce006892eac26.camel@freebsd.org> Subject: Re: msleep_spin is failed to waken up even after wakeup routine is invoked for the same. From: Ian Lepore To: Arpan Palit Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Date: Sun, 23 Feb 2020 12:48:26 -0700 In-Reply-To: References: <890a299c074fc83a02911583531d686257924be8.camel@freebsd.org> Content-Type: text/plain; charset="ASCII" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48QbNG1DZSz41Pq X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.97)[-0.968,0]; ASN(0.00)[asn:16509, ipnet:54.186.0.0/15, country:US]; NEURAL_HAM_LONG(-1.00)[-0.996,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Feb 2020 19:48:31 -0000 On Sun, 2020-02-23 at 12:30 +0530, Arpan Palit wrote: > Thanks for replying Ian. > > The code is similar to the fix structure code as you have mentioned above. > > Its happening after running the test for 5 to 10 iterations, out of 2k > posted command to the hardware only 1 or 2 of them are failing to wake > up properly. mutex lock has been properly introduced in both > wait_for_completion code & interrupt handler part and also the doneflag is > getting set from the interrupt context properly. > > I am running 16 thread of 2k command simultaneously. > > I have verified that from the hardware side the command completion is > proper, its failing in waking up the sleep in software side. > > So I am suspecting some issue with the scheduler which failed to wake up > the sleep in this case. > > Below is the current code structure: > > /* Wait for Completion */ > mtx_lock(cmd_comp_lock); > completion = false; > ret = execute_cmd(cmd); > if (ret == 0) { > while ( !completion && err == 0) > err = msleep(cmd, cmd_comp_lock, PCATCH, "cmd_xfer", hz); > } > mtx_unlock(cmd_comp_lock); > > /* Interrupt handler */ > mtx_lock_spin(cmd_lock); > *//Disable inerrupt > *// error checks > completion = true; > wakeup_one(cmd); > mtx_unlock_spin(cmd_lock); > *//Enable interrupt > That shows two different mutexes being used, which violates the safe pattern for doing conditional wakeups. The same mutex that protects the setting of the 'completion' variable in the interrupt handler must be used to protect the clearing and testing of that variable in the mainline code, to avoid missed wakeups. That means using msleep_spin() with cmd_lock in the mainline code. -- Ian > > On Wed, Feb 19, 2020 at 8:12 PM Ian Lepore wrote: > > > On Wed, 2020-02-19 at 14:13 +0530, Arpan Palit wrote: > > > Hi, > > > > > > I am facing one issue where wakeup rountine call is unable to > > > waken up a > > > msleep_spin routine call with a timeout value of 10 Seconds. > > > > > > The real scenario is as follows: post a hardware command and > > > sleep using > > > msleep_spin routine till interrupt comes, After getting the > > > interrupt > > > > waken > > > up the sleeping process using wakeup_one/wakeup routine call. As > > > there > > > > are > > > more than 2048 command and 16 parallel threads are running, > > > observed randomly *one or two of the posted command* is *timing > > > out* for > > > which the *interrupt has came and also wakeup routine is invoked > > > *after > > > getting the interrupt for the same command. > > > > > > Note: > > > *The issue is not seen when number of commands are less than 2048 > > > with > > > timeout of 10 seconds. > > > *The issue can be seen with less number of commands also when > > > timeout > > > > value > > > 1 second. > > > > > > Can anyone please provide me an optimized way to schedule the > > > process or > > > > a > > > better way to do the scheduling. > > > > > > Thanks, > > > Arpan Palit > > > > > > > Is there any chance this is the classic "missed wakeup" scenario, > > where > > the wakeup happens before the thread enters msleep_spin()? That > > can > > happen with code structured like > > > > enqueue_request(req); > > err = msleep_spin(req, etc); > > /* Handle done req or timeout */ > > > > and a fix is to structure the code using the same idiom required > > for > > pthread_cond_wait() in userland, something like: > > > > req->doneflag = false; > > enqueue_request(req); > > while (!req->doneflag && err == 0) > > err = msleep_spin(req, etc); > > /* Handle done req or timeout */ > > > > and of course on the interrupt handler side you need something like > > > > /* lock mutex, do hardware stuff */ > > req->doneflag = true; > > wakeup(req); > > /* unlock mutex */ > > > > -- Ian > > > > > > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org"