Date: Fri, 23 Mar 2012 00:35:17 -0700 (PDT) From: Sushanth Rai <sushanth_rai@yahoo.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: Improving gcore Message-ID: <1332488117.24609.YahooMailClassic@web180005.mail.gq1.yahoo.com> In-Reply-To: <20120322140151.GB2358@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
What I mean by inconsistent is that, a process with lots of threads takes a while before all threads are suspended. When I look at the resulting core file, the state of some of the shared data is not exactly what I was expecting when I issued the gcore command. It is quite possible that state might have changed even before ptrace() had a chance to issue SIGSTOP. But I am looking at any improvement that can be reasonably done in kernel. As you described suspension are checked at safe points and only when threads reach those, they get suspended. I understand and agree with the reasons behind asynchronous stopping. But the net effect is that threads can potentially run for a short duration before they suspend themselves. So, I am trying to figure out ways to reduce this duration as much as possible. One thing I noticed is that in sig_suspend_threads(), we check if the threads are sleeping interruptibly. If so, they get suspended immediately. Otherwise, set TDF_ASTPENDING and if the thread is running on CPU we send IPI_AST to that CPU. What about the target process's threads that are on the runq ? It looks like the thread will only notice the flag when it is at user->kernel boundary. Can we safely remove them out of the runq ? With respect to PT_SUSPEND, as part of PT_ATTACH request I was thinking of explicitly suspending all the threads by setting TDF_DBSUSPEND instead of posting SIGSTOP. As each thread in the target process calls thread_suspend_check(), it would notice this flag and suspend itself. PT_ATTACH command would then wait until all threads are suspended before returning to the caller. This is the general approach and ofcourse it is missing details at this point. The idea again is to suspend all threads as quickly as possible. I'm running on 7.2. Cursory look at trunk version didn't show major changes in this area. Thanks, Sushanth --- On Thu, 3/22/12, Konstantin Belousov <kostikbel@gmail.com> wrote: > From: Konstantin Belousov <kostikbel@gmail.com> > Subject: Re: Improving gcore > To: "Sushanth Rai" <sushanth_rai@yahoo.com> > Cc: freebsd-hackers@freebsd.org > Date: Thursday, March 22, 2012, 7:01 AM > On Wed, Mar 21, 2012 at 04:35:13PM > -0700, Sushanth Rai wrote: > > Sometimes I have trouble capturing the "correct" state > of a > > multithreaded process using gcore. That is, it looks > like target > > process might have done some work since the time > command was issued > > and the core file was generated. > > > > Looking at the code, gcore calls ptrace(PT_ATTACH...), > which > > internally issues SIGSTOP, and calls waitpid() to wait > until the > > process stops. So, it's quite possible that some > threads that are not > > sleeping interruptibly will continue to run until the > process notices > > the signal. Signals are only checked when a thread that > is tagged to > > handle the signal crosses the user boundary (return > from syscall, > > trap). When the thread finally handles SIGSTOP, it > needs to stop all > > threads, which is done by lighting a flag-bit it each > thread. This > > bit is checked as each thread crosses the user > boundary. So, there > > will always be some state change in the target process > from the time > > SIGSTOP is posted to the time all threads are actually > stopped. > Yes, this is how things work. There are two factors causing > the asynchronous > stopping: > first, other CPUs may execute several threads of the > process, so the > suspension of that other threads require an IPI to be > generated. IPI_AST > handler just returns, which causes kernel->usermode > transition and > possible signal delivery and suspend check. > > second, kernel never allows to suspend thread executing and > blocked in > kernel. Doing otherwise would cause deadlocks, because > executing threads > own resources that are shared with other threads. > > So, the only safe points to suspend the threads is at > kernel->user boundary > or at some sleep points that are not marked as unsafe with > PBDRY flag. > On the other hand, since kernel waits for all threads to > suspend before > reporting the wait(2) event, the usermode state shall be > consistent with > itself, or rather, it shall be not worse then if the threads > reach the > stop point executing asynchronously on different CPUs. > > See the check for p->p_suspcount == p->p_numthreads in > the kern_wait() > function before it decides that the found process is > satisfactory > for wait request. > > > > > I was wondering if I could improve this a bit by > calling PT_SUSPEND on > > all threads, instead of posting SIGSTOP and waiting for > all threads > > to stop. Once the core is generated, unsuspend all > threads. As with > > SIGSTOP, individual thread will only notice suspension > as they cross > > user boundary. But there is no overhead of tagging a > thread to handle > > the signal and that thread doing the suspension. The > idea is to try > > and generate the core file which reflects the running > state of the > > process as closely as possible. > PT_SUSPEND can only be called on the process which you > alread attached to. > So the call to suspend all threads of the just attached > threads is mostly > nop for your purposes. > > > > > Does this sound reasonable ? > I think you need to describe in more details what do you > mean by > inconsistent state of the threads in gcore-generated core > file, before > some conclusion could be made. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1332488117.24609.YahooMailClassic>