Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 17 Feb 2012 00:06:40 -0800
From:      Julian Elischer <julian@freebsd.org>
To:        davidxu@freebsd.org
Cc:        David Xu <listlog2011@gmail.com>, Alexander Kabaev <kan@freebsd.org>, Andriy Gapon <avg@freebsd.org>, threads@freebsd.org, FreeBSD Stable <freebsd-stable@freebsd.org>, Jung-uk Kim <jkim@freebsd.org>
Subject:   Re: pthread_cond_timedwait() broken in 9-stable? (from JAN 10)
Message-ID:  <4F3E0A90.6080400@freebsd.org>
In-Reply-To: <4F3E04AB.2000508@freebsd.org>
References:  <4F3C2671.3090808__7697.00510795719$1329343207$gmane$org@freebsd.org>	<4F3D3E2D.9090100@FreeBSD.org>	<4F3D6FDD.9050808@freebsd.org>	<4F3D89CD.9050309@freebsd.org> <4F3DA27A.3090903@freebsd.org>	<4F3DB3DB.2060603@gmail.com> <4F3DB91A.2090806@freebsd.org>	<4F3DBE90.5030305@gmail.com> <4F3E04AB.2000508@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2/16/12 11:41 PM, Julian Elischer wrote:
> adding jkim as he seems to be the last person working with TSC.
>
>
> On 2/16/12 6:42 PM, David Xu wrote:
>> On 2012/2/17 10:19, Julian Elischer wrote:
>>> On 2/16/12 5:56 PM, David Xu wrote:
>>>> On 2012/2/17 8:42, Julian Elischer wrote:
>>>>> Adding David Xu for his thoughts since he reqrote the code in 
>>>>> quesiton in revision 213098
>>>>>
>>>>> On 2/16/12 2:57 PM, Julian Elischer wrote:
>>>>>> On 2/16/12 1:06 PM, Julian Elischer wrote:
>>>>>>> On 2/16/12 9:34 AM, Andriy Gapon wrote:
>>>>>>>> on 15/02/2012 23:41 Julian Elischer said the following:
>>>>>>>>> The program fio (an IO test in ports) uses pthreads
>>>>>>>>>
>>>>>>>>> the following code (from fio-2.0.3, but its in earlier code 
>>>>>>>>> too)
>>>>>>>>> has suddenly started misbehaving.
>>>>>>>>>
>>>>>>>>>          clock_gettime(CLOCK_REALTIME,&t);
>>>>>>>>>          t.tv_sec += seconds + 10;
>>>>>>>>>
>>>>>>>>>          pthread_mutex_lock(&mutex->lock);
>>>>>>>>>
>>>>>>>>>          while (!mutex->value&&  !ret) {
>>>>>>>>>                  mutex->waiters++;
>>>>>>>>>                  ret = 
>>>>>>>>> pthread_cond_timedwait(&mutex->cond,&mutex->lock,&t);
>>>>>>>>>                  mutex->waiters--;
>>>>>>>>>          }
>>>>>>>>>
>>>>>>>>>          if (!ret) {
>>>>>>>>>                  mutex->value--;
>>>>>>>>>                  pthread_mutex_unlock(&mutex->lock);
>>>>>>>>>          }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It turns out that 'ret' sometimes comes back instantly (on 
>>>>>>>>> my machine) with a
>>>>>>>>> value of 60 (ETIMEDOUT)
>>>>>>>>> despite the fact that we set the timeout 10 seconds into the 
>>>>>>>>> future.
>>>>>>>>>
>>>>>>>>> Has anyone else seen anything like this?
>>>>>>>>> (and yes the condition variable attribute have been set to 
>>>>>>>>> use the REALTIME clock).
>>>>>>>> But why?
>>>>>>>>
>>>>>>>> Just a hypothesis that maybe there is some issue with time 
>>>>>>>> keeping on that system.
>>>>>>>> How would that code work out for you with MONOTONIC?
>>>>>>>
>>>>>>> Jens Axboe, (CC'd) tried both CLOCK_REALTIME and 
>>>>>>> CLOCK_MONOTONIC, and they both had the same problem..
>>>>>>> i.e. random early returns with ETIMEDOUT.
>>>>>>>
>>>>>>> I think we will try move out machine forward to a newer 
>>>>>>> -stable to see if it resolves.
>>>>>> Kan upgraded the machine today to today's 9.x branch tip and 
>>>>>> the problem still occurs.
>>>>>> 8.x does not have this problem.
>>>>>>
>>>>>> I have not got a 9-RELEASE machine to test on.. so I can not 
>>>>>> tell if this came in with the burst of stuff
>>>>>> that came in after the 9.x branch was unfrozen after the 
>>>>>> release of 9.0.
>>>>>>
>>>>>>
>>>>>
>>>> I am trying to reproduce the problem,  do you have complete 
>>>> sample code to test ?
>>>
>>> I'm still looking the exact set
>>> but on my machine (4 cpus) the program from ports sysutils/fio 
>>> exhibits the problem when used with
>>> kern.timecounter.hardware=TSC-low and with the following config file:
>>>
>>> pu05 # cat config.fio
>>>
>>> [global]
>>> #clocksource=cpu
>>> direct=1
>>> rw=randread
>>> bs=4096
>>> fill_device=1
>>> numjobs=16
>>> iodepth=16
>>> #ioengine=posixaio
>>> #ioengine=psync
>>> ioengine=psync
>>> group_reporting
>>> norandommap
>>> time_based
>>> runtime=60000
>>> randrepeat=0
>>>
>>> [file1]
>>> filename=/dev/ada0
>>>
>>> pu05 #
>>> pu05 # fio config.fio
>>> fio: this platform does not support process shared mutexes, 
>>> forcing use of threads. Use the 'thread' option to get rid of this 
>>> warning.
>>> file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=psync, iodepth=16
>>> ...
>>> file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=psync, iodepth=16
>>> fio 2.0.3
>>> Starting 15 threads and 1 process
>>> fio: job startup hung? exiting.
>>> fio: 5 jobs failed to start
>>> Segmentation fault (core dumped)
>>> pu05#
>>>
>>>
>>> The reason 5 jobs failed to start is because the parent timed out 
>>> on them immediately.
>>> It didn't time out on 10 of them apparently.
>>>
>>>
>>> if I set the timer to ACPI-fast it works as expected..
>> maybe following code can check to see if TSC-LOW works by let the 
>> thread run
>> on each cpu.
>>
>> gettimeofday(&prev, NULL);
>> int cpu = 0;
>> for (;;) {
>>      cpuset_t set;
>>      cpu = ++cpu % 4;
>>      CPU_ZERO(&set);
>>      CPU_SET(cpu, &set);
>>      pthread_setaffinity_np(pthread_self(), sizeof(set), &set);
>>      gettimeofday(&cur, NULL);
>>      if ( timercmp(&prev, &cur, >=)) {
>>         abort();
>>    }
>> }
>>
>>

pu05# sysctl kern.timecounter.hardware=TSC-low
kern.timecounter.hardware: ACPI-fast -> TSC-low
pu05# ./test
^C
pu05# cat test.c

#include <stdlib.h>
#include <sys/param.h>
#include <sys/cpuset.h>
#include <pthread_np.h>

#include <sys/time.h>

main()
{
     int cpu = 0;
     struct timeval prev, cur;

     gettimeofday(&prev, NULL);
     for (;;) {
          cpuset_t set;
          cpu = ++cpu % 4;
          CPU_ZERO(&set);
          CPU_SET(cpu, &set);
          pthread_setaffinity_np(pthread_self(), sizeof(set), &set);
          gettimeofday(&cur, NULL);
          if ( timercmp(&prev, &cur, >)) {
             abort();
        }
        prev = cur;
     }
}

pu05# ./test

minutes pass.......

^C
pu05#

so it looks as if the TSC is working ok..
I'm just going to check that the program is actually moving CPU...
yes it is moving around but I can't tell at what speed. (according to 
top).

so we are still left with a question of "where is the problem?"

kernel TSC driver?
generic gettimeofday() code?
pthreads cond code?
the application?








Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F3E0A90.6080400>