sched/hrtimer: hrtimer state machine improvement for SMP cases #17642

wangchdo · 2025-12-23T00:56:50Z

Summary

This PR is also included in #17573. Since #17573 covers both overall scheduler support with hrtimer and related hrtimer improvements for SMP mode, and since hrtimer is an independent module, this change is submitted separately to focus on hrtimer improvements and avoid conflicts with other scheduler-related enhancements such as #17640.

This PR refines the hrtimer state machine to allow ARMED and RUNNING hrtimers to be safely restarted, and fixes several corner-case issues in SMP mode.

The updated hrtimer state machine is shown below and is also documented in the hrtimer module documentation:

Impact

Improments for the hrtimer module

Testing

Test 1 passed (integrated in ostest):

- test implementation:

/****************************************************************************
 * apps/testing/ostest/hrtimer.c
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.  The
 * ASF licenses this file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance with the
 * License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
 * License for the specific language governing permissions and limitations
 * under the License.
 *
 ****************************************************************************/

/****************************************************************************
 * Included Files
 ****************************************************************************/

#include <nuttx/config.h>
#include <nuttx/hrtimer.h>

#include <stdio.h>
#include <sched.h>

#include "ostest.h"

/****************************************************************************
 * Pre-processor Definitions
 ****************************************************************************/

#define NSEC_PER_500MS (500 * NSEC_PER_MSEC)

/* Set a 1ms margin to allow hrtimertest to pass in QEMU.
 *
 * QEMU is a virtual platform, and its timer resolution and scheduling
 * latency may be less precise than on real hardware. Using a larger
 * margin ensures that tests do not fail due to timing inaccuracies.
 *
 * On real hardware (verified on the a2g-tc397-5v-tft board), this
 * margin can be reduced to less than 5 ns because timers are precise
 * and deterministic.
 */

#define NSEC_MARGIN    (NSEC_PER_MSEC)

/* Simple assertion macro for HRTimer test cases */
#define HRTIMER_TEST(expr, value)                                   \
  do                                                                \
    {                                                               \
      ret = (expr);                                                 \
      if (ret != (value))                                           \
        {                                                           \
          printf("ERROR: HRTimer test failed, line=%d ret=%d\n",   \
                 __LINE__, ret);                                    \
          ASSERT(false);                                            \
        }                                                           \
    }                                                               \
  while (0)

/****************************************************************************
 * Private Types
 ****************************************************************************/

/* Structure for HRTimer test tracking */

struct hrtimer_test_s
{
  hrtimer_t timer;    /* HRTimer instance */
  uint64_t  previous; /* Previous timestamp in nanoseconds */
  uint32_t  count;    /* Number of timer expirations */
  uint32_t  period;   /* Expected period between expirations */
  bool      active;   /* True while the test is still running */
};

/****************************************************************************
 * Private Functions
 ****************************************************************************/

/****************************************************************************
 * Name: hrtimer_test_init
 *
 * Description:
 *   Initialize a hrtimer_test_s structure for a new test.
 *
 * Input Parameters:
 *   test_hrtimer - Pointer to the test structure to initialize
 *   period       - Expected timer period in nanoseconds
 *
 * Returned Value:
 *   None
 *
 ****************************************************************************/

static void hrtimer_test_init(FAR struct hrtimer_test_s *test_hrtimer,
                              uint32_t period)
{
  test_hrtimer->previous = 0;
  test_hrtimer->count    = 0;
  test_hrtimer->active   = true;
  test_hrtimer->period   = period;
}

/****************************************************************************
 * Name: test_hrtimer_callback
 *
 * Description:
 *   HRTimer callback function for test.
 *
 *   - Verifies the timer interval is exactly 500ms (nanosecond precision)
 *   - Stops the test after 15 expirations
 *   - Re-arms the timer in absolute mode
 *
 * Input Parameters:
 *   hrtimer - Pointer to the expired HRTimer instance
 *
 * Returned Value:
 *   Timer period in nanoseconds (NSEC_PER_500MS)
 *
 ****************************************************************************/

static uint32_t test_hrtimer_callback(FAR hrtimer_t *hrtimer)
{
  struct timespec ts;
  uint32_t diff;
  uint64_t now;
  int ret;

  FAR struct hrtimer_test_s *test =
    (FAR struct hrtimer_test_s *)hrtimer;

  /* Increment expiration count */

  test->count++;

  /* Get current system time */

  clock_systime_timespec(&ts);
  now = clock_time2nsec(&ts);

  /* Skip comparison for first two invocations */

  if (test->count > 2)
    {
      /* Verify the timer interval is exactly
       * 500ms with nsec resolution
       */

      diff = (uint32_t)(now - test->previous);

      HRTIMER_TEST(NSEC_PER_500MS < diff + NSEC_MARGIN, true);
      HRTIMER_TEST(NSEC_PER_500MS > diff - NSEC_MARGIN, true);
    }

  test->previous = now;

  /* Stop the test after 15 expirations */

  if (test->count  >= 15)
    {
      ret = hrtimer_cancel(hrtimer);
      HRTIMER_TEST(ret, 0);

      test->active = false;
    }

  return test->period;
}

/****************************************************************************
 * Public Functions
 ****************************************************************************/

/****************************************************************************
 * Name: hrtimer_test
 *
 * Description:
 *   Entry point for high-resolution timer functional test.
 *
 *   - Initializes a HRTimer
 *   - Starts it with a 500ms relative timeout
 *   - Verifies subsequent expirations occur at 500ms intervals
 *
 * Input Parameters:
 *   None
 *
 * Returned Value:
 *   None
 *
 ****************************************************************************/

void hrtimer_test(void)
{
  int ret;
  struct hrtimer_test_s test_hrtimer_500ms;

  /* Initialize test structure */

  hrtimer_test_init(&test_hrtimer_500ms, NSEC_PER_500MS);

  /* Initialize the high-resolution timer */

  hrtimer_init(&test_hrtimer_500ms.timer,
               test_hrtimer_callback,
               NULL);

  /* Start the timer with 500ms relative timeout */

  ret = hrtimer_start(&test_hrtimer_500ms.timer,
                      test_hrtimer_500ms.period,
                      HRTIMER_MODE_REL);

  HRTIMER_TEST(ret, OK);

  /* Wait until the test completes */

  while (test_hrtimer_500ms.active)
    {
      usleep(500 * USEC_PER_MSEC);
    }
}

test log on rv-virt:smp64:

NuttShell (NSH)
nsh> 
nsh> uname -a
NuttX 0.0.0 6847a0cc95-dirty Dec 20 2025 12:26:39 risc-v rv-virt
nsh> ostest

(...)

user_main: hrtimer test

End of test memory usage:
VARIABLE  BEFORE   AFTER
======== ======== ========
arena     1fbdec0  1fbdec0
ordblks         9        9
mxordblk  1f73880  1f73880
uordblks    109b0     ff18
fordblks  1fad510  1fadfa8

Final memory usage:
VARIABLE  BEFORE   AFTER
======== ======== ========
arena     1fbdec0  1fbdec0
ordblks         1        9
mxordblk  1fb2cf8  1f73880
uordblks     b1c8     ff18
fordblks  1fb2cf8  1fadfa8
user_main: Exiting
ostest_main: Exiting with status 0

test 2 passed (provided by @Fix-Point )

test implementation

/****************************************************************************
 * apps/examples/hello/hello_main.c
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.  The
 * ASF licenses this file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance with the
 * License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
 * License for the specific language governing permissions and limitations
 * under the License.
 *
 ****************************************************************************/

/****************************************************************************
 * Included Files
 ****************************************************************************/

#include <nuttx/config.h>
#include <stdio.h>
#include <unistd.h>

#include <nuttx/hrtimer.h>

#define HRTIMER_TEST_THREAD_NR (1)
#define HRTIMER_TEST_NR        (1000000)

/****************************************************************************
 * Public Functions
 ****************************************************************************/

static int volatile tcount = 0;
static volatile uint32_t next = 0;

static uint32_t test_callback(FAR struct hrtimer_s *hrtimer)
{

  tcount++;
  up_ndelay(hrtimer->expired % (10 * NSEC_PER_USEC));

  return 0;
}

static uint32_t test_callback_background(FAR struct hrtimer_s *hrtimer)
{
  up_ndelay(hrtimer->expired % NSEC_PER_USEC);
  return 0;
}


static void test1(int tid)
{
  hrtimer_t   timer;
  int         count = 0;
  irqstate_t flags;
  spinlock_t lock;

  if (tid == 0)
    {
      hrtimer_init(&timer, test_callback, NULL);
    }
  else
    {
      hrtimer_init(&timer, test_callback_background, NULL);
    }

  while (count++ < HRTIMER_TEST_NR)
    {
      int ret;
      if (tid == 0)
        {
          uint64_t delay = rand() % (10 * NSEC_PER_MSEC);

          /* Simulate the periodical hrtimer.. */

          flags = spin_lock_irqsave(&lock);

          /* Use as periodical timer */

          ret = hrtimer_cancel(&timer);
          ret = hrtimer_start(&timer, 1000, HRTIMER_MODE_REL);

          spin_unlock_irqrestore(&lock, flags);

          up_ndelay(NSEC_PER_MSEC);

          flags = spin_lock_irqsave(&lock);

          ret = hrtimer_cancel_sync(&timer);
          ret = hrtimer_start(&timer, 1000, HRTIMER_MODE_REL);
          spin_unlock_irqrestore(&lock, flags);

          up_ndelay(NSEC_PER_MSEC);

          hrtimer_cancel_sync(&timer); // stucked here????
          printf("???\n");
        }
      else
        {
          /* Simulate the background hrtimer.. */

          uint64_t delay = rand() % (10 * NSEC_PER_MSEC);

          ret = hrtimer_cancel(&timer);
          ret = hrtimer_start(&timer, delay, HRTIMER_MODE_REL);
        }

      UNUSED(ret);
    }
}

static void* test_thread(void *arg)
{
  while (1)
    {
      test1((int)arg);
    }
  return NULL;
}
/****************************************************************************
 * hello_main
 ****************************************************************************/

int main(int argc, FAR char *argv[])
{
  unsigned int   thread_id;
  pthread_attr_t attr;
  pthread_t      pthreads[HRTIMER_TEST_THREAD_NR];

  printf("hrtimer_test start...\n");

  ASSERT(pthread_attr_init(&attr) == 0);

  /* Create wdog test thread */

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      ASSERT(pthread_create(&pthreads[thread_id], &attr,
                            test_thread, (void *)thread_id) == 0);
    }

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      pthread_join(pthreads[thread_id], NULL);
    }

  ASSERT(pthread_attr_destroy(&attr) == 0);

  printf("hrtimer_test end...\n");
  return 0;
}

test passed log on rv-virt:smp64

nsh> uname -a
NuttX 0.0.0 6847a0cc95-dirty Dec 20 2025 12:26:39 risc-v rv-virt
nsh> 
nsh> hello
???
???
???
(...)

test 3 passed (provided by @Fix-Point )

test implementation

/****************************************************************************
 * apps/examples/hello/hello_main.c
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.  The
 * ASF licenses this file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance with the
 * License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
 * License for the specific language governing permissions and limitations
 * under the License.
 *
 ****************************************************************************/

/****************************************************************************
 * Included Files
 ****************************************************************************/

#include <nuttx/config.h>
#include <stdio.h>

#include <nuttx/hrtimer.h>

#define HRTIMER_TEST_THREAD_NR (CONFIG_SMP_NCPUS * 8)

/****************************************************************************
 * Public Functions
 ****************************************************************************/

static uint32_t test_callback(FAR struct hrtimer_s *hrtimer)
{
  printf("test callback ...\n");
  return 0;
}

static void* test_thread(void *arg)
{
  irqstate_t  flags;
  hrtimer_t   timer;
  spinlock_t  lock = SP_UNLOCKED;
  hrtimer_init(&timer, test_callback, NULL);
  while (1)
    {
      uint64_t delay = rand() % NSEC_PER_MSEC;
      int ret;

      /* Simulate the usage of driver->wait_dog. */

      flags = spin_lock_irqsave(&lock);

      /* The driver lock acquired */

      /* First try, failed. Because hrtimer_start can not ensure the timer being started. */

      ret = hrtimer_cancel(&timer);
      // ret = hrtimer_start(&timer, 10 * NSEC_PER_USEC, HRTIMER_MODE_REL); /* May fail */

      /* This try-loop start should be OK. But it failed again.
       * Besides, we can not sleep or spin in the critical sections.
       */

      while (hrtimer_start(&timer, 10 * NSEC_PER_USEC, HRTIMER_MODE_REL) != OK);
      ret = OK;

      /* Second try, Success, but we can not sleep or spin in the critical section. */

      // ret = hrtimer_cancel_sync(&timer); /* Sleep in critical sections */
      // ret = hrtimer_start(&timer, delay, HRTIMER_MODE_REL);


      spin_unlock_irqrestore(&lock, flags);

      if (ret != OK)
        {
          printf("hrtimer_start failed\n");
        }
      up_ndelay(delay);
    }
  return NULL;
}

/****************************************************************************
 * hello_main
 ****************************************************************************/

int main(int argc, FAR char *argv[])
{
  unsigned int   thread_id;
  pthread_attr_t attr;
  pthread_t      pthreads[HRTIMER_TEST_THREAD_NR];

  printf("hrtimer_test start...\n");

  ASSERT(pthread_attr_init(&attr) == 0);

  /* Create wdog test thread */

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      ASSERT(pthread_create(&pthreads[thread_id], &attr,
                            test_thread, NULL) == 0);
    }

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      pthread_join(pthreads[thread_id], NULL);
    }

  ASSERT(pthread_attr_destroy(&attr) == 0);

  printf("hrtimer_test end...\n");
  return 0;
}

test passed log on rv-virt:smp64

NuttShell (NSH)
nsh> 
nsh> 
nsh> uname -a
NuttX 0.0.0 6847a0cc95-dirty Dec 20 2025 12:33:59 risc-v rv-virt
nsh> hello
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...
test callback ...

(....)

test 4 passed (provided by @Fix-Point )

test implementation

#include <nuttx/config.h>
#include <stdio.h>

#include <nuttx/hrtimer.h>

#define HRTIMER_TEST(expr, value)                                   \
  do                                                                \
    {                                                               \
      ret = (expr);                                                 \
      if (ret != (value))                                           \
        {                                                           \
          printf("ERROR: HRTimer test failed, line=%d ret=%d\n",   \
                 __LINE__, ret);                                    \
        }                                                           \
    }                                                               \
  while (0)

#define HRTIMER_TEST_THREAD_NR (4)

/****************************************************************************
 * Public Functions
 ****************************************************************************/

static volatile int count = 0;
static hrtimer_t    timer;
static spinlock_t   lock = SP_UNLOCKED;

static uint32_t test_callback(FAR struct hrtimer_s *hrtimer)
{
  irqstate_t flags;
  flags = spin_lock_irqsave(&lock);
  up_ndelay(10 * NSEC_PER_USEC);
  count++;
  spin_unlock_irqrestore(&lock, flags);
  return 0;
}

static uint32_t test_callback_back(FAR struct hrtimer_s *hrtimer)
{
  return 0;
}

static void* test_thread(void *arg)
{
  irqstate_t  flags;
  int         tid   = (int)arg;
  int         ret;

  if (tid == 0)
    {
      hrtimer_init(&timer, test_callback, NULL);
    }

  while (1)
    {
      if (tid == 0)
        {
          int tmp_count;
          flags = spin_lock_irqsave(&lock);
          ret = hrtimer_cancel(&timer);
          ret = hrtimer_start(&timer, 0, HRTIMER_MODE_REL);
          spin_unlock_irqrestore(&lock, flags);

          up_ndelay(1000);

          spin_lock_irqsave(&lock);
          ret = hrtimer_cancel(&timer);
          ret = hrtimer_start(&timer, 0, HRTIMER_MODE_REL);
          spin_unlock_irqrestore(&lock, flags);

          ret = hrtimer_cancel_sync(&timer); // timer should be fully-cancelled here.
          tmp_count = count; // Read the count
          printf("tmp_count: %d\n",tmp_count);
          HRTIMER_TEST(tmp_count == count, true);
          ASSERT(tmp_count == count);
        }
      else
        {
          flags = up_irq_save();
          up_irq_restore(flags);
        }
    }

  UNUSED(ret);

  return NULL;
}

/****************************************************************************
 * hello_main
 ****************************************************************************/

int main(int argc, FAR char *argv[])
{
  unsigned int   thread_id;
  pthread_attr_t attr;
  pthread_t      pthreads[HRTIMER_TEST_THREAD_NR];

  printf("hrtimer_test start...\n");

  ASSERT(pthread_attr_init(&attr) == 0);

  /* Create wdog test thread */

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      ASSERT(pthread_create(&pthreads[thread_id], &attr,
                            test_thread, (void *)thread_id) == 0);
    }

  for (thread_id = 0; thread_id < HRTIMER_TEST_THREAD_NR; thread_id++)
    {
      pthread_join(pthreads[thread_id], NULL);
    }

  ASSERT(pthread_attr_destroy(&attr) == 0);

  printf("hrtimer_test end...\n");
  return 0;
}

test passed log on rv-virt:smp64

NuttShell (NSH)
nsh> 
nsh> 
nsh> uname -a
NuttX 0.0.0 b32da5db3a-dirty Dec 23 2025 09:31:26 risc-v rv-virt
nsh> 
nsh> hello
tmp_count: 7648
tmp_count: 7650
tmp_count: 7652
tmp_count: 7654
tmp_count: 7656
tmp_count: 7658
tmp_count: 7660
tmp_count: 7662
tmp_count: 7664
tmp_count: 7666
tmp_count: 7668
tmp_count: 7670
tmp_count: 7672
tmp_count: 7674
tmp_count: 7676
tmp_count: 7678
tmp_count: 7680
(....)

Allow running/armed hrtimer to be restarted to fix hrtimer bug: apache#17567 Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

Update the hrtimer documentation to describe the hrtimer state machine, which is introduced to handle safe cancellation and execution in SMP environments. Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

Improve hrtimer to only enable SMP related logic when in SMP mode to improve performance in non-SMP mode Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

wangchdo requested review from acassis, hartmannathan and xiaoxiang781216 as code owners December 23, 2025 00:56

github-actions bot added Area: Documentation Improvements or additions to documentation Area: OS Components OS Components issues Size: M The size of the change in this PR is medium labels Dec 23, 2025

wangchdo force-pushed the refine_hrtimer branch 3 times, most recently from 742782b to f475319 Compare December 23, 2025 01:28

wangchdo mentioned this pull request Dec 23, 2025

sched/wdog: Fix-up for SMP and Improvements. #17640

Merged

wangchdo force-pushed the refine_hrtimer branch from f475319 to 7c27ef8 Compare December 23, 2025 01:55

wangchdo requested review from anchao and masayuki2009 as code owners December 23, 2025 01:55

github-actions bot added the Board: risc-v label Dec 23, 2025

wangchdo force-pushed the refine_hrtimer branch from 7c27ef8 to 6ba659c Compare December 23, 2025 02:03

github-actions bot removed the Board: risc-v label Dec 23, 2025

wangchdo force-pushed the refine_hrtimer branch 3 times, most recently from 18b7ebc to 5923f82 Compare December 23, 2025 06:58

wangchdo added 2 commits December 23, 2025 15:00

sched/hrtimr: Allow running/armed hrtimer to be restarted

f19832b

Allow running/armed hrtimer to be restarted to fix hrtimer bug: apache#17567 Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

sched/hrtimer: document hrtimer state machine

a923a94

Update the hrtimer documentation to describe the hrtimer state machine, which is introduced to handle safe cancellation and execution in SMP environments. Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

wangchdo force-pushed the refine_hrtimer branch 4 times, most recently from b1ed47e to c91948a Compare December 23, 2025 10:10

github-actions bot added the Board: risc-v label Dec 23, 2025

wangchdo force-pushed the refine_hrtimer branch from c91948a to dc9a2b7 Compare December 23, 2025 10:24

github-actions bot removed the Board: risc-v label Dec 23, 2025

sched/hrtimer: Only enable SMP related logic when in SMP mode

bc161b5

Improve hrtimer to only enable SMP related logic when in SMP mode to improve performance in non-SMP mode Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>

wangchdo force-pushed the refine_hrtimer branch from dc9a2b7 to bc161b5 Compare December 23, 2025 12:44

acassis approved these changes Dec 23, 2025

View reviewed changes

wangchdo mentioned this pull request Dec 25, 2025

sched: Refactor the functional incorrect HRTimer implementation. #17675

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sched/hrtimer: hrtimer state machine improvement for SMP cases #17642

sched/hrtimer: hrtimer state machine improvement for SMP cases #17642

wangchdo commented Dec 23, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sched/hrtimer: hrtimer state machine improvement for SMP cases #17642

Are you sure you want to change the base?

sched/hrtimer: hrtimer state machine improvement for SMP cases #17642

Conversation

wangchdo commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Impact

Testing

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wangchdo commented Dec 23, 2025 •

edited

Loading