Skip to content

Conversation

@techraed
Copy link
Member

@techraed techraed commented Aug 5, 2025

This PR introduces:

  • CI job which runs benchmarks in case run-benchmarks label is set in a PR and specific paths were changed (rs/, benchmarks/, examples/, Cargo.*, and the job config). Also in case the PR is pushed to master, the bench will run.
  • bench-analyzer scripts which reads 2 files with bench data and prints a table with bench statistics. Optionally returned table can be set to the file. Also in case threshold is introduced, if any bench doesn't pass threshold (i.e. current bench differs from the other by threshold value) the script ends with an error

@techraed techraed requested a review from Copilot August 5, 2025 16:46
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a CI workflow for running benchmarks on Rust code changes. The workflow compares benchmark results between the PR branch and the master branch to detect performance regressions.

  • Adds a GitHub Actions workflow that triggers on labeled PRs and pushes to master
  • Introduces a new binary utility for converting benchmark data formats
  • Configures the workflow to compare performance and comment on PRs with results

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File Description
benchmarks/Cargo.toml Adds a new binary target for benchmark data conversion
.github/workflows/bench.yml Implements the complete benchmark CI workflow with comparison logic

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

Test comment from GitHub Actions

@techraed techraed added the run-benchmarks Enables running benchmarks in CI label Aug 6, 2025
@techraed techraed removed A0-pleasereview run-benchmarks Enables running benchmarks in CI labels Aug 6, 2025
@techraed
Copy link
Member Author

github benchamarks action doesn't work with custom benches

adjusting solution to do not working steps on our own

@techraed techraed added run-benchmarks Enables running benchmarks in CI and removed run-benchmarks Enables running benchmarks in CI labels Aug 14, 2025
@techraed techraed removed the run-benchmarks Enables running benchmarks in CI label Aug 14, 2025
@techraed techraed added the run-benchmarks Enables running benchmarks in CI label Aug 14, 2025
@github-actions
Copy link
Contributor

🔬 Benchmark Comparison

Benchmark Current Baseline Change Change % Status
compute 450_513_681_369 450_513_691_329 -9_960 -0.00%
alloc-0 563_779_139 563_795_739 -16_600 -0.00%
alloc-12 567_285_731 567_302_331 -16_600 -0.00%
alloc-143 726_654_326 726_670_926 -16_600 -0.00%
alloc-986 827_640_093 827_656_693 -16_600 -0.00%
alloc-10945 2_018_404_619 2_018_421_219 -16_600 -0.00%
alloc-46367 6_403_666_102 6_403_682_702 -16_600 -0.00%
alloc-121392 16_858_357_860 16_858_374_460 -16_600 -0.00%
alloc-317810 43_146_677_197 43_146_693_797 -16_600 -0.00%
counter_sync 677_720_794 677_860_234 -139_440 -0.02%
counter_async 850_342_667 850_482_107 -139_440 -0.02%
cross_program 2_511_201_218 2_511_573_058 -371_840 -0.01%
redirect 3_612_511_453 3_612_969_613 -458_160 -0.01%

Legend

  • 🚀 Significant improvement (>5% reduction)
  • 👍 Minor improvement (<5% reduction)
  • ✅ No significant change
  • ⚠️ Minor regression (<5% increase)
  • ❌ Significant regression (>5% increase)

🤖 This comment was automatically generated by the benchmark comparison workflow at dc4fbac.

1 similar comment
@github-actions
Copy link
Contributor

🔬 Benchmark Comparison

Benchmark Current Baseline Change Change % Status
compute 450_513_681_369 450_513_691_329 -9_960 -0.00%
alloc-0 563_779_139 563_795_739 -16_600 -0.00%
alloc-12 567_285_731 567_302_331 -16_600 -0.00%
alloc-143 726_654_326 726_670_926 -16_600 -0.00%
alloc-986 827_640_093 827_656_693 -16_600 -0.00%
alloc-10945 2_018_404_619 2_018_421_219 -16_600 -0.00%
alloc-46367 6_403_666_102 6_403_682_702 -16_600 -0.00%
alloc-121392 16_858_357_860 16_858_374_460 -16_600 -0.00%
alloc-317810 43_146_677_197 43_146_693_797 -16_600 -0.00%
counter_sync 677_720_794 677_860_234 -139_440 -0.02%
counter_async 850_342_667 850_482_107 -139_440 -0.02%
cross_program 2_511_201_218 2_511_573_058 -371_840 -0.01%
redirect 3_612_511_453 3_612_969_613 -458_160 -0.01%

Legend

  • 🚀 Significant improvement (>5% reduction)
  • 👍 Minor improvement (<5% reduction)
  • ✅ No significant change
  • ⚠️ Minor regression (<5% increase)
  • ❌ Significant regression (>5% increase)

🤖 This comment was automatically generated by the benchmark comparison workflow at dc4fbac.

@techraed techraed requested a review from vobradovich August 14, 2025 12:28
@github-actions
Copy link
Contributor

🔬 Benchmark Comparison

Benchmark Current Baseline Change Change % Status
compute 450_513_681_369 450_513_691_329 -9_960 -0.00%
alloc-0 563_779_139 563_795_739 -16_600 -0.00%
alloc-12 567_285_731 567_302_331 -16_600 -0.00%
alloc-143 726_654_326 726_670_926 -16_600 -0.00%
alloc-986 827_640_093 827_656_693 -16_600 -0.00%
alloc-10945 2_018_404_619 2_018_421_219 -16_600 -0.00%
alloc-46367 6_403_666_102 6_403_682_702 -16_600 -0.00%
alloc-121392 16_858_357_860 16_858_374_460 -16_600 -0.00%
alloc-317810 43_146_677_197 43_146_693_797 -16_600 -0.00%
counter_sync 677_720_794 677_860_234 -139_440 -0.02%
counter_async 850_342_667 850_482_107 -139_440 -0.02%
cross_program 2_511_201_218 2_511_573_058 -371_840 -0.01%
redirect 3_612_511_453 3_612_969_613 -458_160 -0.01%

Legend

  • 🚀 Significant improvement (>5% reduction)
  • 👍 Minor improvement (<5% reduction)
  • ✅ No significant change
  • ⚠️ Minor regression (<5% increase)
  • ❌ Significant regression (>5% increase)

🤖 This comment was automatically generated by the benchmark comparison workflow at 6eb6ebc.

@github-actions
Copy link
Contributor

🔬 Benchmark Comparison

Benchmark Current Baseline Change Change % Status
compute 450_513_678_049 450_513_691_329 -13_280 -0.00%
alloc-0 563_779_139 563_795_739 -16_600 -0.00%
alloc-12 567_285_731 567_302_331 -16_600 -0.00%
alloc-143 726_654_326 726_670_926 -16_600 -0.00%
alloc-986 827_640_093 827_656_693 -16_600 -0.00%
alloc-10945 2_018_404_619 2_018_421_219 -16_600 -0.00%
alloc-46367 6_403_666_102 6_403_682_702 -16_600 -0.00%
alloc-121392 16_858_357_860 16_858_374_460 -16_600 -0.00%
alloc-317810 43_146_677_197 43_146_693_797 -16_600 -0.00%
counter_sync 677_720_794 677_860_234 -139_440 -0.02%
counter_async 850_342_667 850_482_107 -139_440 -0.02%
cross_program 2_511_201_218 2_511_573_058 -371_840 -0.01%
redirect 3_612_521_413 3_612_969_613 -448_200 -0.01%

Legend

  • 🚀 Significant improvement (>5% reduction)
  • 👍 Minor improvement (<5% reduction)
  • ✅ No significant change
  • ⚠️ Minor regression (<5% increase)
  • ❌ Significant regression (>5% increase)

🤖 This comment was automatically generated by the benchmark comparison workflow at 2be57a7.

1 similar comment
@github-actions
Copy link
Contributor

🔬 Benchmark Comparison

Benchmark Current Baseline Change Change % Status
compute 450_513_678_049 450_513_691_329 -13_280 -0.00%
alloc-0 563_779_139 563_795_739 -16_600 -0.00%
alloc-12 567_285_731 567_302_331 -16_600 -0.00%
alloc-143 726_654_326 726_670_926 -16_600 -0.00%
alloc-986 827_640_093 827_656_693 -16_600 -0.00%
alloc-10945 2_018_404_619 2_018_421_219 -16_600 -0.00%
alloc-46367 6_403_666_102 6_403_682_702 -16_600 -0.00%
alloc-121392 16_858_357_860 16_858_374_460 -16_600 -0.00%
alloc-317810 43_146_677_197 43_146_693_797 -16_600 -0.00%
counter_sync 677_720_794 677_860_234 -139_440 -0.02%
counter_async 850_342_667 850_482_107 -139_440 -0.02%
cross_program 2_511_201_218 2_511_573_058 -371_840 -0.01%
redirect 3_612_521_413 3_612_969_613 -448_200 -0.01%

Legend

  • 🚀 Significant improvement (>5% reduction)
  • 👍 Minor improvement (<5% reduction)
  • ✅ No significant change
  • ⚠️ Minor regression (<5% increase)
  • ❌ Significant regression (>5% increase)

🤖 This comment was automatically generated by the benchmark comparison workflow at 2be57a7.

@techraed techraed merged commit abe0f99 into master Aug 18, 2025
3 checks passed
@techraed techraed deleted the st-benchmarks-ci branch August 18, 2025 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A2-mergeoncegreen run-benchmarks Enables running benchmarks in CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants