Skip to content

Conversation

@dcookspi
Copy link
Collaborator

@dcookspi dcookspi commented Nov 12, 2025

This updates spk's FileMatcher objects to lazily create its internal gitignore object on first call to the matches() method.

The FileMatcher and its gitignore field was accounting for about a quarter of the load/read time for a package build spec. The matching is only carried out when building a package, so setting it up lazily saves time in other parts of spk (e.g. a solve that reads 15k package build specs).

Todo:

  • Add check for empty rules before creating a gitignore
  • Include result to avoid repeated recreation failures
  • Profile vs ArcSwap

@dcookspi dcookspi requested review from jrray and rydrman November 12, 2025 19:40
@dcookspi dcookspi self-assigned this Nov 12, 2025
@dcookspi dcookspi added SPI AOI Area of interest for SPI enhancement New feature or request labels Nov 12, 2025
@codecov
Copy link

codecov bot commented Nov 12, 2025

Codecov Report

❌ Patch coverage is 63.63636% with 12 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ema/crates/foundation/src/spec_ops/file_matcher.rs 62.50% 12 Missing ⚠️

📢 Thoughts on this report? Let us know!

pub struct FileMatcher {
rules: Vec<String>,
gitignore: ignore::gitignore::Gitignore,
gitignore: Arc<Mutex<Option<ignore::gitignore::Gitignore>>>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
gitignore: Arc<Mutex<Option<ignore::gitignore::Gitignore>>>,
gitignore: Arc<Mutex<Option<Result<ignore::gitignore::Gitignore>>>>,

I suggest making it distinguishable between an uninitialized value and a value that failed to initialize, to avoid repeatedly trying to initialize something that will fail.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to include this and store the fail error.

pub struct FileMatcher {
rules: Vec<String>,
gitignore: ignore::gitignore::Gitignore,
gitignore: Arc<Mutex<Option<ignore::gitignore::Gitignore>>>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Profile first, but ArcSwap may be a cheaper way to manage this field. Instead of creating numerous Mutex instances. I'm more concerned about this significantly slowing down builds for packages with a lot of files and calling matches on each file.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a OnceCell to leverage get_or_try_init? Maybe that's been tried, but leaving the note anyway

/// The first call to this will construct an internal gitignore
/// matching object and cache it for subsequent calls.
pub fn matches<P: AsRef<std::path::Path>>(&self, path: P, is_dir: bool) -> Result<bool> {
let mut matcher = self
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before acquiring the lock you could check if self.rules is empty, if so then this could early exit. I'm assuming an empty Gitignore returns false for every input. This may alleviate some of the concerns about the Mutex overhead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check for a FileMatcher with no rules.

… first call to matches() method.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
…hing lazy initialisation.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
…dly trying to initialise it when that will fail

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi dcookspi force-pushed the update-FileMatcher-to-lazy-init-gitignore-object branch from 1c26c2e to f01b410 Compare December 18, 2025 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request SPI AOI Area of interest for SPI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants