Skip to content

Expanded tags and packets to accept any Python data type and use Arrow-based datastore#22

Merged
eywalker merged 28 commits intowalkerlab:mainfrom
eywalker:expanded_packet
Jun 19, 2025
Merged

Expanded tags and packets to accept any Python data type and use Arrow-based datastore#22
eywalker merged 28 commits intowalkerlab:mainfrom
eywalker:expanded_packet

Conversation

@eywalker
Copy link
Contributor

  • Major refactor of pod logic to accept tags and packets with arbitrary Python data type
  • New set of data stores (ArrowDataStore) that accepts arrow tables to be stored, being aware of "semantic_type" metadata to handle special fields such as Path and UUID using registered handlers
  • New hasher for Arrow table that is "semantic_type" aware
  • New logic of "saving" computation results and associated tags into result_store and tag_store using CachedFunctionPod

eywalker added 28 commits June 8, 2025 01:02
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@eywalker eywalker requested a review from Copilot June 18, 2025 21:12
@eywalker eywalker requested a review from brian-arnold June 18, 2025 21:12
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors multiple components to enable support for arbitrary Python data types and integrates a new Arrow‐based datastore. Key changes include renaming of parameters (e.g. replacing “store_name” with “function_name” and “content_hash” with “function_hash”), updating hasher factory methods to use the new API, and modifying various import paths and error messages to reflect the architectural changes.

Reviewed Changes

Copilot reviewed 71 out of 83 changed files in this pull request and generated no comments.

File Description
tests/* Updates to test cases with revised parameter names and updated import paths
src/orcabridge/store/* Refactor of DirDataStore and SafeDirDataStore to use new function-based naming conventions
src/orcabridge/hashing/* Adjustments to hasher implementations and defaults to align with new API, including renaming in factory methods
notebooks/* Notebook examples updated with new import paths and parameter names

@eywalker eywalker merged commit 3e012f9 into walkerlab:main Jun 19, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants