Feature: support rectilinear chunk grid extension #3534
Conversation
github-actions
bot
added
the
needs release notes
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3534 +/- ##
==========================================
+ Coverage 60.90% 61.60% +0.70%
==========================================
Files 86 86
Lines 10174 10640 +466
==========================================
+ Hits 6196 6555 +359
- Misses 3978 4085 +107
🚀 New features to boost your workflow:
|
…ature/rectilinear-chunk-grid
github-actions
bot
removed
the
needs release notes
| With variable chunking, the standard `.chunks` property is not available since chunks | ||
| have different sizes. Instead, access chunk information through the chunk grid: |
There was a problem hiding this comment.
I think it would be better if .chunks just had a different type (tuple of tuples of ints)
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class RectilinearChunkGrid(ChunkGrid): |
There was a problem hiding this comment.
thoughts on just calling this class Rectilinear, and renaming the RegularChunkGrid to Regular? We could keep around a RegularChunkGrid class for compatibility. But I feel like people know these are chunk grids when they import them
There was a problem hiding this comment.
50/50. I think the more descriptive class is useful when looking at a tracebacks. Plus, this is currently in .core so its not meant to be used directly by users.
| ) | ||
|
|
||
| @cached_property | ||
| def _cumulative_sizes(self) -> tuple[tuple[int, ...], ...]: |
src/zarr/testing/strategies.py
Outdated
| chunk_shapes_rle = [ | ||
| [[c, r] for c, r in zip(draw(dim_chunks), draw(repeats), strict=True)] | ||
| for _ in range(ndim) | ||
| ] | ||
| return RectilinearChunkGrid(chunk_shapes=chunk_shapes_rle) |
| @given(data=st.data()) | ||
| async def test_basic_indexing(data: st.DataObject) -> None: | ||
| zarray = data.draw(simple_arrays()) | ||
| @given(data=st.data(), zarray=st.one_of([simple_arrays(), complex_chunked_arrays()])) |
There was a problem hiding this comment.
Because the search space for the standard arrays strategy is so large, i made a different one complex_chunked_arrays that purely checks different chunk grids
with simple_arrays() we are only spending 10% of our time trying RectilinearChunkGrid so using this approach. We should boost number of examples too.
| 2. **Not compatible with sharding**: You cannot use variable chunking together with | ||
| the sharding feature. Arrays must use either variable chunking or sharding, but not both. |
There was a problem hiding this comment.
I hope this is a temporary limitation! There's a natural extension of rectilinear chunk grids to rectilinear shard grids.
There was a problem hiding this comment.
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…arr-python into feature/rectilinear-chunk-grid
…ature/rectilinear-chunk-grid
…ature/rectilinear-chunk-grid
4 participants
(Marking this as a draft for now)
closes: #1595
replaces: #1483
xref: zarr-developers/zarr-extensions#25
TODO:
docs/user-guide/*.mdchanges/