diff --git a/.dockerignore b/.dockerignore index 71c01775..2270f496 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,14 +1,3 @@ -<<<<<<< HEAD -**/*.pyc -**/*.pyo -**/*.log -.git/ -.gitignore -Dockerfile -docker-compose.yml -.env -docs/_build -======= # Docker related development/Dockerfile development/docker-compose*.yml @@ -36,4 +25,3 @@ LICENSE **/.vscode/ invoke*.yml tasks.py ->>>>>>> c5f3eb1 (Cookie initialy baked by NetworkToCode Cookie Drift Manager Tool) diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index f735817d..00000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,212 +0,0 @@ -# Changelog - -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - -## [2.1.0] - -### Changed - -- [284](https://github.com/networktocode/diffsync/pull/284) - Update pyyaml to 6.0.2 in poetry.lock by @gsnider2195 -- [286](https://github.com/networktocode/diffsync/pull/286) - Update GitHub Actions runner version by @mjbear -- [289](https://github.com/networktocode/diffsync/pull/289) - Remove upper limit on Packaging by @jdrew82 - -## New Contributors - -- @gsnider2195 made their first contribution in https://github.com/networktocode/diffsync/pull/284 -- @mjbear made their first contribution in https://github.com/networktocode/diffsync/pull/286 -- @jdrew82 made their first contribution in https://github.com/networktocode/diffsync/pull/289 - -## [2.0.1] - -### Changed - -- #276 - Removed upper version bound for `structlog` dependency - -### Fixed - -- #281 - Properly deprecated `DiffSync` class name -- #273 - Properly capitalized `DiffSync` in documentation -- #273 - Removed more mentions of `DiffSync` in favor of `Adapter` -- #274 - Fixed doc section title for getting started -- #269 - Fixed wording for a couple of docstrings -- #265 - Fixed readthedocs build - -## [2.0.0] - -### Changed - -- **BREAKING CHANGE** #236/240 - Upgrade to Pydantic v2. - -## [1.10.0] - 2023-11-16 - -### Fixed - -- #249 - Fixes natural deletion order flag -- #247 - Fixes underspecified typing_extensions dependency - -### Changed - -- #247 - Deprecates Python 3.7 - -## [1.9.0] - 2023-10-16 - -### Added - -- #220 - Implement DiffSyncModelFlags.NATURAL_DELETION_ORDER. - -### Changed - -- #219 - Type hinting overhaul - -## [1.8.0] - 2023-04-18 - -### Added - -- #182 - Added `get_or_add_model_instance()` and `update_or_add_model_instance()` APIs. -- #189 - Added note in `README.md` about running `invoke tests`. -- #190 - Added note in `README.md` about running `invoke build`. - -### Changed - -- #77/#188 - `sync_from()` and `sync_to()` now return the `Diff` that was applied. -- #211 - Loosened `packaging` and `structlog` library dependency constraints for broader compatibility. - -## [1.7.0] - 2022-11-03 - -### Changed - -- #176 - Remove pytest-redislite in favor of pytest-redis. -- #174 - Update Dockerfile to install build-essential - -### Added - -- #174 - Add methods to load data from dictionary and enable tree traversal -- #174 - Add a `get_or_none` method to the DiffSync class -- #168 - Add 'skip' counter to diff.summary() -- #169/#170 - Add documentation about model processing order -- #121/#140 - Add and configure renovate -- #140 - Add renovate configuration validation to the CI - -### Fixed - -- #149 - Limit redundant CI concurrency - -## [1.6.0] - 2022-07-09 - -### Changed - -- #120 - Dropped support for Python 3.6, new minimum is Python 3.7 - -## [1.5.1] - 2022-06-30 - -### Added - -- #111 - Added example 6, regarding IP prefixes. - -### Changed - -- #107 - Updated example 5 to use the Redis backend store. - -### Fixed - -- #115 - Fixed ReadTheDocs rendering pipeline -- #118 - Fixed a regression in `DiffSync.get(modelname, identifiers)` introduced in 1.5.0 - -## [1.5.0] - 2022-06-07 - -### Added - -- #106 - Add a new, optional, backend store based in Redis - -## [1.4.3] - 2022-03-03 - -### Fixed - -- #101 - Revert changed introduced in #90 that affected `DiffElement.action` - -**NOTE**: this change is a breaking change against DiffSync 1.4.0 through 1.4.2, but was necessary to restore backward compatibility with DiffSync 1.3.x and earlier. Apologies for any inconvenience this causes. - -### Changed - -- #103 - Update development dependencies - -## [1.4.2] - 2022-02-28 - -**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. - -### Fixed - -- #100 - Added explicit dependency on `packaging`. - -## [1.4.1] - 2022-01-26 - -**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. - -### Fixed - -- #95 - Removed optional dependencies on `sphinx`, `m2r2`, `sphinx-rtd-theme`, `toml`. - -## [1.4.0] - 2022-01-24 - -**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. - -### Added - -- #53 - Add a new example based on pynautobot and Nautobot REST API -- #59 - Add proper documentation published in Read the doc -- #68 - Cleanup Readme, add link to new documentation site -- #70 - Add `add_or_update()` method to DiffSync class that requires a DiffSyncModel to be passed in and will attempt to add or update an existing object -- #72 - Add core engine section in docs and rename example directories -- #75 - Add support for Structlog v21 in addition to v20. -- #80 - Add support for an existing Diff object to be passed to `sync_to()` & `sync_from()` to prevent another diff from being calculated. -- #81 - Add a new example based on PeeringDB -- #83 - Add support for Python 3.10 -- #87 - Add new model flags : `SKIP_UNMATCHED_BOTH`, `SKIP_UNMATCHED_SRC` & `SKIP_UNMATCHED_DST` to match the behavior of the global flags - -### Changed - -- #62 - Update CI Token -- #69 - Replace Travis CI with Github Actions to run unit tests -- #82 - Update lock file with latest versions. -- #90 - Convert list of actions (`create`, `update`, `delete`) to proper Enum - -### Fixed - -- #51 - Update minimum Pydantic version due to security advisory GHSA-5jqp-qgf6-3pvh -- #63 - Fix type in Readme - -## [1.3.0] - 2021-04-07 - -### Added - -- #48 - added optional `callback` argument to `diff_from`/`diff_to`/`sync_from`/`sync_to` for use with progress reporting. - -## [1.2.0] - 2020-12-08 - -### Added - -- #45 - minimum Python version lowered from 3.7 to 3.6, also now tested against Python 3.9. - -## [1.1.0] - 2020-12-01 - -### Added - -- #37 - added `sync_complete` callback, triggered on `sync_from` completion with changes. -- #41 - added `summary` API for Diff and DiffElement objects. -- #44 - added `set_status()` and `get_status()` APIs so that DiffSyncModel implementations can provide details for create/update/delete logging - -### Changed - -- Now requires Pydantic 1.7.2 or later -- #34 - in diff dicts, changed keys `src`/`dst`/`_src`/`_dst` to `-` and `+` -- #43 - `DiffSync.get_by_uids()` now raises `ObjectNotFound` if any of the provided uids cannot be located; `DiffSync.get()` raises `ObjectNotFound` or `ValueError` on failure, instead of returning `None`. - -### Fixed - -- #44 - On CRUD failure, do not generate an extraneous "success" log message in addition to the "failed" message - -## [1.0.0] - 2020-10-23 - -Initial release diff --git a/REVIEW_RECOMMENDATIONS.md b/REVIEW_RECOMMENDATIONS.md deleted file mode 100644 index a2a6a522..00000000 --- a/REVIEW_RECOMMENDATIONS.md +++ /dev/null @@ -1,633 +0,0 @@ -# DiffSync Project Review - Performance & Capability Recommendations - -## Executive Summary - -This document outlines opportunities for performance improvements and new capabilities in the DiffSync library. The review identified several optimization opportunities and feature enhancements that would significantly improve the library's efficiency and usability. - ---- - -## Performance Optimizations - -### 1. Batch Operations in Store Implementations - -**Current Issue:** -- `get_by_uids()` in both `LocalStore` and `RedisStore` performs individual lookups in a loop (N+1 query problem) -- This is inefficient for large datasets - -**Recommendation:** -```python -# In LocalStore.get_by_uids() -def get_by_uids(self, *, uids: List[str], model: Union[str, "DiffSyncModel", Type["DiffSyncModel"]]) -> List["DiffSyncModel"]: - if isinstance(model, str): - modelname = model - else: - modelname = model.get_type() - - # Batch lookup - single dict access per model - model_dict = self._data[modelname] - results = [] - missing = [] - for uid in uids: - if uid in model_dict: - results.append(model_dict[uid]) - else: - missing.append(uid) - - if missing: - raise ObjectNotFound(f"{modelname} {', '.join(missing)} not present in {str(self)}") - return results -``` - -**For RedisStore:** -- Use `mget()` for batch retrieval instead of individual `get()` calls -- Consider using Redis pipelines for multiple operations - -**Impact:** O(N) → O(1) for batch lookups, significant improvement for large datasets - ---- - -### 2. Optimize RedisStore.get_all_model_names() - -**Current Issue:** -- Uses `scan_iter()` which iterates through all keys -- Comment in code: `# TODO: optimize it` - -**Recommendation:** -- Use Redis Sets to track model names -- Maintain a set key like `{store_label}:_models` that gets updated on add/remove -- This reduces complexity from O(N) scan to O(1) set retrieval - -**Implementation:** -```python -def get_all_model_names(self) -> Set[str]: - """Get all the model names stored.""" - model_key = f"{self._store_label}:_models" - models = self._store.smembers(model_key) - return {m.decode() for m in models} if models else set() - -def add(self, *, obj: "DiffSyncModel") -> None: - # ... existing add logic ... - # After successful add: - model_key = f"{self._store_label}:_models" - self._store.sadd(model_key, modelname) -``` - -**Impact:** Eliminates full key scan, improves performance for stores with many objects - ---- - -### 3. Optimize diff_object_list() Dictionary Conversion - -**Current Issue:** -- Converts lists to dictionaries on every call, even when lists are already small -- Creates unnecessary intermediate data structures - -**Recommendation:** -- Cache dictionary conversions when possible -- Use sets for faster intersection operations -- Consider lazy evaluation for large datasets - -**Implementation:** -```python -def diff_object_list(self, src: List["DiffSyncModel"], dst: List["DiffSyncModel"]) -> List[DiffElement]: - diff_elements = [] - - # Use sets for faster lookups if lists are large - if len(src) > 100 or len(dst) > 100: - dict_src = {item.get_unique_id(): item for item in src} - dict_dst = {item.get_unique_id(): item for item in dst} - combined_uids = set(dict_src.keys()) | set(dict_dst.keys()) - combined_dict = {uid: (dict_src.get(uid), dict_dst.get(uid)) for uid in combined_uids} - else: - # For small lists, current approach is fine - dict_src = {item.get_unique_id(): item for item in src} if not isinstance(src, ABCMapping) else src - dict_dst = {item.get_unique_id(): item for item in dst} if not isinstance(dst, ABCMapping) else dst - combined_dict = {} - for uid in dict_src: - combined_dict[uid] = (dict_src.get(uid), dict_dst.get(uid)) - for uid in dict_dst: - combined_dict[uid] = (dict_src.get(uid), dict_dst.get(uid)) - - # ... rest of method -``` - -**Impact:** Reduces memory allocations and improves performance for large datasets - ---- - -### 4. Add Caching Layer for Frequently Accessed Models - -**Recommendation:** -- Implement an optional LRU cache for model lookups -- Cache can be enabled via adapter configuration -- Useful for repeated diff/sync operations on same datasets - -**Implementation:** -```python -from functools import lru_cache -from typing import Optional - -class Adapter: - def __init__( - self, - name: Optional[str] = None, - internal_storage_engine: Union[Type[BaseStore], BaseStore] = LocalStore, - enable_cache: bool = False, - cache_size: int = 128, - ) -> None: - # ... existing init ... - self._cache_enabled = enable_cache - if enable_cache: - self._get_cached = lru_cache(maxsize=cache_size)(self._get_uncached) - else: - self._get_cached = self._get_uncached - - def _get_uncached(self, model, identifier): - return self.store.get(model=model, identifier=identifier) - - def get(self, obj, identifier): - if self._cache_enabled: - return self._get_cached(obj, identifier) - return self.store.get(model=obj, identifier=identifier) -``` - -**Impact:** Significant speedup for repeated operations on same data - ---- - -### 5. Parallel/Async Diff Calculation - -**Recommendation:** -- Add optional parallel processing for independent model types -- Use `concurrent.futures` or `asyncio` for I/O-bound operations -- Process top-level models in parallel when they have no dependencies - -**Implementation:** -```python -from concurrent.futures import ThreadPoolExecutor, as_completed -from typing import List - -def calculate_diffs_parallel(self, max_workers: int = 4) -> Diff: - """Calculate diffs with parallel processing for independent model types.""" - self.diff = self.diff_class() - - # Process independent model types in parallel - with ThreadPoolExecutor(max_workers=max_workers) as executor: - futures = {} - for obj_type in intersection(self.dst_diffsync.top_level, self.src_diffsync.top_level): - future = executor.submit( - self.diff_object_list, - src=self.src_diffsync.get_all(obj_type), - dst=self.dst_diffsync.get_all(obj_type), - ) - futures[future] = obj_type - - for future in as_completed(futures): - diff_elements = future.result() - for diff_element in diff_elements: - self.diff.add(diff_element) - - return self.diff -``` - -**Impact:** Significant speedup for large datasets with multiple independent model types - ---- - -## New Capabilities - -### 1. Incremental Diff Calculation - -**Feature:** -- Only calculate diffs for models that have changed since last sync -- Track last sync timestamp/metadata per model -- Useful for large datasets where only small portions change - -**Implementation:** -```python -class IncrementalDiff(Diff): - """Diff that tracks changes since last sync.""" - - def __init__(self, last_sync_metadata: Optional[Dict] = None): - super().__init__() - self.last_sync_metadata = last_sync_metadata or {} - - def should_diff_model(self, model: "DiffSyncModel", last_sync_time: Optional[datetime]) -> bool: - """Determine if model needs diffing based on last sync time.""" - if not last_sync_time: - return True - # Compare model's last modified time with last sync time - # This requires models to track modification time - return True # Placeholder -``` - -**Impact:** Dramatically reduces diff calculation time for large, mostly-static datasets - ---- - -### 2. Diff Filtering and Querying - -**Feature:** -- Filter diffs by model type, action type, or custom predicates -- Query diffs to find specific changes -- Export filtered diffs - -**Implementation:** -```python -class Diff: - def filter(self, predicate: Callable[[DiffElement], bool]) -> "Diff": - """Return a new Diff containing only elements matching the predicate.""" - filtered = Diff() - for element in self.get_children(): - if predicate(element): - filtered.add(element) - return filtered - - def filter_by_action(self, action: str) -> "Diff": - """Filter diffs by action type (create, update, delete).""" - return self.filter(lambda e: e.action == action) - - def filter_by_type(self, model_type: str) -> "Diff": - """Filter diffs by model type.""" - return self.filter(lambda e: e.type == model_type) - - def query(self, **criteria) -> List[DiffElement]: - """Query diffs using criteria like type, action, keys, etc.""" - results = [] - for element in self.get_children(): - match = True - if 'type' in criteria and element.type != criteria['type']: - match = False - if 'action' in criteria and element.action != criteria['action']: - match = False - if 'keys' in criteria: - for key, value in criteria['keys'].items(): - if element.keys.get(key) != value: - match = False - break - if match: - results.append(element) - return results -``` - -**Impact:** Enables more sophisticated diff analysis and selective syncing - ---- - -### 3. Diff Export/Import - -**Feature:** -- Export diffs to JSON/YAML for persistence or sharing -- Import diffs from files -- Useful for reviewing diffs before applying, or applying same diff to multiple targets - -**Implementation:** -```python -class Diff: - def to_dict(self) -> Dict: - """Export diff to dictionary (already exists as dict()).""" - return self.dict() - - def to_json(self, filepath: Optional[str] = None) -> str: - """Export diff to JSON string or file.""" - import json - data = self.dict() - json_str = json.dumps(data, indent=2, default=str) - if filepath: - with open(filepath, 'w') as f: - f.write(json_str) - return json_str - - @classmethod - def from_json(cls, json_str: str) -> "Diff": - """Import diff from JSON string.""" - import json - data = json.loads(json_str) - diff = cls() - # Reconstruct diff from data - # This would require additional implementation - return diff -``` - -**Impact:** Enables diff persistence, review workflows, and batch operations - ---- - -### 4. Transaction Support for Sync Operations - -**Feature:** -- Rollback capability if sync fails -- Atomic sync operations -- Checkpoint/resume for long-running syncs - -**Implementation:** -```python -class Adapter: - def sync_from_transactional( - self, - source: "Adapter", - rollback_on_failure: bool = True, - **kwargs - ) -> Tuple[Diff, bool]: - """Perform sync with transaction support.""" - checkpoint = self._create_checkpoint() - try: - diff = self.sync_from(source, **kwargs) - return diff, True - except Exception as e: - if rollback_on_failure: - self._restore_checkpoint(checkpoint) - raise - finally: - self._cleanup_checkpoint(checkpoint) - - def _create_checkpoint(self) -> Dict: - """Create a checkpoint of current state.""" - return { - 'models': self.dict(), - 'timestamp': datetime.now() - } - - def _restore_checkpoint(self, checkpoint: Dict) -> None: - """Restore state from checkpoint.""" - self.load_from_dict(checkpoint['models']) -``` - -**Impact:** Critical for production use where data integrity is paramount - ---- - -### 5. Validation Hooks - -**Feature:** -- Pre-sync validation callbacks -- Custom validation rules -- Validation errors prevent sync - -**Implementation:** -```python -class Adapter: - def __init__(self, *args, **kwargs): - # ... existing init ... - self._pre_sync_validators: List[Callable[[Diff], bool]] = [] - self._post_sync_validators: List[Callable[[Diff], bool]] = [] - - def add_pre_sync_validator(self, validator: Callable[[Diff], bool]) -> None: - """Add a validator that runs before sync.""" - self._pre_sync_validators.append(validator) - - def sync_from(self, source: "Adapter", **kwargs) -> Diff: - diff = self.diff_from(source, **kwargs) - - # Run pre-sync validators - for validator in self._pre_sync_validators: - if not validator(diff): - raise ValidationError("Pre-sync validation failed") - - # ... existing sync logic ... - - return diff -``` - -**Impact:** Enables data quality checks and prevents invalid syncs - ---- - -### 6. Diff Statistics and Analytics - -**Feature:** -- Detailed statistics about diffs -- Change impact analysis -- Performance metrics - -**Implementation:** -```python -class Diff: - def get_statistics(self) -> Dict: - """Get detailed statistics about the diff.""" - stats = { - 'total_elements': len(self), - 'by_action': {}, - 'by_type': {}, - 'largest_changes': [], - 'affected_models': set(), - } - - for element in self.get_children(): - # Count by action - action = element.action or 'no-change' - stats['by_action'][action] = stats['by_action'].get(action, 0) + 1 - - # Count by type - stats['by_type'][element.type] = stats['by_type'].get(element.type, 0) + 1 - - # Track affected models - stats['affected_models'].add(element.type) - - return stats - - def get_change_impact(self) -> Dict: - """Analyze the impact of changes.""" - # Determine which models will be affected by cascading changes - # This could be useful for understanding sync scope - pass -``` - -**Impact:** Better visibility into sync operations and change management - ---- - -### 7. Dry-Run Mode - -**Feature:** -- Preview what would change without actually applying changes -- Useful for testing and validation - -**Implementation:** -```python -class Adapter: - def sync_from_dry_run( - self, - source: "Adapter", - **kwargs - ) -> Tuple[Diff, Dict]: - """Perform a dry-run sync, returning what would change.""" - diff = self.diff_from(source, **kwargs) - - # Simulate sync without actually modifying data - changes = { - 'would_create': [], - 'would_update': [], - 'would_delete': [], - } - - for element in diff.get_children(): - if element.action == DiffSyncActions.CREATE: - changes['would_create'].append(element) - elif element.action == DiffSyncActions.UPDATE: - changes['would_update'].append(element) - elif element.action == DiffSyncActions.DELETE: - changes['would_delete'].append(element) - - return diff, changes -``` - -**Impact:** Essential for production environments where changes need review - ---- - -### 8. Conflict Resolution Strategies - -**Feature:** -- Handle conflicts when both source and destination have changed -- Configurable conflict resolution strategies (source wins, dest wins, merge, manual) - -**Implementation:** -```python -class ConflictResolutionStrategy(enum.Enum): - SOURCE_WINS = "source_wins" - DEST_WINS = "dest_wins" - MANUAL = "manual" - MERGE = "merge" - -class DiffElement: - def has_conflict(self) -> bool: - """Check if this element represents a conflict.""" - # A conflict exists when both source and dest have changed - # and the changes are different - return ( - self.source_attrs is not None - and self.dest_attrs is not None - and self.source_attrs != self.dest_attrs - and self.action == DiffSyncActions.UPDATE - ) - - def resolve_conflict(self, strategy: ConflictResolutionStrategy) -> Dict: - """Resolve conflict using specified strategy.""" - if strategy == ConflictResolutionStrategy.SOURCE_WINS: - return self.source_attrs - elif strategy == ConflictResolutionStrategy.DEST_WINS: - return self.dest_attrs - # ... other strategies -``` - -**Impact:** Handles real-world scenarios where both sides have changed - ---- - -## Code Quality Improvements - -### 1. Address TODOs - -**Current TODOs:** -- `diffsync/__init__.py:257` - Enforce that only attrs in `_attributes` can be updated -- `diffsync/helpers.py:153-154` - Validation checks for model consistency -- `diffsync/diff.py:219` - Implement `__eq__()` for Diff class -- `diffsync/diff.py:256,259` - Refactor `add_attrs()` method -- `diffsync/store/redis.py:79` - Optimize `get_all_model_names()` - -**Recommendation:** Prioritize and address these TODOs, especially the Redis optimization which has a direct performance impact. - ---- - -### 2. Add Type Hints for Better IDE Support - -**Recommendation:** -- Ensure all public methods have complete type hints -- Use `typing.Protocol` for callback types -- Add return type hints where missing - ---- - -### 3. Improve Error Messages - -**Recommendation:** -- Add more context to error messages (e.g., which model, which operation) -- Include suggestions for common errors -- Add error codes for programmatic error handling - ---- - -## Testing Recommendations - -### 1. Performance Benchmarks - -**Recommendation:** -- Add benchmark tests for large datasets (10K+, 100K+ models) -- Track performance regressions -- Compare performance of different store implementations - ---- - -### 2. Load Testing - -**Recommendation:** -- Test with very large datasets -- Test concurrent operations -- Test memory usage under load - ---- - -## Documentation Improvements - -### 1. Performance Tuning Guide - -**Recommendation:** -- Document performance characteristics of different stores -- Provide guidance on when to use caching -- Best practices for large datasets - ---- - -### 2. Advanced Usage Examples - -**Recommendation:** -- Examples for each new capability -- Real-world use cases -- Integration patterns - ---- - -## Priority Recommendations - -### High Priority (Immediate Impact) -1. ✅ Batch operations in `get_by_uids()` (LocalStore & RedisStore) -2. ✅ Optimize `RedisStore.get_all_model_names()` -3. ✅ Add dry-run mode -4. ✅ Add transaction support - -### Medium Priority (Significant Value) -5. ✅ Diff filtering and querying -6. ✅ Diff export/import -7. ✅ Validation hooks -8. ✅ Parallel diff calculation - -### Low Priority (Nice to Have) -9. ✅ Incremental diff calculation -10. ✅ Conflict resolution strategies -11. ✅ Diff statistics and analytics -12. ✅ Caching layer - ---- - -## Implementation Notes - -- All new features should be backward compatible -- Consider feature flags for experimental features -- Maintain comprehensive test coverage -- Update documentation for all new features -- Follow existing code style and patterns - ---- - -## Conclusion - -The DiffSync library is well-architected and functional. The recommended improvements focus on: -1. **Performance**: Batch operations, caching, parallel processing -2. **Reliability**: Transactions, validation, dry-run -3. **Usability**: Filtering, export/import, statistics -4. **Production Readiness**: Error handling, conflict resolution - -These improvements would make DiffSync more suitable for production use with large datasets and complex synchronization scenarios. - - diff --git a/changes/+main.housekeeping b/changes/+main.housekeeping deleted file mode 100644 index 3433adf6..00000000 --- a/changes/+main.housekeeping +++ /dev/null @@ -1 +0,0 @@ -Rebaked from the cookie `main`. diff --git a/docs/admin/release_notes/version_1.0.md b/docs/admin/release_notes/version_1.0.md index 00375d77..1cbcdfe2 100644 --- a/docs/admin/release_notes/version_1.0.md +++ b/docs/admin/release_notes/version_1.0.md @@ -1,40 +1,11 @@ # v1.0 Release Notes -!!! warning "Developer Note - Remove Me!" - Guiding Principles: - - - Changelogs are for humans, not machines. - - There should be an entry for every single version. - - The same types of changes should be grouped. - - Versions and sections should be linkable. - - The latest version comes first. - - The release date of each version is displayed. - - Mention whether you follow Semantic Versioning. - - Types of changes: - - - `Added` for new features. - - `Changed` for changes in existing functionality. - - `Deprecated` for soon-to-be removed features. - - `Removed` for now removed features. - - `Fixed` for any bug fixes. - - `Security` in case of vulnerabilities. - - -This document describes all new features and changes in the release `1.0`. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Release Overview -- Major features or milestones -- Achieved in this `x.y` release -- Changes to compatibility with Nautobot and/or other apps, libraries etc. - -## [v1.0.0] - 2025-12-03 - -### Added - -### Changed +Initial release of DiffSync. -### Fixed +## [v1.0.0] - 2020-10-23 -- [#123](https://github.com/networktocode/diffsync/issues/123) Fixed Tag filtering not working in job launch form. +Initial release diff --git a/docs/admin/release_notes/version_1.1.md b/docs/admin/release_notes/version_1.1.md new file mode 100644 index 00000000..8223834e --- /dev/null +++ b/docs/admin/release_notes/version_1.1.md @@ -0,0 +1,24 @@ +# v1.1 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.1.0] - 2020-12-01 + +### Added + +- #37 - added `sync_complete` callback, triggered on `sync_from` completion with changes. +- #41 - added `summary` API for Diff and DiffElement objects. +- #44 - added `set_status()` and `get_status()` APIs so that DiffSyncModel implementations can provide details for create/update/delete logging + +### Changed + +- Now requires Pydantic 1.7.2 or later +- #34 - in diff dicts, changed keys `src`/`dst`/`_src`/`_dst` to `-` and `+` +- #43 - `DiffSync.get_by_uids()` now raises `ObjectNotFound` if any of the provided uids cannot be located; `DiffSync.get()` raises `ObjectNotFound` or `ValueError` on failure, instead of returning `None`. + +### Fixed + +- #44 - On CRUD failure, do not generate an extraneous "success" log message in addition to the "failed" message + diff --git a/docs/admin/release_notes/version_1.10.md b/docs/admin/release_notes/version_1.10.md new file mode 100644 index 00000000..0cc6f1d6 --- /dev/null +++ b/docs/admin/release_notes/version_1.10.md @@ -0,0 +1,16 @@ +# v1.10 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.10.0] - 2023-11-16 + +### Fixed + +- #249 - Fixes natural deletion order flag +- #247 - Fixes underspecified typing_extensions dependency + +### Changed + +- #247 - Deprecates Python 3.7 diff --git a/docs/admin/release_notes/version_1.2.md b/docs/admin/release_notes/version_1.2.md new file mode 100644 index 00000000..574c6e24 --- /dev/null +++ b/docs/admin/release_notes/version_1.2.md @@ -0,0 +1,11 @@ +# v1.2 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.2.0] - 2020-12-08 + +### Added + +- #45 - minimum Python version lowered from 3.7 to 3.6, also now tested against Python 3.9. diff --git a/docs/admin/release_notes/version_1.3.md b/docs/admin/release_notes/version_1.3.md new file mode 100644 index 00000000..55bd70e2 --- /dev/null +++ b/docs/admin/release_notes/version_1.3.md @@ -0,0 +1,11 @@ +# v1.3 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.3.0] - 2021-04-07 + +### Added + +- #48 - added optional `callback` argument to `diff_from`/`diff_to`/`sync_from`/`sync_to` for use with progress reporting. diff --git a/docs/admin/release_notes/version_1.4.md b/docs/admin/release_notes/version_1.4.md new file mode 100644 index 00000000..ee696354 --- /dev/null +++ b/docs/admin/release_notes/version_1.4.md @@ -0,0 +1,62 @@ +# v1.4 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.4.3] - 2022-03-03 + +### Fixed + +- #101 - Revert changed introduced in #90 that affected `DiffElement.action` + +**NOTE**: this change is a breaking change against DiffSync 1.4.0 through 1.4.2, but was necessary to restore backward compatibility with DiffSync 1.3.x and earlier. Apologies for any inconvenience this causes. + +### Changed + +- #103 - Update development dependencies + +## [v1.4.2] - 2022-02-28 + +**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. + +### Fixed + +- #100 - Added explicit dependency on `packaging`. + +## [v1.4.1] - 2022-01-26 + +**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. + +### Fixed + +- #95 - Removed optional dependencies on `sphinx`, `m2r2`, `sphinx-rtd-theme`, `toml`. + +## [v1.4.0] - 2022-01-24 + +**WARNING** - #90 inadvertently introduced a breaking API change in DiffSync 1.4.0 through 1.4.2 (#101); this change was reverted in #102 for DiffSync 1.4.3 and later. We recommend not using this release, and moving to 1.4.3 instead. + +### Added + +- #53 - Add a new example based on pynautobot and Nautobot REST API +- #59 - Add proper documentation published in Read the doc +- #68 - Cleanup Readme, add link to new documentation site +- #70 - Add `add_or_update()` method to DiffSync class that requires a DiffSyncModel to be passed in and will attempt to add or update an existing object +- #72 - Add core engine section in docs and rename example directories +- #75 - Add support for Structlog v21 in addition to v20. +- #80 - Add support for an existing Diff object to be passed to `sync_to()` & `sync_from()` to prevent another diff from being calculated. +- #81 - Add a new example based on PeeringDB +- #83 - Add support for Python 3.10 +- #87 - Add new model flags : `SKIP_UNMATCHED_BOTH`, `SKIP_UNMATCHED_SRC` & `SKIP_UNMATCHED_DST` to match the behavior of the global flags + +### Changed + +- #62 - Update CI Token +- #69 - Replace Travis CI with Github Actions to run unit tests +- #82 - Update lock file with latest versions. +- #90 - Convert list of actions (`create`, `update`, `delete`) to proper Enum + +### Fixed + +- #51 - Update minimum Pydantic version due to security advisory GHSA-5jqp-qgf6-3pvh +- #63 - Fix type in Readme diff --git a/docs/admin/release_notes/version_1.5.md b/docs/admin/release_notes/version_1.5.md new file mode 100644 index 00000000..3f292c82 --- /dev/null +++ b/docs/admin/release_notes/version_1.5.md @@ -0,0 +1,26 @@ +# v1.5 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.5.1] - 2022-06-30 + +### Added + +- #111 - Added example 6, regarding IP prefixes. + +### Changed + +- #107 - Updated example 5 to use the Redis backend store. + +### Fixed + +- #115 - Fixed ReadTheDocs rendering pipeline +- #118 - Fixed a regression in `DiffSync.get(modelname, identifiers)` introduced in 1.5.0 + +## [v1.5.0] - 2022-06-07 + +### Added + +- #106 - Add a new, optional, backend store based in Redis diff --git a/docs/admin/release_notes/version_1.6.md b/docs/admin/release_notes/version_1.6.md new file mode 100644 index 00000000..2b4b8080 --- /dev/null +++ b/docs/admin/release_notes/version_1.6.md @@ -0,0 +1,11 @@ +# v1.6 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.6.0] - 2022-07-09 + +### Changed + +- #120 - Dropped support for Python 3.6, new minimum is Python 3.7 diff --git a/docs/admin/release_notes/version_1.7.md b/docs/admin/release_notes/version_1.7.md new file mode 100644 index 00000000..82f51b8b --- /dev/null +++ b/docs/admin/release_notes/version_1.7.md @@ -0,0 +1,25 @@ +# v1.7 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.7.0] - 2022-11-03 + +### Changed + +- #176 - Remove pytest-redislite in favor of pytest-redis. +- #174 - Update Dockerfile to install build-essential + +### Added + +- #174 - Add methods to load data from dictionary and enable tree traversal +- #174 - Add a `get_or_none` method to the DiffSync class +- #168 - Add 'skip' counter to diff.summary() +- #169/#170 - Add documentation about model processing order +- #121/#140 - Add and configure renovate +- #140 - Add renovate configuration validation to the CI + +### Fixed + +- #149 - Limit redundant CI concurrency diff --git a/docs/admin/release_notes/version_1.8.md b/docs/admin/release_notes/version_1.8.md new file mode 100644 index 00000000..c3a01b3f --- /dev/null +++ b/docs/admin/release_notes/version_1.8.md @@ -0,0 +1,18 @@ +# v1.8 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.8.0] - 2023-04-18 + +### Added + +- #182 - Added `get_or_add_model_instance()` and `update_or_add_model_instance()` APIs. +- #189 - Added note in `README.md` about running `invoke tests`. +- #190 - Added note in `README.md` about running `invoke build`. + +### Changed + +- #77/#188 - `sync_from()` and `sync_to()` now return the `Diff` that was applied. +- #211 - Loosened `packaging` and `structlog` library dependency constraints for broader compatibility. diff --git a/docs/admin/release_notes/version_1.9.md b/docs/admin/release_notes/version_1.9.md new file mode 100644 index 00000000..be125c72 --- /dev/null +++ b/docs/admin/release_notes/version_1.9.md @@ -0,0 +1,15 @@ +# v1.9 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v1.9.0] - 2023-10-16 + +### Added + +- #220 - Implement DiffSyncModelFlags.NATURAL_DELETION_ORDER. + +### Changed + +- #219 - Type hinting overhaul diff --git a/docs/admin/release_notes/version_2.0.md b/docs/admin/release_notes/version_2.0.md new file mode 100644 index 00000000..3227218c --- /dev/null +++ b/docs/admin/release_notes/version_2.0.md @@ -0,0 +1,28 @@ +# v2.0 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +Major release with Pydantic v2 upgrade and subsequent bug fixes. + +## [v2.0.0] + +### Changed + +- **BREAKING CHANGE** #236/240 - Upgrade to Pydantic v2. + +## [v2.0.1] + +### Changed + +- #276 - Removed upper version bound for `structlog` dependency + +### Fixed + +- #281 - Properly deprecated `DiffSync` class name +- #273 - Properly capitalized `DiffSync` in documentation +- #273 - Removed more mentions of `DiffSync` in favor of `Adapter` +- #274 - Fixed doc section title for getting started +- #269 - Fixed wording for a couple of docstrings +- #265 - Fixed readthedocs build diff --git a/docs/admin/release_notes/version_2.1.md b/docs/admin/release_notes/version_2.1.md new file mode 100644 index 00000000..4d051d1e --- /dev/null +++ b/docs/admin/release_notes/version_2.1.md @@ -0,0 +1,19 @@ +# v2.1 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v2.1.0] + +### Changed + +- [284](https://github.com/networktocode/diffsync/pull/284) - Update pyyaml to 6.0.2 in poetry.lock by @gsnider2195 +- [286](https://github.com/networktocode/diffsync/pull/286) - Update GitHub Actions runner version by @mjbear +- [289](https://github.com/networktocode/diffsync/pull/289) - Remove upper limit on Packaging by @jdrew82 + +## New Contributors + +- @gsnider2195 made their first contribution in [PR #284](https://github.com/networktocode/diffsync/pull/284) +- @mjbear made their first contribution in [PR #286](https://github.com/networktocode/diffsync/pull/286) +- @jdrew82 made their first contribution in [PR #289](https://github.com/networktocode/diffsync/pull/289) diff --git a/docs/admin/release_notes/version_2.2.md b/docs/admin/release_notes/version_2.2.md new file mode 100644 index 00000000..405c5742 --- /dev/null +++ b/docs/admin/release_notes/version_2.2.md @@ -0,0 +1,9 @@ +# v2.2 Release Notes + +This document describes all new features and changes in the release. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Release Overview + +## [v2.2.0] - 2025-12-08 + +Remove Python 3.9 support as it's EOL. diff --git a/mkdocs.yml b/mkdocs.yml index eba42596..d43b8366 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -132,6 +132,19 @@ nav: - Release Notes: - "admin/release_notes/index.md" - v1.0: "admin/release_notes/version_1.0.md" + - v1.1: "admin/release_notes/version_1.1.md" + - v1.2: "admin/release_notes/version_1.2.md" + - v1.3: "admin/release_notes/version_1.3.md" + - v1.4: "admin/release_notes/version_1.4.md" + - v1.5: "admin/release_notes/version_1.5.md" + - v1.6: "admin/release_notes/version_1.6.md" + - v1.7: "admin/release_notes/version_1.7.md" + - v1.8: "admin/release_notes/version_1.8.md" + - v1.9: "admin/release_notes/version_1.9.md" + - v1.10: "admin/release_notes/version_1.10.md" + - v2.0: "admin/release_notes/version_2.0.md" + - v2.1: "admin/release_notes/version_2.1.md" + - v2.2: "admin/release_notes/version_2.2.md" - Developer Guide: - Extending the Library: "dev/extending.md" - Contributing to the Library: "dev/contributing.md" diff --git a/pyproject.toml b/pyproject.toml index 5359332c..115b43d2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "diffsync" -version = "2.1.1a0" +version = "2.2.1a0" description = "Library to easily sync/diff/update 2 different data sources" authors = ["Network to Code, LLC "] license = "Apache-2.0"