Speed improvements. #55
Merged
+516
−313
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Move imports inside function calls to reduce startup time and memory usage - Remove unused imports including multiprocessing, glob, traceback, and rich modules - Consolidate import statements and remove redundant module references - Maintain all functionality while improving performance through lazy loading - Reduce initial module load time by importing only when needed in each command function
…ove CLI options - Introduce --workers option to control number of parallel processes for CLI - Implement ProcessPoolExecutor for parallel file processing in dispatch module - Add worker pipeline caching to avoid repeated initialization overhead - Implement proper stdout/stderr capture and redirection per worker process - Modify verbosity handling to adjust stanza logging levels appropriately - Update morphotag pipeline to accept pre-initialized NLP pipeline instance - Add pre-download of stanza resources to prevent interleaved downloads - Limit workers for GPU-intensive tasks like transcription to prevent memory issues - Replace manual regex operations with pre-compiled patterns for better performance - Remove debug breakpoints and add graceful error handling for edge cases - Update progress reporting to work correctly with parallel execution model
…ve CLI processing order - Replace full-matrix dynamic programming with Hirschberg-style divide-and-conquer algorithm for linear space complexity - Reduce memory usage significantly for large sequence alignments while maintaining same edit-distance results - Add size-based sorting in CLI dispatcher to process largest files first, preventing late stragglers in parallel processing - Maintain backward compatibility with existing alignment API and match functions - Optimize progress tracking and cleanup for alignment operations
…ontext handling - Add tokenizer_context parameter to morphoanalyze function to maintain pre-parallelization behavior - Implement NLP pipeline caching using lang/retokenize/mwt combinations to avoid redundant rebuilds - Extract language conversion logic into _lang_alpha2 static method for better organization - Add MWT signature hashing to support custom multi-word token configurations - Introduce _build_nlp method to centralize pipeline construction with proper processor chaining - Create _get_or_create_nlp method to manage cached pipeline instances efficiently - Restore tokenizer post-processing functionality that was lost during parallelization changes - Add validation to prevent code-switching documents with custom MWT lists
- Add progress callback mechanism to worker tasks to track pipeline processing stages - Implement multiprocessing-safe progress queue to collect real-time progress updates - Update progress display to show detailed processing stages instead of generic waiting/processing states - Replace basic completion tracking with granular progress reporting from pipeline operations - Add proper cleanup and resource management for progress tracking infrastructure - Maintain backward compatibility with existing error handling and logging features
Member
|
Woah thanks! I will review this and get back to you. DP being in Thanks again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
batchalign --helpdoes not take 20 seconds.