Feature Request: Support for noGIL-Based Parallelism

[pperanich]

Summary

Enable support for true parallelism using Python’s experimental noGIL mode, which allows multi-threaded execution in a single process without the performance penalties of multiprocessing.

Background

Currently, ezmsg leverages multiprocessing to achieve parallelism due to Python’s Global Interpreter Lock (GIL), which traditionally prevents true multithreading. While multiprocessing enables parallel execution by spawning separate interpreter processes, it comes with:

• High resource overhead (due to memory duplication and IPC costs).
• Increased complexity in managing shared state and inter-process communication.

With noGIL now integrated experimentally into Python, true multithreading without these penalties is becoming a reality. Adopting noGIL-based parallelism in ezmsg could provide significant performance improvements while reducing system overhead.

Proposal

• Investigate and prototype an optional noGIL execution mode for ezmsg in supported Python versions.
• Identify areas in ezmsg that could benefit from thread-based execution over multiprocessing.
• Consider a runtime flag or configuration option to allow users to choose between multiprocessing and noGIL threading dynamically.
• Ensure compatibility with existing modules and integrations.

Benefits

• Lower resource overhead by avoiding the need for multiple Python processes.
• Faster inter-thread communication vs. inter-process communication.
• Future-proofing ezmsg as noGIL becomes more mainstream.