Skip to content

Create Compiled Build Procedure #860

@MicahGale

Description

@MicahGale

Motivation

As discussed in #432, I am seriously considering rewriting our parser engine in a compiled language, most likely Rust, due to serious performance restrictions we are currently running into. This issue is meant to document how we should go about making a hybrid code-base, and developing a robust development and deployment process.

Criteria

In montepy's scope we have previously identified some criteria for adding a new language to the project:

The use of another language in the code base goes against the principle that it should be easy to contribute. However, there could be cases where the advantages of another language could justify this violation. Here are guidelines for when it could be appropriate:

  1. There needs to be a clear and significant benefit.
  2. The rewrite needs to be limited in scope to a module that few developers will modify.
    1. Only the input parser seems to meet this criteria.
      There must be a dependable build system that allows deployment to PyPI.

I think I would add a few more thoughts from my experience:

  1. It must be easy for new developers to start developing and testing other parts of the code-base.
  2. We must be fast to add support for new python releases.

Deploy Procedure

I think based on the above we would need to do the following:

  1. have a build tool that integrates well with PyPA tools, e.g., pip and build. I believe maturin (for Rust and pyo3) meets this requirement.
  2. The CI workflow must build and test against all python versions and supported platforms
  3. The source tar balls should be buildable via python -m build on its own.
  4. We need to be ready for new python releases (i.e., don't be vtk).
    1. Start testing against dev releases of python ASAP (probably around June)
    2. Prepare a new PyPI/conda release the week of the official python release.
    3. I think we could setup GHA cron jobs to get ready for this or at least remind us.

External dependencies

The other thing is how much do averse do we want to be to compiled (rust) side dependencies? Do we go similarly austere like we do with Python dependencies? On one hand we will be distributing wheels so most of those dependencies will be invisible to the users. However, we will also then be liable for that dependency breaking everything for us.

Perhaps we still have some level of dependency austerity, but be a little less restrictive and just do reasonable vetting for reliability and long term support.

Task List

  1. select a language and python wrapper interface. My vote is for Rust and pyo3
  2. Create a buildable dummy library.
  3. Look into what NumPy does.
  4. Determine Supported platforms. I suggest: 1. All tier 1 Cpython platforms (Linux, Windows, Mac)
    2. Web-assembly (WASM) (for Jupyter lite)
    3. Tier 2 Cpython platforms? (aarch-linux (ARM), x86-apple, clang x86-linux?)
  5. Create test and build and deploy CI workflows.
  6. Create deploy reminder cron jobs.
  7. Create workflow for developers easily getting workable dev environment. Ideally perhaps falling back to the last pip installed version for the compiled section if not being built locally?
  8. 5. What do we need to do under the licenses of upstream compiled dependencies for distributing them compiled in a wheel?

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDcode improvementA feature request that will improve the software and its maintainability, but be invisible to users.dependenciesPull requests that update a dependency fileperformance 🐌Issues related to speed and memory

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions