Add Kubernetes support via SkyPilot #2083

romilbhardwaj · 2025-12-08T06:11:41Z

Hi Monarch community 👋

This PR adds support for running Monarch workloads on Kubernetes and cloud VMs via SkyPilot.

Adds a new SkyPilotJob class that provisions Monarch workers on K8s (or any of the 20+ cloud providers supported by SkyPilot):

job = SkyPilotJob(
    meshes={"workers": 2},
    resources=sky.Resources(cloud=sky.Kubernetes(), accelerators="H200:1"), # SkyPilot resources object https://skypilot.readthedocs.io/en/latest/reference/yaml-spec.html#resources
    cluster_name="my-cluster",
    file_mounts={"/local/path": "/worker/path"}, # For any code/data sync
)

With this I was able to run the getting started example from Monarch docs on my multi-node Coreweave Kubernetes cluster with H200s. Full example in examples/skypilot_getting_started.py.

Notes:

Cold start: Currently builds Monarch from source on workers to keep client/worker versions in sync (~7-10 min). A pre-built container image would reduce this to ~30 seconds.
Networking: Assumes workers are in the same network as the client. If client-worker communication is not very chatty, SkyPilot's SSH tunneling could remove this requirement.

This is an early PR - I'm looking for feedback on the direction. I'm a maintainer of the SkyPilot project and was trying to run Monarch on my k8s cluster. Figured an integration with SkyPilot would be a good fit.

Happy to iterate on the implementation if this aligns with where you'd like Monarch to go.

Summary: This update introduces two new optional parameters, `workdir` and `file_mounts`, to the `SkyPilotJob` class. The `workdir` parameter allows users to specify a local directory to sync with the cluster, while `file_mounts` enables additional file mounts by mapping remote paths to local paths. These enhancements improve the flexibility and usability of job configurations in SkyPilot.

- Add SKY_README.md with comprehensive documentation: - Architecture overview - Implementation details - Usage examples - Troubleshooting guide - Networking considerations for Kubernetes - Add python/examples/skypilot_getting_started.py: - Example script demonstrating Monarch actors on SkyPilot - Supports multiple clouds (Kubernetes, AWS, GCP, Azure) - Configurable via command-line arguments - Update skypilot.py: - Add host mesh initialization wait - Improve logging for debugging - Fix worker command environment variable setup

meta-cla · 2025-12-08T06:11:47Z

Hi @romilbhardwaj!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

johnwhumphreys · 2025-12-09T02:32:09Z

@romilbhardwaj - I just saw this; sorry for missing it all day. I'll check it out in the morning and circle back :).

colin2328 · 2025-12-09T18:52:10Z

very cool! is this ready for review? (if so push it out of draft state?)

examples/skypilot/skypilot_job.py

colin2328 · 2025-12-09T19:02:00Z

python/monarch/_src/job/skypilot.py

+sudo apt-get update
+
+# Install system dependencies
+sudo apt-get install -y \


skypilot requires new system level dependencies?

hey @colin2328, these are dependencies for monarch. SkyPilot's dependencies are handled by the SkyPilot runtime during setup.

They're needed because this PR currently builds Monarch from source on the workers to keep client/worker versions in sync during development. Once this is merged and a Monarch wheel is released to PyPI, the setup simplifies to just pip install torchmonarch on workers.

The current approach is a temporary workaround for the chicken-and-egg problem: SkyPilotJob lives in Monarch source, but there's no released wheel with it yet.

Question on architecture: How should new schedulers extend Monarch's JobTrait?

Should they:

Live in Monarch source (python/monarch/_src/job/)

Pro: Clean import path, UX for users (from monarch.job import SkyPilotJob)

Con: Requires building from source during dev until wheel is released, may need version pinning after release?

Live in a separate package (user code)

Pro: Don't need to rebuild monarch

Con: Different import pattern (from <userpkg> import SkyPilotJob?)

Something else I'm missing.

Currently I have the SkyPilot implementation in python/monarch/_src/job/skypilot.py, but I can refactor it. Any guidance here would be appreciated 🙏

cc: @zdevito as this is an interesting question.

My intuition is that they should live in python/monarch/_src/job even though writing a job or changing the API will requires building from source for testing purposes. The testing cost is only paid by the developer, not the end user.

IIUC, keeping it in a separate package will still require installation on the workers while the code is in development. However it will not require building rust libraries which are probably the main source of the build latency.

It would be nice for JobTrait implementers to not require to build rust binaries while testing their code. I wonder if simply a cp of the new files into site-packages dir on the workers would work, though it seems like a hack.

Job implementations should be buildable on top of Monarch's public APIs. So, in general, I would favor the latter pattern, which also has good DevEx. I don't think we should accumulate many job implementations in core Monarch.

Note that this is distinct from the question of whether these are part of the Monarch "ecosystem" packages. In other words, we might maintain them in Monarch itself, but build and distribute them as separate packages.

We have previously talked about using pip flavors to bundle such dependencies for this purpose, e.g.,

$ pip install torchmonarch[skypilot]

etc.

romilbhardwaj · 2025-12-10T21:57:36Z

Thanks @colin2328 @johnwhumphreys - I'll polish the PR and get it ready for review. Have a question about the architecture here, happy to do whatever aligns with Monarch's recommended design.

shayne-fletcher · 2025-12-11T04:03:13Z

examples/skypilot/skypilot_ddp.py

+import os
+import sys
+
+# Set timeouts before importing monarch


am working on enabling

from monarch.config import configure configure( host_spawn_ready_timeout="300s", message_delivery_timeout="300s", mesh_proc_spawn_max_idle="300s" )

to replace the use of env-vars here.

Thanks @shayne-fletcher! That'll be super helpful for backends where cold starts can be > 30s.

romilbhardwaj · 2025-12-11T07:57:24Z

Monarch build was slowing down worker launches, so I've refactored along the lines of option 2 in 20d36e8. Let me know if this is not the right direction :)

Tested on Coreweave k8s cluster with H200s. PR is still a draft, will finish up the examples (torchtitan and ddp) + readme and mark ready for review soon!

colin2328 · 2025-12-11T18:00:46Z

@romilbhardwaj "the author does not have a valid CLA on file." can you acept the CLA? and publish?

meta-cla · 2025-12-11T19:24:28Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

ahmadsharif1 · 2025-12-11T22:06:53Z

examples/skypilot/skypilot_getting_started.py

+
+Usage:
+    # Run on Kubernetes:
+    python getting_started.py --cloud kubernetes --num-hosts 2


s/getting_started/skypilot_getting_started?

ahmadsharif1 · 2025-12-11T22:18:38Z

examples/skypilot/__init__.py

@@ -0,0 +1,23 @@
+"""


I will let @colin2328 and @johnwhumphreys chime in as well, but IMO this package should probably go in a subdir here:

https://github.com/meta-pytorch/monarch/tree/49547b120c8902e4383b48072529b85c81187d78/python/monarch/_src/job

I think the package dir structure for independent contributors should have job/contrib or job/providers

WDYT?

Agree, it seems like JobTrait implementations should be in the monarch package. Some more discussion here.

We do something similar for community contributed clouds in SkyPilot: they are included as a part of sky.clouds module.

However for Monarch, it introduces the overhead of build monarch workers from source to keep client/worker versions in sync (discussion, example of extra build steps that a worker needs to run).

Any recommendations on how to add a new module to monarch.job while still letting workers use wheels from torchmonarch-nightly? Trying to avoid having workers build from src during dev/testing of this PR.

Is this resolved now that we have moved the module outside in a separate dir?

python/monarch/job/__init__.py

meta-codesync · 2025-12-11T22:42:12Z

@colin2328 has imported this pull request. If you are a Meta employee, you can view this in D88994552.

ahmadsharif1 · 2025-12-11T23:17:53Z

examples/skypilot/skypilot_ddp.py

+try:
+    import sky
+except ImportError:
+    print("ERROR: SkyPilot is not installed. Run: pip install skypilot[kubernetes]")


Maybe we should add this as an optional dependency in pyproject.toml if we move skypilot_job.py from examples to the job dir

ahmadsharif1 · 2025-12-12T17:24:56Z

I am also wondering how to test this when we move this outside of the examples dir and into the main package.

@romilbhardwaj How does skypilot do end to end testing in its CI? Maybe we should have a minikube/kind test setup in our CI to test this if we expose this to users. We don't want silent breakages if users are going to use this.

romilbhardwaj · 2025-12-13T01:24:50Z

Cleaned up the readme, added an architecture diagram and included a DDP notebook inspired by the slurm example.

Tested on coreweave k8s with H200s.

This PR is ready for review now! cc @ahmadsharif1 @colin2328 @johnwhumphreys

romilbhardwaj · 2025-12-13T02:21:37Z

@romilbhardwaj How does skypilot do end to end testing in its CI? Maybe we should have a minikube/kind test setup in our CI to test this if we expose this to users. We don't want silent breakages if users are going to use this.

Great point. For our release testing, our CI spins up k8s clusters on EKS, GKE, CW and Nebius and runs our smoke test suite which includes model training examples from pytorch, ray, nemo and other popular frameworks.

For lightweight CPU-only tests, some parts of our CI uses kind (via sky local up) to create a local k8s cluster. If it is of interest, shouldn't be too hard to edit this example to run on CPUs only for the CI tests.

romilbhardwaj · 2025-12-13T02:26:21Z

Added some flags to allow the example to run on CPU-only k8s clusters

docs/source/examples/README.rst

ahmadsharif1

Looks good for the most part. You can wait for Colin to take a look as well.

I think he wants this to be part of the release, so you can address them and merge if you want

docs/source/index.md

ahmadsharif1 · 2025-12-18T18:44:08Z

examples/skypilot/__init__.py

@@ -0,0 +1,23 @@
+"""


Is this resolved now that we have moved the module outside in a separate dir?

ahmadsharif1 · 2025-12-18T18:47:57Z

examples/skypilot/monarch_getting_started.sky.yaml

+  python -c "import monarch; print(f'Monarch installed: {monarch}')"
+  python -c "import sky; print(f'SkyPilot installed: {sky}')"
+  
+  # Configure SkyPilot to use in-cluster Kubernetes context


I am curious as to what permissions the driver pod has. How can it launch worker pods using sky? Where are those permissions configured?

By default, SkyPilot automatically creates a service account with minimal permissions it needs and attaches it to the pod. You can find the permissions here: https://docs.skypilot.co/en/latest/cloud-setup/cloud-permissions/kubernetes.html

These permissions also allow SkyPilot running inside the driver pod to create worker pods.

Optionally, the user can override and specify their own custom service account SkyPilot config.

ahmadsharif1 · 2025-12-18T18:49:29Z

examples/skypilot/monarch_getting_started.sky.yaml

+  # Configure SkyPilot to use in-cluster Kubernetes context
+  # This allows the driver pod to launch nested SkyPilot clusters
+  unset SKYPILOT_IN_CLUSTER_CONTEXT_NAME
+  sky api start


Is there a way to view the YAML that skypilot itself generates and applies to kubernetes (that contains the permissions, etc.)? Just curious.

Yep! If you run sky launch with export SKYPILOT_DEBUG=1, the terminal output will include a long k8s manifest that will be applied. It's also stored in SkyPilot internal state db.

ahmadsharif1 · 2025-12-18T18:50:56Z

examples/skypilot/skypilot_getting_started.py

@@ -0,0 +1,269 @@
+#!/usr/bin/env python3


Sorry about this, but maybe rename this to example_driver.py or something more descriptive?

examples/skypilot/skypilot_job.py

ahmadsharif1 · 2025-12-18T19:13:36Z

examples/skypilot/skypilot_job.py

+            logger.error(f"Failed to launch SkyPilot cluster: {e}")
+            raise RuntimeError(f"Failed to launch SkyPilot cluster: {e}") from e


What if line 215 succeeds and line 222 fails? Do we still need to clean that up?

ahmadsharif1 · 2025-12-18T19:14:17Z

examples/skypilot/skypilot_job.py

+        logger.info(f"SkyPilot cluster '{cluster_name}' launched successfully")
+
+        # Wait for the job to be RUNNING (setup complete, run started)
+        self._wait_for_job_running(cluster_name, job_id, timeout=JOB_TIMEOUT)


Again, should you clean something up if this timeout fails if there is state on the kubernetes cluster

examples/skypilot/skypilot_job.py

ahmadsharif1 · 2025-12-18T20:05:04Z

examples/skypilot/skypilot_job.py

@@ -0,0 +1,457 @@
+"""


I am also wondering if this package should be in a subdir outside of the notebooks, etc. I'll leave it to you

I think that'd be a good idea, especially if we later want to support @mariusae's suggestion of using pip flavors like pip install torchmonarch[skypilot]

romilbhardwaj · 2025-12-19T20:14:07Z

Thanks for the reviews @ahmadsharif1! I'll wait for a bit for @colin2328's feedback before working through the comments and doing a round of final testing.

Also I noticed the linter was failing - do I need to take any action? Couldn't find instructions to run the linter myself.

colin2328 · 2025-12-19T20:41:42Z

docs/source/index.md

 - [Demo notebook](https://github.com/meta-pytorch/monarch/blob/main/examples/presentation/presentation.ipynb)
 - [DevX Pytorch tutorial](https://docs.pytorch.org/tutorials/intermediate/monarch_distributed_tutorial.html)
 - [Lightning Monarch blog](https://lightning.ai/meta-ai/environments/large-scale-interactive-training-with-monarch)
+- [Monarch on Kubernetes using Skypilot](https://github.com/meta-pytorch/monarch/tree/main/examples/skypilot)


we can drop this, leave I'll just link to the examples in index.rst

colin2328 · 2025-12-19T20:42:28Z

@romilbhardwaj , thanks. I'll fix the lints and push for you. please address the comments first, rest is fine to me.

Romil Bhardwaj and others added 10 commits December 5, 2025 06:49

Add SkyPilot integration for job launching

6ffd98c

fixes

b2344bf

Working example

a740ae1

fix

efa313f

updates

35b6e0e

cleanup

6a75678

cleanup

fd310d3

updates

8511e5a

romilbhardwaj mentioned this pull request Dec 8, 2025

TorchX Kubernetes Support #2039

Closed

romilbhardwaj marked this pull request as draft December 8, 2025 06:16

colin2328 reviewed Dec 9, 2025

View reviewed changes

johnwhumphreys added the kubernetes label Dec 9, 2025

romilbhardwaj added 3 commits December 11, 2025 01:31

Extract SkyPilotJob from monarch src

20d36e8

remove stale changes

e23bd3f

Add DDP and titan examples

40f3a6a

shayne-fletcher reviewed Dec 11, 2025

View reviewed changes

Update README.md

2132e3c

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 11, 2025

ahmadsharif1 reviewed Dec 11, 2025

View reviewed changes

romilbhardwaj added 5 commits December 11, 2025 16:19

Clean up, add run_getting_started

0b1e5fd

renaming

3cda869

Add DDP notebook

32ee2d3

Readme updates

ca7014a

Updates

ffe74f5

romilbhardwaj added 7 commits December 12, 2025 23:58

fix mermaid doc

8514504

Docs updates

2d83527

updates

4d6ed27

updates

3f3e890

Add notes on how to set resources and num nodes

be7818e

fix ssh command

0a443b3

Update jupyter commands

5fcf775

romilbhardwaj marked this pull request as ready for review December 13, 2025 01:24

Add CPU-only support

56e7c8c

romilbhardwaj requested review from ahmadsharif1, colin2328 and shayne-fletcher December 16, 2025 16:35

ahmadsharif1 reviewed Dec 17, 2025

View reviewed changes

docs/source/examples/README.rst Outdated Show resolved Hide resolved

ahmadsharif1 approved these changes Dec 18, 2025

View reviewed changes

romilbhardwaj added 2 commits December 20, 2025 01:08

update docs

236a01a

review comments

8207b92

colin2328 reviewed Dec 19, 2025

View reviewed changes

		logger.error(f"Failed to launch SkyPilot cluster: {e}")
		raise RuntimeError(f"Failed to launch SkyPilot cluster: {e}") from e

Add Kubernetes support via SkyPilot #2083

Are you sure you want to change the base?

Add Kubernetes support via SkyPilot #2083

Uh oh!

Conversation

romilbhardwaj commented Dec 8, 2025

Uh oh!

meta-cla bot commented Dec 8, 2025

Action Required

Process

Uh oh!

johnwhumphreys commented Dec 9, 2025

Uh oh!

colin2328 commented Dec 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj commented Dec 10, 2025

Uh oh!

shayne-fletcher Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj commented Dec 11, 2025

Uh oh!

colin2328 commented Dec 11, 2025

Uh oh!

meta-cla bot commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

meta-codesync bot commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahmadsharif1 commented Dec 12, 2025

Uh oh!

romilbhardwaj commented Dec 13, 2025

Uh oh!

romilbhardwaj commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romilbhardwaj commented Dec 13, 2025

Uh oh!

Uh oh!

ahmadsharif1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

romilbhardwaj Dec 10, 2025 •

edited

Loading

shayne-fletcher Dec 11, 2025 •

edited

Loading

romilbhardwaj commented Dec 13, 2025 •

edited

Loading

romilbhardwaj Dec 19, 2025 •

edited

Loading

colin2328 commented Dec 19, 2025 •

edited

Loading