feat: add eval_config support to RL API client #271
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Member
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
manveerxyz
reviewed
Dec 31, 2025
- Use BaseConfig.from_sources for eval config precedence instead of manual if-statements - Require owner/name format for --eval-envs (same as training environments) - Rename EvalConfig.eval_base_model to base_model for proper underscore mapping
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa1b26d886
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
JannikSt
added a commit
that referenced
this pull request
Jan 3, 2026
* Implement commands for hosted RL * Hosted RL * Allow for user to use just Usage: prime rl [OPTIONS] ENVIRONMENTS... | COMMAND [ARGS]... Manage RL training runs. By default, 'prime rl <environments>' runs 'prime rl run <environments>'. ╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ run Create an RL training run with specified environments and model. │ │ models List available models for RL training. │ │ runs List your RL training runs. │ │ stop Stop an RL training run. │ │ delete Delete an RL training run. │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ to start a run * Support tomls on prime rl cmd * Minor fix * Cleanup references to RFT * Minor improvements * Fix ruff * Match post rft run schema to new backend * Refactor delete_run method to remove return value and simplify success handling in RLClient and related command. * Fix/prime rl list (#267) * quick fix for prime rl list when no name set * remove truncation of id in prime rl list * Add support for run_config * feat: add eval_config support to RL API client (#271) * feat: add eval_config support to RL API client * Remove accidentally committed test files * feat: add logs command for RL runs * fix: move time import to top, add rl_config example * feat: add --watch flag and improve log streaming * fix: allow built-in envs like reverse-text, update example * feat: add --eval-* options to rl run command * fix: strip ANSI escape codes from logs output * fix: increase poll interval to 5s, add rate limit handling * fix: filter progress bars from logs output, remove redundant --watch flag * fix: keep 100% progress bar completion lines in logs * fix: address review comments - simplify log follow, warn on unused eval options * fix: handle log rotation in follow mode when tail window is full * fix: always use overlap detection for log follow to handle fast growth with rotation * feat: add [eval] section support in TOML config files * fix: improve progress bar filtering to remove empty lines * fix: require owner/name format for environments, remove example config * fix: use from_sources for eval config merging, require owner/name format - Use BaseConfig.from_sources for eval config precedence instead of manual if-statements - Require owner/name format for --eval-envs (same as training environments) - Rename EvalConfig.eval_base_model to base_model for proper underscore mapping * prime registry support (#215) * custom image registry for sandboxes * prime images * --image typo * linux/amd64 * updated to not build locally * full image path * rm emojis * remove inline * image status * full image path * add cleanup * adjust scope output * bug bot stuff * validate_output_format * bug bot comment * update prime images list * limit platform * bump timeout * add closed beta info * Chore/bump version 0.5.8 (#270) * bump version to 0.5.8 * bump versions * Fix: Update eval sample field (#265) * Update eval sample field. * Update docs. * Fix: Remove trailing comma from API token URL (#273) Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: sami <sami@primeintellect.ai> --------- Co-authored-by: Johannes Hagemann <johannes@primeintellect.ai> Co-authored-by: JannikSt <JannikSt@users.noreply.github.com> Co-authored-by: Jannik Straube <info@jannik-straube.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds missing
eval_configsupport to the RL API client.Changes:
eval_configfield toRLRunmodeleval_configparameter tocreate_run()payload["eval"]in request bodyRelates to #256
Note
Adds evaluation support to RL runs and a new logs experience.
eval_configtoRLRun, plumbseval_configthroughRLClient.create_run()aspayload["eval"][eval]config (envs, interval, num_examples, rollouts_per_example, base_model) and corresponding--eval-*flags; validates env/eval slugs; surfaces eval settings in run creation outputprime rl logscommand with--tailand--follow; cleans output by stripping ANSI and collapsing progress bars; handles rate limitingget_logs(run_id, tail_lines)method used by CLIWritten by Cursor Bugbot for commit aa1b26d. This will update automatically on new commits. Configure here.