Skip to content

Warn users when simulating campaigns without an objective#750

Open
AVHopp wants to merge 6 commits intomainfrom
bug/simulte_scenario_requires_objective
Open

Warn users when simulating campaigns without an objective#750
AVHopp wants to merge 6 commits intomainfrom
bug/simulte_scenario_requires_objective

Conversation

@AVHopp
Copy link
Collaborator

@AVHopp AVHopp commented Feb 13, 2026

Fixes #749 by warning users that there are campaign without an objective instead of crashing.

simulate_scenarios will now run for the case described above, but warn the users about the missing objective(s). The code from the original bug now produces the following warning

(baybe) mXXXXX@LTDE0243486 baybe % python bug_test.py
/Users/mXXXXX/Programming/BayesianBE/baybe/baybe/simulation/scenarios.py:215: UserWarning: The following scenario(s) have campaigns without an objective and will not track any targets during simulation: ['Random Search']. If this was not intended, make sure to define an objective for each campaign.

and the following result dataframe:

        Scenario  Random_Seed  Initial_Data  Iteration  Num_Experiments
0  Random Search         1337           NaN          0                1
1  Random Search         1337           NaN          1                2
2  Random Search         1337           NaN          2                3

When also using a campaign that actually has an objective, the resulting dataframe looks as follows:

         Scenario  Random_Seed  Initial_Data  Iteration  Num_Experiments t_Measurements  t_IterBest  t_CumBest
0   Random Search         1337           NaN          0                1            NaN         NaN        NaN
1   Random Search         1337           NaN          1                2            NaN         NaN        NaN
2   Random Search         1337           NaN          2                3            NaN         NaN        NaN
3  With Objective         1337           NaN          0                1          [0.5]         0.5        0.5
4  With Objective         1337           NaN          1                2          [0.5]         0.5        0.5
5  With Objective         1337           NaN          2                3          [0.5]         0.5        0.5

@AVHopp AVHopp self-assigned this Feb 13, 2026
@AVHopp AVHopp requested a review from Scienfitz as a code owner February 13, 2026 11:44
Copilot AI review requested due to automatic review settings February 13, 2026 11:44
@AVHopp AVHopp requested a review from AdrianSosic as a code owner February 13, 2026 11:44
@AVHopp AVHopp added the bug Something isn't working label Feb 13, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes issue #749 by enabling simulate_scenarios to work with campaigns that don't have an objective (e.g., campaigns using RandomRecommender). The fix automatically propagates an objective from the first campaign that has one to any campaigns lacking an objective, allowing target tracking during simulation.

Changes:

  • Modified simulate_scenarios to propagate objectives from campaigns that have them to campaigns that don't
  • Added comprehensive tests for both successful objective propagation and failure when no objective exists
  • Updated CHANGELOG to document the fix

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
baybe/simulation/scenarios.py Added objective propagation logic using attrs.evolve() and updated docstring to document the new behavior
tests/test_simulation.py Added two tests: one for successful simulation with mixed objective/no-objective campaigns, and one to verify proper error handling when all campaigns lack objectives. Also updated import from baybe.targets.numerical to baybe.targets
CHANGELOG.md Documented the fix in the "Fixed" section

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@AVHopp AVHopp marked this pull request as draft February 15, 2026 17:38
@AVHopp AVHopp changed the title Propagate objective in simulate_scenarios if campaign does not have one Warn users when simulating campaigns without an objective Feb 15, 2026
@AVHopp AVHopp force-pushed the bug/simulte_scenario_requires_objective branch from f6e8588 to ccc494b Compare February 15, 2026 22:16
@AVHopp AVHopp force-pushed the bug/simulte_scenario_requires_objective branch from ccc494b to d441747 Compare February 15, 2026 22:17
@AVHopp AVHopp marked this pull request as ready for review February 15, 2026 22:21
@AVHopp AVHopp requested a review from Copilot February 15, 2026 22:21
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

AVHopp and others added 2 commits February 16, 2026 11:26
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
return pd.concat(dfs, ignore_index=True)

# Warn about campaigns without objectives
_no_obj_keys = [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the warning should be thrown in simulate_experiment where the original error was thrown. This place here can catch and count them and rethrow a combined warning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

simulate_scenarios requiring campaign to have an objective

2 participants