Merge pull request #720 from avigna/JOSS_pythonSubmit

Joss python submit

Author: ilyamandel

docs/compasHPC.md

@@ -79,8 +79,8 @@ to your `~/.bash_profile` or equivalent.
   Make a copy of the `stroopwafelInterface.py` file in the `defaults/` folder into your current directory, and set the `run_on_helios` parameter to `True`.
   Set any other stroopwafel parameters as you see fit.
 
-  If you have many non-default COMPAS arguments, you are encouraged to set them in a `pythonSubmit.py` file in the same directory, 
-  and set the `usePythonSubmit` parameter to `True`.
+  If you have many non-default COMPAS arguments, you are encouraged to set them in a `runSubmit.py` file in the same directory, 
+  and set the `usePythonSubmit` parameter to `True`, and adapt `stroopwafelInterface.py` accordingly, given that is still under development.
 
   See sampling.md for details.
 

docs/docker.md

@@ -41,22 +41,23 @@ To see all available versions, go to the TeamCOMPAS docker hub page [here](https
 
 ### Running
 
-COMPAS can still be configured via command line arguments passed to the COMPAS executable or via a `pythonSubmit.py` file.
+COMPAS can still be configured via command line arguments passed to the COMPAS executable or via a `runSubmit.py` file.
+The `runSubmit.py` file read the options defined in the `compasConfigDefault.yaml`, which overwrite the COMPAS defaults. 
 
-#### Run pythonSubmit.py
+#### Execute runSubmit.py
 
-To run COMPAS via a `pythonSubmit.py` file, the command is a little more complex.
+To run COMPAS via a `runSubmit.py` file, the command is a little more complex.
 
 ```
 docker run                                                  \
     --rm                                                    \
     -it                                                     \
     -v $(pwd)/compas-logs:/app/COMPAS/logs                  \
-    -v $(pwd)/pythonSubmit.py:/app/starts/pythonSubmit.py   \
+    -v $(pwd)/runSubmit.py:/app/starts/runSubmit.py         \
     -e COMPAS_EXECUTABLE_PATH=/app/COMPAS/bin/COMPAS        \
     -e COMPAS_LOGS_OUTPUT_DIR_PATH=/app/COMPAS/logs         \
     teamcompas/compas                                       \
-    python3 /app/starts/pythonSubmit.py                     
+    python3 /app/starts/runSubmit.py                     
 ```
 
 Breaking down this command:
@@ -74,7 +75,7 @@ short for [-i and -t](https://docs.docker.com/engine/reference/run/#foreground)
 `-v <path-on-host>:<path-in-container>`
 [Bind mounts](https://docs.docker.com/storage/bind-mounts/)
 mount `<path-on-host>` to `<path-in-container`>
-This time we not only want to get the output from COMPAS on the host machine, we also want to supply a `pythonSubmit.py` to the container from the host machine.
+This time we not only want to get the output from COMPAS on the host machine, we also want to supply a `runSubmit.py` to the container from the host machine.
 
 `-e VAR_NAME=value`
 [Environment variables](https://docs.docker.com/engine/reference/run/#env-environment-variables)
@@ -83,13 +84,13 @@ set the environment variable `VAR_VAME` to `value`
 `teamcompas/compas`
 the image to run
 
-`python3 /app/starts/pythonSubmit.py`
+`python3 /app/starts/runSubmit.py`
 the command to run when the container starts
 
 
 #### Run the COMPAS executable
 
-To run the COMPAS executable directly (i.e. without `pythonSubmit.py`)
+To run the COMPAS executable directly (i.e. without `runSubmit.py`)
 ```
 docker run                                  \
     --rm                                    \
@@ -136,12 +137,12 @@ More info on `docker run` [here](https://docs.docker.com/engine/reference/run/)
 
 NOTE 1:
 
-Two new environment variables have been added, both of these apply to `pythonSubmit.py` only and are non-breaking changes.
+Two new environment variables have been added, both of these apply to `runSubmit.py` only and are non-breaking changes.
 
-`COMPAS_EXECUTABLE_PATH` is an addition to the default `pythonSubmit.py` that overrides where `pythonSubmit.py` looks for the compiled COMPAS.
+`COMPAS_EXECUTABLE_PATH` is an addition to the default `runSubmit.py` that overrides where `runSubmit.py` looks for the compiled COMPAS.
 This override exists purely for ease-of-use from the command line.
 
-`COMPAS_LOGS_OUTPUT_DIR_PATH` is also an addition to the default `pythonSubmit.py` that overrides where logs are placed.
+`COMPAS_LOGS_OUTPUT_DIR_PATH` is also an addition to the default `runSubmit.py` that overrides where logs are placed.
 The override exists because the mounted directory (option `-v`) is created before COMPAS runs. COMPAS sees that the directory where it's supposed to put logs already exists, so it created a different (i.e. non-mapped) directory to deposit logs in.
 
 
@@ -154,16 +155,16 @@ All container output will be hidden.
 An example where this would be useful is if you were running 4 instances of COMPAS at once.
 You could copy/paste the following into the terminal...
 ```
-docker run --rm -d -v $(pwd)/compas-logs/run_0:/app/COMPAS/logs -v $(pwd)/pythonSubmitMMsolar_01.py:/app/starts/pythonSubmit.py teamcompas/compas python3 /app/starts/pythonSubmit.py &
+docker run --rm -d -v $(pwd)/compas-logs/run_0:/app/COMPAS/logs -v $(pwd)/runSubmitMMsolar_01.py:/app/starts/runSubmit.py teamcompas/compas python3 /app/starts/runSubmit.py &
 
-docker run --rm -d -v $(pwd)/compas-logs/run_1:/app/COMPAS/logs -v $(pwd)/pythonSubmitMMsolar_02.py:/app/starts/pythonSubmit.py teamcompas/compas python3 /app/starts/pythonSubmit.py &
+docker run --rm -d -v $(pwd)/compas-logs/run_1:/app/COMPAS/logs -v $(pwd)/runSubmitMMsolar_02.py:/app/starts/runSubmit.py teamcompas/compas python3 /app/starts/runSubmit.py &
 
-docker run --rm -d -v $(pwd)/compas-logs/run_2:/app/COMPAS/logs -v $(pwd)/pythonSubmitMMsolar_03.py:/app/starts/pythonSubmit.py teamcompas/compas python3 /app/starts/pythonSubmit.py &
+docker run --rm -d -v $(pwd)/compas-logs/run_2:/app/COMPAS/logs -v $(pwd)/runSubmitMMsolar_03.py:/app/starts/runSubmit.py teamcompas/compas python3 /app/starts/runSubmit.py &
 
-docker run --rm -d -v $(pwd)/compas-logs/run_3:/app/COMPAS/logs -v $(pwd)/pythonSubmitMMsolar_04.py:/app/starts/pythonSubmit.py teamcompas/compas python3 /app/starts/pythonSubmit.py
+docker run --rm -d -v $(pwd)/compas-logs/run_3:/app/COMPAS/logs -v $(pwd)/runSubmitMMsolar_04.py:/app/starts/runSubmit.py teamcompas/compas python3 /app/starts/runSubmit.py
 ```
 
-...which would run 4 separate instances of COMPAS, each with its own `pythonSubmit.py` file and logging directory, and all console output supressed.
+...which would run 4 separate instances of COMPAS, each with its own `runSubmit.py` file and logging directory, and all console output supressed.
 
 You may want to check the console output to see how far into the run COMPAS is.
 The command for this is `docker logs <container_id>`.
@@ -235,7 +236,7 @@ Make COMPAS using a specific makefile (more below) and as many cores as possible
 [RUN](https://docs.docker.com/engine/reference/builder/#run) docs
 
 Dockerfiles will usually end with a `CMD` directive that specifies what command should run when the container is started.
-COMPAS doesn't have a `CMD` directive because some users will want to run the executable directly and some will want to use `pythonSubmit.`.
+COMPAS doesn't have a `CMD` directive because some users will want to run the executable directly and some will want to use `runSubmit.`.
 [CMD](https://docs.docker.com/engine/reference/builder/#cmd) docs
 
 

docs/getting_started.md

@@ -8,8 +8,9 @@
     + [1.3 Setting up the Makefile and Compiling](#13-setting-up-the-makefile-and-compiling)
     + [1.4 Installing Python](#14-installing-python)
   * [2. Evolving your first binary](#2-evolving-your-first-binary)
-    + [2.1 Running COMPAS from a grid file](#21-running-compas-from-a-grid-file)
-    + [2.2 Examining detailed output](#22-examining-detailed-output)
+    + [2.1 Running COMPAS out-of-the-box](#21-running-compas-out-of-the-box)
+    + [2.2 Running COMPAS from a grid file](#22-running-compas-from-a-grid-file)
+    + [2.3 Examining detailed output](#23-examining-detailed-output)
   * [3. Further queries](#3-further-queries)
 
 ## 1. Installing COMPAS and Dependencies
@@ -124,15 +125,53 @@ If you do not have python3 installed, install it by following the instructions b
 
 
 ## 2. Evolving your first binary
-To start using COMPAS, you will need the python script `pythonSubmitDemo.py`, which specifies all the program options (physics assumptions, output types) and runs COMPAS in the terminal. Although the primary functionality of COMPAS is to evolve a whole population of binary stars rapidly, for now, let's focus on evolving a single stellar system and examining the detailed output.
+
+### 2.1 Running COMPAS out-of-the-box
+To start using COMPAS, and after the source code has been compiled, you need to go to `$COMPAS_ROOT_DIR/preProcessing/` and run the `runSubmit.py`, which reads the options from `compasConfigDefault.yaml` and overwrite the COMPAS defaults when specified.
+The `compasConfigDefault.yaml` specifies all the program options (physics assumptions, output types) and runs COMPAS in the terminal. 
+Your output should be similar to following.
+
+	$ python3 runSubmit.py
+	python_version = 3
+	compas_executable_override None
+	grid_filename None
+	$COMPAS_ROOT_DIR/src/COMPAS --LONG-COMMAND-LINE
+
+	COMPAS v02.25.10
+	Compact Object Mergers: Population Astrophysics and Statistics
+	by Team COMPAS (http://compas.science/index.html)
+	A binary star simulator
+
+	Start generating binaries at DATE
+
+	0: Unbound binary: (Main_Sequence_>_0.7 -> Neutron_Star) + (Main_Sequence_>_0.7 -> Main_Sequence_>_0.7)
+
+	Generated 1 of 1 binaries requested
+
+	Simulation completed
+
+	End generating binaries at DATE
+
+	Clock time = 0.264009 CPU seconds
+	Wall time  = 0000:00:00 (hhhh:mm:ss)
+
+If that is the case, you have succesfully installed COMPAS and ran your fist binary.
+
+As of v02.25.10, the `runSubmit.py` and `compasConfigDefault.yaml` combo are the preferred way to set up runs using python, instead of the prevously used `pythonSubmit.py` (and their adaptations).
+We will eventually update the documentation to reflect this, but for now the references to `pythonSubmit.py` or `pythonSubmitDemo.py` refer to the combined options and commands python script that has been tested and used in releases previous than v02.25.10, including those of the methods paper.
+The `pythonSubmit.py`, `pythonSubmitDemo.py`, and their functionality, will eventually be fully replaced by the `runSubmit.py` and `compasConfigDefault.yaml` combo. 
+
+### 2.2 Running COMPAS from a grid file
+
+We will now try to run a binary using a grid file. For this, you will need the python script `pythonSubmitDemo.py`, which specifies all the program options (physics assumptions, output types) and runs COMPAS in the terminal. For now, let's focus on evolving a single stellar system and examining the detailed output.
 
 To start, change to the `examples/methods_paper_plots/detailed_evolution/` directory:
 
     cd $COMPAS_ROOT_DIR/examples/methods_paper_plots/detailed_evolution/
 
-Here, you will find the script `pythonSubmitDemo.py` for this demo.
+Here, you will find the script `pythonSubmitDemo.py` for this demo. This file is an older version of the `runSubmit.py` and `compasConfigDefault.yaml`
+
 
-### 2.1 Running COMPAS from a grid file
 In population synthesis, the initial stellar population is usually generated by drawing the primary mass, secondary mass, semi-major axis, and eccentricity from their respective distributions specified in the program options. However, we illustrate COMPAS's ability to specify a grid of initial values for single and binary star evolution using COMPAS's grid functionality.
 
 An example grid file, `Grid_demo.txt`, has been included in the current `detailed_evolution` directory. Open it with a text editor to view it:
@@ -144,7 +183,7 @@ An example grid file, `Grid_demo.txt`, has been included in the current `detaile
 
 It should be clear that this grid file specifies a binary of zero-age main sequence stars with primary mass 35.4 Msol, secondary mass 29.3 Msol, metallicity 0.001, zero eccentricity, semi-major axis of 1.02 AU, and kick velocities for each component. For more detailed documentation of COMPAS's grid functionality for both single and binary stars, please see [Specifications](./COMPAS_Doc.pdf).
 
-To tell the python submit script to take its input from this grid file, you usually need to open `$COMPAS_ROOT_DIR/preProcessing/pythonSubmit.py` with a text editor, and specify the grid filename `grid_filename = 'Grid_demo.txt'`.  And to print the time evolution of binary properties, we need to turn on detailed output: `detailed_output = True`.  COMPAS can produce logfiles of different types: HDF5, CSV, TSV, and TXT, which can be chosen by editing the line `logfile_type = 'HDF5'` (the default type is HDF5).  For this demo, this has all been done for you in the file `pythonSubmitDemo.py` found in the current directory.
+To tell the python submit script to take its input from this grid file, you usually need to open `$COMPAS_ROOT_DIR/preProcessing/compasConfigDefault.yaml` with a text editor, and specify the grid filename `grid: 'Grid_demo.txt'`.  And to print the time evolution of binary properties, we need to turn on detailed output: `detailed-output: True`.  COMPAS can produce logfiles of different types: HDF5, CSV, TSV, and TXT, which can be chosen by editing the line `logfile-type = 'HDF5'` (the default type is HDF5). For this demo, you can use the `pythonSubmitDemo.py` found in the current directory; this is a good option for v02.25.10 and previous releases. Alternatively, for more recent releases, you could choose to try and adapt the same options from the `$COMPAS_ROOT_DIR/preProcessing/compasConfigDefault.yaml` and the run via `$COMPAS_ROOT_DIR/preProcessing/runSubmit.py` script.
 
 Now let's run COMPAS!
 
@@ -172,7 +211,7 @@ Now let's run COMPAS!
 Congratulations! You've just made a binary black hole. And it didn't even take a second.
 
 
-### 2.2 Examining detailed output
+### 2.3 Examining detailed output
 The COMPAS run just now produces a new directory `COMPAS_Output`, inside which you will find the following files/directories (here we assume `logfile_type = 'h5'` in the python submit file):
 
 * `COMPAS_Output.h5`: The primary output file, containing hdf5 data groups for the relevant output physics. By default, and for a sufficiently large simulation, this will include `BSE_Common_Envelopes`, `BSE_Double_Compact_Objects`, `BSE_Supernovae`, and `BSE_System_Parameters`. 

examples/methods_paper_plots/chirpmass_distribution/pythonSubmit.py

@@ -8,6 +8,11 @@
 import subprocess
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)

examples/methods_paper_plots/detailed_evolution/pythonSubmitDemo.py

@@ -8,6 +8,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 #### NOTE: For this demo, we use the Grid_demo.txt grid file. 
 #### The values in that file will override many of the defaults
 #### listed here, but in order to reproduce the example in Fig. 9

examples/methods_paper_plots/fig_5_HR_diagram/pythonSubmit.py

@@ -5,6 +5,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)

examples/methods_paper_plots/fig_6_max_R/pythonSubmit.py

@@ -5,6 +5,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)

examples/methods_paper_plots/fig_8_initial_core_final_mass_relations/pythonSubmitDefaults.py

@@ -5,6 +5,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)

examples/methods_paper_plots/fig_8_initial_core_final_mass_relations/pythonSubmitFryerRapid.py

@@ -5,6 +5,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)

examples/methods_paper_plots/fig_8_initial_core_final_mass_relations/pythonSubmitMandelMueller.py

@@ -5,6 +5,11 @@
 import ntpath
 from subprocess import call
 
+#### DISCLAIMER: This script uses the `pythonSubmit.py` format
+#### that has been replaced by the `runSubmit.py` and 
+#### `compasConfigDefault.yaml` combo as of v02.25.10.
+#### The `pythonSubmit.py` format will eventually become deprecated.
+
 # Check if we are using python 3
 python_version = sys.version_info[0]
 print("python_version =", python_version)