Add automated database schema diagram generation#187
Conversation
Co-authored-by: DevSecNinja <14926452+DevSecNinja@users.noreply.github.com>
Co-authored-by: DevSecNinja <14926452+DevSecNinja@users.noreply.github.com>
Co-authored-by: DevSecNinja <14926452+DevSecNinja@users.noreply.github.com>
DevSecNinja
requested review from
Copilot
and removed request for
DevSecNinja
github-actions
bot
added
documentation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #187 +/- ##
=======================================
Coverage 71.82% 71.82%
=======================================
Files 11 11
Lines 802 802
Branches 124 124
=======================================
Hits 576 576
Misses 163 163
Partials 63 63
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds an automated GitHub Actions workflow to generate and publish a database schema diagram for the backend’s SQLAlchemy/SQLite models, improving visibility into the data model over time.
Changes:
- Introduces a new workflow to generate a PNG schema diagram via SchemaCrawler and upload it as an artifact (and currently also commit it to the repo).
- Adds a helper script to generate a temporary SQLite DB from SQLAlchemy models for SchemaCrawler input.
- Updates architecture documentation to reference the automated diagram.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
ARCHITECTURE.md |
Documents where the auto-generated schema diagram can be found. |
.github/workflows/schema-diagram.yml |
New workflow to build a sample DB, generate the diagram, upload artifact, and commit the PNG. |
.github/scripts/create-sample-db.sh |
New script that installs backend deps and creates a SQLite DB from SQLAlchemy metadata. |
.github/workflows/schema-diagram.yml
Outdated
| - name: Commit schema diagram to repository | ||
| run: | | ||
| git config --local user.email \ | ||
| "github-actions[bot]@users.noreply.github.com" | ||
| git config --local user.name "github-actions[bot]" | ||
| git add schema-diagram.png | ||
| git diff --staged --quiet || \ | ||
| git commit -m "Update database schema diagram [skip ci]" | ||
| git push |
There was a problem hiding this comment.
This workflow adds an inline multi-line run: | block for the git commit/push step. Repo guidance says multi-line workflow scripts must be extracted into an executable under .github/scripts/ (see .github/copilot-instructions.md:158-164). Move this logic into a dedicated script (e.g., .github/scripts/commit-schema-diagram.sh) and call it with a single-line run:.
.github/workflows/schema-diagram.yml
Outdated
| git add schema-diagram.png | ||
| git diff --staged --quiet || \ | ||
| git commit -m "Update database schema diagram [skip ci]" | ||
| git push |
There was a problem hiding this comment.
The workflow pushes an extra commit back to main, which will trigger all other push-based workflows again (e.g., .github/workflows/docker-test.yml runs on every push to main). The [skip ci] marker in the commit message is not used anywhere else in this repo’s workflows, so it won’t prevent those runs. Consider removing the auto-commit (rely on the artifact), or changing the approach (e.g., open a PR / commit to a separate branch) to avoid cascading CI executions on every model change.
There was a problem hiding this comment.
Removed the auto-commit step. The workflow now only uploads the diagram as an artifact (90-day retention) to avoid triggering cascading CI executions. Changed permissions to contents: read and updated documentation accordingly. (commit cd62a6e)
|
|
||
| # Change to backend directory | ||
| cd "$BACKEND_DIR" | ||
|
|
||
| # Activate virtual environment if it exists, otherwise create it | ||
| if [ ! -d "venv" ]; then | ||
| echo "Creating virtual environment..." | ||
| python3 -m venv venv | ||
| fi | ||
|
|
||
| source venv/bin/activate | ||
|
|
||
| # Install dependencies | ||
| echo "Installing dependencies..." | ||
| pip install -q -r requirements.txt | ||
|
|
There was a problem hiding this comment.
This script creates/activates a backend/venv and installs dependencies on every run. In CI this adds time and duplicates the existing .github/scripts/install-backend-deps.sh pattern used elsewhere. Consider reusing the existing dependency-install script (and/or avoid creating a venv inside the repo) to keep workflows consistent and faster.
| # Change to backend directory | |
| cd "$BACKEND_DIR" | |
| # Activate virtual environment if it exists, otherwise create it | |
| if [ ! -d "venv" ]; then | |
| echo "Creating virtual environment..." | |
| python3 -m venv venv | |
| fi | |
| source venv/bin/activate | |
| # Install dependencies | |
| echo "Installing dependencies..." | |
| pip install -q -r requirements.txt | |
| INSTALL_SCRIPT="$PROJECT_ROOT/.github/scripts/install-backend-deps.sh" | |
| # Install backend dependencies using the shared script when available | |
| if [ -x "$INSTALL_SCRIPT" ]; then | |
| echo "Using shared backend dependency install script..." | |
| (cd "$PROJECT_ROOT" && "$INSTALL_SCRIPT") | |
| # Activate virtual environment if it was created by the install script | |
| cd "$BACKEND_DIR" | |
| if [ -d "venv" ]; then | |
| # shellcheck disable=SC1091 | |
| source venv/bin/activate | |
| fi | |
| else | |
| echo "Shared install script not found; falling back to local venv setup..." | |
| cd "$BACKEND_DIR" | |
| # Activate virtual environment if it exists, otherwise create it | |
| if [ ! -d "venv" ]; then | |
| echo "Creating virtual environment..." | |
| python3 -m venv venv | |
| fi | |
| # shellcheck disable=SC1091 | |
| source venv/bin/activate | |
| # Install dependencies (only when using the local fallback) | |
| echo "Installing dependencies..." | |
| pip install -q -r requirements.txt | |
| fi |
There was a problem hiding this comment.
Updated to reuse the existing .github/scripts/install-backend-deps.sh script instead of duplicating the dependency installation logic. The script now calls the shared install script which is consistent with other workflows. (commit cd62a6e)
.github/scripts/create-sample-db.sh
Outdated
| python3 -m venv venv | ||
| fi | ||
|
|
||
| source venv/bin/activate | ||
|
|
||
| # Install dependencies | ||
| echo "Installing dependencies..." | ||
| pip install -q -r requirements.txt |
There was a problem hiding this comment.
The script hardcodes python3 and pip, which can bypass the interpreter configured by actions/setup-python (the repo’s other CI scripts use python -m pip ...). Use the runner-selected python (and python -m pip) for consistency and to avoid inadvertently mixing Python versions/environments.
| python3 -m venv venv | |
| fi | |
| source venv/bin/activate | |
| # Install dependencies | |
| echo "Installing dependencies..." | |
| pip install -q -r requirements.txt | |
| python -m venv venv | |
| fi | |
| source venv/bin/activate | |
| # Install dependencies | |
| echo "Installing dependencies..." | |
| python -m pip install -q -r requirements.txt |
There was a problem hiding this comment.
Changed to use python instead of python3 for consistency with the interpreter configured by actions/setup-python. (commit cd62a6e)
|
@copilot Fix the feedback please |
…thon Co-authored-by: DevSecNinja <14926452+DevSecNinja@users.noreply.github.com>
All feedback addressed in commit cd62a6e:
|
|
@copilot Also fix the failing pipeline runs. See linking errors. |
| # yamllint disable-line rule:truthy | ||
| on: | ||
| workflow_dispatch: null |
There was a problem hiding this comment.
The yamllint suppression comment is on its own line, but disable-line only applies to the line it’s on. In this repo other workflows use on: # yamllint disable-line rule:truthy, so this file may fail YAML linting. Move the comment onto the on: line (or use an appropriate disable-next-line directive) to match the existing pattern.
| # Create a temporary Python script to generate the database | ||
| cat > /tmp/create_db.py << 'EOF' | ||
| """Generate sample SQLite database from SQLAlchemy models.""" | ||
| from app.models.database import Base | ||
| from sqlalchemy import create_engine | ||
|
|
||
| # Create engine and database |
There was a problem hiding this comment.
This script writes to a predictable path /tmp/create_db.py. Using a fixed temp filename can fail if the file already exists and is generally unsafe; prefer mktemp (and a trap cleanup) to create a unique temp file and ensure it’s removed even on errors.
| from app.models.database import Base | ||
| from sqlalchemy import create_engine | ||
|
|
||
| # Create engine and database | ||
| engine = create_engine('sqlite:///sample_schema.db') | ||
| Base.metadata.create_all(engine) | ||
|
|
||
| print("Sample database created successfully at backend/sample_schema.db") |
There was a problem hiding this comment.
If backend/sample_schema.db already exists, Base.metadata.create_all() won’t remove/modify existing tables/columns, which can produce a stale schema diagram on reruns. Remove any existing DB file before creating it (or explicitly create it at a fresh path) so the diagram always reflects the current models.
| from app.models.database import Base | |
| from sqlalchemy import create_engine | |
| # Create engine and database | |
| engine = create_engine('sqlite:///sample_schema.db') | |
| Base.metadata.create_all(engine) | |
| print("Sample database created successfully at backend/sample_schema.db") | |
| from pathlib import Path | |
| from sqlalchemy import create_engine | |
| from app.models.database import Base | |
| # Determine database path in current working directory (backend/) | |
| db_path = Path("sample_schema.db") | |
| # Remove existing database file to avoid stale schemas | |
| if db_path.exists(): | |
| db_path.unlink() | |
| # Create engine and database | |
| engine = create_engine(f"sqlite:///{db_path}") | |
| Base.metadata.create_all(engine) | |
| print(f"Sample database created successfully at {db_path.resolve()}") |
Co-authored-by: DevSecNinja <14926452+DevSecNinja@users.noreply.github.com>
Schema Diagram GitHub Actions Workflow
.github/workflows/schema-diagram.ymlworkflow file.github/scripts/create-sample-db.shto generate databaseinstall-backend-deps.shscript instead of duplicating logicpythoninstead ofpython3for consistency with setup-pythoncontents: read(no longer need write)@v17.6.0) for SchemaCrawler-ActionImplementation Complete
The schema diagram workflow is ready to run on GitHub. It will:
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.