feat(dbt): Migrate dbt producer to PostgreSQL and validate CI/CD integration

[roller100]

Summary

This PR completes the PostgreSQL migration for the dbt producer compatibility tests and validates full GitHub Actions CI/CD integration.

Changes

1. PostgreSQL Migration

• ✅ Migrated from DuckDB to PostgreSQL 15 for dbt-ol compatibility
• ✅ Added docker-compose.yml for local PostgreSQL testing
• ✅ Updated profiles.yml, requirements.txt, and test scenarios
• ✅ Renamed scenario: csv_to_duckdb_local → csv_to_postgres_local

2. GitHub Actions Integration

• ✅ Added PostgreSQL 15 service container to workflow
• ✅ Fixed validation path (producer-dir: 'producer' instead of 'producer/dbt')
• ✅ Successfully tested with workflow_dispatch
• ✅ Matrix testing: dbt 1.8.0 with OpenLineage 1.23.0 and 1.39.0

3. Test Results

• ✅ 22 OpenLineage events generated successfully
• ✅ All dbt operations complete (seed, run, test)
• ✅ Core facets validate correctly (schema, dataSource, sql, columnLineage, etc.)
• ✅ GitHub Actions workflow passing

4. Documentation

• ✅ Updated README.md with local vs GitHub Actions testing workflows
• ✅ Enhanced SPECIFICATION_COVERAGE_ANALYSIS.md with PostgreSQL details
• ✅ Documented custom dbt facets and validation warnings
• ✅ Added cross-references between documentation files

Known Validation Warnings (Expected)

The dbt integration emits custom facets that generate expected validation warnings:

• dbt_version - Captures dbt-core version
• dbt_run - Captures dbt execution metadata

These are vendor-specific extensions to OpenLineage with valid schemas. The warnings indicate the validator doesn't recognize these custom facets, not that the events are invalid. This is documented and accepted behavior.

Workaround (Temporary)

• fail-for-new-failures temporarily disabled in main_pr.yml for feature branch testing
• Reason: Baseline from main branch has no dbt entries, so all dbt results are flagged as 'new failures'
• Resolution: Will be resolved when merged to main and baseline includes dbt results

Testing

Local Testing:

docker-compose up -d
python test_runner/cli.py run-scenario --scenario csv_to_postgres_local

GitHub Actions:

• Workflow run: https://github.com/BearingNode/openlineage-compatibility-tests/actions/runs/19471889344
• Result: ✅ All tests passed

Compliance

• ✅ All commits properly attributed: roller100 (BearingNode) contact@bearingnode.com
• ✅ Follows automated test standards
• ✅ Documentation comprehensive for contributors
• ✅ PostgreSQL setup validated locally and in CI/CD

Feedback Welcome

This PR represents our implementation of dbt producer compatibility testing meeting the repository's automated test standards. Any feedback on the approach, documentation, or test coverage is greatly appreciated!

[tnazarew]

Hi, thanks for the PR! I left some comments, mostly about cleanup because there are lot of files that I think should not be in the PR

[tnazarew]

why change the status of tests in here?

[tnazarew]

appreciate examples but we don't want to keep output events in the repository :)

[tnazarew]

I think that's old, since we migrated to Postgres

[tnazarew]

this file seems to be doing way too much stuff, the test should only be dbt execution with configured transport so we have events for validation step, no validation should actually happen here.

while in case of spark or hive we need to have some logic defined here, in case of dbt it could be enough to have shell script with some dbt-ol + argumets, we should only ensure that we produce ol events into some location that is available for, maybe we will need to split the events into separate files though

[tnazarew]

I see that you use validation script but there is also validation in the test scenario file, is there some reason for it?

[tnazarew]

jsonl?

[tnazarew]

I think something here is duplicated

[roller100]

Hi @tnazarew,

Thank you very much for your prompt and thorough review! We really appreciate the detailed feedback. Here's a summary of what we've addressed:

✅ Completed in Commit 3ea4557 (Address PR #186 review feedback)

Cleanup Items

1. test_output event files (#2541285905) - ✅ Deleted all 22 event output files, added .gitkeep
2. test.py removal (#2541315324) - ✅ Deleted entire test.py (was violating community pattern - dbt doesn't need test.py since cli.py handles execution)
3. docker-compose.yml - ✅ Removed (GitHub Actions provides PostgreSQL service container)
4. future/ directory (#2541348656) - ✅ Removed duplicate scripts, kept design docs for potential TSC discussion
5. generated-files/report.json (#2541272805) - ✅ Reverted manual changes (should be auto-generated by CI/CD)
6. dbt_producer_report.json - ✅ Deleted (should not be committed)

Pattern Alignment

• Duplicate validation (#2541319098) - ✅ Resolved by removing test.py; validation now only happens via validate_ol_events.py in CI/CD workflow
• Followed community pattern: test scenarios don't need test.py when CLI runner handles execution

✅ Completed in Commit 59d2f11 (Align with community conventions)

7. Scenario naming - ✅ Renamed csv_to_postgres_local → csv_to_postgres (aligned with Spark/Hive patterns)
8. README.md removal - ✅ Removed scenario README (community uses scenario.md only, verified against Hive/Spark scenarios)
9. Documentation updates (#2541289936) - ✅ Updated all docs to reflect CI/CD service container deployment, corrected PostgreSQL references

📊 Summary

• Files Changed: 10 files (6 deleted, 3 modified, 1 created)
• All 7 review comments: Addressed
• Pattern Compliance: Now aligned with Spark/Hive community conventions

📝 Clarification on .gitignore patterns

.gitignore jsonl pattern (#2541321601) - We're ignoring both .json and .jsonl extensions intentionally:

Why both formats?

1. OpenLineage file transport creates JSONL - When configured with type: file and append: true, the OpenLineage client writes events to a single file in JSONL (JSON Lines) format, where each event is one line (source)
2. Our CLI then splits into individual JSON files - The validation framework expects separate JSON files (one per event), so our CLI reads the JSONL file and splits it into event_001.json, event_002.json, etc.

Process flow:

dbt-ol → openlineage_events.jsonl (JSONL format, multiple events)
       → cli.py splits into → event_001.json, event_002.json, ... (individual JSON files)
       → validate_ol_events.py validates individual JSON files

The .gitignore patterns cover both intermediate (.jsonl) and final (.json) output files generated during local testing.

---

Let us know if anything else needs attention. Thanks again for the guidance!

[roller100]

Hi @tnazarew,

Hope you're doing well.

We believe we have addressed all 7 review comments in commits 3ea4557 and 59d2f11:

• ✅ Removed test output files, added .gitkeep
• ✅ Deleted test.py (eliminated duplicate validation)
• ✅ Removed docker-compose.yml (CI uses service container)
• ✅ Cleaned up future/ directory
• ✅ Reverted manual report.json changes
• ✅ Renamed scenario to csv_to_postgres
• ✅ Updated all PostgreSQL documentation

The PR has been ready for re-review for little while now. Would you please be able to take another look when you have a chance?

If there are any blockers or concerns we should be aware of, please let us know and we'll work to resolve them.

Thanks for your time and guidance on this.