generate_schema_name handling.

[jaredvivints]

Hi,

I've got a situation where I can't add the results of this package to a custom schema in Snowflake. We're using generate_schema_name to push custom folders into specific schemas. I for the life of me can't figure out how to make this work wanting the dbt_artifacts objects to go into our src_dbt_artifact schema.

Here is our generate_schema_name
{% macro generate_schema_name(custom_schema_name, node) -%}

{%- set default_schema = target.schema -%}
{%- if custom_schema_name is none -%}

    {{ default_schema }}

{%- else -%}

    {{ custom_schema_name | trim }}

{%- endif -%}

{%- endmacro %}

Here is our dbt_project.yml config with the models block.

models:
dp_dbt:
dbt_artifacts:
+database: "{{ target.name + '_DB' }}" # optional, default is your target database
+schema: src_dbt_artifact # optional, default is your target schema
staging:
+database: "{{ target.name + '_DB' }}" # optional, default is your target database
+schema: src_dbt_artifact # optional, default is your target schema
sources:
+database: "{{ target.name + '_DB' }}" # optional, default is your target database
+schema: src_dbt_artifact # optional, default is your target schema

Would love any thoughts. I know the docs say "Be aware of any impact that generate_database_name and generate_schema_name macros may have on the final database/schema."

I'm just wondering how to handle this scenario.

Thanks.

[T-Dunlap]

Hey @jaredvivints! I don't think you'll need to edit the generate_database_name or generate_schema_name macros... you should be able to handle this in the configs of the dbt_project.yml file. Did you define the variables section?

I think it should look something like this...

vars:
  dbt_artifacts:
    dbt_artifacts_database: "{{ target.name + '_DB' }}"  # optional, default is your target database
    dbt_artifacts_schema: src_dbt_artifact # optional, default is 'dbt_artifacts'
    dbt_artifacts_create_schema: true # optional, true|false set to false if you don't have privileges to create schema, default is true

models:

  dbt_artifacts:
    +schema: src_dbt_artifact 
    +database: "{{ target.name + '_DB' }}"
    staging:
      +schema: src_dbt_artifact

[jaredvivints]

Thanks for this.

I've updated our dbt_project.yml with the vars. We're still getting the data pushing to our default sf schema. Nothing changed.

[T-Dunlap]

@jaredvivints - Ahh, as we discussed a couple weeks ago, it looks like the package has changed since I last used it. Are you getting an error message, or is it running fine, just outputting results into the wrong schema? If you do have an error message, can you paste the logs here?
@jared-rimmer - I noticed that you've recently added to the code base. Do you know what the issue might be?

[jack-goslin]

This is happening to me as well. I set the schema to dbt_artifacts and it always prepends the target profile schema name to whatever schema name I specify in the dbt_project.yaml.

vars:
  distant_future: '9999-01-01 00:00:00.000'
  distant_past: '1900-01-01 00:00:00.000'
  'dbt_date:time_zone': 'America/Chicago'
  dbt_artifacts:
    dbt_artifacts_database: engineering
    dbt_artifacts_schema: dbt_artifacts

on-run-end:
  - "{{ dbt_artifacts.upload_results(results) }}"


models:
  dbt_artifacts:
    +database: engineering
    +schema: dbt_artifacts
    staging:
      +database: engineering
      +schema: dbt_artifacts

Partial output:

19:43:43  1 of 36 START sql incremental model engineering.staging_dbt_artifacts.exposures  [RUN]
19:43:43  2 of 36 START sql incremental model engineering.staging_dbt_artifacts.invocations  [RUN]
19:43:43  3 of 36 START sql incremental model engineering.staging_dbt_artifacts.model_executions  [RUN]
19:43:43  4 of 36 START sql incremental model engineering.staging_dbt_artifacts.models ... [RUN]
19:43:43  5 of 36 START sql incremental model engineering.staging_dbt_artifacts.seed_executions  [RUN]
19:43:43  6 of 36 START sql incremental model engineering.staging_dbt_artifacts.seeds .... [RUN]
19:43:43  7 of 36 START sql incremental model engineering.staging_dbt_artifacts.snapshot_executions  [RUN]
19:43:43  8 of 36 START sql incremental model engineering.staging_dbt_artifacts.snapshots  [RUN]

[jack-goslin]

It seems like it's using the default target name in the profiles.yaml and prepending that to the schema. Anyone know how to set the target just for the package?