Skip to content

Conversation

@chrowe
Copy link

@chrowe chrowe commented Mar 13, 2025

This PR contains several changes that I made to get things working on my M1 Mac.

The main thing was ignoring .DS_Store files that MacOS creates that were blocking the ingestor from finding bucket folders to import.

Other changes

  1. Updated healthcheck to use ${POSTGRES_USER} instead of postgres because I was getting repeated "User not found errors".
  2. Add .csv.gz as a file extention since that is how AWS generates csv files by default and Pandas can read them, so this avoids having to extract them.
  3. Use arm64 Docker images to improve performance
  4. Implement Dask for the ingestor to speed up large imports. I was trying to ingest ~64 million rows, so it was taking a really long time without this.
  5. Git ignore everything in user_input_data so I can store other data there while developing.
  6. Comment out ingest_inventory_files() after the import finished while trying to get refresh_materialized_views to finish running. Still have not gotten it to work for the large bucket I am trying to visualize because I am running out of disk space.

@chrowe chrowe changed the title Macos-fixes MacOS fixes + more Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant