Snowflake Data Migration

About the Client

Culture machine is a social media analytics company focused on brand promotion and analytics. With a product called Intelligent machine. Having numerous giants like Mattel as their customers. Culture machine also owns numerous video blogging facebook pages & YouTube channels.

Business Challenges

  • Large amount of social networking history data
  • Connecting Amazon AWS source to Snowflake Cloud warehouse
  • Different Source file Formats to be handled and migrated to snowflake
  • Inconsistency in the source data and file nomenclature
  • Scheduling the daily jobs to migrate the data from Amazon AWS to Snowflake warehouse
  • Data validation of history data after the migration

Business Requirements

  • Perform History/Daily Data migration from heterogeneous sources like Amazon AWS, Postgres, Crawler output into Snowflake Cloud Data warehouse.
  • Build new data warehouse in Snowflake Cloud.
  • Create Scripts/Code for migration and Schedule and monitor the same on the Snowflake.
  • Validate the migration on the Snowflake Cloud Data warehouse.