Senior Data Engineer - Jobs in Fredericton

Job Description

Job DescriptionThe Senior Data Engineer will serve as a liaison between the Appcast core development team and downstream analysts on the Data and Insights team. The senior data engineer sets up the data engineering infrastructure, using the tools of your choice. From the architecture you establish, you will then code automated processes that extract data from disparate sources; efficiently build and manage transformations into a cloud data warehouse; and build pipelines that load data into tools used by economists, data scientist, data analysts, and many other stakeholders. SQL and Python/R skills are a must. As is extensive experience managing data warehouses and ELT/ETL processes in an OLAP database.Job ResponsibilitiesMove data between systems:

• Extraction – Appcast has several disparate internal proprietary, external proprietary and external public sources that all serve our customers in several ways. A senior data engineer’s primary job is to build automated processes that extract data from these disparate sources for downstream use by data analysts.
• Transform – The senior data engineer will transform data from disparate sources into distinct tables using a relational data model, so that analysts can pull them together in one unified workflow.
• Load – The senior data engineer will need to load the new tables into a cloud data warehouse using efficient, low-latency methods.

Manage cloud data warehouse:

• Monitor performance – As data needs grow, the senior data engineer will need to monitor essential performance queries used by downstream analysts to ensure data is being delivered in a timely manner that does not cause slowdowns.
• Data quality – Ensuring both internal and external data sources are accurate and up to date to provide the best quality for our clients and media partners.

Serve data to downstream users (“democratize data”):

• Liaison with the Appcast core software development team on data extraction from its proprietary database.
• Data visualization/dashboards - Partner with data analysts to ensure BI tools (e.g., Tableau) are accurate and up to date.
• Data permissions/access - Manage the onboarding and offboarding of data permissions, who is given access to the cloud data warehouse and ensuring proper security protocols are being followed.

Set data strategy for company:

• Control cost – Work with procurement specialists to keep negotiated contracts controlled with external data vendors; research/keep up to date with new data vendors that can provide Appcast with a competitive advantage in our competitive market
• Data awareness/education - Educate analysts on basic data “hygiene” on how to manage and clean data, and where to access data sources
• Data security and governance – Develop a deep understanding of Appcast data, work with internal stakeholders to decide on what data is proper to share with clients/media due to sample size and quality
• Pioneer the adoption of sophisticated ETL/ELT frameworks and platforms such as Apache Airflow and DBT to enhance data workflow automation.
• Serve as the primary senior data engineering expert, advising senior management and other internal stakeholders on technological advancements and strategic investments in data infrastructure.