Data Engineer

Requirements

• Develop, build, optimize and implement pipeline, data ingestion, storage, wrangling, cataloguing, quality, security features from various data sources on cloud
• Monitor and ensure the smooth operation of data pipelines between OneTrust and external Club platforms (ESPs, CRMs, etc.).
• Automate jobs (ingestion & pipelines), notifications and reports
• Assist in the ongoing data integration tasks (e.g., connectors, APIs) between OT and various business systems.
• Write quality code and participate in code reviews
• Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD
• Continuously improve systems through performance enhancements and cost reductions in compute and storage
• Prioritize to manage ad-hoc requests in parallel with ongoing sprints
• Participate with the team to execute sound solutions and approaches to meet business expectations in an efficient manner; deliver and execute the technical and product roadmap for data engineering across a variety of technologies.
• Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
• Experience working in AWS, Redshift, Glue and basic API integrations
• Prior exposure to Data Ingestion and Curation work (such as working with Data Lakehouse)
• Knowledge in SQL for purpose of data analysis/investigation