Deskripsi Pekerjaan
Are you a seasoned Data Engineer looking to elevate your career with a dynamic, forward-thinking organization? Vault Outsourcing is expanding its elite data team and we are seeking a highly skilled Databricks Data Engineer to join our remote operations. This is a unique opportunity to work on complex data architectures while enjoying the convenience of an Australian Dayshift schedule.
At Vault Outsourcing, we pride ourselves on being more than just a service provider; we are a strategic partner for global clients. As our new Data Engineer, you will play a pivotal role in designing, implementing, and maintaining robust data pipelines, leveraging the full power of the Databricks Unified Data Analytics Platform. If you are passionate about cloud-native engineering, data lakehouse architectures, and performance optimization, we want to hear from you.
Tanggung Jawab
- Design, develop, and optimize scalable ETL/ELT pipelines using Databricks and Apache Spark.
- Build and maintain high-performance data lakehouse architectures to support advanced analytics and reporting.
- Collaborate with cross-functional teams to integrate disparate data sources into a centralized environment.
- Perform comprehensive performance tuning and troubleshooting for Spark jobs and SQL queries.
- Implement data governance, security, and quality controls within the Databricks environment.
- Work closely with Data Scientists to operationalize machine learning models and data sets.
- Provide technical leadership and mentor junior engineers on best practices for cloud data engineering.
Kualifikasi
- Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field.
- Minimum of 3-5 years of experience in data engineering, with a strong focus on Databricks and Apache Spark.
- Expertise in Python, PySpark, and SQL for complex data manipulation.
- Proven experience working with cloud platforms (AWS, Azure, or GCP).
- Strong understanding of Data Modeling, Data Warehousing, and Lakehouse concepts (Delta Lake).
- Familiarity with CI/CD pipelines and version control systems like Git.
- Excellent communication skills with the ability to thrive in a remote, Australian-timezone aligned team.
- Databricks Certified Professional certification is highly preferred.