About this opportunity
Compunnel, Inc. lists this Data Engineer - III opportunity in san francisco, California. Review the employer’s description below for duties, qualifications and application requirements.
Job description
We are seeking a Data Engineer – III to design, build, and maintain scalable, secure, and repeatable data pipelines across multiple platforms.
This role involves working with large datasets to transform raw data into actionable insights and collaborating with cross-functional teams to support data-driven decision-making.
Key Responsibilities
Design, develop, and maintain robust data pipelines to ingest, transform, catalog, and deliver curated, trusted data from diverse sources.
Participate in Agile ceremonies and follow Scaled Agile practices as defined by the program team.
Deliver high-quality data products and services aligned with Safe Agile principles.
Proactively identify and resolve issues related to data pipelines and analytical data stores.
Implement monitoring and alerting for data pipelines and stores, including auto-remediation strategies.
Apply a security-first, test-driven, and automation-focused approach to data engineering.
Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data needs and deliver appropriate infrastructure and tools.
Stay current with emerging technologies and recommend tools and frameworks to enhance data engineering processes.
Required Qualifications
Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent experience.
3+ years of experience with Python and PySpark.
2+ years of experience with Databricks, Collibra, and Starburst.
Experience using Jupyter notebooks for coding and unit testing.
Hands-on experience with relational and NoSQL databases, including STAR and dimensional modeling.
2+ years of experience with modern data stacks (e.g., S3, Spark, Airflow, Lakehouse architectures, real-time databases).
Experience with cloud data warehouses such as Redshift or Snowflake.
Strong background in traditional ETL and Big Data engineering, both on-premises and in the cloud.
Experience building end-to-end data pipelines for unstructured and semi-structured data using Spark.
Data engineering experience in AWS, including familiarity with relevant services and tools.
Preferred Qualifications
Experience working in secure environments with access to confidential supervisory information.
Familiarity with compliance and data governance requirements in regulated industries.
#J-18808-Ljbffr
Worksite address
san francisco, CA, 94199, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.