About this opportunity
Compunnel, Inc. lists this Data Engineer opportunity in san francisco, California. Review the employer’s description below for duties, qualifications and application requirements.
Job description
The Data Engineer will be responsible for collecting, parsing, managing, analyzing, and visualizing large datasets to transform information into actionable insights. The role involves building scalable, repeatable, and secure data pipelines across various platforms. The engineer will ensure that data workflows support multiple users while maintaining high performance, security, and reliability.
Key Responsibilities
Design, develop, and maintain robust and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted, and quality data into the Common Data Platform.
Actively participate in Agile ceremonies and follow Scaled Agile processes defined by the CDP Program team.
Deliver high-quality data products and services following SAFe Agile practices.
Identify and resolve issues related to data pipelines and analytical data stores.
Implement monitoring and alerting for pipelines and data stores, enabling auto-remediation to ensure uptime and reliability.
Apply a security-first approach with strong testing and automation practices.
Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data needs and provide tools and infrastructure.
Stay updated with modern data engineering trends and evaluate new tools, frameworks, and technologies to improve efficiency.
Required Skills
Bachelor’s degree in computer science, Information Systems, or related field, or equivalent experience.
2+ years’ experience with Databricks, Collibra, and Starburst.
3+ years’ experience with Python and PySpark.
Experience using Jupyter notebooks for coding and unit testing.
Strong recent experience with relational and NoSQL data stores, STAR and Dimensional Modeling methods.
2+ years’ experience with a modern data stack (S3, Spark, Airflow, Lakehouse architecture, real‑time databases).
Experience with cloud data warehouses such as RedShift or Snowflake.
Overall data engineering experience across traditional ETL and Big Data (on‑prem or cloud).
Experience building end‑to‑end pipelines for unstructured and semi‑structured data using Spark.
Ability to work with confidential supervisory information (Must meet Protected Individual requirements).
Education
Bachelor’s degree in computer science, Information Systems, or related field (or equivalent experience).
#J-18808-Ljbffr
Worksite address
san francisco, CA, 94199, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.