About this opportunity
Smart IT Frame LLC lists this Apache Iceberg Engineer opportunity in sunnyvale, California. Review the employer’s description below for duties, qualifications and application requirements.
Job description
We are looking for an experienced Apache Iceberg Engineer to design, develop, and optimize large-scale data lakehouse solutions leveraging Apache Iceberg. The ideal candidate will have expertise in big data processing frameworks (Apache Spark, Flink, Presto, Trino, Hive) and cloud-based data platforms like AWS S3, Google Cloud Storage, or Azure Data Lake Storage . You will work closely with data engineers, data scientists, and DevOps teams to ensure efficient, scalable, and reliable data architecture.
Key Responsibilities:
Design, implement, and optimize Iceberg-based data lake architectures for large-scale datasets.
Develop data ingestion, transformation, and query optimization pipelines using Spark, Flink, or Presto/Trino .
Ensure ACID compliance, schema evolution, and partition evolution in Iceberg tables.
Implement time travel, versioning, and snapshot management for historical data analysis.
Optimize metadata management and query performance in Iceberg-based data lakes .
Integrate Apache Iceberg with cloud storage solutions (AWS S3, GCS, ADLS ) and data warehouses.
Implement best practices for data governance, access control, and security within an Iceberg-based environment.
Troubleshoot performance issues, metadata inefficiencies, and schema inconsistencies in Iceberg tables.
Collaborate with DevOps, ML engineers, and BI teams to enable smooth data workflows.
Required Qualifications:
Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field .
3+ years of experience in Big Data, Data Engineering, or Cloud Data Warehousing .
Hands-on experience with Apache Iceberg in a production environment.
Strong expertise in Apache Spark, Flink, Trino, Presto, or Hive for big data processing.
Proficiency in SQL and distributed query engines .
Experience working with cloud storage solutions (AWS S3, GCS, ADLS) .
Knowledge of data lakehouse architectures and modern data management principles .
Familiarity with schema evolution, ACID transactions, and partitioning techniques .
Experience with Python, Scala, or Java for data processing.
Preferred Qualifications:
Experience in real-time data processing using Flink or Kafka.
Understanding of data governance, access control, and compliance frameworks .
Knowledge of other data lake frameworks like Delta Lake (Databricks) or Apache Hudi .
Hands-on experience with Terraform, Kubernetes, or Airflow for data pipeline automation.
Seniority level
Mid-Senior level
Employment type
Full-time
Job function
Other
Industries
Software Development and IT Services and IT Consulting
#J-18808-Ljbffr
Worksite address
sunnyvale, CA, 94087, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.