About this opportunity
Suncap Technology lists this Collibra Data Lineage Automation Engineer / McLean, VA- 6+ months opportunity in mc lean, Virginia. Review the employer’s description below for duties, qualifications and application requirements.
Job description
Data Lineage Automation EngineerWe are seeking a highly experienced Data Lineage Automation Engineer to lead the design and implementation of automated end-to-end lineage solutions across a highly heterogeneous enterprise data ecosystem. This role requires deep technical expertise in lineage frameworks, experience across cloud and legacy environments, and a strong AI foundation to support intelligent metadata extraction and traceability.Key Responsibilities:Lead the implementation of automated data lineage across a complex data estate that includes:Cloud platforms (e.g., Snowflake, AWS)Legacy relational databases and ETLsNoSQL data storesBI/reporting platforms (e.g., Tableau, Power BI)Implement or extend frameworks such as Spline, OpenLineage, or similar open frameworks to support active lineage captureBuild connectors, extractors, or agents where necessary to bridge gaps between systems and lineage frameworksIntegrate with metadata platforms (e.g., Collibra) to publish lineage in a consumable formatApply AI/ML techniques to infer lineage where automation is incomplete (e.g., handling Java based ETLs), using logs, query patterns, or usage metadataDevelop reusable lineage components for operational reuse across domainsGuide stakeholders on best practices for lineage standardization, storage, and useRequired Skills & Experience:Proven experience delivering automated data lineage solutions across hybrid architecturesHands-on expertise with Spline, OpenLineage, Marquez, or comparable lineage frameworksDeep understanding of metadata capture, ETL process tracing, and query execution mappingStrong AI/ML background — particularly in metadata intelligence, natural language processing for code parsing, or pattern detectionExperience integrating lineage with data governance tools (e.g., Collibra, Alation, etc.)Strong programming background in Python, Scala, or JavaDeep familiarity with SQL and query logs from systems like Snowflake, SQL Server, Oracle, MongoDB, etc.Big Plus Skills:Experience with third-party commercial data lineage solutions a plus (evaluations and implementations)Prior work in regulated environments (e.g., financial services, healthcare)Familiarity with event-based architectures for real-time lineage propagationKnowledge of data mesh or domain-driven lineage strategiesIdeal Candidate:Has successfully implemented automated lineage at enterprise scaleOperates at the intersection of data engineering, metadata management, and AICan act as a technical thought partner to architecture teams and governance leadsBrings the mindset of automation-first and reuse-oriented design
Worksite address
mc lean, VA, 22101, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.