About this opportunity
Sharp Decisions lists this Senior Databricks Engineer opportunity in jersey city, New Jersey. Review the employer’s description below for duties, qualifications and application requirements.
Job description
Jersey City, NJ — Hybrid (3 Days Onsite)
Contract to Hire
W2 Only — No Third Parties 1
8+ Years Experience
Azure Databricks, Python, PySpark, SQL
This role is responsible for designing, developing, and deploying cloud data solutions as part of a broader cyber data initiative focused on building out a security and risk data platform for Information Security. The engineer will develop data feeds and contribute to the buildout of a Cybersecurity Data Lake — centralizing data and establishing effective data governance around data sources and lineage using Databricks. This individual will collaborate with developers, data owners, governance leads, and business analysts across IT and aligned application stakeholders.
? Critical Requirements
Financial services or similarly regulated environment experience strongly preferred
Responsibilities
Design, develop, test, and support cloud data solutions focusing on data ingestion, data quality, tuning, and performance of upstream cybersecurity data sources using Databricks
Provide indirect leadership and mentorship to junior team members; lead data feed development efforts and design initiatives
Participate in development meetings to align priorities and objectives, assign tasks, and share experiences across applications under development
Consult with technology and development teams to coordinate integration of applications within the broader software ecosystem
Capture and document metadata for Key Data Elements (KDEs) to ensure accuracy and completeness for Data Quality (DQ) rules and daily dataset processing
Work with the data architecture team to align KDEs to logical data models, develop physical data structures, and document physical data names, definitions, and types
Partner with data owners and stakeholders to create technical requirements and DQ rules around data elements needed in the data warehouse
Partner with Business Management and Data Owners to identify critical metrics and data fields required for metric dashboards
Establish views to encapsulate data for downstream consumption and ensure alignment with DQ rules established on metadata
Utilize IBM TWS production job scheduling and adhere to standards around daily scheduling and batch monitoring of production jobs
Identify and resolve data quality issues including inaccuracies and incomplete information; implement improved procedures and processes
Qualifications & Skills
Azure & Cloud Platform
Azure Databricks & Unity Catalog
Azure Data Factory
Azure Data Lake Storage
Azure Functions
Azure Event Grid
Azure Log Analytics & Monitor
Development & Data Engineering
Python / Java
SQL Server, Oracle / PL-SQL, T-SQL
ETL / ELT
REST API & Microservices
CI/CD — Git, Jenkins, Azure DevOps
Collibra DQIM/DQE, IBM Infosphere DA
Security Domain & Compliance
Strong knowledge of enterprise Information Security data domains: Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, and Threat Detection
Data knowledge of PaaS/SaaS products including ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, and Azure Active Directory
Understanding of information security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA)
? Nice to Have
Knowledge of Microsoft Sentinel for SIEM
Knowledge of IBM Tivoli Workload Scheduler
Exposure to Power BI for data visualization and reporting
#J-18808-Ljbffr
Worksite address
jersey city, NJ, 07390, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.