Lead Software Engineer - Databricks

We have an opportunity to impact your career and provide an adventure where you

can push the limits of what's possible.

As a Lead Software Engineer-Databricks at JPMorgan Chase within our Corporate

Technology team, you are an integral part of an agile team that works to

enhance, build, and deliver trusted market-leading technology products in a

secure, stable, and scalable way. As a core technical contributor, you are

responsible for conducting critical technology solutions across multiple

technical areas within various business functions in support of the firm’s

business objectives.

Job

Responsibilities

• * Lead the architecture and delivery of high-throughput, low-latency data
• pipelines on Databricks using Apache Spark (Core, SQL, Structured Streaming),
• driving performance, reliability, and scalability.
• * Establish and evolve Lakehouse patterns with Delta Lake (ACID transactions,
• schema evolution, time travel, Z-ordering, compaction) to ensure performant,
• maintainable data platforms at scale.
• * Own Databricks cluster strategy and configuration, including runtime
• selection, autoscaling, driver/executor sizing, Spark configurations, init
• scripts, cluster policies, pools, and instance profiles.
• * Orchestrate and automate pipelines and jobs using Databricks Workflows,
• integrating with AWS eventing and orchestration services as needed.
• * Design secure ingestion and transformation frameworks leveraging Databricks
• services, including Delta or unmanaged table design, ingestion task creation,
• and Airflow DAGs to produce trusted and refined datasets.
• * Enforce data quality, lineage, and governance using Unity Catalog and/or AWS
• Glue Catalog, embedding expectations and validation directly into pipelines.
• * Drive Spark and Databricks performance engineering and tuning (partitioning
• and file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory
• control, job right-sizing, and liquid clustering/partitioning keys) to
• optimize cost and throughput.
• * Build and maintain reusable libraries, frameworks, and APIs in Python and/or
• Java, ensuring strong unit, integration, and data validation test coverage.
• * Implement CI/CD for data projects using Git-based workflows, Terraform-based
• infrastructure deployments and environment promotion, and automated releases;
• champion engineering standards, code reviews, and enterprise-authorized
• AI-assisted engineering practices (e.g., code review/refactoring, test
• acceleration, and incident/root-cause analysis) with consistent validation
• (secure coding, peer review, automated testing) and reuse of proven patterns.
• * Drives team adoption of enterprise-authorized AI-assisted engineering
• practices within the work environment to improve code quality, delivery
• speed, and operational outcomes (e.g., AI-assisted code review/refactoring,
• test strategy acceleration, incident/root-cause analysis support), while
• establishing consistent validation standards (secure coding, peer review,
• automated testing) and promoting reuse of effective patterns across the team.
• * Applies knowledge of tools within the Software Development Life Cycle
• toolchain, including enterprise-authorized AI-assisted development and
• automation capabilities, to improve the value realized by automation.
• Required qualifications, capabilities, and skills:
• * Formal training or certification on software engineering concepts and 5+
• years applied experience.
• * Advanced experience in software engineering and data engineering, including
• significant production delivery with Apache Spark on Databricks and/or AWS
• EMR.
• * Advanced hands-on Databricks expertise across Delta Lake, Unity Catalog,
• Workflows, Repos/notebooks, and SQL Warehouses, including cluster
• configuration and optimization.
• * Proven ability to architect, build, and operate reliable ETL/ELT data
• pipelines (batch and streaming), including schema design/evolution, SLAs, and
• reliability engineering practices.
• * Deep Spark performance tuning skills, with experience diagnosing bottlenecks
• and optimizing jobs for scalability, cost, and runtime efficiency.
• * Strong programming proficiency in Python and/or Java for data processing,
• platform tooling, and automation.
• * Strong SQL and analytics data modeling expertise, including dimensional/star
• schema design and Lakehouse best practices.
• * Demonstrated experience leading effective use of approved AI-assisted
• software development tools (coding, code review, test acceleration,
• troubleshooting), including setting team expectations and validation
• standards for correctness, performance, and security of AI outputs.
• * Strong responsible-AI and security-first engineering mindset, including data
• sensitivity awareness, secure handling of inputs/outputs, roles/instance
• profiles, secrets management, encryption at rest/in transit, network
• controls, and adherence to resiliency and security expectations; experience
• coaching teams on safe, compliant adoption within delivery practices.
• * Demonstrated experience leading effective use of approved AI-assisted
• software development tools (e.g., for coding, code review, test acceleration,
• troubleshooting) with the ability to set team expectations for validating AI
• outputs for correctness, performance, and security.
• * Strong understanding of responsible AI use in engineering workflows,
• including data sensitivity considerations, secure handling of inputs/outputs,
• and adherence to resiliency and security expectations; experience coaching
• engineers on safe, compliant adoption within delivery practices.
• Preferred qualifications, capabilities, and skills:
• * Experience with Delta Live Tables and advanced governance (catalogs, grants,
• auditing) in Databricks.
• * AWS networking knowledge (VPC, subnets, routing, security groups) and data
• egress controls.
• * Experience with Terraform for Infra deployments
• * Cost optimization experience: autoscaling strategies, spot vs on-demand,
• auto-termination, storage layouts and compaction.
• * Familiarity with Airflow, Genie, Streamlit and React
• * Observability for data systems (freshness/completeness metrics, lineage,
• SLAs, alerting).
• * Demonstrated leadership in code quality, reviews, testing strategy, CI/CD,
• and technical mentorship; excellent communication with stakeholders.
• JPMorganChase, one of the oldest financial institutions, offers innovative
• financial solutions to millions of consumers, small businesses and many of the
• world’s most prominent corporate, institutional and government clients under the
• J.P. Morgan and Chase brands. Our history spans over 200 years and today we are
• a leader in investment banking, consumer and small business banking, commercial
• banking, financial transaction processing and asset management.
• We offer a competitive total rewards package including base salary determined
• based on the role, experience, skill set and location. Those in eligible roles
• may receive commission-based pay and/or discretionary incentive compensation,
• paid in the form of cash and/or forfeitable equity, awarded in recognition of
• individual achievements and contributions. We also offer a range of benefits and
• programs to meet employee needs, based on eligibility. These benefits include
• comprehensive health care coverage, on-site health and wellness centers, a
• retirement savings plan, backup childcare, tuition reimbursement, mental health
• support, financial coaching and more. Additional details about total
• compensation and benefits will be provided during the hiring process.
• We recognize that our people are our strength and the diverse talents they bring
• to our global workforce are directly linked to our success. We are an equal
• opportunity employer and place a high value on diversity and inclusion at our
• company. We do not discriminate on the basis of any protected attribute,
• including race, religion, color, national origin, gender, sexual orientation,
• gender identity, gender expression, age, marital or veteran status, pregnancy or
• disability, or any other basis protected under applicable law. We also make
• reasonable accommodations for applicants’ and employees’ religious practices and
• beliefs, as well as mental health or physical disability needs. Visit our FAQs
• [https://careers.jpmorgan.com/us/en/how-we-hire/faqs] for more information about
• requesting an accommodation.
• JPMorgan Chase & Co. is an Equal Opportunity Employer, including
• Disability/Veterans
• Our Corporate Technology team relies on smart, driven people like you to develop
• applications and provide tech support for all our corporate functions across our
• network. Your efforts will touch lives all over the financial spectrum and
• across all our divisions: Global Finance, Corporate Treasury, Risk Management,
• Human Resources, Compliance, Legal, and within the Corporate Administrative
• Office. You’ll be part of a team specifically built to meet and exceed our
• evolving technology needs, as well as our technology controls agenda.