Principal Research Scientist - AI Scaling & Optimization

PRINCIPAL RESEARCH SCIENTIST – AI SCALING & OPTIMIZATION

P-1227

ABOUT DATABRICKS AI

At Databricks, we are obsessed with enabling data teams to solve the world’s

toughest problems, from security threat detection to cancer drug development, by

building and running the world’s best data and AI platform. The Databricks AI

Research organization enables companies to develop AI models and agents using

their own data, with technologies ranging from post-training open source LLMs to

developing advanced multi-agent architectures. Databricks AI is committed to the

belief that a company’s AI models and agents are just as valuable as any other

core IP, and that high-quality AI should be available to all.

ABOUT THE SCALING RESEARCH TEAM

The Databricks AI Scaling team focuses on pushing the boundaries of large

language model (LLM) training and inference efficiency beyond what is required

to support existing models. The team explores novel avenues for scaling and

efficiency improvements across algorithms, systems, and infrastructure,

requiring researchers who can both drive independent research agendas and dive

deep into low‑level implementation details with engineering partners.

ROLE SUMMARY

As a Principal Research Scientist – Scaling, you will lead a team of world‑class

researchers and engineers to advance the state of the art in large‑scale machine

learning, focusing on post-training, RL and inference efficiency, optimization,

and scaling. You will define and execute a research roadmap that advances the

Databricks AI platform and delivers tangible improvements to how customers

train, serve, and adapt LLMs at scale, working closely with product, data, and

engineering leaders to bring cutting‑edge methods into production.

THE IMPACT YOU WILL HAVE

* Lead and grow a multidisciplinary research team focused on foundational and

applied AI problems, with a particular emphasis on LLM scaling, efficiency,

and systems performance.

* Define the scaling research roadmap in alignment with Databricks’ strategic

objectives, prioritizing advances in foundation model efficiency and

large‑scale training and inference.

* Drive algorithmic innovations for large‑scale neural network training and

inference, including novel optimizers, low‑precision techniques, and model

adaptation methods, and guide your team in rigorous empirical validation

against state‑of‑the‑art approaches.

* Optimize end‑to‑end ML systems for distributed training and RL, memory

efficiency, and compute efficiency through close collaboration with core

systems and platform teams, ensuring that research ideas translate into

performant, reliable infrastructure.

* Partner with product and engineering to translate research breakthroughs,

especially around scaling and efficiency, into customer‑impacting

capabilities in the Databricks AI platform.

* Foster a culture of scientific excellence and openness, including

high‑quality research practices, reproducible experimentation, and effective

internal knowledge sharing across Databricks AI.

* Represent Databricks AI research externally through top‑tier publications,

conference talks, and collaborations with academia and the open‑source

community, with a focus on optimization and efficiency for large‑scale

models.

* Mentor and develop talent, providing both technical guidance (research

agendas, experimentation, implementation) and career development support for

research scientists and engineers.

WHAT YOU WILL DO

* Define and lead independent research programs on foundation model efficiency,

covering topics such as optimizer design, low‑precision training/inference,

scalable model architectures, and efficient adaptation methods.

* Oversee the design and execution of large‑scale experiments, including

benchmarking against state‑of‑the‑art methods and evaluating trade‑offs in

quality, latency, throughput, and cost.

* Work hands‑on with your team on high‑quality, efficient code in Python and

PyTorch for research implementation, rapid prototyping, and integration with

Databricks’ production systems.

* Collaborate with distributed systems and infra teams to push the limits of

distributed training, parallelism strategies, memory management, and hardware

utilization for LLMs and other large models.

* Establish metrics, evaluation protocols, and best practices for

scaling‑focused research (e.g., training efficiency, inference cost, energy

usage) and drive their adoption across Databricks AI.

* Champion responsible and robust deployment of scaling innovations, ensuring

that model behavior, reliability, and safety remain first‑class

considerations.

WHAT WE LOOK FOR

* Proven ability to lead a research team to develop novel techniques for

foundation model efficiency and related topics, with a strong track record of

industry impact.

* Deep expertise in at least one of: generative AI, LLMs, distributed ML

systems, model optimization, or responsible AI, with a strong emphasis on

scaling and efficiency for large‑scale neural networks.

* Hands on leadership - strong programming skills and demonstrated ability to

write high‑quality, efficient code in Python and PyTorch for research

implementation and experimentation.

* Demonstrated ability to translate research innovation into scalable product

capabilities in partnership with product and engineering teams.

* Excellent communication, leadership, and stakeholder management skills, with

experience influencing cross‑functional roadmaps and aligning research with

business impact.

NICE TO HAVE

* Prior work at the intersection of systems and ML, such as distributed

training frameworks, compiler and kernel optimization for deep learning

workloads, or memory‑/compute‑efficient model design.

* Strong industry and academic network in large‑scale ML, with ongoing

collaborations or service (e.g., PC/area chair) at top conferences in ML and

systems.

* A strong record of research impact—such as first‑author publications at top

ML/systems conferences (e.g., ICLR, ICML, NeurIPS, MLSys), influential

open‑source contributions, or widely used deployed systems—especially in

optimization or efficiency.

Pay Range Transparency

Databricks is committed to fair and equitable compensation practices. The pay

range(s) for this role is listed below and represents the expected salary range

for non-commissionable roles or on-target earnings for commissionable roles.

Actual compensation packages are based on several factors that are unique to

each candidate, including but not limited to job-related skills, depth of

experience, relevant certifications and training, and specific work location.

Based on the factors above, Databricks anticipates utilizing the full width of

the range. The total compensation package for this position may also include

eligibility for annual performance bonus, equity, and the benefits listed above.

For more information regarding which range your location is in visit our page

here

[https://www.databricks.com/sites/default/files/2024-08/us-pay-zone-mapping.pdf].

Local Pay Range

$270,000—$350,000 USD

About Databricks

Databricks is the data and AI company. More than 10,000 organizations worldwide

— including Comcast, Condé Nast, Grammarly, and over 50% of the Fortune 500 —

rely on the Databricks Data Intelligence Platform to unify and democratize data,

analytics and AI. Databricks is headquartered in San Francisco, with offices

around the globe and was founded by the original creators of Lakehouse, Apache

Spark™, Delta Lake and MLflow. To learn more, follow Databricks on Twitter

[https://twitter.com/databricks], LinkedIn

[https://www.linkedin.com/company/databricks] and Facebook

[https://www.facebook.com/databricksinc].

Benefits

At Databricks, we strive to provide comprehensive benefits and perks that meet

the needs of all of our employees. For specific details on the benefits offered

in your region click here

[https://docs.google.com/document/d/154un3e8Xav4BceOSlcYFZRGEuQI54xMxVydRwQn54eQ/edit? usp=sharing].

Our Commitment to Diversity and Inclusion

At Databricks, we are committed to fostering a diverse and inclusive culture

where everyone can excel. We take great care to ensure that our hiring practices

are inclusive and meet equal employment opportunity standards. Individuals

looking for employment at Databricks are considered without regard to age,

color, disability, ethnicity, family or marital status, gender identity or

expression, language, national origin, physical and mental ability, political

affiliation, race, religion, sexual orientation, socio-economic status, veteran

status, and other protected characteristics.

Compliance

If access to export-controlled technology or source code is required for

performance of job duties, it is within Employer's discretion whether to apply

for a U.S. government license for such positions, and Employer may decline to

proceed with an applicant on this basis alone.