Lead Data Engineer Full-time

Lead Data Engineer

The Lead Data Engineer for the UT Data Hub improves university outcomes and advances the UT mission to transform lives for the benefit of society by increasing the useability and value of institutional data. You will lead senior data engineers and data engineers to create complex data pipelines within UT's cloud data ecosystem in support of academic and administrative needs. In collaboration with our team of data professionals, you will help build and run a modern data hub to enable advanced data-driven decision making for UT. You will leverage your creativity to solve complex technical problems and build effective relationships through open communication within the team and outside partners.

Responsibilities

Your duties will include:

• Design, architect, and deliver production-grade, scalable data pipelines and AI-ready data platforms using Databricks, AWS cloud-native services and modern data engineering frameworks.
• Lead end-to-end implementation of lakehouse data pipelines, ensuring performance, reliability, and cost efficiency.
• Champion industry best practices for data engineering.
• Conduct and participate in peer code reviews to maintain code quality and consistency across the team.
• Proactively identify and resolve bottlenecks in data ingestion, transformation, and orchestration processes using Databricks Delta Live Tables, Spark optimization techniques, and workflow automation.
• Implement systems for data quality, observability, governance, and compliance using tools such as Unity Catalog, Delta Lake, and data validation frameworks.
• Lead technical knowledge-sharing sessions on topics such as AI/ML integration, data lakehouse architecture, and emerging data technologies.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent professional experience.
• 5+ years of experience designing, implementing, and optimizing complex, production-grade data pipelines or enterprise-scale data platforms.
• 5 years of experience in cloud-based data engineering using Databricks and Amazon Web Services (AWS) (e.g., Glue, S3, Lambda, Redshift).
• 3+ years of experience managing or leading teams of data and/or software engineers, including mentorship, performance management, and project delivery.
• Expertise in Python, PySpark, and SQL, with strong understanding of data modeling, stored procedures, and scalable data transformations.
• Proven experience architecting and implementing ETL/ELT solutions across relational, non-relational, and lakehouse environments (e.g., Delta Lake, Parquet, or Iceberg).
• Experience designing and managing CI/CD pipelines and infrastructure as code (IaC) using tools such as Databricks Repos, CDK, Terraform, or GitHub Actions.
• Demonstrated knowledge of test-driven development (TDD) and data quality frameworks, ensuring reliability and reproducibility across data workflows.
• Deep understanding of data governance, security, and compliance standards in cloud environments.
• Excellent analytical, problem-solving, and debugging skills across distributed data systems.
• Proven ability to communicate complex technical concepts clearly to both technical and non-technical audiences.
• Experience supervising, mentoring, and guiding junior team members on technical and professional development.

Salary Range

$125,000 – $143,712

Working Conditions

• May work around standard office conditions
• Repetitive use of a keyboard at a workstation
• Use of manual dexterity (ex: using a mouse)

Work Shift

• Monday – Friday 8am-5pm; Occasional nights or weekends may be required

Required Materials

• Resume/CV
• 3 work references with their contact information; at least one reference should be from a supervisor
• Letter of interest