Senior Engineer - IT

Guardian Life · Chennai, Tamil Nadu, India · Onsite

Posted

Job Description: Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks and Py Spark Build robust ETL/ELT workflows for batch and near real‑time data processing Write optimized and efficient SQL queries for data transformation, validation, and analysis Work with large-scale distributed datasets using Spark-based processing frameworks Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture Ensure data quality, integrity, and consistency through validations and monitoring Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.) Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions Troubleshoot pipeline failures and implement proactive monitoring and alerting Maintain proper documentation for data models, transformations, and workflows Required Skills & Experience Strong hands-on experience in: Databricks (Lakehouse platform, notebooks, jobs, clusters) Py Spark / Apache Spark Advanced SQL (joins, window functions, performance tuning) Solid understanding of ETL concepts, data warehousing, and data modeling Experience building scalable data pipelines for structured and semi-structured data Familiarity with data lake / delta lake architecture Good understanding of performance optimization techniques in Spark and SQL Experience with version control (Git) and deployment practices Strong problem-solving and analytical skills Good to Have Experience with cloud platforms (AWS preferably) Exposure to CI/CD pipelines (Jenkins) Knowledge of Delta Lake, Unity Catalog, or similar governance tools Understanding of data quality frameworks / monitoring tools Experience working in Agile / Scrum teams Location: This position can be based in any of the following locations: Chennai Current Guardian Colleagues: Please apply through the internal Jobs Hub in Workday

Apply on Guardian Life's site