Sr. Bigdata Engineer

UST · Hyderabad, Telangana, India · Full Time · Onsite

Posted

Role Description UST Tittle: Lead I - Data Engineering Who We Are At UST, we help the world’s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact—touching billions of lives in the process. Visit us at . Summary UST is looking for Sr. Big Data Engineer The Opportunity Hands-on senior engineer and technical lead who has personally built and shipped large-scale, production-grade ETL pipelines on AWS. The resource will architect and deliver batch/streaming pipelines, optimize cost and performance at scale, and set the team's engineering standards. Required Minimum 10 years in data engineering / big data is mandatory. Mandatory Qualifications Production experience (not POCs) required in all of: - AWS Glue — Glue ETL jobs, Data Catalog, crawlers, job bookmarks, dynamic frames; tuning for cost/runtime. - Amazon EMR — Cluster lifecycle (transient, EMR Serverless / on EKS), Spark tuning, instance fleets / spot. - Py Spark — Data Frame APIs, UDFs, partitioning, join strategies, skew handling, and Spark memory/executor tuning. - AWS Step Functions — Orchestrating multi-stage ETL with retries, error handling, and parallel/map states. - Production ETL delivery — Delivered ETL at scale with data quality controls, schema evolution, idempotency, and backfills. Core Skills Apache Spark internals; Parquet/ORC/Avro and Iceberg/Hudi/Delta; partitioning and small-file management - S3 data lake, Athena, Redshift/Spectrum, Lake Formation; Lambda, SQS/SNS, Event Bridge; Kinesis/MSK (plus) Expert Python and strong SQL; IAC (Terraform / CDK / Cloud Formation); CI/CD; Git; Docker - Dimensional modelling, Lakehouse / medallion architectures, batch and CDC ingestion Production-Readiness Expectations - Reliability (retries, checkpointing, safe re-runs, SLAs/SLOs) Observability (Cloud Watch/Datadog metrics, logging, lineage) Data quality (validation frameworks, late/malformed data handling) Cost and performance optimization at multi-terabyte scale Security/compliance (PII, encryption, masking, access governance) Operational ownership (on-call, incident response, RCA) Leadership Mentor engineers, run design/code reviews, and lead architecture decisions. Partner with data scientists, analysts, and product to deliver multi-quarter initiatives. Nice To Have AWS certifications; streaming (Kafka/Kinesis/Structured Streaming/Flink); Airflow/MWAA; dbt; regulated-domain experience (financial / credit / rental data). Education Bachelor's or Master's in CS/Engineering or equivalent practical experience. What We Believe We’re proud to embrace the same values that have shaped UST since the beginning. Since day one, we’ve been building enduring relationships and a culture of integrity. And today, it's those same values that are inspiring us to encourage innovation from everyone, to champion diversity and inclusion and to place people at the centre of everything we do. Humility We will listen, learn, be empathetic and help selflessly in our interactions with everyone. Humanity Through business, we will better the lives of those less fortunate than ourselves. Integrity We honour our commitments and act with responsibility in all our relationships. Equal Employment Opportunity Statement UST is an Equal Opportunity Employer. We believe that no one should be discriminated against because of their differences, such as age, disability, ethnicity, gender, gender identity and expression, religion, or sexual orientation. All employment decisions shall be made without regard to age, race, creed, colour, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law. UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance. To support and promote the values of UST. Comply with all Company policies and procedures Skills AWS, AWS CDK, AWS Cloud Formation, AWS Glue, AWS Step Functions, Amazon Athena, Amazon Cloud Watch, Amazon EMR, Amazon Event Bridge, Amazon SNS, Amazon S3

Apply on UST's site