Data Engineer

Clicksoft Solutions · Bengaluru, Karnataka, India · Full Time · Hybrid

Posted

Data Engineer | U S Staffing Position: Data Engineer Location: Bengaluru, Karnataka Experience: 3–6 Years Employment Type: Full-Time Work Mode: On-site / Hybrid Shift: Day Shift Industry: IT / Software Development Job Summary : We are looking for an experienced Data Engineer with strong hands-on expertise in building and maintaining scalable data pipelines and data processing solutions.The ideal candidate should have solid experience with Python, SQL, ETL/ELT, Apache Spark, Databricks, cloud platforms (AWS/Azure), and modern data warehousing technologies. The candidate will work closely with data analysts, data scientists, software engineers, and business teams to deliver reliable and high-quality data solutions. Key Responsibilities : Design, develop, and maintain scalable ETL/ELT data pipelines.Collect, transform, integrate, and process large volumes of structured and unstructured data.Develop data-processing solutions using Python, SQL, and Apache Spark/Py Spark.Build and optimize data pipelines using Databricks.Design and maintain data models, data warehouses, and data lake solutions.Develop complex SQL queries, stored procedures, and data transformations.Integrate data from databases, APIs, cloud storage, applications, and third-party sources.Perform data cleansing, validation, transformation, and quality checks.Monitor data pipelines and troubleshoot failures and performance issues.Optimize data-processing workloads for performance, scalability, and cost.Implement data quality, governance, security, and access-control standards.Work with cloud platforms such as AWS or Microsoft Azure.Develop and maintain workflow orchestration using Apache Airflow / Azure Data Factory / AWS Glue.Implement automated deployment and CI/CD processes for data pipelines.Collaborate with Data Analysts, Data Scientists, BI Developers, and application teams.Create and maintain technical documentation for pipelines, data models, and architecture.Participate in Agile/Scrum meetings, code reviews, and technical discussions. Required Technical Skills : Programming & Database Python SQLPy Spark / Apache Spark Advanced SQL queries and performance optimization Data Engineering ETL / ELT Data Pipelines Data Integration Data Modeling Data Warehousing Data Lakes / Lakehouse architecture Batch and streaming data processing Cloud AWS or Azure AWS: S3, Glue, Lambda, Redshift, EMR Azure: Data Factory, ADLS, Synapse Analytics, Databricks Big Data & Modern Data Platforms Apache Spark Databricks Delta Lake Kafka knowledge is preferred Snowflake experience is an advantage Databases Postgre SQL / My SQL / SQL Server / Oracle Snowflake / Redshift / Azure Synapse No SQL knowledge is an advantage Tools Git Airflow Docker CI/CD Jenkins / Azure Dev Ops / Git Hub Actions Required Qualifications Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.3–6 years of professional Data Engineering experience.Strong hands-on experience with Python and SQL.Experience developing production-level ETL/ELT pipelines.Strong knowledge of Spark/Py Spark and Databricks.Hands-on experience with AWS or Azure cloud services.Good understanding of data warehousing and data modeling concepts.Experience handling large datasets and optimizing data-processing workloads.Strong analytical and troubleshooting skills.Good written and verbal communication skills.Ability to work effectively with cross-functional teams. Preferred Skills Snowflake Kafka Delta Lake Apache Airflowdbt Kubernetes Terraform Power BI / Tableau Real-time/streaming data pipelines Data governance and security Cloud certification in AWS or Azure Key Skills Data Engineering | Python | SQL | Py Spark | Apache Spark | Databricks | ETL | ELT | Data Pipelines | AWS | Azure | Snowflake | Airflow | Kafka | Data Warehousing | Data Lake | Delta Lake | Data Modeling | CI/CD | Git

Apply on Clicksoft Solutions's site