Senior Automation QA Engineer (Big Data Platform)

Essential functions

• Migration Support: Actively participate in the migration of the Enterprise Data Platform from HDFS/Hive to Cloud Storage.

• Pipeline Development & Validation: Implement and validate Spark-based ETL/ELT jobs (Scala/Python) to facilitate data movement to cloud storage.

• Automation: Develop and maintain robust automated tests and integration scripts, primarily utilizing Python.

• Data Integrity: Conduct functional testing and code validation to ensure the quality and stability of data pipelines.

• Verification: Test migration outcomes to confirm data accuracy, completeness, and reliability; perform critical data quality checks across systems.

• Analysis: Use SQL and analytics tools (e.g., Jupyter Notebooks, Apache Superset) to analyze and validate processed or migrated data and interpret performance metrics.

• Collaboration: Partner closely with Data Engineers to troubleshoot issues, validate fixes, and maintain exceptionally high data quality standards.

Qualifications

• Proven experience as an Automation QA Engineer, ideally within Big Data or large-scale data platform projects.

• Strong background in functional testing, code validation, and end-to-end data verification.

• Big Data Tech Stack: Hands-on experience with Apache Spark, Flink, HDFS, and a strong preference for experience with Apache Iceberg.

• Cloud Proficiency: Familiarity with AWS or equivalent major cloud environments.

• Coding: Proficiency in Python for writing and maintaining robust integration and automation tests.

• Solid understanding of data analytics—capable of analyzing results and validating transformations using SQL.

• Experience in validating the quality and consistency of data and tools used within pipelines.

• Excellent communication and collaboration skills within distributed team settings.

We offer

• Opportunity to work on bleeding-edge projects

• Work with a highly motivated and dedicated team

• Competitive salary

• Flexible schedule

• Professional development opportunities

About us

Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.