Senior Automation QA Engineer (AI)

Essential functions

• Build AI-Specific Test Suites: Design and implement automated frameworks to test LLM responses, focusing on accuracy, tone, and groundedness in RAG pipelines.
• Agent Logic Validation: Verify that autonomous agents correctly select tools, manage conversational memory, and follow multi-step planning sequences.
• RAG Performance Testing: Evaluate the retrieval quality of Vector Databases (Vertex AI Vector Search) using metrics like Context Precision and Context Recall.
• Integrations Testing: Ensure seamless communication between Python-based AI services and Java/Spring backend systems via automated API tests.
• Guardrail & Security Testing: Automate the detection of "jailbreaking" attempts, PII leakage, and toxic content in AI outputs.
• Benchmarking: Conduct latency and cost-per-token analysis to ensure the AI platform meets enterprise-grade performance SLAs.
• Continuous Testing: Integrate AI evaluation steps into GCP-native CI/CD pipelines (Cloud Build).
• Documentation: Maintain detailed test plans and defect reports specifically tailored to the nuances of non-deterministic AI behavior.

Qualifications

• Languages: Python (Advanced/Expert), familiarity with Java is a plus.

• AI Frameworks: Experience with LLM evaluation frameworks (e.g., RAGAS, DeepEval, or G-Eval).

• Cloud Infrastructure: Deep expertise in Google Cloud Platform (GCP), specifically Vertex AI, Cloud Run, and BigQuery.

• Databases: Hands-on experience testing against Vector Databases (e.g., Vertex AI Vector Search, Pinecone).

• Tools: Pytest, Postman/Newman, Git, and Docker.

• Methodology: Strong experience in Agile/Scrum and "Shift-Left" testing mentalities

Would be a plus

• Experience with Google ADK or Vertex AI Agent Builder.
• Prior work in IT services/consulting for Fortune 500 clients.
• Knowledge of fine-tuning evaluation (comparing base models vs. fine-tuned versions).
• Experience with Prompt Engineering specifically for creating "synthetic" test data.

We offer

• Flexible working hours (full-time).
• One "Flex Day" off per month – eligible after six months with the company.
• 10 business days of vacation.
• Swiss Medical health coverage.
• 100% remote work.
• Permanent contract with salary review every four months (in ARS).
• Access to Udemy and Platzi for professional training.
• Employee Assistance Program (financial, nutritional, psychological support, etc.).
• Fully covered English classes during working hours.
• Discounts on Club de Beneficios and Samsung products.
• Birthday day off.

About us

Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.