Data Scientist

Essential functions

• Canonical Guideline Development: Author and maintain canonical evaluation guidelines that standardize task definitions, rating criteria, and edge-case handling. These assets will be designed to scale across languages and markets, with the support of multi-lingual experts. You will ensure guidelines are clear, complete, and adaptable.

• Task Setup & Configuration: Collaborate with partners to configure evaluation tasks, including platform setup, workflow design, annotator assignment, and quality control mechanisms. Ensure task configurations align with research design specifications.

• Pilot Design & Execution: Design and run pilot evaluations to validate task setups, identify guideline ambiguities, calibrate annotator understanding, and surface issues before full-scale deployment. Analyze pilot results and iterate on guidelines and configurations accordingly.

• Monitoring & Data Quality: Develop and implement monitoring frameworks to track evaluation progress, annotator performance, inter-rater agreement, and data quality in real time. Flag anomalies and implement corrective actions to maintain data integrity across markets.

• Cross-Linguistic Execution Support: Work collaboratively with cross-functional partners, multi-lingual annotators, and language specialists to adapt evaluation guidelines and workflows for linguistic and cultural nuance. Ensure consistent quality standards are met across all target languages.

• Data Pipeline & Delivery: Manage the end-to-end data pipeline from raw annotations to clean, analysis-ready datasets. Ensure data is properly structured, documented, and delivered to downstream research and engineering consumers.

Qualifications

• 3+ years of experience in a data science, applied research, or evaluation operations role, with hands-on experience managing annotation or evaluation pipelines.

• Advanced degree (MS/PhD) in Data Science, Statistics, Computational Linguistics, Information Science, or a related field.

• Proficiency in Python and experience with data processing, statistical analysis, and visualization libraries (e.g., pandas, NumPy, scipy, matplotlib, seaborn).

• Experience developing and maintaining annotation guidelines or evaluation protocols for human labeling tasks.

• Comfortable computing and interpreting inter-rater reliability metrics (e.g., Cohen's kappa, Krippendorff's alpha) and other data quality indicators.

• Demonstrated ability to collaborate with annotation operations services, vendor teams, or distributed study participants .

• Able to work independently as well as collaboratively with minimal direction.

• Organized, highly attentive to detail, and manages time well.

Would be a plus

• Experience operating evaluation or annotation pipelines across multiple languages or markets.

• Familiarity with annotation platforms and task management tools (e.g., Label Studio, Scale AI, or similar).

• Experience with SQL and large-scale data infrastructure (e.g., Spark, Hadoop, or cloud-based analytics platforms).

• Prior experience in AI safety, responsible AI, content moderation, or trust and safety domains.

• Experience designing quality assurance frameworks for crowdsourced or distributed annotation work.

• General familiarity with localization workflows or working with language service providers.

We offer

• Opportunity to work on cutting-edge projects

• Work with a highly motivated and dedicated team

• Competitive salary

• Flexible schedule

• Benefits package - medical insurance, vision, dental, etc.

• Corporate social events

• Professional development opportunities

• Well-equipped office

About us

Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.