Data Platform Engineer

Essential functions

• Medallion Architecture & Event-Driven Ingestion: Design and implement a medallion data architecture on AWS S3 with IAM-enforced tier promotion rules, replacing manual ingestion triggers with event-driven pipelines and data discovery logic that guarantees complete capture.

• Harmonization & Field Data Gateway: Build an adapter layer that converts heterogeneous sources into a canonical format preserving temporal metadata, sensor calibration and spatial context, plus a gateway for large-scale real-world telemetry with automated PII scrubbing and sensor frequency alignment.

• Versioning, Lineage & Leakage-Free Splits: Deploy a centralized metadata registry with immutable dataset versioning and branching per tier, enforce episode-level partitioning for strict train/validation splits, and establish provenance chains from raw data through processed datasets and training runs to model artifacts.

• Quality Gates & Observability: Build automated schema validation gates and configurable circuit breakers that stop corrupted data from advancing between tiers, and a unified dashboard tracking ingestion and rejection rates, validation failures, augmentation throughput and latency.

• Governed Augmentation & Training-Readiness Contracts: Integrate a data augmentation pipeline as a managed flow stage with full lineage from source data through augmentation parameters to output, and define with ML teams the contracts for schema completeness, temporal windowing, annotation quality thresholds and approval gates.

• Pilot to Platform: Prove the end-to-end toolchain, from data ingestion to safety model training, on a first pilot workflow such as wheel-loader stockpile management, then scale it as the foundation for further autonomy use cases.

Qualifications

• 5+ years of experience building production-grade data pipelines and data platforms.

• AWS Data Services: Deep expertise with S3, EventBridge, Step Functions, FSx for Lustre, CloudWatch and IAM, with IaC proficiency and GitOps practices.

• Distributed Processing & Python: Hands-on experience with Ray or Spark and strong Python for building ETL/ELT pipelines at scale.

• Formats & Event-Driven Ingestion: Solid knowledge of columnar and binary formats (Parquet, HDF5) and event-driven ingestion architectures.

• Versioning & Lineage: Practical experience with LakeFS, DVC, Delta Lake or equivalent dataset versioning and lineage tools.

• Data Quality Frameworks: Proven experience designing schema validation, circuit breakers and rejection-rate monitoring.

• Experiment Tracking: Familiarity with MLflow or an equivalent artifact registry and experiment tracking system.

Would be a plus

• Robotics Sensor Data: Experience with autonomous systems data formats such as ROS bags, LiDAR, IMU and camera streams.

• Lakehouse Background: Prior work with medallion or lakehouse architectures.

• Generative Augmentation: Exposure to tools such as NVIDIA Cosmos, Stable Diffusion-based pipelines or similar.

• Safety-Critical Domains: Experience in regulated or safety-critical data environments such as automotive, aerospace or industrial autonomy.

We offer

• Opportunity to work on bleeding-edge projects
• Work with a highly motivated and dedicated team
• Competitive salary
• Flexible schedule
• Benefits package - medical insurance, sports
• Corporate social events
• Professional development opportunities
• Well-equipped office

About us

Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.