Model Evaluator – DevOps
About this role
About TuringTuring is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research. About the Role: We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents. In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development. What does day-to-day look like: - You'll review three AI coding agents that were each given the same real developer request. - You read each agent's full run: the commands it ran, the files it changed, what it checked and its final message. - Then you judge which run did the job best. For every task you - Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication. - Give each run an overall score from 1 to 5 and rank the three. - Write a short rationale explaining your ranking, with evidence from the runs. Required Skills: - Minimum 3+ years of overall experience - Strong experience with at least one of the following languages: Python, JavaScript/TypeScript, Java, Go, PHP or similar. - Proficiency with Shell commands, Git, Docker, and basic software pipeline setup. - Comfortable running, modifying, and testing real-world projects locally. Assessment & Selection Process - Candidates must complete three practice evaluation tasks as part of the selection process. - Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales. - A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning. - Candidates will be onboarded only after all three practice tasks have been approved. IMPORTANT — STRICT NO-AI POLICY - DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS. - ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE. - COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION. Perks of Freelancing With Turing - Work in a fully remote environment. - Opportunity to work on cutting-edge AI projects with leading LLM companies. Offer Details - Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST. - Employment type : Contractor assignment (no medical/paid leave) - Duration of contrac : 1 month; [expected start date is next week] - Payment model: $25 per approved task - Location: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LA Evaluation Process (approximately 75 mins) : - Assessment based review
Application listings
Choose a source listing and check its country requirements.