Role Overview
As a Data Annotator for Rubrics and LLM Evaluation, you will review and assess AI-generated responses using structured rating criteria and detailed project guidelines. You will evaluate output quality, accuracy, relevance, safety, and overall usefulness to support the development of reliable AI systems.
This role is suited to detail-oriented professionals who can apply consistent judgment, document clear rationale, and work independently with structured evaluation workflows. Projects may include Portuguese and English content across domains such as Legal, Finance, Healthcare, and other professional subject areas.
Location: Remote
Work Type: Freelance / Independent Contractor
Schedule: Flexible Hours
Content Evaluation and Annotation
- Review, analyze, and evaluate AI-generated content according to project guidelines.
- Rate responses using established rubrics for quality, accuracy, relevance, safety, and completeness.
- Identify, classify, and categorize issues using established annotation criteria.
- Apply annotation standards consistently across assigned tasks.
- Document findings and provide supporting rationale when required.
- Identify supporting evidence within content and record findings accurately.
- Escalate unclear, ambiguous, or complex cases when appropriate.
- Handle confidential project information with professionalism and discretion.
Quality and Process Compliance
- Follow detailed project instructions and established workflows.
- Perform quality checks and validation before submitting completed tasks.
- Maintain accuracy, consistency, and adherence to project standards.
- Meet established productivity and quality expectations.
- Identify recurring issues and flag them appropriately.
- Contribute to maintaining reliable and high-quality project outputs.
Required Qualifications
- Professional-level proficiency in Portuguese.
- Strong English reading-comprehension and written-communication skills.
- Excellent analytical, critical-thinking, and problem-solving abilities.
- Exceptional attention to detail and ability to follow structured guidelines.
- Reliable computer and stable internet connection.
- Ability to work independently and manage deadlines effectively.
Preferred Background
Candidates with education, training, or professional experience in one or more of the following areas are encouraged to apply:
- Legal Studies
- Finance, Accounting, or Economics
- Healthcare, Nursing, Medicine, or Life Sciences
- Linguistics
- Communications
- Translation and Localization
- STEM Disciplines
- Research, Data Analysis, or Content Evaluation
Experience in data annotation, LLM evaluation, content review, quality assurance, moderation, localization, research, or related fields is preferred.
Why Join Agents Only Technologies Inc.?
- 100% remote, flexible, project-based work.
- Access to multilingual language and AI-data operations projects.
- Develop advanced evaluation, annotation, and quality-assurance skills.
- Gain exposure to specialized content across Legal, Finance, Healthcare, and other professional domains.
- Work within structured operational and quality-focused workflows.
- Potential opportunities to qualify for reviewer, quality-assurance, and specialized project roles.
- Priority consideration for future projects based on performance and business needs.
To proceed with your application, please complete our online application form using the link below. The process takes approximately 5–10 minutes to complete.
Application Form
Complete the application form
We appreciate your interest in joining Agents Only Technologies Inc. and look forward to reviewing your application.
