$60–120/hr · Mercor · Part time, 40 hours a week
Design research evaluation tasks and grade AI scientific reasoning.
What you would do
- Author research evaluation problems with correct methodology
- Grade AI analysis for methodological and statistical rigor
- Assess whether results would survive peer review
- Flag design, sample, or measurement issues
- Explain research standards and evaluation
Who they want
- Graduate degree in science discipline with research experience
- Clear written communication explaining your reasoning
- Comfort with ambiguous tasks and flagging unclear instructions
- Recent experience explaining methodological decisions
- Ability designing studies and interpreting data
What the interview asks about
1.Designing problems revealing methodology
Good tasks expose whether AI understands methods or mimics patterns.
For example: “Design a problem where textbook answers would be wrong given subtle constraints.”
2.Grading conclusions against evidence
AI reaches defensible conclusions from weak evidence; you judge if evidence actually supports claims.
For example: “AI applies statistical tests correctly but draws broad conclusions from small, non-representative samples.”
3.Identifying high-confidence wrong results
Internally consistent but wrong results hide from calculation checking; peer reviewers catch through design critique.
For example: “AI develops elegant solution with correct math but misses key problem constraints.”
A task you may get
Design a research evaluation problem in your field with reference answer. Grade sample AI responses identifying methodological issues.
How to prepare
- Gather recent research problems from literature or grants
- Identify common methodological mistakes in your field
- Prepare peer review feedback explaining flawed methods
The facts
- Pay
- $60–120/hr
- Hours
- Part time, 40 hours a week
- Where
- Remote
- Field
- Life, Physical, and Social Science
- Role type
- Talent network
- Posted
- 9/15/2026
We wrote this page from the public Mercor listing. It may be out of date, so read the full posting before you apply.