Over the past year I have been building an AI-powered research assistant that helps us conduct literature reviews and synthesize scientific knowledge. Here are a few lessons learned along the way.

1. Start with a knowledge graph

Raw papers are hard for an LLM to reason about. We first extract structured triples โ€” entity โ†’ relation โ†’ entity โ€” and store them in a knowledge graph. Query time dropped dramatically once retrieval happened on the graph instead of on raw text.

2. Evaluation is everything

1
2
3
4
5
6
def evaluate_retrieval(predictions, ground_truth, k=10):
    hits = 0
    for pred, truth in zip(predictions, ground_truth):
        if truth in pred[:k]:
            hits += 1
    return hits / len(predictions)

A simple hits@k benchmark caught regressions that informal testing completely missed.

3. Abstracts lie, methods donโ€™t

When synthesizing evidence, we force the model to cite the methods section rather than the abstract. This single change improved factuality on our internal test set by over 15%.

Checklist for a new tool

  • Extract entities and relations into a graph
  • Set up hits@k evaluation from day one
  • Ground every generated claim in methods text