OpenAIRE Graph Evaluator: Research paper classifier and LLM-as-Judge pipeline
A multi-model LLM-as-Judge pipeline that builds annotation datasets to train Echo-DSRN intent classifiers — entry for the OpenAIRE AI Hackathon 2026, co-organised by OpenAIRE and Alien Intelligence.
The app streams scientific paper metadata from the OpenAIRE Graph API, classifies research intent with a 98M-parameter Echo-DSRN model (fast CPU inference), and enlists multiple LLMs as judges to build a golden consensus dataset.
Features
- OpenAIRE Stream — live publication metadata via the OpenAIRE Graph API
- Collab Hub — community annotations with LLM judge consensus; versioned rounds with read-only archives (e.g. v0.1.3 archive)
- Free Text — ad-hoc classification of custom titles and abstracts
- Saved History — local browser storage with optional server-side sync
- Model Card — architecture specs, parameter census, and consumption code
Hackathon Deliverable
This project is the artifact submission for the OpenAIRE AI Hackathon 2026.
The accompanying story explains the question, journey, insight, and what others can reuse.