Learning Commons • Platform • Docs
Evaluators help you to measure the attributes of LLM-generated text through the lens of learning science:
- Optimize features: Sharpen AI-generated content so it aligns with pedagogy and your goals.
- Maintain performance: Use evaluators as product analytics to confirm LLM output stays on target.
- Select models: Compare candidate models by testing their output against the evaluators.
| Folder | Contents |
|---|---|
evals |
Evaluators (with prompts, input/output schemas, and Python notebooks), grouped by family: Text Complexity, Feedback, Academic Standards Alignment, and Durable Skills |
sdks |
Python and TypeScript |
demos |
Sample apps that demonstrate SDK use |
datasets |
Expert-annotated datasets used to create and validate evaluators |
scripts |
Repo check harness — run python3 scripts/check.py --fix before committing (see scripts/README) |
LICENSE |
Open source license details |
Check out the Quickstart to start using Evaluators.
| Access method | When to use | How to use |
|---|---|---|
| Evaluators playground | For a quick demo of how evaluators work | 1. Open the Evaluators playground 2. Choose an evaluator family and provide your inputs 3. Click Evaluate, then expand each result for its score and reasoning |
| SDK | To integrate into your TypeScript or Python project | 1. Install: npm install @learning-commons/evaluators or pip install learning-commons-evaluators2. Import an evaluator, pass your API key, and call evaluate3. See the TypeScript and Python SDK READMEs for details |
| Python notebook | For quick prototyping | 1. Start Jupyter Lab: jupyter lab (opens at http://localhost:8888)2. Browse into evals/ and open the evaluator you want to try3. Copy your text into the last code cell and run it You can also use an IDE with Python and Jupyter support, such as VS Code with Microsoft's Python and Jupyter extensions. |
| Local | To evaluate inputs locally on your machine | Clone this repository and follow the directions for your platform: - Setup on Mac/Linux - Setup on Windows |
| Evaluator family | Evaluator |
|---|---|
| Text Complexity - Docs | Grade Level Appropriateness - Docs |
| Background Knowledge Demands - Docs | |
| Vocabulary Complexity - Docs | |
| Sentence Structure - Docs | |
| Meaning Directness - Docs | |
| Purpose Clarity - Docs | |
| Organizational Structure - Docs | |
| Reference Knowledge Demands - Docs | |
| Graphics - early access, docs pending | Graphics Complexity - early access, docs pending |
| Graphics Accuracy - early access, docs pending | |
| Feedback - Docs | Strength Acknowledgment - Docs |
| Revision Actionability - Docs | |
| Student Response Specificity - Docs | |
| Revision Accuracy - Docs | |
| Revision Manageability - Docs | |
| Withholding Answers - Docs | |
| Tone Appropriateness - Docs | |
| Academic Standards - Docs | Math Standards Alignment - Docs |
| Durable Skills - Docs | Critical Thinking - Docs |
If you have questions or feedback, open an issue or reach out to us at [email protected]. To report a security issue, please disclose responsibly by contacting us at [email protected].
Sign up for a Learning Commons Platform account to stay up-to-date about the latest updates and releases.
Use of the resources provided in this repository is subject to our Terms of Use.