Python Prediction Scorer v0.3.0

Scoring predictions is useful, but non-trivial. It can take quite a bit of work to implement and test. In order to make it easier, I have started working on Python Prediction Scorer, an open source library to score and compare predictions.

Current features

Score predictions using the Brier scoring rule on questions

  • with two or more alternatives;
  • where the order matters; and
  • where predictions can be updated over time.

Developer experience:

  • Fully type hinted
  • Object-oriented and (hopefully) user-friendly API
  • Immutability where possible

Planned features

There are many things I would like to add, such as:

  • More scoring rules
  • Calibration scoring
  • Hosted (REST) API to make it easier to use, especially in non-Python projects
  • Integrations with tools that people use to make predictions, such as Roam Research

Request for comments

This project is in its early stages, and I would love feedback on

  • what features you would like to see added;
  • improvements to the public API;
  • what tools you would like to see integrations with; and
  • code improvements.

Email me on or open an issue on GitHub.