Python Prediction Scorer v0.3.0
Published 2020-05-24.
Scoring predictions is useful, but non-trivial. It can take quite a bit of work to implement and test. In order to make it easier, I have started working on Python Prediction Scorer, an open source library to score and compare predictions.
Current features
Score predictions using the Brier scoring rule on questions
- with two or more alternatives;
- where the order matters; and
- where predictions can be updated over time.
Developer experience:
- Fully type hinted
- Object-oriented and (hopefully) user-friendly API
- Immutability where possible
Planned features
There are many things I would like to add, such as:
- More scoring rules
- Calibration scoring
- Hosted (REST) API to make it easier to use, especially in non-Python projects
- Integrations with tools that people use to make predictions, such as Roam Research
Request for comments
This project is in its early stages, and I would love feedback on
- what features you would like to see added;
- improvements to the public API;
- what tools you would like to see integrations with; and
- code improvements.