Machine Learning Metrics
Machine Learning Metrics
Introduction
In Machine Learning projects whether classical machine learning, deep learning, computer vision, speech processing, NLP, or any other ML project we keep building different models with different datasets. But how to know that for a particular problem model X is the best one? For that, we need to evaluate these models against certain metrics. What metrics we pick, depends upon the problem statement, data imbalance, type of data, etc. In this article, we will explore an exhaustive list of ML Metrics.
From various sources, benchmarking platforms, and research papers, I have noted 330+ metrics for evaluating Machine Learning models. I keep working on this page, expanding and defining these terms on a regular basis. Some of these metrics go over my head but I am keeping it here on this page because they have been used by the practitioners in some of their projects. After experimenting with those I will expand them here. Some of the metrics are obvious to Data Scientists. But, even if you are new to Data science it gives you an idea about these metrics.
MRR
MRR stands for “Mean Reciprocal Rank,” and it is a metric commonly used in information retrieval and evaluation tasks, including those in natural language processing (NLP). MRR is used to assess the effectiveness of ranking algorithms or systems in presenting relevant information to users. MRR is often applied to tasks such as question answering, search engines, and recommendation systems. A higher MRR indicates that relevant results tend to appear higher in the ranked lists, which suggests better user experience.
Step1: Ranking of Results: Imagine you have a system that retrieves a list of possible answers or documents in response to a user’s query or question. These results are usually ranked based on their perceived relevance to the query.
Step2: Reciprocal Rank: For each query or question, the reciprocal rank of the first correct (relevant) result in the ranked list is calculated as 1 divided by the position of that correct result. If the correct result is the second item, the reciprocal rank would be 1/2; if it’s the fifth, the reciprocal rank would be 1/5, and so on.
Step3: Mean Reciprocal Rank (MRR): To calculate the MRR, you take the average of the reciprocal ranks across all queries or questions in your evaluation dataset. The formula for MRR is:
MRR = (1/N) * ∑(1/rank_i)
N is the total number of queries or questions.
rank_i is the position of the correct result for the i-th query.
mrr_at_1
mrr_at_3
mrr_at_5
mrr_at_10
mrr_at_100
mrr_at_1000
MAP
MAP stands for “Mean Average Precision”. MAP focuses on evaluating the effectiveness of ranking algorithms or systems
Step1: Ranking of Results: You have a system that retrieves a ranked list of possible answers or documents in response to a user’s query or question.
Step2: Precision and Recall: Precision is the ratio of relevant items retrieved to the total number of items retrieved. Recall is the ratio of relevant items retrieved to the total number of relevant items in the dataset. These two metrics are often in tension with each other; increasing precision might result in lower recall and vice versa.
Step3: Average Precision (AP) for Each Query: For each query or question, you calculate the precision at each position in the ranked list where a relevant item is retrieved. You then calculate the average of these precision values, resulting in the Average Precision (AP) for that query.
Step4: Mean Average Precision (MAP): To calculate the MAP, you take the average of the Average Precision values across all queries or questions in your evaluation dataset. The formula for MAP is:
MAP = (1/N) * ∑(AP_i)
N is the total number of queries or questions.
AP_i is the Average Precision for the i-th query.
map_at_1
map_at_3
map_at_5
map_at_10
map_at_100
map_at_1000
Mean average precision (mAP):
A measure of the average precision over a range of IoU thresholds. It is also used in object detection and segmentation tasks.
mAP@IoU thresholds
The mean average precision over a range of intersection over union (IoU) thresholds.
Accuracy:
The percentage of test samples that are correctly classified.
acc_norm
accuracy_cardiffnlp/tweet_topic_multi
accuracy_cardiffnlp/tweet_topic_single
accuracy_cosinus
accuracy_euclidean
accuracy_manhattan
accuracy_tweet_eval/emoji
accuracy_tweet_eval/emotion
accuracy_tweet_eval/hate
accuracy_tweet_eval/irony
accuracy_tweet_eval/offensive
accuracy_tweet_eval/sentiment
Precision:
The fraction of predicted positive samples that are actually positive.
precision_entity_span
precision_at_1
precision_at_3
precision_at_5
precision_at_10
precision_at_100
precision_at_1000
precision_macro
Recall:
The fraction of actual positive samples that are predicted positive.
recall_entity_span
recall_at_1
recall_at_3
recall_at_5
recall_at_10
recall_at_100
recall_at_1000
recall_macro
R@P
“R@P” stands for “Recall at Precision.” It’s a metric used to evaluate the performance of information retrieval systems, such as search engines or question-answering models, in terms of their ability to retrieve relevant documents or answers.
R@P is a combination of recall and precision. It evaluates how well a system can maintain a specified precision level while retrieving relevant items. It’s typically expressed as “R@X,” where “X” represents the desired precision level.
For example, R@5 measures the recall when precision is equal to 0.5 (50%). It calculates the percentage of relevant items retrieved when the system is limited to a precision level of 50%.
Mathematically, R@X is calculated by finding the recall at the point where precision reaches the specified level X. This is done by examining the retrieved items in descending order of relevance until the desired precision level is achieved.
R@P is a useful metric in tasks where both recall (finding all relevant items) and precision (avoiding irrelevant items) are important. It provides insights into how well a system balances these two aspects when retrieving information.
R@P=50
R@P=75
R@P=90
F1 score:
The harmonic mean of precision and recall.
f1_entity_span
f1_macro
f1_micro
f1_weighted
f1 (seqeval)
f1 macro
f1@m
f1@m (absent)
f1@m (present)
f1@o (absent)
f1@o (present)
f1neg
f1pos
Hit@n
Hit@1:
- The percentage of queries for which the correct document is ranked first.
- The percentage of images for which the correct class is in the top k predictions. It is also used in image classification tasks.
Hit@5:
The percentage of queries for which the correct document is ranked among the top 5 documents.
Log-loss:
The negative log likelihood of the predicted labels.
Brier score:
The mean squared difference between the predicted probabilities and the actual labels.
Confusion matrix:
A table that shows the true and predicted labels for each class.
MSE
Root mean squared error (RMSE):
A measure of the average squared difference between the predicted and actual values.
Mean absolute error (MAE):
A measure of the average absolute difference between the predicted and actual values.
R-squared:
A measure of the proportion of the variance in the target variable that is explained by the model.
Cohen’s kappa:
A measure of agreement between two raters, taking into account chance agreement.
Matthews correlation coefficient (MCC):
A measure of the accuracy of a binary classifier that takes into account both the true positive rate and the false positive rate.
Area under the curve (AUC):
A measure of the performance of a binary classifier at all possible thresholds.
Precision-recall curve:
A graphical representation of the precision and recall of a binary classifier at all possible thresholds.
ROC curve:
A graphical representation of the true positive rate and the false positive rate of a binary classifier at all possible thresholds.
Mean squared logarithmic error (MSLE):
A measure of the average squared difference between the logarithms of the predicted and actual values.
Mean absolute percentage error (MAPE):
A measure of the average percentage difference between the predicted and actual values.
Root mean square logarithmic error (RMSLE):
A measure of the average squared difference between the logarithms of the predicted and actual values, after taking the square root.
Normalized discounted cumulative gain (NDCG):
A measure of the ranking performance of a model.
Precision at k (P@k):
The percentage of the top k predictions that are correct.
The percentage of predicted positive samples that are ranked among the top k predictions.
Precision@k = (# of relevant items retrieved among top-k recommendations) / (# of total predicted relevant items)
For example, if a system retrieves 5 relevant items among the top-10 recommendations out of a total of 25 predicted relevant items, then the recall@10 would be 0.20 or 20%.
In other words, precision@k measures how well the system performs in predicting relevant items among the top-k recommendations. It is useful when we are interested in evaluating the ranking quality of recommendations rather than their absolute number.
Recall at k (R@k):
The percentage of actual positive samples that are ranked among the top k predictions.
Recall@k = (# of relevant items retrieved among top-k recommendations) / (# of total relevant items)
For example, if a system retrieves 5 relevant items among the top-10 recommendations out of a total of 20 relevant items, then the recall@10 would be 0.25 or 25%.
In other words, recall@k measures how well the system performs in retrieving relevant items among the top-k recommendations. It is useful when we are interested in evaluating the ranking quality of recommendations rather than their absolute number.
F1-score at k (F1@k):
The harmonic mean of precision at k and recall at k.
Interpretability:
A measure of how easy it is to understand and explain the predictions of a model.
Fairness:
A measure of how the model treats different groups of people.
Inception score:
A measure of the diversity and quality of the generated images by a generative adversarial network (GAN).
Frechet Inception distance:
A measure of the similarity between the generated images by a GAN and the real images.
Wasserstein distance:
A measure of the distance between the distributions of the generated images and the real images.
Inception score:
A measure of the diversity and quality of the generated text by a language model.
Perplexity:
A measure of how difficult it is to predict the next word in a sequence.
BLEU score:
A measure of the similarity between the generated text and the reference text. BLEU More..
bleu4
bleu4_answer_extraction
bleu4_question_answer_generation
bleu4_question_answering
bleu4_question_generation
Intersection over union (IoU):
A measure of the overlap between two regions. It is typically used in object detection and segmentation tasks.
Top-5 error rate:
The percentage of images for which the correct class is not in the top 5 predictions. It is typically used in image classification tasks.
EM - Exact Match
EM used in NLP task evaluation, particularly in tasks like question answering and text generation. The EM metric measures the percentage of predictions that exactly match the ground truth or reference answers.
If the model’s answer matches the reference answer word-for-word, then the EM score for that particular instance is 1. If the answers do not match exactly, the EM score is 0. The EM score is then calculated as the ratio of instances where the model’s answer matches the reference answer exactly to the total number of instances in the evaluation dataset.
ROUGE
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): ROUGE is another metric used for evaluating text summarization and generation tasks. It measures the overlap of n-grams between the generated text and the reference text. ROUGE More..
rouge-l
rouge-2
rouge-lsum
rouge_l_answer_extraction
rouge_l_question_answer_generation
rouge_l_question_answering
rouge_l_question_generation
METEOR
Metric for Evaluation of Translation with Explicit ORdering: METEOR is a metric that combines multiple measures including precision, recall, stemming, and synonymy. It aims to provide a balanced evaluation of machine translation quality.METEOR More..
meteor_answer_extraction
meteor_question_answer_generation
meteor_question_answering
meteor_question_generation
HEQD
HEQD stands for Hierarchical Edit Distance. It is a metric used to evaluate the quality of text summarization. It is based on the edit distance between the ground truth summary and the generated summary, taking into account the hierarchical structure of the summary.
The HEQD metric is calculated as follows:
HEQD = 1 - (ED / L) where
ED is the edit distance between the ground truth summary and the generated summary.
L is the length of the ground truth summary.
The HEQD metric is a more accurate measure of the quality of text summarization than the BLEU metric, as it takes into account the hierarchical structure of the summary.
Example:
Ground truth summary:
- The cat is on the mat.
- The dog is chasing the ball.
Generated summary:
- The cat and the dog are playing.
The HEQD metric for this example would be:
HEQD = 1 - (2 / 2) = 0.5
This means that the generated summary is 50% similar to the ground truth summary.
The HEQD metric is a relatively new metric, and it is not yet as widely used as the BLEU metric. However, it is a promising metric for evaluating the quality of text summarization.
Perplexity
In NLP and machine learning, perplexity is a measure of how well a language model predicts the next word in a sequence. It is calculated as the inverse of the probability of the model’s predictions. A lower perplexity indicates that the model is better at predicting the next word.
The perplexity is calculated as follows:
Perplexity = \(exp( \frac{-1}{N * \sum(log(p(w_i)))})\)
where
N is the number of words in the test set.
\(w_i\) is the i-th word in the test set.
\(p(w_i)\) is the probability of the model predicting word \(w_i\).
The perplexity metric is a more robust measure of the quality of a language model than the accuracy metric. This is because the accuracy metric only measures how well the model predicts the correct word, while the perplexity metric also takes into account the probability of the model’s predictions.
Here is an example of how the perplexity metric can be used to evaluate the quality of a language model:
Let’s say we have a language model that is trained on a corpus of text. We want to evaluate the quality of the model by using the perplexity metric. We take a test set of text and calculate the perplexity of the model’s predictions on the test set. If the perplexity is low, then the model is good at predicting the next word. If the perplexity is high, then the model is not good at predicting the next word.
The perplexity metric is a useful tool for evaluating the quality of language models. It is a more robust measure of quality than the accuracy metric, and it can be used to compare the performance of different language models.
Pass@n
pass@1
pass@10
Pass@100
The Pass@100 metric is calculated by measuring the percentage of queries for which the correct answer is among the top 100 candidates retrieved by the foundation language model (FLM) from a large corpus of documents. It reflects the ability of the FLM to retrieve relevant information from a large-scale knowledge source, which is essential for downstream tasks such as question answering, summarization, and dialogue. The higher the Pass@100 score, the better the FLM is at retrieving knowledge 1.
If you are interested in calculating Pass@100 for your own data, you can use binomial distribution. The formula for calculating Pass@100 using binomial distribution is as follows:
\[P(X >= k) = 1 - \sum_{i=0}^{k-1} [nCi * p^i * (1-p)^{n-i}]\]where X is a binomial random variable representing the number of correct answers in the top 100 candidates, n is the total number of queries, p is the probability of getting a correct answer in one query, and k is the minimum number of correct answers required to achieve Pass@100.