Resources
/
Model Accuracy vs Model Performance

Which is more important: model performance or model accuracy?

6

 Min Read

With more industries leveraging AI and ML for critical use cases, how can we be sure that ML models are trustworthy? Further, are they actually doing what they are supposed to be doing and doing it well? To answer this question, let’s take a look at model accuracy vs model performance and which is more important.

What is the definition of model performance?

Model performance is an overall assessment of how well your ML model is doing. To determine performance, MLOps teams use common ML model performance metrics, like:‍

Classification metrics 

  • Accuracy
  • True positive rate (recall)
  • False positive rate
  • Precision
  • F1 Score
  • Log loss
  • Area under curve (AUC)

Regression metrics

  • Coefficient of determination (R-squared)
  • Mean squared error (MSE)
  • Mean absolute error (MAE)
  • Mean  absolute percentage error (MAPE)
  • Weighted mean absolute percentage error (WMAPE)
  • Mean reciprocal rank (MRR)
  • Root mean square error (RMSE)

Ranking metrics

  • Mean average precision (MAP)
  • Normalized discounted cumulative gain (NDCG)

Statistical metrics

  • Pearson correlation coefficient
  • Coefficient of determination (R2) 

Computer vision metrics

Natural language processing metrics

  • Bilingual evaluation understudy (BLEU)
  • Metric for evaluation of translation with explicit ordering (METEOR)
  • Recall-oriented understudy for gisting evaluation (ROGUE)
  • Perplexity 

Model drift metrics

Within each of these categories are several individual measurements and performance indicators. While we didn’t list every single one, it’s worth noting that there are many data points that go into measuring model performance, and your choice of model performance metrics depends on your model use case.

One consideration that is becoming increasingly important in measuring model performance is model drift. Model drift refers to the decay of a model’s predictive ability as a result of real-world changes. The main concern with model drift is that as the model ages, it becomes less accurate because the input data no longer reflects what the model was trained on. This leads to an erosion of trust in the model. In order to combat performance degradation, it’s important to monitor the model and detect model drift in a timely fashion.

Why is model performance important?

Naturally, model performance is important because it’s a reflection of how well your ML solution is accomplishing the task it was designed for.

What is the definition of model accuracy?

Model accuracy is the model performance metric that measures the percentage of correct predictions or classifications. In other words:

$$Accuracy=\frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$$

As we alluded to above, accuracy is simply one metric on the spectrum of performance in machine learning. Overall model performance is more important than model accuracy, since model accuracy only takes one metric into account. However, model accuracy is a particularly useful single metric, so let’s dig a little deeper. 

Why is model accuracy important?

Model accuracy is important because it can help gauge a model's ability to process, understand, and predict. If model accuracy is off, it’s a pretty big warning sign that there might be some serious issues with your model. 

Think of it like a fever. Sometimes it might require a little bit of attention like rehydrating. Other times, it might be a symptom of a more serious illness. Model accuracy and model performance are the same way. Good model accuracy alone isn’t enough to prove you have a trustworthy (healthy) model, but it is a good sign. Having an inaccurate model could mean you just need to implement a quick fix or it might mean completely rebuilding your unhealthy model. 

All said and done, it is difficult to overstate the importance of model monitoring over the course of a model lifecycle. 

What is a good accuracy for machine learning? 

Good accuracy in machine learning is somewhat subjective and depends on the use for the model. Generally speaking, industry standards for good accuracy is above 70%. However, depending on the model objectives, good accuracy may demand 99% accuracy and up. For example, if your model relates to life and death scenarios, you probably need the highest accuracy possible; on the other hand, if your model suggests eCommerce recommendations, lower accuracy is often tolerated in order to speed model deployment. 

Improve model performance with Fiddler

Machine learning and artificial intelligence aren’t brand new, and neither are the complexities that come with it. ML teams depend on AI observability platforms and processes to ensure their models are both accurate and high-performing.