modelEval {CORElearn}R Documentation

Statistical evaluation of predictions

Description

Using predictions of given model produced by predict.CoreModel and correct labels, computes some statistics evaluating the quality of the model.

Usage

modelEval(model, correctClass, predictedClass, predictedProb=NULL, costMatrix=NULL, priorClProb = NULL, avgTrainPrediction = NULL) 

Arguments

model The model structure as returned by CoreModel, or NULL if some other predictions are evaluated.
correctClass A vector of correct class labels for classification problem and function values for regression problem.
predictedClass A vector of predicted class labels for classification problem and function values for regression problem.
predictedProb An optional matrix of predicted class probabilities for classification.
costMatrix Optional cost matrix can provide nonuniform costs for classification problems.
priorClProb If model=NULL a vector of prior class probabilities shell be provided in case of classification.
avgTrainPrediction If model=NULL mean of prediction values on training set shell be provided in case of regression.

Details

The function uses the model structure as returned by CoreModel, predictedClass and (optionally) predictedProb returned by predict.CoreModel. Predicted values are compared with true values and some statistics are computed measuring the quality of predictions.

Optional cost matrix can provide nonuniform costs for classification problems. For regression problem this parameter is ignored. The costs can be different from the ones used for building the model in CoreModel and predicting with the model in predict.CoreModel. If no costs are supplied uniform costs are assumed where necessary.

If a non CORElearn model is evaluated, one should set model=NULL, and a vector of prior of class probabilities priorClProb shell be provided in case of classification, and in case of regression avgTrainPrediction shell be mean of prediction values on training set.

Value

For classification problem function returns list with the components

accuracy classification accuracy
averageCost average classification cost
informationScore statistics measuring information in the predicted probabilities
AUC Area under the ROC curve
predictionMatrix matrix of miss-classifications
sensitivity sensitivity for two class problems
specificity specificity for two class problems
brierScore Brier score of predicted probabilities
kappa Kappa statistics measuring randomness of the predictions
MSE Mean Squared Error
RMSE Relative Mean Squared Error
MAE Mean Absolute Error
RMAE Relative Mean Absolute Error

Author(s)

Marko Robnik-Sikonja, Petr Savicky

References

Igor Kononenko, Matjaz Kukar: Machine Learning and Data Mining: Introduction to Principles and Algorithms. Horwood, 2007

See Also

CORElearn, CoreModel, predict.CoreModel.

Examples

# use iris data

# build random forests model with certain parameters
model <- CoreModel(Species ~ ., iris, model="rf", 
              selectionEstimator="MDL",minNodeWeight=5,rfNoTrees=100)

# prediction with node distribution
pred <- predict.CoreModel(model, iris, rfPredictClass=FALSE)

# Model evaluation
mEval <- modelEval(model, iris[["Species"]], pred$class, pred$prob)
print(mEval)

[Package CORElearn version 0.9.22 Index]