The experiments performed by Caruana and Niculescu-Mizil were very comprehensive and beyond the scope of my project. In my experiments, I have not calibrated any of the algorithms using Platt Scaling or Isotonic regression.

The authors used different software packages for different learning algorithms. For example the authors have used the Breiman-Cutler implementation of random forests since it produced better results. Similarly they have used SVMLight for SVMs. While conducting my experiments, I have only used the algorithms provided by Weka. The use of different implementations and lack of calibration may have contributed to the different results obtained by me. Since I did not perform any calibration, boosted trees did not perform as well as expected.

Data sets Used:
The following 7 data sets were used by me
Adult, Bacteria, California Housing, Cover_type, HS(Indian pine92), letter p1, letter p2

Algorithms Evaluated:
The following algorithms were evaluated
Naive Bayes, Random Forests, Decision trees, Boosted trees, Bagged trees, Boosted stumps, ANNs, SVMs, Logistic regression and KNN.


Performance Metrics Used:
Accuracy, F-score, ROC, RMSE, APR(average precision)
For some metrics such as accuracy higher values are desirable while for some measures such as RMSE, lower scores are desirable. For such scores I have inverted the score( 1-RMSE) in order to maintain uniformity among the metrics.




Continue to "results"
Back
Back to project main page
Back to Varun's home page