DTREG Benchmarks of Predictive Model Methods

Benchmarks of Predictive Model Methods The following table shows the results for various types of predictive models applied to a large number of benchmarks. Note that different types of models work best for different types of data.

All of these benchmarks are classification problems (i.e., the target variable is categorical). Some types of models work better with regression problems where the target variable is continuous.

The following predictive models were tested using DTREG:

• Single tree - Classical single decision trees.
• TreeBoost - DTREG implementation of Jerome Friedman's Stochastic Gradient Boosting ("MART").
• Decision Tree Forests - DTREG implementation of Leo Breiman's "Random Forest"™ algorithm.
• SVM - Support Vector Machine.
• ANN - Multilayer Perceptron artifical neural network.
• PNN - Probabilistic Neural Network.
• GMDH - GMDH Polynomial Neural Network.
• CCNN - Cascade Correlation Neural Network.
• RBF - Radial Basis Function Neural Network.
• LDA - Linear Discriminant Analysis.
• K-Means - K-Means Clustering.
• Linear Regression - Traditional (OLS) linear regression.
• Logistic Regression - Regression adapted for binary classifications.

The numbers in the table are the percent misclassification (error) for each method on each benchmark. Smaller error values imply greater accuracy. The misclassification percentages were computed using 10-fold cross-validation for all methods except for Decision Tree Forests which used Out-Of-Bag (OOB) validation and Probabilistic Neural Networks which used Leave-One-Out (LOO) validation.

Logistic regression could not be performed for benchmark problems that have more than two target categories. Some of the other methods were not suitable for a few of the benchmarks.

Many of the benchmark data sets came from the UCI Machine Learning Repository.

The best result for each benchmark is shown in bold face type. The worst result is in italics.