In this note, we will review what we did last week, show how to use pipelines in sklearn, use sklearn's cross_val_score, look at forward stepwise feature selection, and be introduced to the Python library modeled to look like R
Although there will be presented very formal mathematics within probability, with measures and the like, we are free to be as formal or informal as we prefer. The idea of presenting us with these formal definitions is to show us exactly what these different concepts mean.
The most important for this course is that we learn how do calculate probabilities on the computer, and can use computers as statistical tools for analysis.
Also, goal is to show how ML is done in the industry, using data through pipelines, instead of loose numpy arrays.
Review¶
In [2]:
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
%matplotlib inline
In [3]:
#!tail -n 14 data/adult.names #doesnt work for me on windows
colnames = [
'age',
'workclass',
'fnlwgt',
'education',
'education-num',
'martial-status',
'occupation',
'relationship',
'race',
'sex',
'cap-gain',
'cap-loss',
'hours-per-week',
'native-country',
'income'
]
In [4]:
def read_data(n=None):
colnames = [
'age',
'workclass',
'fnlwgt',
'education',
'education-num',
'martial-status',
'occupation',
'relationship',
'race',
'sex',
'cap-gain',
'cap-loss',
'hours-per-week',
'native-country',
'income'
]
df = pd.read_csv('IN-STK5000-Notebooks-2020/data/adult.data.gz', names=colnames)
# Here you would usually do more checks
# is n bigger than our data set
# how big sample size is needed, and so on
# but for now, we just assume the general case that n is acceptable
if n:
df = df.sample(n)
df.index = range(n)
target = (df.income == ' >50K')*1
df.pop('income')
return df, target
In [5]:
features, target = read_data(2000)
Sometimes your training data may not include all the possible values for some categories. If this new category is included in your test set, then your model may blow up.
Pipelines¶
Let's look at using a pipeline in sklearn.
In [6]:
cat_columns = ['sex', 'education', 'race']
cont_columns = ['age', 'education-num']
In [7]:
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import OneHotEncoder, StandardScaler
In [8]:
cat_trans = Pipeline(steps=[('onehot', OneHotEncoder(drop='first'))])
# if you were to not drop this first column, it would still work
# but sklearn tries to restrict some values, so its a good idea to not trust this..?
In [9]:
cont_trans = Pipeline(steps=[('scale', StandardScaler())])
feature_trans = ColumnTransformer(
transformers=[('categorical', cat_trans, cat_columns),
('continuous', cont_trans, cont_columns)])
classifier = Pipeline(steps=[('feature_transform', feature_trans),
('classifier', KNeighborsClassifier(n_neighbors=35))])
In [10]:
classifier
Out[10]:
In [11]:
from sklearn import set_config #this is somewhat a gimmick
set_config(display='diagram')
In [12]:
classifier
Out[12]:
This could be pretty useful for seeing the structure of your pipelines.
Cross-validation¶
In [13]:
from sklearn.model_selection import cross_val_score
# ?cross_val_score
In [14]:
cv_scores = cross_val_score(classifier, features, target, cv=20)
In [15]:
cv_scores
Out[15]:
In [66]:
cv_scores.mean()
Out[66]:
In [67]:
plt.hist(cv_scores, bins=5)
Out[67]:
In [54]:
from sklearn.utils import resample
In [63]:
plt.hist([resample(cv_scores).mean() for _ in range(500)])
Out[63]:
This can be run over and over, and every time it will give slightly different results.
In [65]:
plt.hist([resample(cv_scores).mean() for _ in range(500)])
None #this is the same thing, it's just the None hides the extra infomation about the array
In [69]:
ks = list(range(10, 100, 10))
In [74]:
score_results = [cross_val_score(Pipeline(steps=[('feature_transform', feature_trans),
('classifier', KNeighborsClassifier(n_neighbors=k))]), features, target, cv=5)
for k in ks]
In [76]:
score_results
Out[76]:
In [77]:
plt.plot(ks, [s.mean() for s in score_results])
Out[77]:
In [78]:
plt.plot(ks, [s.mean() for s in score_results])
plt.plot(ks, [s.min() for s in score_results])
plt.plot(ks, [s.max() for s in score_results])
Out[78]:
Green:max, Blue:mean, Orange:min
In [81]:
from sklearn.model_selection import GridSearchCV
In [83]:
param_grid = {'classifier__n_neighbors':ks} # the name of the step and the parameter
grid_search = GridSearchCV(classifier, param_grid, cv=10)
In [84]:
fit_result = grid_search.fit(features, target)
This finds the best parameters, which build your best model.
In [87]:
fit_result.best_estimator_.get_params()['steps']
Out[87]:
Here we see, the alg. chose n_neighbors=70 as the best number of neighbors.
In [88]:
dummies = pd.get_dummies(features)
In [89]:
dummies
Out[89]:
In [90]:
for column in dummies:
corr = dummies[column].corr(target)
print(f'{column}: {corr}')
Correlation will work if you have something that looks like a linear.
In [94]:
from sklearn.tree import DecisionTreeClassifier
In [102]:
columns = list(features.columns)
selected_features = []
scores = []
N = 10
while len(selected_features) < N:
best_score = pd.Series([0])
best_feature = None
for feature in columns:
score = cross_val_score(DecisionTreeClassifier(max_depth=10),
pd.get_dummies(features[selected_features + [feature]]), target)
if score.mean() > best_score.mean():
best_feature = feature
best_score = score
print(f'{best_feature}: {best_score} ({best_score.mean()})')
columns.remove(best_feature)
selected_features.append(best_feature)
scores.append(best_score)
In [104]:
plt.plot([s.mean() for s in scores])
plt.plot([s.min() for s in scores])
plt.plot([s.max() for s in scores])
Out[104]:
Let's see what results we get for a completely random data set. Hint: there should be no correlation between features!
In [105]:
import numpy
In [107]:
N = 200
random_features = pd.DataFrame(numpy.random.normal(size=(N,N)))
random_target = numpy.random.choice([0, 1], size=N)
In [109]:
from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(random_features, random_target, test_size=0.2)
In [112]:
columns = list(Xtr.columns)
selected_features = []
scores = []
N = 5
while len(selected_features) < N:
best_score = pd.Series([0])
best_feature = None
for feature in columns:
score = cross_val_score(DecisionTreeClassifier(max_depth=10),
pd.get_dummies(Xtr[selected_features + [feature]]), ytr)
if score.mean() > best_score.mean():
best_feature = feature
best_score = score
print(f'{best_feature}: {best_score} ({best_score.mean()})')
columns.remove(best_feature)
selected_features.append(best_feature)
scores.append(best_score)
In [113]:
plt.plot([s.mean() for s in scores])
Out[113]:
In [114]:
model = DecisionTreeClassifier(max_depth=10).fit(Xtr[selected_features], ytr)
In [115]:
from sklearn.metrics import accuracy_score
In [117]:
accuracy_score(yte, model.predict(Xte[selected_features]))
Out[117]:
When doing feature selection:
- Use a pipeline
- Cross validate
- Use your brain to see if the features make sense
In [120]:
cv_scores_r = cross_val_score(DecisionTreeClassifier(max_depth=10), random_features[selected_features], random_target, cv=30)
In [121]:
cv_scores_r
Out[121]:
In [122]:
plt.hist(cv_scores_r)
Out[122]:
R-ish-Library¶
In [126]:
import statsmodels.formula.api as smf
In [127]:
features
Out[127]:
In [129]:
X = features.copy()
In [132]:
X['target'] = target
In [133]:
X
Out[133]:
In [138]:
sm_fit = smf.logit("target ~ age + Q('education-num') + sex + race", data=X).fit() #logistic regression
In [139]:
sm_fit.summary()
Out[139]: