Add SGD

[nineil]

SGD Classifier and Regressor added

The default parameters are:

'alpha': [0.000001, 0.00001, 0.0001, 0.001, 0.01], 'penalty': ['l1', 'l2', 'elasticnet']

[dan-blanchard]

I should probably have mentioned this before, but we strictly follow the PEP8 style guidelines for our code. I know you're using PyCharm, and there's supposed to be a way to enable PEP8 checking while you're editing.

[dan-blanchard]

No need for this extra blank line here.

[dan-blanchard]

To ensure that there are unit tests that use SGDClassifier and SGDRegressor, please add SGDRegressor and RescaledSGDRegressor to tests/configs/test_regression1.template.cfg, and SGDClassifier to tests/configs/test_sparse.template.cfg

I also just realized you haven't add a rescaled version of SGDRegressor to learner.py (because I forgot to tell you to 😄). You'll just want to add the following three lines around line 340 or so:

@rescaled
class RescaledSGDRegressor(SGDRegressor):
    pass

All that's different with the rescaled versions of the regressors is that the predictions are rescaled and constrained to better match the training set.

[dan-blanchard]

Not that it has to be done for this PR, but one thing we also talked about as necessary for really being able to use SGDClassifier is the need for kernel approximation support. I imagine we will implement this by adding another field to the config files that specifies what type of Sampler you would like to use, and then just making that transformation get applied after self.feat_selector.fit_transform() is called in Learner.train().

[dan-blanchard]

You've got an extra ' at the end of this line before the ] that is causing this unit test to fail. Take that out and everything should be fine, I think.

[nineil]

I have corrected that error.

Hope that now works.

[dan-blanchard]

It looks like somewhere around here you're going to want to set self._model_kwargs['loss'] to either log or modified_huber for SGDClassifier, because getting probabilities is only supported with those loss types, and we typically want the prediction probabilities. It's currently failing unit tests for this reason, as you can see here.

[nineil]

Oks.

Added. So now, our default loss function will be ‘log’