12/10/2019 (2 hours)

• examined data format
• found tools in pandas and sklearn for loading and processing data
• implemented util functions for processing data into training and test sets

12/11/2019 (2 hours)

• read about bag-of-words model
• played around with DataFrame
• implemented bag-of-words with CountVectorizer

12/12/2019 (1 hour)

• ran Logistic Regression
• produced confusion matrix using code from lab
• 42% accuracy only!
• selecting more ML algos to experiment with

12/13/2019 (1 hour)

• produced confusion matrix with pandas
• exploring NLP methods to get better predictions with text data

12/15/2019 (1 hour)

• implemented KNN and SVM
• read about TFIDF

12/17/2019 (8-10 hours)

• implemented Naive Bayes and Random Forest
• added pipeline with TFIDF for each algo
• tried not counting stop words => worse performance
• GridSearchCV to tune parameters for each of 5 classifiers
• A LOT of test runs for tuned parameters and fixing stuff (most time-consuming!)
• calculated AUC scores
• added stemming to vectorizer to reduce # features
• plotted ROC curves for each classifier against random guessing
• exploring bigram models

12/18/2019:

(4 hours)

• implemented bag-of-bigrams model
• generated ROC curve plots for bigram model with different classifiers
• tried bigram + TFIDF => worse performance
• calculated coefficients for words, selecting top words with highest pos/neg correlations with '1' label
• worked on presentation

(5 hours)

• refactored code for 2 separate models
• put back calculations for coeffs and TFIDF values for both models
• exported coeffs and TFIDF values (i.e., word importances) to Excel and CSV files
• final reruns, performance records, and plot figures for all model + classifier pairs
• updated README

• Working With Text Data. https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html.
• Shaikh, J. (2017, July). Machine Learning, NLP: Text Classification Using scikit-learn, Python and NLTK. https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a.
• Harlalka, R. (2018, June). Choosing the Right Machine Learning Algorithm. https://hackernoon.com/choosing-the-right-machine-learning-algorithm-68126944ce1f.
• Gelé, A. (2016). OMG! NLP with the DJIA and Reddit!. https://www.kaggle.com/ndrewgele/omg-nlp-with-the-djia-and-reddit.
• Ganesan, K. How to Use Tfidftransformer & Tfidfvectorizer?. https://kavita-ganesan.com/tfidftransformer-tfidfvectorizer-usage-differences/#.Xfm7sdZKjBI.
• Sun, J. (2016, August). Daily News for Stock Market Prediction, Version 1. Retrieved [2019, November 16] from https://www.kaggle.com/aaron7sun/stocknews.

• run_unigram.py: runs 5 classifiers with the bag-of-words + TFIDF model.
• run_bigram.py: runs 5 classifiers with the bag-of-bigrams model. ROC curve plots will be recreated from each run, thus may be different from the results recorded in the next section (e.g., for Random Forest).
• stocknews: contains labeled data.
• figs: contains ROC curve plot figures.
• idf: contains TFIDF values for all words and bigrams in the vocabulary of the given text data. A number of most and least important words are exported into separate files for ease of reference.
• coeffs: contains coefficients for all words and bigrams in the vocabulary of the given text data. 30 words with highest/lowest correlations with the '1' label are provided in separate files for the unigram model, and 300 words for the bigram model.

Logistic Regression: 42.59%
KNN: 51.06%
Naive Bayes: 49.47%
SVM: 44.97%
Random Forest: 44.97%

Logistic Regression: 48.68%
KNN: 50.53%
Naive Bayes: 50.79%
SVM: 44.44%
Random Forest: 51.59% (range 50-52%)

Logistic Regression:
best score: 0.5418994413407822, best params: {'base__C': 0.001, 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
accuracy = 50.79%
ROC AUC = 0.454

KNN: best score: 0.5543140906269398, best params: {'base__n_neighbors': 32, 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
accuracy = 52.65%
ROC AUC = 0.536

Naive Bayes:
best score: 0.5406579764121664, best params: {'base__alpha': 0.01, 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
accuracy = 51.59%
ROC AUC = 0.488

SVM:
best score: 0.5418994413407822, best params: {'base__alpha': 0.01, 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)} accuracy = 50.79
ROC AUC = 0.454

Random Forest:
best score: 0.5450031036623215, best params: {'base__max_features': 31, 'base__n_estimators': 80, 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
ROC AUC = 0.526

------------
Logistic Regression
------------
Predicted    1
True
0          186
1          192
accuracy = 50.79%
ROC AUC = 0.468

------------
KNN
------------
Predicted   0    1
True
0          63  123
1          57  135
accuracy = 52.38%
ROC AUC = 0.534

------------
Naive Bayes
------------
Predicted   0    1
True
0          56  130
1          52  140
accuracy = 51.85%
ROC AUC = 0.511

------------
SVM
------------
Predicted    1
True
0          186
1          192
accuracy = 50.79%
ROC AUC = 0.506

------------
Random Forest
------------
Predicted   0    1
True
0          44  142
1          44  148
accuracy = 50.79%
ROC AUC = 0.523

------------
Logistic Regression
------------
Predicted   0    1
True
0          66  120
1          45  147
accuracy = 56.35%
ROC AUC = 0.591

------------
KNN
------------
Predicted  0    1
True
0          2  184
1          3  189
accuracy = 50.53%
ROC AUC = 0.507

------------
Naive Bayes
------------
Predicted   0    1
True
0          17  169
1          21  171
accuracy = 49.74%
ROC AUC = 0.481

------------
SVM
------------
Predicted   0    1
True
0          89   97
1          69  123
accuracy = 56.08%
ROC AUC = 0.554

------------
Random Forest
------------
Predicted   0    1
True
0          55  131
1          49  143
accuracy = 52.38%
ROC AUC = 0.531