Persisting data in sklearn

0 votes
I'm using scikit-learn to cluster text documents. I'm using the classes CountVectorizer, TfidfTransformer and MiniBatchKMeans to help me do that. New text documents are added to the system all the time, which means that I need to use the classes above to transform the text and predict a cluster.

How should I store the data on disk? Should I simply pickle the vectorizer, transformer and kmeans objects? Should I just save the data? If so, how do I add it back to the vectorizer, transformer and kmeans objects?
Jul 10, 2019 in Python by ana1504.k
• 7,910 points
727 views

1 answer to this question.

0 votes
If you want to find some fixed cluster centers on a training set and then re-use them later to compute cluster assignments for new data then pickling the models (or just saving the vocabulary of the vectorizer and the other models constructors parameters and the cluster center positions) is ok.

If what you want is doing clustering with new data, you might want to retrain the whole pipeline using the union of the new data + the old data to make it possible for the vocabulary of the vectorizer to build new features (dimensions) for the new words and let the clustering algorithm find cluster centers that better match the structure of the complete dataset.
answered Jul 10, 2019 by SDeb
• 13,300 points

Related Questions In Python

+3 votes
5 answers

How to read multiple data files in python

Firstly we will import pandas to read ...READ MORE

answered Apr 6, 2018 in Python by DeepCoder786
• 1,720 points
15,126 views
0 votes
1 answer

Unique identification for data items in Python

Try the UUID module of Python. For example, ...READ MORE

answered Apr 17, 2018 in Python by Nietzsche's daemon
• 4,260 points
1,120 views
0 votes
2 answers

Extracting data from a JSON file in Python

Here is what i found and was ...READ MORE

answered Nov 27, 2018 in Python by Rupali
29,616 views
0 votes
1 answer

Numpy data in csv

Use this -  numpy.savetxt("data.csv", arr, delimiter=",") READ MORE

answered Jul 3, 2018 in Python by Hamartia's Mask
• 1,580 points
793 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 4,364 views
0 votes
1 answer
0 votes
1 answer

Read & write data in mifare card

Hi, you can check out the following ...READ MORE

answered Nov 29, 2018 in Python by SDeb
• 13,300 points
2,050 views
0 votes
1 answer

Get bbox in data coordinates in matplotlib

I'm not sure how you got the ...READ MORE

answered Jun 18, 2019 in Python by SDeb
• 13,300 points
4,743 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP