last day (15 days later) » 

9:54 AM
0
A: Using Word2Vec in scikit-learn pipeline

İsmail DurmazStep 1. MultinomialNB FeaturesSelection.w2v is a dict and it does not have fit or fit_transform functions. Also MultinomialNB needs non-negative values, so it doesn't work. So I decided to add a pre-processing stage to normalize negative values. from sklearn.preprocessing import MinMaxScaler nb_...

 
LdM
Thanks Ismail. It does not work for me as I get this error: AttributeError: 'dict' object has no attribute 'itervalues' which misses in self.dim.I would have also one question about the word2vec in that code. Does it work for you using X in model = gensim.models.Word2Vec(X, size=100) or you need to use training_sentence instead of X?
 
@LdM I have already tested and it is worked. However, accuracy is between 54% - 57%. I think you should add more stages to improve accuracy.
 
LdM
Sorry Ismail, but I am still getting the error AttributeError: 'dict' object has no attribute 'itervalues'. How did you fix it/handle it?
 
@LdM I added steps. Please look at the second step.
 
LdM
unfortunately the change has not fixed the issue. Now I am getting a new error: AttributeError: 'int' object has no attribute 'transform' due to: nb_pipeline.fit(DataPrep.... ) This happens also for the other learners when I use FeatureSelection.MeanEmbeddingVectorizer(FeatureSelection.w2‌​v)
 
9:54 AM
@LdM can you update last changes, I can check what did you do?
 
LdM
please see the updated question. I have tried with other learners. I used the parts you mentioned in your answer.
 
@LdM I mentioned that MultinomialNB is not working for negative values. I have updated my answer with adding MinMaxScaler to normalize values 0 to 1. Please follow that solution for MultinomialNB pipeline.
 
LdM
Still getting the same error. I think there is a problem with DataPrep. Have you changed anything there? I know you mentioned the issue with NB so I have tried with a different learner (still in the link I shared) and it gives me the same error. I think there is a problem with DataPrep at this point
 
@LdM There are some errors like SGDClassifier n_iter is not recognised, I have replaced with max_iter. And remove metrics. package name on some lines. Also, there are lots of warnings. I executed successfully with changes.
Hi
 
LdM
Hi Ismail, thanks a lot for your time and help with my question and problem
 
9:55 AM
You are welcom, I hope that I can help your
 
LdM
May I ask you to show me how you applied the metrics and replaced max_iter?
 
this is FeatureSelection.py source code
this is classifier.py source code
I have tested now they are running
You can replace with your files
It needs a long time process whole file. I recommend that run with partially
or run on a jupyter notebook
 
LdM
Thanks a lot. I will check if it works with my file as well. I very much appreciated your help. Just one more thing: you mentioned about the poor performance and for more stages that might improve it. Could you please tell me if you were referring to some specific techniques ?
I will do it
 
I think it is better which focusing on words instead of letters. Words more descriptive way that classifies a text as fake or not. Also, use stop words to eliminate meaningless words.
 
LdM
thanks a lot for the advice, Ismail. I am new in using word2vec and doing some text classification. Where is the step in the code that I should replace in order to focus on words?
 
10:18 AM
I don't know about word2vec. It focusses only letters, maybe there is no option for words. You can increase the size of Word2Vec.
 
LdM
ok, thanks a lot Ismail. I am trying to figure it out why I am still getting the error with my file (which has similar format to that one used in the link) :(
 
create a virtualenv or anaconda environment
maybe you should upgrade your packages
 
LdM
I do not know how to create a virtualenv. Yes, maybe I should upgrade the packages
 
I recommend that investigate anaconda and virtualenv advantages
 

last day (15 days later) »