« first day (1 day earlier)    last day (119 days later) » 

2:47 PM
Howdy @ShubhamSharma
So essentially, this process works, but only on small datasets of 1-2k tweets. Last night I moved my script and all its parts to my gaming PC with 32GB of 3000hz RAM, ran the full tweet dataframe of 40k tweets and it crashed this morning in PyCharm with a memory error for the part we're working on data = [tuple(x) for x in data]
This is the specific block of code that fails:

# get a list of words without stopwords
words = hourly_tweet_df['full_tweet'].astype(str).str.extractall(r'(\w+)')[0]
# Remove stopwords and create frequency table THIS TAKES A LONG TIME TO RUN
print('**MEMORY_HEAVY** running remove stopwords and create frequency table...')
df = words[~words.isin(z)].str.get_dummies().sum(level=0)
df = df.loc[:, df.max().sort_values(ascending=False).index]
print('dataframe created')
 
3:50 PM
Hi @GeordiAlm
 
Hi @ShubhamSharma
 
At which part its crashing?
Is the below code executing without crashing?
# get a list of words without stopwords
words = hourly_tweet_df['full_tweet'].astype(str).str.extractall(r'(\w+)')[0]
# Remove stopwords and create frequency table THIS TAKES A LONG TIME TO RUN
print('**MEMORY_HEAVY** running remove stopwords and create frequency table...')
df = words[~words.isin(z)].str.get_dummies().sum(level=0)
df = df.loc[:, df.max().sort_values(ascending=False).index]
print('dataframe created')
 
4:07 PM
So, everything runs fine in a tweet subset of 1-2k tweets, but the memory heavy line (the part where it crashes) is df = words[~words.isin(z)].str.get_dummies().sum(level=0)
 
Just noticed a small trick to improve the speed..
You can use:
table = words[~words.isin(stopwords)].groupby(level=0).value_counts().unstack(fill_value=0)
instead of
table = words[~words.isin(stopwords)].str.get_dummies().sum(level=0)
But this will still crash as there are lots of words to be encoded..
Can you tell me in general what is the average length of len(words[0]) that is the number of words found per row?
I guess you should try:
table = words[~words.isin(stopwords)].groupby(level=0).value_counts().unstack(fill_value=0)
And let me know if that crashes.
 
Hmm, the last hasgtag I did was for elonmusk, and that yielded 13k total tweets for a two hour timeframe before tweepy api timed out
 
Otherwise we have to process the dataframe in chunks.
 
Okay! I will run that. I'm creating some new data right now (essentially I couldn't find intraday data for dogecoin + #dogecoin, so I decided to switch the analysis to Bitcoin + #elonmusk) - but tweepy's api kept kicking me out after a few hours worth of tweets. I should be getting about 20k tweets in a few minutes, so I will run ```table = words[~words.isin(stopwords)].groupby(level=0).value_counts().unstack(fill_value=0)
``` on that new dataset.
intraday financial data*
 
Cool! #elonmusk seems trending :P
 
4:20 PM
I was a little let down when I found a company that already does this, but it still seems like a cool little personal project to do at a smaller scale
 
yeah its cool!
 

« first day (1 day earlier)    last day (119 days later) »