« first day (3634 days earlier)      last day (1297 days later) » 

AAB
6:10 AM
Hi all
have a question related to scrapy
say I want to parse all links in a website
Should I be using CrawlSpider class.
or can I use Spider class and use parse method to follow all links?
Any advantage of using CrawlSpider?
 
6:55 AM
@roganjosh it's ok. no harm. I will try my best to improve the community.
 
 
2 hours later…
8:48 AM
My sister works in recruitment for python developers and has asked me to write her a LinkedIn post giving resources for learning. What's the general feeling towards Real Python? It's not in our recommended sources - is it reasonably vetted?
I've found some articles there that have been helpful to me, but I don't have a feeling for it as-a-whole
 
9:02 AM
Mmm, I can see stuff like this:
Feb 26 at 18:30, by Andras Deak
Probably, although I've seen a low-quality page there (about lambdas)
I might have to cherry-pick articles
 
9:15 AM
Howdy
 
cbg
 
What ya'll point of view on R vs Python?
Is R superior to Python in any way?
I am struggling to find any up to date arguments for R being superior to Python in some way
 
Funnily enough, I've been having to learn R this last week. The reason is for r-markdown and ggplot because it's the only decent way for us to build reports in a standard format. I don't know how I'd replicate that output in python
 
Can't you just use Jupyter Notebook and then shift between markdown - python output - plots ... etc.?
Then e.g. export file as pdf, html, or whatever you prefer.
 
Potentially I could
 
9:25 AM
Wouldn't that be a possible way to replicate the output in question?
 
<shrug/> I've never had to auto-generate reports with markdown until last week so it's not something I've thought about too much, I can just cargo-cult the company's existing approach that happens to be in R :P
I don't think an R vs. Python argument is particularly helpful, though. What's the outcome going to be if I told you that it was better at frobnicating bizbaz? Are you suddenly going to need to learn the language and start frobnicating bizbaz?
 
Yes :)
I have no idea what frobnicating bizbaz means by the way, haha
 
I was surprised to learn that frobnication is actually a word. When I was only a few weeks into learning python (and programming at all) I came across a docstring - "optionalplotz wants to frobnicate the bizbaz first" or something close to that and I just remember thinking "what on Earth have I got myself into?"
 
Haha, "and that's how a wish to learn programming unraveled into me joining a obscure cult of hooded people, whose mantra while summoning the underworld is 'frobnicate the bizbaz' "
 
9:42 AM
My first programming computer was affectionately known as The Frobnicator in honour of that "wut" moment :)
The case also lit up blue, which any good programmer knows means that it runs 25% faster
heh, it's none other than our friend PEP8
 
Hmm, does that mean the contributors of PEP8 consist of people from the frobnicate the bizbaz cult?
 
I don't think it's really a cult. Everyone will need a foobang at some point in their life, it's just not been your time yet
 
10:21 AM
Goodbye for now, perhaps our lives will intertwine in the future somehow -- sorry that sounded way too dramatic, I need to learn how to curb myself, jezz.
 
You mean "see you when I next visit chat in my browser"? :P
 
10:53 AM
why doesnt DoubleLinkedList need to inherit the class Node there in order to use it?
(like class DoubleLinkedList(Node):)
 
@SebastianNielsen why are you asking about R here?
 
@Permian ...why should it? Would be pretty dumb if every class that wants to use another class would have to inherit from it, wouldn't it?
 
yeah it would
 
11:49 AM
dpaste.org/Fsv1 Found 0 images belonging to 0 classes. stuck on this from 4 hours. plz help
 
I'm going to need more context than that. Is it an exception with a traceback?
 
its not an exception. i can share a picture if you w'd like.
 
Couldn't I just run it with any directory of images? I still don't understand the problem properly.
 
crystal ball says it's a CWD problem
 
My first thought is that it can't find the directory of images
 
11:54 AM
@roganjosh the problem may be this only but I even tried os.path.abspath("train_images")
@Aran-Fey CWD?
 
mmm, what is "Found 0 images belonging to 0 classes."? It kinda looks like an error to me but I'm not familiar with jupyter
 
@VisheshMangla current working directory
 
oops, that was easy
 
@VisheshMangla NO code as text.
Sep 14 at 20:11, by Andras Deak
@VisheshMangla please stop posting images of code
 
Ah, I missed that, sorry
 
11:56 AM
> Each subdirectory in this directory will be considered to contain images from one class
^ pretty sure that's your problem, you have no subdirectories
 
@Aran-Fey I ve an equivalent code that I wrote an year back, I just changed the path of images. It's like one of your template code what you say it as.
there too I don't have any subfolders but I get images loaded there
 
Please take a look at this question: stackoverflow.com/questions/64062890/…
 
@Aran-Fey you seem to be right, there I was using os.getcwd() in the code. That must be the reson so. Thanks
 
 
1 hour later…
user13415013
1:12 PM
Hey guys, I have scraped 1 million python question from stackoverflow :D
 
Rob
Hmm, might be too late to tell you about their archive then :)
 
@Rob oh we told them...
it was 1. API, 2. data dump, 3. scrape the rest if you must API still, but why are you even doing this
which is just to say, be careful not to waste your time ;)
 
Rob
Ah, hah
 
ultimate conclusion:
yesterday, by nerd
@Kevin Thanks but not understand, I am sticking to scraping.
 
<David Attenborough voice> The life cycle of a Rob is complex. He'll only emerge from hibernation on the Python chat channel when roused by an SO API question :P
 
1:26 PM
cbg, what is the best way to import data from a number of csv files to a db?
 
Depends what the db is
 
A loop?
 
ok I should have researched more before I did that but I used pd.read_html and pd.to_csvto initially create the csv files, I realized there is pd.to_sql now
sqlite database btw
 
Oh, for sqlite then just use executemany
 
so no need for an ORM? I saw some SO posts and like Andras suggested it was mostly loop based solutions
 
1:29 PM
Aggregate into a nested list and use executemany. It's borked for postgres
 
thanks for the suggestion guys
 
@python_learner where do you feel that an ORM fits into this? (that's a genuine question, not being standoff-ish, because I don't know your setup)
 
I saw this before I asked here pandas.pydata.org/pandas-docs/stable/reference/api/… and it said "Databases supported by SQLAlchemy [1] are supported" so I was expecting a pandas based solution, didnt know loops were actually ok
 
You wouldn't need loops if you have the df already. It might be worth taking a step back and explaining the context
Merging and/or joining dfs just to be able to make use of that method might be a waste of time
 
sure, as I mentioned above, I scraped a couple tables from a site using pd.read_htmland then saved them to csv using pd.to_csv, it turns out I need these csv in a db thats it. So I can either run the scraper again and use to_sql or convert the existing csv to db, do I make sense?
 
1:35 PM
I was being facetious
 
Well, there's a part which is lost there. is the question equal to merge all CSV files into one DataFrame and then insert it to database?
 
it does make sense, but there's no reason for the intermediate CSVs. Just build the single df
 
I just want all the csv values in a db, so if I have 200 rows per csv, so 5 files is 1000 rows, I need the 1000 rows in the db
 
But why make the CSVs at all?
 
I scraped them before not knowing I will need a db (its my mistake, I didnt analyze the format I would need)
 
1:38 PM
the question is: are you looking to append the data inside one table within the database?
 
yes, I dont have a db as of now, so the code I write will create the db and append the data
 
@python_learner ok, so we're just talking about CSVs because of a prior decision
This is simple enough. Are all the CSVs of the same format?
 
yeah, all are exactly the same to the rows and columns
 
so make it single dataframe
Join all in one. to make it easier to load the single df directly to the database.
 
I think pandas is overkill. Can you give the first 3 lines of 1 of the CSVs?
 
1:40 PM
thanks, sorry if this was a simple question, I just got carried away after
sure, hold on
 
@roganjosh pandas? You mean the universal converter?
 
can I just paste here or pastebin?
 
@python_learner you tell me. If it's not gonna get garbled, post here. Otherwise, it might be safer just to do it off-site
 
Or too many lines
Go for off-site
 
ok, pastebin it is
 
1:42 PM
I don't know whether you have a bajillion columns :P
 
bajillion o--<
 
pastebin.com/JWqGzw8g, no syntax highlighting for csv?
 
you're just short of a bajillion columns
 
laurel, thats why I asked
 
So the csv is an XY problem, right? Or is it a given now?
 
1:46 PM
the question can be "convert csv to db", so not an xy?
 
Not sure what you're asking. I think they could have ditched the CSV step but that's where we are
 
@roganjosh yeah, that
 
Let's deal with the CSV problem first, and then get rid of that step after
 
Thanks
 
yeah I will have to change my scraper to directly add to db now, I need old data in db and hence the csv to db part
 
1:48 PM
@python_learner since you are using pandas from the start to read_html . so you can use pandas.concat which will combine all into one. then you can load it to the database at once.
 
thanks @αԋɱҽԃαмєяιcαη I can try that, I already am using pandas, but I wanted to know best practice, not to make the same mistake again
 
@python_learner does this work? (I have in my head there's an exception) before we move on to the SQLite
import csv
import os

files = os.listdir('some_dir')
data = []
for file in files:
    with open(file, 'r') as infile:
        this_file = list(csv.reader(data))
        data.extend(this_file)
 
thank you, let me try, I was trying something in the mean time with pd.to_sql
 
user13415013
@AndrasDeak, It was my scraping project without using api, :), now its done
 
user13415013
@Rob , thanks, I was aware , just trying my own stuff without api, but next time i will use api :)
 
1:55 PM
@python_learner I use pandas a lot. I don't like to_sql
There are quirks in the python bindings to different sql engines and I don't want to pass that all off to pandas
 
@python_learner pandas is an analysis and manipulation tool where you've just used a small function of it which use requests & bs4 library on background to parse HTML structure. something like i don't need to use pandas.to_csv in case if there's a library can be used to write to csv (csv library .. etc)
 
@roganjosh had to change list(csv.reader(data)) to list(csv.reader(infile)), I get a list now
@αԋɱҽԃαмєяιcαη pandas is a lot, I just dont have any use case where I can use it to the fullest, no number stuff :) so exploring as of now
 
Perfect. Yeah, I knew there was an issue in the code, but thanks for spotting it. So now you have a 2D list?
 
yeah, I do
you type codes off of your mind? thats cool
 
technically we all do
Well, the ones who don't primarily copy-paste ;)
 
2:03 PM
@python_learner well, I certainly couldn't test it with my lazy attitude :P
 
@python_learner without an MCVE he has no other option
 
fair enough :)
 
or did you post that 3-line csv?
 
yeah it on the pastebin, not really an mcve though
 
To test it would be to build files and directories etc.
 
2:04 PM
OK, we'll split the blame
 
It wasn't an MCVE but it was enough for me to understand
Anyway
 
its a csv after all
 
We're good to go for the sqlite insert now
 
so now I refer sqlite docs and create table and executemany?
 
yup. I was gonna draft something out but I'd be happier if you just did it from your own research
 
2:06 PM
yeah you have helped me enough, I have done sqlite inserts before so that wont be a problem
 
sqlite is the one in the stdlib, right? That seemed straightforward enough to me when I looked at the docs
 
thanks all
yeah its stdlib
maybe a random question, but how do you guys be in this room during your day job?
I lurk mostly but many names are familiar and I see them most of the time
 
I don't think pandas has anything to add, btw. Not only do I distrust to_sql but I think the concatentation of multiple CSVs is slower than using the stdlib csv module
 
I thought pandas had some C code that did it fast
I have not looked at the source code of pandas or python, I dont know C or C code
 
Did what fast?
 
2:08 PM
Pandas is "95% python" according to github. Probably cython then.
 
anything pandas does, I thought library functions was better than my own way
 
@python_learner I have flexible times (I'm in academia)
 
csv is also a library function so nvm what I said
 
No. You can write seriously borked code in pandas
 
@AndrasDeak I remember you mentioned you use python for work not a dev though
 
2:10 PM
yeah
Right now I should be writing a report :P So I'll make coffee.
 
rbrb :)
 
@python_learner the same as how people during work post a questions and get answers :D
 
Until recently, I've been a contractor, so they can't tell me how to use my time. Now I have a job, I still will watch Room 6 (but not the main feed) because I learn a lot from here and it's a requirement that I stay on top of tech
 
in all fairness you can post come back and check every 15 mins :), chances are the same question is posted on reddit discrod etc
@roganjosh I dont understand when people here start talking maths, but I guess its cool if you know maths
not meaning you in specific though
 
@python_learner I don't always follow things, either. I can make a judgement call on whether it's sensible, regardless
 
2:14 PM
Phantom technique is lovely within Python rogan :P
 
You mean just watching as other people talk?
 
I hope I will gather as much lurking here :p
 
Also, I've not seen you for a while @αԋɱҽԃαмєяιcαη . I hope you've been well
 
@roganjosh kinda thing such bajillion which i personally liked :P. yea am recovering and all is good. hope you are fine as well. and all nice guys here as well.
 
dont mean to hijack this, but did any of you guys attend the hangouts?
I wanted to but 9:30 pm for me is pretty much dinner and sleep :/
 
2:19 PM
@αԋɱҽԃαмєяιcαη glad to hear it! "bajillion" is an overlooked unit of measurement. It's less than np.inf but more than basically everything else. I think I need a PR to python
 
We are some kind of nocturnal creatures
 
@python_learner None of those present, I believe. Inspector, holdenweb, PaulMcG are the main figures I know who attended.
 
ahh, I will try to attend the oct 3 one, its just like another virtual class :)
 
@python_learner I didn't but I would have liked to. I've met others from the room already for different reasons
 
@roganjosh am not Numpy user yet . so bajillion is new to my knowledge :p
 
2:25 PM
@αԋɱҽԃαмєяιcαη Just in case my sarcasm hasn't come across, "bajillion" is made up @αԋɱҽԃαмєяιcαη :)
 
2:59 PM
@roganjosh Indeed. Prepare to be mildly amused!
 
@toonarmycaptain am I supposed to retro-actively be amused or is there something else coming? :P
 
I think it was a threat, going forward
 
You and your speculative threats, these last few days! You're gonna need an honour-guard :P
 
Just because I'm paranoid doesn't mean I'm not being followed...
 
cron is too predictable but I'm thinking you need a "boo" message played at random times in the evening :)
 
3:17 PM
That reminds me of a project that I thought of a week or two ago. If augmented reality is sufficiently developed now, one could write a tamagotchi-like AR game where you watch your animals walk around your house. And then after a few weeks of gameplay animals start to disappear. And again after a while you start seeing weird things, like your pets being scared of something off-screen. And eventually you can see an animal being snatched behind/under a cupboard by something.
of course the game is advertised as a tamagotchi-like app, the rest is an easter egg :P
 
"a tamagotchi-like app"? I think you've missed the bus, sir
 
Are you saying that games where all you do is tend to a farm and pet animals are out of fashion? I haven't been keeping up with the times but I doubt that.
 
Well.. yeah, I... but.... <retreats to cave>
 
3:56 PM
ok so what should i do if no one answers my question on stackoverflow?, dont say "edit" i edited it more than 5 times now
 
put a bounty on it
 
4:10 PM
It's still close-worthy even if you know what SSTV is and how it works. You're either asking people to help you find a library that does the work for you, which is off-topic, or to write the whole code for you, which is also off-topic/too broad
 
yeah but, i just asked if there is any library that can help me, i even mentioned a module which does the opposite
 
like I said, library recommendations are off-topic
 
ok well how can i do tht with python?, can anyone lead me the way? or something like that
i am sorry if i offended or something ,i talk like this
 
4:28 PM
You didn't offend anyone, @Praveen. It still doesn't mean that you can't ask questions with a Minimal, Reproducible Example
 
 
1 hour later…
Ugh. I posted that to SOCVR too.
delvable now
 
o/
I've been working in Python for sometime but while troubleshooting I discovered a fundamental problem with my understanding.
I noticed that givenm = 2, print(((m+1)^2)) gives a different answer from print(((m+1)*(m+1))). Clearly I'm missing something.
My MATLAB background makes me predisposed to view those as the same yet Python views them differently.
 
^ is bitwise xor. The power operator is **
 
@Aran-Fey I am an idiot. You are right.
Oh the shame!
 
5:45 PM
It's funny, because ** is what fortran uses for power, and MATLAB has very strong fortran roots.
 
 
2 hours later…
8:02 PM
Can someone who's closer to kids these days tell me if Chuck Norris memes are still relevant to late teenagers?
 
90% sure they're not
 
@AndrasDeak Using my sister's late teenagers as an example: not really, no. I don't think they even know who he is.
 
My 15-year-old (living in US) definitely knows who Chuck Norris is
 
:( Sic transit gloria mundi. Thanks for the feedback, guys. Any chance there's a replacement immortal figure? I saw Shaggy being an all-powerful being a few years ago...
@MattDMo but do they also know that Chuck Norris counted to infinity, twice?
 
absolutely
 
8:06 PM
Good kid :)
 
but he just said Chuck Norris isn't really that popular these days
even though the flu has to get Chuck Norris shots every year
 
hehe
 
A couple years ago I would watch this video with my kids to cheer them up if they were feeling down.
 
thanks, wife appreciated that :D
 
 
2 hours later…
11:08 PM
littletable is getting cleaned up. Just pushed version 1.1.0, with rich integration for pretty tabular output. And converted crappy HowToUseLittletable.txt to Markdown-ified HowToUseLittletable.md
 
11:18 PM
@PaulMcG blew the chance to get rid of that capcase file name ;)
 

« first day (3634 days earlier)      last day (1297 days later) »