@AndrasDeak @taritgoswami Checking first is a little slower, since .add checks anyway. The only reason to do an explicit check is if you need to do some extra stuff based on the result of the test. I've sometimes wished that .add returned a bool that lets you know if the added item is a dupe.
Yeah, actually a list was given, which contains colourful socks, denoted with numbers, to find the number of paired socks. E.g; [1,2,1,3,2,1] - here we have socks of 3 different colours and as we have 1 pair of 1 and 1 pair of 2 we have total of 2 pairs of socks.
To solve this I took a set, scanned through the list and at a time if the sock is not in that set I added it(that's the need to check the condition) , and if it was there already I have increased the `no_of_paired_socks` and `remove()` the element.
How to delete multiple columns in Pandas Dataframe?
0
I am cleaning my large datasets which have 393 columns and 4838 rows. I want to keep only 56 columns and need to delete the remaining columns. I know the 56 columns name. I am trying to delete using drop function but it is taking too long. I would like to do in a loop for deleting the unwanted columns or any other easy method. I've seen some examples but this doesn't answer my question.
@AndrasDeak Oh yeah, that's right, just realized, sorry.
user11285244
when you use webdriver , which takes longer? ,page loading or getting element. I mean I want to escape TimeoutException. where should I put driver.implicitly_wait()? Webdriver.Firefox() _____ driver.get(url)______driver.find_element_by_id() ______means putting area.
Hey if I want to present a histogram that looks like this: [1.1117e+04, 3.1900e+02, 4.6000e+01, 5.0000e+00, 0.0000e+00, 1.0000e+00], so 99.9% are in one bin. Is it ok to be able to visualize it better to only show the first bin with finer resolution? Do I even mention that one outlier in the last bin?
@santimirandarp Ok. I'd do if number % 2 == 0: print(number), rather than using continue. There's nothing wrong with continue, but it's generally better to avoid it, when practical. Loops with lots of continue or break statements are harder to analyse than loops without such disruptions to the control flow. But sometimes continue does make the code easier to read, and less indented.
@Hakaishin Sweeping unwanted data under the rug isn't a good practice. But I guess it depends on the cause of the outlier. You could display all data and cleaned data. And you can certainly present a zoomed in view of the 1st bin to show important structure. But don't ask me how to do it in matplotlib, or whatever.
not really. Yeah I guess I will do it like this, show all and then pruned
user11285244
Can I ask a question?
user11285244
when you use webdriver , which takes longer? ,page loading or getting element. I mean I want to escape TimeoutException. where should I put driver.implicitly_wait()? Webdriver.Firefox() _____ driver.get(url)______driver.find_element_by_id() ______means putting area.
the problem is why I'm uncomfy to do that, is that the outlier is a bug, that I and my supervisors know about. They told me it is fine and to just leave it like it is, because the time for the project is over and because it happens probably very rarely, as my plots indicate is true. But I'm kinda afraid the professor will ask what is that outlier and I will have to tell him it's a known bug and we didn't have the time to fix it. :(
Which sheds quite a bad light on the project, I feel like. But maybe I'm overthinking and he won't ask
@AtsushiSakata What's the problem? Are you not able to find an element because the webdriver object has not finished booting? You should favour an explicit wait with an expected condition in that context... But I don't completely follow your question
@AtsushiSakata That looks like the same question you posted a little while ago. Please don't repeat yourself like that. If you really need to repeat a question (eg because it got overlooked in the middle of a busy conversation), you can link to it.
user11285244
@Sam my program get a lot of urls using for roop, then get elements from each page using webdriver.
@Hakaishin It's a slippery slope. So many old results in the social sciences are rubbish because of rejected inconvenient data. Most of the time, that was just due to poor statistics methodology, not deliberate dishonesty.
:45989116 url_detail=url+"func=PCTopQuickSearch"
all_info_list=[]
driver=webdriver.Firefox()
driver.implicitly_wait(20)
try:
for i in range(100):
single_info_list=[]
driver.get(url_detail)
try:
driver.implicitly_wait(5)
url_1=driver.find_element_by_id("corpNameLink["+str(i)+"]").get_attribute("href")
if url_1 not in all_info_list:
all_info_list.append(url_1)
except:
pass
driver.quit()
return all_info_list
Yeah it's a super slippery slope and I want to do the right thing. So how do I present this, in a manner that shows there are outliers, but also focuses on the majority of the data?
@AtsushiSakata So apparently The TimeoutException will be thrown when the page is not loaded within specific time. I can see you have an implicit wait of 5 seconds before attempting to grab the element. Are you certain the page is fully loading within this time frame? You should really consider explicit waits. Implicit waiting isn't always reliable IMO
@Hakaishin Sorry, I should read the full convo first.
user11285244
@Sam explicit wait takes too much time implicit is shorter, just
@AtsushiSakata Explicit waits take as long as they need to, to resume normal execution of the program. Your implicit wait is likely not waiting long enough.
Hi. whats is the point of events emitter in node if I have emitter s and listeners in pairs? instead of hey this thing triggered, couldl just just call the function associated to it Is is better when async stuff triggers at random?
@PM2Ring thanks for letting me to join your python room.Can we talk over skype or zoom or something? I have a kind of offer to you. It takes like 10 minutes talk over skype.
@JasminParent Fair enough. There are a few different ways to organise event handling. Different frameworks provide different strategies. And I guess there are some benefits in discussing this stuff without referring to a specific framework, or specific language. OTOH, that keeps the discussion rather abstract, because we can't give concrete examples.
@AtsushiSakata The translation of the code I added is "perform a wait until an element (of the DOM) which has an id=myDynamicElement has been found. What you need to do is find an element which appears in all of the URLs you are visiting (I'm assuming you are scraping web pages from the same website). A usual safe bet is to grab an element from a navigation bar
@JasminParent I don't know Node, but I have used frameworks that use that emitters & listeners pattern. I'm not clear on your actual question. Are you saying "why does the emitter send a message that the listener receives and acts on"?
@SardorIslomov I'm sorry, I don't know you, and I don't give out my contact details to strangers.
@JasminParent You mean why doesn't the emitter just perform the desired action, instead of sending a message and having the listener perform the desired action?
question, does it make sense to club a lot of functions that logically speaking belong to a sort of data processing pipeline? It would make for one BIG class if i do make a class out of it.
but ive never done OOP, but it just dawned on me this pipeline does make a lot of sense logically speaking
@JasminParent Ok. This pattern is flexible. You can have multiple emitters broadcasting event signals, and multiple listeners. And they can do their stuff without worrying about synchronization, since events always have some kind of timestamp associated.
@ParitoshSingh Sounds reasonable. If all the functions make sense as methods. But methods can call functions defined outside the class. Don't just jam everything into the class. If a function could be generally useful with objects that aren't instances of your class, it may make more sense to define it as a separate function.
flask.pocoo.org/docs/1.0/patterns/packages I've been reading this which states its a good idea to have larger applications as packages.. When I do this and freeze my requirements.txt file I get the added -e {github link} requirement. This seems to be causing some problems however when trying to dockerise my application. My docker file attempts to download all dependancies from the requirements.txt file which fails on the github link as it's a private repo
Problems are arising because the repo is private. Is my flow completely wrong?
I think what I'd do is instead of creating a requirements.txt and using it to setup the env in the container, to instead just put the whole wheelhouse in there
i think the command is just pip wheel -w wheelhouse
and then copy that folder into the docker image
then docker can do pip install wheelhouse/* and you're done
I can't test it right now on this pc, but if you run into problems I can try to help
MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl is not a supported wheel on this platform.
The command '/bin/sh -c pip install wheelhouse/*' returned a non-zero code: 1
you can probably still just copy your own wheel over
just not the complete wheelhouse
only authservice
lots of ways to get it to work afterwards. you can delete the line for authservice from the requirements.txt and just install it explicitly afterwards, for example
So I'm seeing something I don't understand. When I import a class using from v.w.x import y as vwxy and from x import y as xy, and try isinstance, I get different answers depending on how I imported the class, and my error message (which is testing passed type) says "expected type y, got type vwxy instead".
@PM2Ring I searched to know more about itertools.groupby() and found a good explanation here but I am not getting how this line is working(actually I'm not getting the working of lambda ):
What looks like is happening is when I import a class differently from how I import it where the isinstance is happening, I get isinstance evaluating to False when I would expect it to evaluate to True.
I will note that that repl session is occurring in folder dionysus/ where class_.py and student.py are in dionysus/dionysus_app/student.py and dionysus/dionysus_app.class_.py respectively
@toonarmycaptain Ah, yes, in that case it actually does matter how you import the class. As far as python is concerned, student and dionysus_app.student are two distinct modules, with two distinct Student classes.
You know what, this may be a case of Pycharm resolves my plain imports where a python repl in terminal won't...
>>> os.getcwd()
'C:\\Users\\david\\OneDrive\\Programming\\PycharmProjects\\dionysus'
>>> from student import Student
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'student'
>>> from dionysus_app.student import Student
@Aran-Fey Fair enough, but unless cwd changes, I'm not able to import two different ways AFAICT. So as long as I don't mess with cwd and uniformly import throughout my application, this shouldn't be an issue?
@Hakaishin I'd use full plot + inset. If you know there's a bug don't hide its effects. Be honest. What if the prof asks why there's no artifact due to the bug and you have to confess that you manipulated the data?
@JasminParent why not ask JS questions in the JS room?
cbg. I hate asking questions on SO, I get either: (i) downvoted or (ii) question is being marked to be closed. And all that happens without any feedback at all. Awful. Mathematics is so much better. I think I was never downvoted on maths.
Thanks Andras, yeah that is good advice. I dont know why but I freaked out way too much, thought figuring this out is much harder than it was. Actually now I made an analysis of the outliers and the “bug“ is actually a difficulty of the problem itself and not a clear case bug.
But for a proper analysis of an approach you need to show cases that work and failure modes, and I did now a proper analysis of the failure modes and the kind of "bug" I thought happens often actually is very rare, like my supervisor thought 5 /10000. And the other failures are also very understandable.
@isquared-KeepitReal I took a quick glance at two of your questions and both had problems. this one is basically only answerable because it has a MCVE - if it weren't for the code, I'd have no idea what you're asking. And this one has the same problem, only worse - no clue what command you entered to get that error message. Your questions seem - for lack of a better word - incoherent.
But yeah, SO gets so much garbage that people can't be bothered to provide feedback all the time. If you want feedback, you have to ask for it
Let's be honest here: We're programmers. We have trouble even just naming variables. Nobody should expect us to have the kind of literary skill that's required to sugarcoat "your question sucks, you need to improve X and Y and Z" for the newbies.
@PM2Ring Yeah the code works, but how it's working? groupby() takes the elements and the function to group with, but you haven't provided any function here