Hi guys, I am having a problem with some web-scraping using Python, Selenium, and Beautiful soup. I am attempting to get some url's from a page source, but I am unsure how to return just the URL portion of the HTML, rather I am having to return a much larger chunk, because the URL is linked as an attribute of style. I have uploaded an imgur to highlight the url I am looking for. i.sstatic.net/U7U0L.jpg If anyone is able to talk with me about this I would greatly appreciate it.
@RaguvaranR They "are good" because they set up guidelines to what can be asked. No one like a cluttered chat. We are python programmers, we don't answer questions about unix here.
@RaguvaranR 1) Even with the current rules that forbid off-topic questions we still get an annoying number of off-topic questions. The rules are fine - or maybe they need to be more strict. 2) I'm pretty sure none of us can answer your question.
But the first 100 days i have no volume data.... so i cand delete rows with volume - or put a value like mean instead or some kind interpolation
what do you think ? any suggests ?
I have to clean and format the data and make some study... i made some plots date vs hig, date vs open, volume vs market cap ... any idea of interesting plots ?
In [48]: a
Out[48]: [1, 2, 3, -4, -5, -6]
In [49]: li = [(-abs(x),x) for x in a]
In [50]: li
Out[50]: [(-1, 1), (-2, 2), (-3, 3), (-4, -4), (-5, -5), (-6, -6)]
In [51]: max(li)
Out[51]: (-1, 1)
same in terms of the key, different only because you do not get a tuple as output. But yes, that's how it's working off of the hierarchy during comparison when it comes to tuples. one value at a time, left before right.
@DeveshKumarSingh so, what part confuses you here? Sounds like you're clear on the hierarchy comparison that tuples do, but are wondering about the internals of how it chooses a max? did i read that correctly?
@EduardoGutierrez both are valid options. you'd have to choose one. One thing to consider is how big is 100 rows in comparison to your total number of rows. If you are using mean from 20 values to fill up 100 rows for example, you'll most likely get a biased dataset. At that ratio, it makes sense to drop the rows, ( or the column too in some cases)
sorting happens on the first element, then the second element is considered for (1, 0, 99) and (1, 1, 0) and that's it right, the third element isn't considered here
@ParitoshSingh thanks this is my full file: In concrete this :https://coinmarketcap.com/es/currencies/bitcoin/historical-data/?start=20130428&end=20170905
id like to know also which plots can be helpful and what statistical functions like mean, standard desviation, interpolation...would also be useful to make a study... We are cleaning and formating/analysing data
yes I realized when I typed, so a layman way to understand is to sort and pick the maximum or minimum, to think about max or min where a tuple is considered as a key
@EduardoGutierrez plot all of the ones you can think of, and see. When it comes to making a study, results tell you best.
yes, in layman terms you can think of it that way. because by definition, the max will be the last value in ascending order, or first value in descending order.
that's why its max after all.
@EduardoGutierrez Also, you can read up on what other people have used for similar studies in the past. searching for "time series" or "stock market analysis" may give you some ideas on what you can use. But i would still suggest trying the different plots out, and seeing what comes out of it.
Got it, and it case of multiple keys, we only use inner keys to resolve conflicts between tuples where the previous keys are the same, and once an ordering is obtained, we don't care anymore
Aah, man this is mind bending for me, because I am used to seeing max or min from the list of numbers point of view, but that ordering is not obeyed here, instead the ordering is from the tuple
@Aran-Fey Is that because the first elements breaks all ties, so the second element will never be touched while calculating the max?
@Aran-Fey Isn't that just the life of a programmer? Working too hard on one part of your project, suddenly it's a new project, and then you've got little parts to that project, and then those become new projects, and it continues forever...
Not all questions (actually, most questions) need questions on main. With the quality of questions people tend to ask, you're more likely than not setting up innocent people for downvotes.
@connectyourcharger if they'd have found the answer by reading SO main they wouldn't be asking here. When you ask for help you are vulnerable to what people you expect to know more than you are saying. If they tell you to go ask on SO you won't contemplate it, you will go and ask, even if it's a bad question.
And frankly, sending people away when there are people here who are willing to hear them out is a disservice to both parties. If I want to send them away I can do that just fine myself.
Hello! We appreciate you coming to ask a question here, but that's what the main website is for. What you should do: First, research your question on the internet and on Stack Overflow. If you can't find a solution here, come back here to this chat room and we'll try to help you. We may direct you to the main site at some point, and at that point it would be best to ask a question on Stack Overflow. Thanks, and have a great day :)
Sending askers here to SO main will just make SO main worse and the users more frustrated. The reason we try to help is to try and teach the askers something, which is not SO main is for. Fishing and whatnot.
@connectyourcharger how about leaving new users alone and let regulars and room owners do what they want?
At which point we tell them to go away and learn. If you start posting that condescending paragraph to everyone asking a question you'll be much more of a problem than the actual people asking newbie questions.
I mean yes, it could get repetitive, but it helps ensure that the quality of questions are better on SO, because the user has to go through other helpful steps before asking a question
And chances are they will never actually have to ask the question
The rules of the room outline that it's OK to ask a question in here, and new users who don't read the whole rules won't understand that if their question meets SO's standards, it's better off there
Frankly, that rule needs to go right to the top of the "Asking a Question" section
We should make an effort to clean up that section of the rules
This question right now that you would've sent to SO main wasn't up to SO's standards and doesn't need asking on SO. It's as simple as "your design is off, use instance attributes". They couldn't have known, and we can tell them in 5 minutes if we hear them out.
@connectyourcharger we have problems with users not reading the rules, but you're the first one to think that people's programming questions here are a problem. They are not.
What and how you've been discussing this comes across as you just wanting to send everyone away without even listening to them on the basis of "they should know better".
I think we're getting nowhere. I don't really want to have an argument - I respect your views and opinions - I just think it needs to be more clearly established the types of questions that are OK to ask in here, rather than on SO.
I've wasted way more time debating this with you than the amount of time I would've spent convincing the last asker that they need instance attributes and guide them how to use them
and coaching somebody into providing the necessary details in an interactive fashion is a lot more productive than on SO main, which is one of the reasons we have different rules here
(though sometimes we fail anyway, because some people are in way over their head)
@DeveshKumarSingh When you use a key function with max (or min, sort, or sorted), Python is doing an invisible Schwartzian transform on the data. So the transformed values are used in the comparisons needed to find the max (or do the sorting) but you never see those transformed values, only the original values.
To get a good feel for what's going on, I recommend using a proper def function for the key function, so you can put some print calls in it that tell you what arg the key function is called with & what it returns.
@PM2Ring Thanks @PM2Ring Yes, there was a question asked here about it: stackoverflow.com/questions/56182445/… But I didn't know what it meant before but using an explicit function for a key and having print statements inside it to see what's going on is a good idea
FWIW, Python guarantees that the key function is only called once on each original value, even though a given (transformed) value will be used in several comparisons before sorting is completed. If the key function is expensive to compute, and the list contains lots of dupes, it can be worthwhile to give the key function a cache.
good morning good guys, looking for some luck: any of you ever worked with Python wrapper of Pipx40 library from Pickering vendor? I'm able to communicate with acquisition board with .NET and C but Python refuses to play with me...
@PM2Ring yes, we decorate the arguments with the output of the key function, sort via the updated keys via comparision, and then undecorate the arguments to get the actual value? Is that what this means
Even if they didn't do the transform, the output we expect from max and min would have been the same, just that the function run would have been more time-taking and inefficient
@DeveshKumarSingh Yes, calling the key function on both values every time you do a comparison when sorting would be very inefficient. With max, it's not so bad, you just have to remember the current max & its transformed value.
To be specific, from how the wikipedia article reads, we cannot call it a Schwartzian transform in case of min-max i think(? correct me if wrong, since no sorting happens for min-max). I just am worried you keep wanting to mix min-max with sort here, and makes me regret bringing up sorting in the context.
@grandangelo Maybe not here, but you might have some luck on the main SO site, if you clearly give all the relevant details, eg OS, and library & Python versions, and any error messages. It might be that Pipx40 only works on Python 2, and you're using Python 3, or vice versa.
@ParitoshSingh Well, the low-level details of how the Schwartzian transform is done are invisible to us. It probably uses C data structures that we can't access directly. But at the Python level, the key function of sort and max or min works exactly the same way. And the same goes for itertools.groupby.
@PM2Ring Aye, i suppose at this point its the terminology that leaves me confused. the article very heavily implies "Schwartzian transform" would necessarily involve a sort step. Would it then be wrong to call it a Schwartzian transform in context of a max calculation? (where sure, the same dict-like structure can be employed to avoid recalculating a dupe key, but no sorting happens)
@DeveshKumarSingh No worries. As a room owner, I feel obliged to keep up with the transcripts. I don't closely read stuff I don't know, like Pandas or Django, but I still skim it, to make sure people are behaving properly. ;)
, sir, thank you very much. Looking for ctypes call differences between Python 2 and 3, I found on SO a post suggesting to take care of string encoding. Adding a encode('ascii') to my call fixed my issue. You saved my day, after hours of trials with no luck :-)
@ParitoshSingh If you have a list containing dupes, the key function does get called on the dupes. But it only gets called once on each original value. So if you sort a list of 16 values, the key func gets called 16 times, even if all those values are identical. You can verify that by putting a print call in your key func.
So that then just makes me think there's no memoization* (for the key calculation) happening during a max call, huh. You're right, the print displayed multiple times
from operator import itemgetter
s, n = map(int, input().split())
lst = list()
counter = 0
for i in range(n):
x, y = map(int, input().split())
lst.append([x, y])
lst = sorted(lst, key=itemgetter(0))
for i in range(len(lst)):
if lst[i][0] < s:
s += lst[i][1]
counter = 'YES'
else:
counter = 'NO'
break
print(counter)
@RaphX You can make the sort more efficient. The sorted function creates a copy of the original list, and then sorts it in-place. So it's more efficient to just sort lst in-place
And I would store the x & y values in a list of tuples, rather than a list of lists. That's mostly a matter of style; it won't make a noticeable speed difference, but it will save a few bytes of RAM.
@RaphX Actually, you don't need enumerate there, since you don't really need the indices. You can just do:
for x, y in lst:
And you can make your flag-setting logic more efficient. There's no need to do counter = 'YES' each time he wins.
I think you may be able to improve the key function, to maximize the strength bonus as soon as possible.
s, n = map(int,input().split())
a = [map(int,input().split())for i in range(n)]
a = sorted(a)
print(a)
for x,y in a:
if s > x:
s += y
else:
print("NO")
break
else:
print("YES")
>>> sorted([map(int, ["14"]) for _ in range(3)])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-42-f4066b4913b5> in <module>
----> 1 sorted([map(int, ["14"]) for _ in range(3)])
TypeError: '<' not supported between instances of 'map' and 'map'
@RaphX a = [tuple(map(int,input().split())) for i in range(n)]
@RaphX And you forgot the key function for sort, but that probably doesn't matter. I'd probably use lambda t: (t[0], -t[1]) So if 2 or more items have the same x value, he fights the one with the highest y value first.
Does anyone know what stack I should use if I have to make a web app which has the CRUD functions, data science package support like pandas, or similar packages like dplyr, data.table, etc?
I have an app in R shiny but need to add more functions to it.