« first day (3726 days earlier)      last day (1207 days later) » 

12:18 AM
@JansthcirlU my suggestion is to poke the answerer with a couple of follow up questions or open a new one
 
@cs95 that's twice wrong
1. they just posted that here for us to see the answer to their original question; no follow-ups necessary. 2. suggesting to post follow-up questions to answerers is how you get chameleon questions
 
Disagree with 1. I think it is perfectly ok to follow up with an answerer if you need a little more help getting their solution to work on your data. OTOH if your actual case is such that the answer cannot be applied at all without significant changes that's a hint the MCVE was broken to begin with
 
that would be 2, not 1
 
@JansthcirlU - BS is definitely the "go-to" package for HTML, but sometimes pyparsing can be helpful in extracting complex text expressions. (Note that pyparsing's makeHTMLTags gives you parsing expressions that will also handle attributes annd other unforeseen HTML variations.) Here is a pyparsing approach:
import pyparsing as pp

# define a scanner to find "<td> ... </td>" text
td_start, td_end = pp.makeHTMLTags("td")
td_scanner = td_start + td_start.tag_body("body") + td_end

# use a parse action with an HTML stripper to remove any embedded tags from td tag body
def cleanup_body(t):
    html_strip = (pp.anyOpenTag + ... + pp.anyCloseTag).suppress()
    t["body"] = html_strip.transformString(t["body"])
td_scanner.addParseAction(cleanup_body)

# try it out!
sample = """<td><span>some span text</span>some td text</td>"""
Prints 'some td text'
It will fail if there are <td> tags embedded in the <td> tag body, so if that is a possibility, then that will take much more effort.
 
@AndrasDeak 1 and 2 tie into each other. Understand what the XY problem is, fix your MCVE, then re-ask your question again, this time more accurately. What I would explicitly discourage here is making massive edits to your question to invalidate the existing answers.
 
12:29 AM
OK, but no for the first half. JansthcirlU asked here first, got no answer, asked on main, got an answer, accepted it, then came back to report that the problem is solved.
 
Ah, apologies. Missed that context
 
 
6 hours later…
AAB
6:59 AM
hi all
need some suggestion and advice on scraping and a website that hosts the data.
I am using scrapy and to remove duplicates while scraping I am using a pipeline class which inserts the product name into a set. This works fine but I feel I see the memory usage get high really fast
 
A set of product names certainly won't eat up your memory, so you'll have to show us some code
 
AAB
@Aran-Fey Its a machine with 1GB
 
It could have 10 MB and a set of strings still wouldn't matter
 
AAB
If I run 2 -3 spiders in parallel
I cant even do ls on the machine.
each spider should have 2000-3000 unique products at max.
 
7:15 AM
Yeah, that code snippet doesn't help one bit. You posted exactly the part of the code that I already told you isn't the problem
 
AAB
@Aran-Fey so I should look at the code that inserts to database or the spider itself?
 
I don't know, but if I had to guess, I'd say probably the spider
 
AAB
@Aran-Fey I have left the default concurrent settings, could that be the reason?
 
No idea, I don't know scrapy
 
AAB
@Aran-Fey okay thanks if the set is not eating up memory then it has to be in the settings, I am not using a list or dict or any datastructure in the spiders. Anyways let me reduce concurrency and check.
 
8:09 AM
@AAB maybe use a sqlite database?
should be a tad bit slower, but it won't give you memory issues.
And slowness shouldn't matter as much since your spider will already have throtling and other stuff in place
 
8:42 AM
recbg
 
cbg
 
Hi all
I have a question about list.
I have this list as example: 3.2.6, 3.3.24, 3.3.25, 3.2.7
means it contains version as string.
sorry, the correct list is:
`3.2.6, 3.2.24, 3.2.25, 3.2.7`
 
And the question is?
 
When I'm trying to sort it I get:
`3.2.34, 3.2.35, 3.2.6, 3.2.7`
 
Because you are sorting them as strings.
 
8:51 AM
any idea how I can get in real version order:
`3.2.6, 3.2.7, 3.2.34, 3.2.35`
 
>>> 25 > 6
True
>>> "25" > "6"
False
 
Yes got it. How can I convert this list to present me the required output?
 
Convert each entry to a list/tuple of integers, sort and convert back.
 
is there a way to it all in one line?
*do it
 
For a string s you can produce such a list with [int(d.strip()) for d in s.split('.')]
Why do you want to do it all in one line? Specifically because you want the code to be hard to read?
 
8:57 AM
so the code you gave me turns it into tuple?
I saw this thread:
https://stackoverflow.com/questions/41516633/how-to-convert-version-number-to-integer-value
Sorry, I think this one is more usefull:
https://stackoverflow.com/questions/12255554/sort-versions-in-python/12255578
 
No, that produces a list, which is good enough.
If you want to accept the full range of possible semantic versions then you should probably check out [the `semver` library](https://pypi.org/project/semver/)
Perhaps think a little harder about what you need to do and stop looking for ready-made code. The latter isn't programming except in the most superficial sense, and won't help you to develop coding skills.
1. Convert list of strings to list of lists of integers; 2. Sort the list; 3 Convert it back to strings.
You could wrap that all in a function, which would keep things tidier.
 
what about using distutils.version?
73
Q: Sorting a list of dot-separated numbers, like software versions

ZackI have a list containing version strings, such as things: versions_list = ["1.1.2", "1.0.0", "1.3.3", "1.0.12", "1.0.2"] I would like to sort it, so the result would be something like this: versions_list = ["1.0.0", "1.0.2", "1.0.12", "1.1.2", "1.3.3"] The order of precedence for the digits...

look like it works for my case. what do you thing on that?
I do wonder what is the diff between using key=LooseVersio to key=StrictVersion
key=LooseVersion to key=StrictVersion
 
9:30 AM
pastebin.com/jWDGfMmS <<this code prints approx 400 words of string, i want to print only 40 or so how can i do that?
 
@Praveen <unclear> What exactly do you want to do? Do you want to print the first 40 or something?
 
yes
i want to print first 40 lines
 
Then just make a counter and break if it reaches 40.
Learn Python properly.
Even easier, slice the list.
 
@A.Man treat versions as tuples of integers, which fits schemes of major [minor [patch [...]]]. Compare (3, 2) > (3, 1, 12).
 
 
1 hour later…
AAB
10:50 AM
@shad0w_wa1k3r Let me confirm if the issue is due to something else or the set datastructure if so then will use it.
 
11:13 AM
I guess this was trending on reddit
 
11:23 AM
Not really related to anyone, but I solved my own jupyter problem.
 
11:37 AM
@user202729 "All these worlds are yours. Except Europa. Attempt no landing there"?
oh wait... wrong Jupiter :p
 
11:53 AM
hey guys, just trying to do a logistic regression on my dataset. It worked so far and now I want to see, how important the single features in my set are. So f.e. age -> How does age correlate to my outcome which is either buy or not buy (1,0)

I used `model.coef_[0]` and got several outputs. How can I now interpret these? So f.e. I got ["-1.555665","0.433334","....]

Thanks in advance!
 
12:09 PM
cabbage
 
12:28 PM
@grumpyp Sounds like Principal Component Analysis. In PCA, the respective contributions of each component are re-synthesized into linear combinations of the inputs to find those that are most significant. Often one will plot a scatterplot of the first two (coef[0] and coef[1]) to get a visualization of these components.
PCA is good for filtering out correlated inputs, such as if you had two columns of data, one being "age_in_years" and another being "age_in_months" - these values would clearly be correlated, and when trying to reduce the model's input dimensions, you should be able to drop one of them.
For each component (say you have properties X, Y, and Z), the coefficient is the triple a, b, c for the weights to compute the linear combination of the X, Y, and Z values. That is, each component is found by evaluating aX + bY + cZ. Look at the inputs you gave to your analysis function to interpret each tuple of weights.
For example, if your inputs were the list attrs=["height", "age", "weight" ,"postal_code"], you should get back coefficients of 4-tuples, and for the 0'th (most significant) component, the four values would correspond to the respective weight or significance of the corresponding input attribute. You could do dict(zip(attrs, coef[0])) to create a dict of each input and its respective weight. The impact of "age" would be the second value in this tuple.
To compute the synthesized component values as a scatterplot, for each data tuple in your input set you would use an expression like sum(a*b for a,b in zip(coef[0], data_tuple)) to get the X value and the same with coef[1] to get the Y value. I would bet 50 cents that pandas has a function that will do this for you on a df.
 
@PaulMcG cheers for that detailed answer buddy! Let me work with that a bit and I may get back to you! Thanks!!
 
1:26 PM
@PaulMcG I'll look into that, thanks!
talking about yesterday's pyparsing comment, that is
 
Ah! okay
 
 
3 hours later…
4:45 PM
sorry someone can help with a model of machine learning ? (already implemented in tensorflow)
 
 
2 hours later…
6:17 PM
Ahh, the joys of string formatting with SQL queries - f"... BETWEEN PARSE_DATETIME('{start.strftime("%Y-%m-%d")}', 'YYYY-MM-DD') AND ...". [weeps internally]
 
I can't tell if you're upset because string formatting is unnecessary there (i.e. you saw some n00b write vulnerable code for the billionth time) or because it's necessary (i.e. the database framework won't let you insert a variable in a query at that location)
 
That it's totally unnecessary and open to injection. There's gotta be at least 3 transitions between strings and datetimes in that code, all for getting the sweet, sweet f-string syntax
Although, I suspect that it would fail if you did try injection in that particular case
Looking at PEP 249 it looks like db connections should be able to take datetimes by default, without requiring conversion to strings. So unless I'm misreading that first tip, there should be some assurance that this will work across all SQL dialects
Although this question pertains to Amazon Athena and apparently that doesn't take parameterized queries. Wut?
 
Nice
 
6:32 PM
There's our monthly reminder that the world is a strange place
 
I shall now proceed to eat all of my Xmas hats. I'm truly surprised by that one.
 
 
1 hour later…
7:46 PM
@roganjosh is that "queries with functions" an inappropriate replacement?
 
It's a fiddly work-around for sure and I'm not quite sure why it's needed. But I've spent some time researching, and the OP has just informed me that it does now take parameterized queries, so it's possible that the Python wrapper takes care of that for you (next thing for me to check now)
If anything, it's interesting that the question has made me read in more detail around the AWS infrastructure, so that's not a totally bad thing :)
Sweet, they did implement the DB spec in some way
 
that's third-party, right?
 
Looks like it. For context, the AWS wrapper for Redshift was only released around a month ago
I haven't quite dug out how exactly they work on escaping query inputs yet, but it seems to be pretty well-written as a 3rd party extension, from what I'm scanning through
I guess I should really say "engine" rather than "wrapper". Or maybe "binding". I don't know the exact terminology that's most appropriate. In any case, nice work that a 3rd party came along and made something that abides by the PEP and get around that limitation :)
 
 
2 hours later…
10:02 PM
Are you guys aware of any image rotation that happens when we convert image to bytes using BytesIO and base64 for decoding. I converted my images and when taking them back to original image, they seemed to be rotated 90 degrees.
 
the elusive rot90 encoding
 
Images can be rotated using metadata, which your python program probably ignores
 
@Aran-Fey That means, its the problem of me reading the image or the bytes?
@AndrasDeak What could this be
 
@CoolCloud a joke
 
@CoolCloud Uh, neither? The problem is that you're not reading/processing the image's metadata, only the pixel data
 
10:07 PM
Oh like that you mean.
 
So I supposed there might be no other way using bytes to read those data right?
 
I think "orientation" is the relevant key
 
Let me try it out
 
10:19 PM
*adapt. I don't think it's gonna work out well if they cargo-cult that
 
I'm surprised we have to do that metadata stuff manually. PIL seems to be taking a "batteries excluded" kind of approach
Ok, that's a lie, I'm actually not really surprised
 
10:41 PM
@Aran-Fey Only interms of bytes I suppose.
 

« first day (3726 days earlier)      last day (1207 days later) »