« first day (3078 days earlier)      last day (1861 days later) » 

12:00 AM
I literally have TBs of data available to explore
 
Before we even consider this, is memory an issue?
ok, that was pretty rapidly answered :)
 
for now I'm more worried about processor power since I'm not planning an upgrade at least for a couple of months and all my systems are non-compatible with gpu acceleration ML
 
There is a bit of wizardry in fully jumping into numpy's dtype possibilities, but if you have a good understanding of that, it makes working with strings and other datatypes much clearer and less full of headaches
 
I have 24 GB of ram available on my system so for temp memory it's not the end of the world
 
I think we might be ambling around the edges of the issue here. What is it exactly that you want to do?
 
12:04 AM
I'm interested in seeing if strings via numpy can allow me to handle larger datasets with limited resources, and eventually when I get better hardware that just means that I can work faster with it
 
But that depends on what you want to do.
Off the top of my head, I couldn't say exactly what I would use numpy for tbh beyond basic examples. It's only when the requirement is set that I could try make a call.
 
I've had a couple of tutorials where I already saw like 30 minute epoch cycles in TF if I didn't greatly limit the sample size. With some of the other numpy-driven optimizations I've done I've been seeing orders of magnitude speed ups and want to see if I can replicate that with string operations for text analysis projects
An example of such a project (besides toy models and tuts for me to gain understanding) is going to be a ML chatbot where I want to try and blend a couple of different models and work on some potentially novel ideas I have for boosting systems.
I have another one I might do where I'm dealing with a relatively small corpus but I'm testing out some ideas related to criticality and might have some really nasty combinatorial expansion with that corpus so its a case where streamlining the processing really makes sense.
 
This is too high-level. I'm not trying to be obstinate (genuinely) but we really need some concrete examples of what you want to do with strings to be able to move forwards
 
So the small corpus, are you familiar with this diagram:
 
I have seen that type of diagram before, yes
 
user11093202
12:18 AM
Hmm I havent
 
It's a map of something like 30k-60k interconnections between biblical passages. I want to try and see if I can do something like a latent semantic analysis (its something like a nearest neighbor for words to build semantic neighborhoods) but instead of just examining nearest neighbors I'm going to also compare each interconnection.
 
user11093202
?????
 
user11093202
english plz
 
@Programmer It is in English and I understand what they are saying
 
user11093202
Okkkk I kind of understand
 
12:20 AM
I haven't worked out the details of the algorithm on that yet, but depending on how deep I compare interconnections that can have a combinatorial type explosion in computational complexity.
 
So there was no need for the abrupt and condescending interjection
 
user11093202
How are you going to draw
 
This is interesting because I am in communications with the people that made the map, the test case is small in memory but computationally taxing, and I might be able to map talking about something in a semantically meaningful way
 
user11093202
ya but somehow draw it with graphics how you going to do it
 
That said, I'm personally not going to ask again @Skyler what any of this has to do with the efficiency of strings in numpy calculations. You asked if we had experience and clearly several of us do, but you've only given an outline of what you're trying to achieve and no actual code steps that might be improved by choosing numpy over regular python
 
user11093202
12:23 AM
how though
 
user11093202
With tkinter or turtle
 
user11093202
hello
 
Or any of the other ways of representing the data, like bibviz.com. Are you just blurting things out here?
 
With this example, lets say I had basically a list (or np array) of each sentence in the text, their sentence number, as well as a map of each interconnection (haven't acquired this yet so structure still uncertain), and I compute something like the nearest neighbor (in number of words) for each word in a fed list of strings vs. a numpy string array, then start doing that with interconnections. Would numpy strings be a use case where I could see performance improvements
by performance I refer to faster computation
 
No idea. Numpy arrays don't work faster than lists if all the processing is just done by iteration and/or indexing. There really is no answer to your questions, sorry. You need to make consideration of whether the problem can be formulated in terms of vectorized calculations and that can sometimes take some serious time
 
user11093202
12:32 AM
But how are you going to draw it?
 
user11093202
with tkinter or turtle
 
Sometimes problems can obviously be vectorized if we start with at least the base Python code. But with nothing to go on, it's truly impossible to say whether something might be faster in numpy
 
user11093202
why are you ignoring me
 
I dont really know much about numpy strings and string arrays which is why I'm trying to get an idea of how and when numpy strings use cases do arise
 
user11093202
hello@Code-Apprentice lets talk have question
 
user11093202
12:36 AM
hello
 
@Skyler Make it work in pure python and maybe you'll find your own context where they apply :) It's just too early to say, honestly.
 
12:52 AM
A warning would've been fine. Kicking that user seems excessive. Unless I'm mistaken and they left on their own.
 
1:20 AM
did Programmer get booted?
or whats the context for @coldspeed
 
1:31 AM
btw guys, is there a book as good as pandas for everyone but on numpy
 
 
3 hours later…
4:04 AM
cbg folks
whats the standard for browsing and interacting with your sql db on linux
 
 
2 hours later…
6:17 AM
@Skyler numpy and strings aren't ideal. You need fixed-length strings for improvements, and no such thing as string operations like replace or startswith etc. for string arrays. Num in numpy stands for numerical and strings are not numbers
Pandas has much better handling of string data
@coldspeed I'm pretty sure everyone was asleep but your valued insight on room moderation is greatly appreciated
 
6:58 AM
any reason people don't use f'hi {name}!' strings more?
they seem much better than 'hi {}!'.format(name)
 
7:15 AM
0
Q: c += map(lambda n: n *2, c) kills the Python 3 shell. Why?

NitinPython 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> c = [1, 2] >>> c += map(lambda n: n *2, range(1, 3)) >>> c [1, 2, 2, 4] >>> c = [1, 2] >>> c += map(lambda n: n ...

interesting pathological case... of course this was downvoted when I first opened it.
because it does not ask "How can I use pandas to calculate the powerball numbers of week 51"
 
i wonder what the rationale behind downvoting it are
it has a MCVE, an actual, answerable question, and doesn't look like it was asked before
maybe people are just annoyed by "why is behavior different between python 2 and 3" type questions.
anyway, interesting read =)
 
wim
meh, not that interesting. just a dumb infinite loop.
I would be more interested to know how to calculate the powerball numbers tbh
 
@wim use pandas
 
wim
thank you sir but please help me with the codes i am beginner in python
map + lambda is an ugly combo and just confuses matters here
the infinite loop is really just this:
for elem in c:
    c.append(elem * 2)
 
"how to get 3.2 lakh crore rupees with pandas?"w
 
wim
7:28 AM
now it doesn't seem so pathological or surprising, does it?
 
@wim dict iterator crashes...
>>> x = {'a': 5}
>>> for k in x:
...     x[k + 'a'] = 42
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration
 
huh, is there a way to iterate over a dynamic dict?
 
@towc no.
 
but what if I want to hurt myself?
 
this is also one porting gotcha when migrating Python 2 for i in k.keys() to Python 3 :P
@towc you will hurt yourself. but not the way you intended.
 
7:39 AM
damnit
I can always c += map(lambda x: x*2, c), that should do the trick
 
not with that , no.
 
Have you tried constructing additional pylons?
 
>>> c = [1, 2]
>>> c += map(lambda x: x*2, c),
>>> c
[1, 2, <map object at 0x7fd1063f3860>]
@code_dredd I am a zerg.
spawns more overlords
 
@AnttiHaapala I am also zerg ._.
 
who's not zerg raise their hand
 
7:42 AM
Just look for those trying to construct additional pylons ._.
 
o/
filthy protoss, reporting in
can confirm, trying to build pylons
 
Get 'em!
 
D=
my life for aiur more pylons
 
I am playing with the pylons project... but mainly xel'naga tech, i.e. pyramids
 
Reminds me of the "my wife for hire" meme
 
7:44 AM
how have I missed this?! how have I not ever misheard that :d
 
works a lot better in SC1 where protoss voices sounded like they were pushed through an oldschool synthesizer
we feel you peasants
 
Never played starcraft...
 
never too late =)
Part 1 even got a remaster not too long ago
 
in 2019 Stack Overflow Moderator Election Chat, yesterday, by Andras Deak
I prefer red alert and total annihilation in the real-time strategy genre
 
@Arne the worst spent 9.90 ever...
 
7:57 AM
I also turned it off, but I assume some people can't look at low rez graphics
 
The grim fandango remaster didn't work on my system :(
Bah, more than that
 
8:20 AM
@AndrasDeak Sorry if I'm missing something, but why the sarcasm?
Also, "pretty sure everyone was asleep" is a pretty strong assumption to make
 
I'm serious. I wish you had gotten the diamond so that you could right all the wrongs done here.
 
8:41 AM
"Kicking that user seems excessive" <- that's a pretty strong assumption you made right there
 
cbg
After executemany, I want to get lastrowid for each recording, any good ideas to reach that?
I using MYSQL + MySQLdb.
 
Nothing wrong with making assumptions if they're clearly labeled as such
 
Trying to Crawl Price from notebooksbilliger.de/asus+vz239he not working by XPATH_PRICE = '//div[@id="product_detail_price"]//content()'
how can I crawl the price ?
 
9:06 AM
guys pls
need helpo
iam practicant i lose job otherwise
 
Hi Marc
Being pushy usually pushes people away instead of attracting them. If someone who can help sees your message, they will reach out on their own.
 
Okay I am sorry
Can you maybe help me by this?
 
You're being pushy again. If I can help, I'll say so
 
9:36 AM
Hi everyone. Newbie's here, so please bear with me. When swapping 2 variables I always do this:

temp = var1
var1 = var2
var2 = temp

But in Python, we can do this

var1, var2 = var2, var1

Why both variables are not equal to `var2`? Does this mean the above line is executed simultaneously?
 
@AndrasDeak nah, you don't :) but thanks, and I shouldn't have jumped to conclusions myself.
I should've phrased it as "was that user kicked?" instead, no offence or disrespect was intended.
 
9:53 AM
@Anastasiya-Romanova秀 No, it doesn't really happen simultaneously. What happens is that the right hand side var2, var1 creates a tuple, and then the elements of that tuple are assigned to var1 and then var2 in that order
the important part is that the right side is evaluated entirely before any assignments happen
 
10:06 AM
@Aran-Fey Ahh, I see. That makes sense. Thank you...
 
No, we answer questions we can answer.
 
10:32 AM
@Dr.Marc Your question lacks environment details (using scrapy? beautifulsoup?) as well as the URL. In any case, such questions / issues have only 1 solution, you need to keep trying (& executing) around the particular element you need scraped.
Eventually you would hit the sweet spot & get your desired (raw) data.
 
10:43 AM
@Dr.Marc don't even start with that
also don't ask for help with your fresh questions on the main site as per the rules
 
I didn't scroll up :|
Thursday cbg!
 
cbg
 
on what instances is .copy() method useful for?
 
mutable containers
and whatnot :P
 
yea i'm learning python but
I see a list like numbers = [1,2,3]
why not just numbers_copied = numbers
numbers_copied = numbers.copy() seems kinda weird
 
10:48 AM
because mutability
read nedbatchelder.com/text/names.html and let us know if you still have this question
 
i see thanks ill look it up
mutable stuff are like lists right
immutable are tuples
 
well it's the other way around (lists are mutable, tuples are immutable), but yeah
 
thanks so the terms
mutable and immutable
to clarify dictionaries are mutable also?
 
yup, but you can probably google this
 
and tuples are the only immutable objects?
 
10:51 AM
not at all
I suggest reading what I linked first, then googling "python (im)mutable built-in types" or whatever
 
cool thanks
 
google can't explain mutability to you but it can list the mutable or immutable types for you
 
if i make an application in python right now
and not make updates to the code for 10 years
what are the chances it'll stop working
due to compatibility issues
 
Greetings everyone! :3
 
0%, if you don't update anything (OS/Python/etc.)
 
10:57 AM
cbg
 
@DirtyBit hello
 
i remember some games i used to play back in windows 95
 
they probably still run on windows 95
 
yeah but on later operating systems
i wonder if it'll cause the program to not even work properly anymore
cause i was hoping to just leave it be after i make it
 
Oh yes, that it will
 
10:58 AM
Shouldn't the chances be increasing with the passage of time? :o
 
yeah probably would lol but do you guys see
 
python being obsolete in the next couple decades
 
okay?
 
lol. I see it acing, only.
 
10:59 AM
That's optimistic though
 
cool i'm just worried it'll be like the other languages created before to be more or less useless now
 
Are you hitting on PHP? ;)
 
a lot like delphi
 
I use fortran for work, and I heard COBOL is still relevant. Nothing gets useless.
 
yeah perhaps still useful but i just plan to laern 1 language
so gotta pick the one that will last me probably until i'm about to retire lol
 
11:02 AM
then learn the one that'll be designed in 10 years
 
D:
 
knowing only one programming language for life is as useful as speaking one natural language for life
 
if you know english you'll live
i am optimistic about python though
 
you'll also live if you know only one programming language
just don't go being a tourist in weird lands
 
So are you guys like in favor of "Jack of all trades, master of none"?
or maybe not.
 
11:04 AM
yeah i was just worried since i'm investing a lot learning a language
since i'm next in line managing my dad's company
i don't want to build something that'll just fail in a couple of years
 
Does the company focus on printing programming textbooks?
 
close but no
 
Learn C. I guarantee that it will be around in the long run.
 
i prefer to master a few things
but i feel engineering and programming is a killer combo
I heard C is hard
and much longer code
 
Wait, you want convenient and future-proof? That's some quick scope creep.
 
11:07 AM
yes isn't python
convenient and future-proof
since everyone is in it
 
the problem with the future is that it hasn't happened yet
 
on the AI and machine learning stuff
 
you're in for a treat
have you heard of python3 vs python2? Who's going to run migrations like those in your company?
or the countless unmaintained libraries?
 
no one unless hired
 
For what it's worth they "promise" not to make breaking changes like that.
 
11:08 AM
that is a big problem
 
deprecations happen in any language though
 
you should probably think of any code as a liability, not an asset
 
right my parents tell me to just hire people but i find it fun
 
get rid of any tech that isn't strictly needed
 
if you want fun, become an expert in Malbolge
 
11:09 AM
yeah probably code will get outdated
in many cases really
i was just thinking it's useful
i do the stock market also
and the dynamic behavior is good to study
and i know pyhton is good with AI and stuff
 
Please save some newlines for the rest of us, and put one sentence into one message.
 
how old are you? Feel free not to answer
 
sorry force of habit :P chat spam in games
I'm 24
 
what's your background?
 
industrial engineer, stock market, e-commerce, and at my working for my dad
so I optimize things mainly
 
11:12 AM
and in these things, what was your role?
 
i'm system analyst and developer
 
I don't know what skills are required by an industrial engineer
 
and I sell to retail
 
@Pherdindy what do you program in?
or what do you develop?
 
I only use VBA currently learning python
office systems, web scraping mainly
and math models
statitics, operations research, and etc.
 
11:14 AM
learn haskell
 
what's that?
 
Oh you are good in Web scraping? I really need help at a web scraping problem
 
I know matlab
 
@Pherdindy google it
 
Good advices all around
 
11:15 AM
i'm not I have limited web scraping knowledge
But I pay people to get some code done and I improve it
 
code in functional programming languages never goes obsolete because nobody is going to run it anyway (oblig)
 
Each site has different methods based on the codes I got made lol
go to fiverr.com ask them to make for you some of them are good
 
@Dr.Marc you were kicked because I asked you not to ask about your question on main in parallel.
 
@towc thanks i'll check
 
11:17 AM
I just have scripts for 2 sites www.alibaba.com and www.lazada.com.ph where I sell
I do statistics on them to find a gap in the market
Then I buy in wholesale and sell locally
 
u doing some profit by this?
 
Yeah pretty good profits in e-commerce
I sell in the south east asian amazon
I feel it's ripe to explode more in a short while
alibaba purchased the company 2 years ago
 
@Pherdindy oh, now I know why your user was familiar. You were struggling to get around the safeguards on lazada.com.ph to scrape what they didn't want you to scrape
 
> And I found a way to maximize reviews
 
Well I learned they had an API
 
11:19 AM
Oh yeeeeah
 
so it's perfectly legal
 
yup yup
@Pherdindy so now all that's left is the fake reviews you make for yourself?
 
I learned reviews are not in their algorithm
It's based on sales on the past 30 days so not really important for me
 
Buyers ignore the fake reviews too I think. They just skip them automatically.
 
they removed all the fake reviews I made
But I did it poorly then
 
11:21 AM
I'd have loved for that to be "I removed all the fake reviews"
you should probably learn haskell after all
 
well they improved their policies
just this month fake reviews are grave offenses
but glad I made it before that happened
yeah might be useful i'll check which one i'll try out
is haskell the next big thing?
I saw this "bye bye python post" about it
 
you can ask the haskell room
 
@AndrasDeak nice
@Pherdindy I thought you didn't want a next big thing
you wanted "a thing that will still be a thing"
haskell is arguably somewhat of a thing, and I don't think that will change
it's not big by any standards
but really good for mathsy people
 
yeah I was just wondering if it'll make python obsolete
I just saw a random site
 
ok, you're not listening to me
 
11:32 AM
with that title
Since i'm planning to learn python I was just wondering if it'll make it obsolete because I want "a thing that will still be a thing"
Yeah probably haskell is like matlab which focuses on a field right?
If that's the case python is still a good choice for my purpose :D I think i'll stick to it
 
great
 
11:53 AM
the reference by ned batchelder is great paints a picture to abstract concepts to beginners like me. the references at python.org are just not beginner friendly at all
 
Oh wow, the documentation is not meant to teach the basics?
stop the press :P
 
12:10 PM
hard to understand the phrasing a python.org when I read it lol
I gave up and just watched some tutorials
 
real pros learn python starting from the C API
 
I was reading this
just reading the first few paragraphs is alien to me
 
Not that I can guarantee that it would've helped, but have you read the first 8 chapters?
 
No but some concepts are introduced for the first time like in 9.1
Despite having some background on the basics the way it was phrased is hard to comprehend
Would prefer something less abstract
Since I have to keep re-reading lines over and over
 
@AndrasDeak I don't know about you, but I compiled to assembly and reverse-engineered, to really see what's going on, without any abstraction bs
 
12:21 PM
So far the fastest learning i've got was watching videos lol
mosh hamedani's python course can jump start beginners like me
 
I'm still far from convinced you're not 9 years old
 
No need for name calling. Plus if they were younger than 13 then their account would have to be nuked on account of COPPA.
 
ok, how about 14
 
irrelevant
 
Every user in this room is a pulsing orb of energy existing outside of time and you can't prove otherwise
 
12:35 PM
I'm a power system engineer performing power flow analyses on various networks. A network is made up of stations and lines (or nodes and edges). I want to visualize the flow in each line (edge), and the voltage in each station (edge). I can show this with numbers next to the nodes/edges.
Do you have a suggestion for a tool I can use for this?
 
Demeaning others by implying that they're children is not a constructive means of communication.
rhubarb for a while
 
Ideally, I'd like this result to look something like this, without the transformer and generator symbols (the circles).
 
NetworkX and matplotlib are capable of generating images with labeled nodes/edges, if python-course.eu/networkx.php is to be believed.
 
But I'm satisfied with something looking like the output from igraph, if I can get the information from the simulations directly inserted.
 
draw.io
 
12:44 PM
@Kevin I've looked through the documentation, but nowhere can I see an example with information next to an edge.
 
Me neither, but it's such a basic feature that I assumed that every graph visualization library in existence would support it. But I have a terrible track history of guessing what counts as a basic feature, so maybe not.
 
@Dr.Marc I can draw diagrams in various programs, but I want to show the simulation results from Python too. Is that possible using draw.io?
 
Oh sorry I am not able to answer this question.
 
Looking at igraph's site, it claims to have Python bindings, so I expect that you could insert data from simulations into it.
I suspect I'm misinterpreting the problem because if you've used igraph before and it produced exactly the output you want, then you wouldn't need to come in here asking for graph library recommendations
 
@Kevin my comment was quite misleading... I would be satisfied with how the graph is drawn, if only I could add the necessary information to nodes and edges.
 
12:59 PM
What I'm hearing is "Using igraph I can generate a graph that matches my network. The layout is correct but the nodes and edges are not labeled". From the igraph tutorial I can see that it can label nodes. I don't know whether edges can be labeled.
 
@Kevin That's pretty much it. But I want more information than just a single label.
 
stackoverflow.com/questions/44077403/… implies that edge labels are possible but the code is in R so I don't know what the Python equivalent is
I'm not sure what "more information than just a single label" means. Surely a label can contain however much text you care to put into it?
 
sorry, wasn't meant to be demeaning. Just a reality check for the person
well, a bit demeaning, sure
 
If you need your edge to display both voltage and amps, then just do edge.label = "voltage: {}; amps: {}".format(voltage, amps) or whatever
Ain't no law that you can only do one
 
Uh, newline characters are allowed in csv fields, right? Right?
I'm dealing with a lazy CSV import feature that can't seem to handle newlines, and I just wanna make sure I'm not in the wrong before I start cursing it and its devs
 
1:07 PM
stackoverflow.com/questions/566052/… says that newlines can be in csv fields under certain conditions
TLDR: you have to quote the field.
 
thanks, that's exactly what I needed
2019 and people can't import csv correctly smh
 
The bad news is that there's no official CSV standard so if a csv importer fails to import your csv, you can't even yell at them for not following the standard
 
shhhhhh
let me be mad, I wanna be mad (at them)
 
Ok, let me backpedal. There is a de facto standard, and that standard is "whatever Excel can consume and produce". Excel can consume and produce csvs with embedded newlines, so any parser that can't is a bit pitiful
 
Apparently parsing 300 rows of csv data takes longer than 5 minutes. Who knew?
I'll probably end up throwing a chair before this day is over
 
1:36 PM
I parse Amazon prices but now get "EUR 29,99" in my PRICE Variable.

How can I now get the EUR and the whitespace out of this?

I want it to be 29,99.

In Python
Please dont kick me out for asking questions here. I have been Time limited on 90 Minutes on stackoverflow.
 
There are several possible solutions. The easiest one is to slice off the first four characters: price = price[4:]
 
And I want to have an IF before so if its not "EUR ..." it should not be cutted
Ok thx thats 1/2 of my solution already
 
>>> s = 'EUR 29,99'
>>> s
'EUR 29,99'
>>> s.split(' ')
['EUR', '29,99']
>>> s.split(' ')[1]
'29,99'
 
You can determine whether a string starts with "EUR " by doing price.startswith("EUR ")
 
Thanks for your fast and good answers
 
1:42 PM
Parsing plaintext from websites can be tricky because you never know if one page out of a thousand renders its prices as "29.99€" instead of "EUR 29,99" and then suddenly any solution depending on splitting or blind slicing will fail to work
 
So I guess checking first is when parsing unreliable things is better
what about this
>>> s = 'EUR 29.99'
>>> t = '29.99€'
>>> s.split(' ')[-1:]
['29.99']
>>> t.split(' ')[-1:]
['29.99\xe2\x82\xac']
 
print(SALE_PRICE)
if not SALE_PRICE is None:
if SALE_PRICE.startswith("EUR"):
SALE_PRICE.replace("EUR ", '')
print(SALE_PRICE)
ends in
EUR 119,90
EUR 119,90
:4 didnt work for me
*$:
*4:
 
Ahhh forget that... just compile a nice regex to get your floats out
 
Yikes
It actually was a syntax problem
yes i wrote your solution down i will now implement this
 
Note that SALE_PRICE.replace("EUR ", '') does nothing because replace() does not modify the string in-place. You have to assign the result to a variable.
In general, nothing can modify a string in-place because strings are immutable
 
1:51 PM
oh okay
nice to know
 
A lot of new users are confused by the fact that many functions behave this way. They write code like x = 1.5; int(x); print(x) and are surprised to find that x is still a float.
 
Eh, wrong window
 

« first day (3078 days earlier)      last day (1861 days later) »