« first day (3594 days earlier)      last day (1369 days later) » 

12:34 AM
yo @smci I appreciate your reply. I'm convinced the best speed benefits will come from tuning up the DE, which we'll be doing this week, but any advice on the code will of course be positive too, if you wanna take a look.
You can skip information on comments if you want. The code is very minimal.
https://pastebin.com/xTKKHaCV
 
1:19 AM
@PedroSpinola Task Manager's CPU usage is just a calculation from the System Idle Process (which calculates the unused percentage of the CPU). In other words: its the percentage of time the system is not running idle tasks (its not an actual calculation of total CPU usage which would be odd as a single number given multiple cores)
^ once again I say - I know way too much about Windows
and for completeness: you'd use some form of wmic cpu get loadpercentage or typeperf or Powershell with Get-WmiObject Win32_PerfFormattedData_PerfOS_Processor to calculate accurate usage values. Or just right-click in Task Manager (performance) and select "Logical processors" (in graph options) to see a good estimation of usage per core (if you hand calculate changes between starting and stopping program)
@AndrasDeak That's because Task Manager is not suppose to be used to monitor performance. Use the correct tools people :P ;)
 
 
1 hour later…
2:47 AM
I don't think that code I sent you is working @smci. Taking a look at it now.
 
 
4 hours later…
6:20 AM
indexNames = a_df[
(a_df['col1'] == '-1.00') | (a_df['col1'] == '-1') | (a_df['col2'] == '-1') | (
a_df['col3'] == '-1') | (
a_df['col4'] == '-1.00') | (a_df['col4'] == '-1') | (a_df['col5'] == '-1.00') | (a_df['col6'] == '-1')].index
a_df.drop(indexNames, inplace=True)

I am using above code to remove all the columns in pandas df with value '-1.00' and '-1' but it is dropping other column also which does not the these values for the above columns any idea why
?
new to pyhon is this the right way to do this ?
 
6:36 AM
Have you tried testing this with a simpler setup, e.g. a 3x3 dataframe from which you drop just '-1'?
 
7:03 AM
@bhavesh I'm not much of a pandas user but the index part seems a bit weird to me. I'd do
indices_drop = a_df[(a_df['col1'] == '-1.00') | (a_df['col1'] == '-1') | (a_df['col2'] == '-1') | (a_df['col3'] == '-1') | (a_df['col4'] == '-1.00') | (a_df['col4'] == '-1') | (a_df['col5'] == '-1.00') | (a_df['col6'] == '-1')]
and then a_keep = a_df.loc[:, ~index_drop] (or something similar if this is incorrect, I haven't tested it) and if needed, a_df = a_keep. It will create a new dataframe rather than modifying the original one, but this might not be a problem
depends on whether you're changing your dataframe in a function without returning it, for instance
but as Miyagi said you should always play with a small dataframe to see what works and why not, and then you can use that to give us reproducible code that doesn't do what it should
 
ok , will try this
 
Also, do your columns really have to be strings? If they are all numbers you could use a numeric dataframe, in which case -1 and -1.0 would be equal (and your columns would only contain either, depending on whether the column is integral or not)
 
that's already the designed schema can't changed that now :(
 
Have you guys ever seen this kind of spurious error with tensorflow on Windows? 2020-08-18 00:30:27.321368: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
 
7:46 AM
no
 
8:04 AM
If I have an object and it calls (). Does it call the attribute called func? Or what does it call?
 
@Hakaishin you mean when you do obj()?
it calls __call__
 
@AndrasDeak yes
shouldn't I see __call__ in the debugger?
I somehow don't which corresponds with the behaviour of nothing happening, but I'm not sure if there really is no call or if I just don't see it in the debugger
 
@Hakaishin put backticks around it to protect from markdown
@Hakaishin I would presume you see it just like any other funcion call. Did you put a breakpoint inside __call__?
 
no somehwere where I have the object, then I can inspect it, I find args, callback id, job_id, kwards yada yada, but no __call__
 
@Hakaishin how are you inspecting it?
>>> class Foo:
...     def __call__(self, *args):
...         return f'called with {args}'
...
... f = Foo()
... '__call__' in dir(f)
True
you are probably inspecting its __init__
 
8:21 AM
I'm trying to get dsbowen.github.io/flask-worker flask-worker to work, but the job somehow never finishes. I get a loading page as expected, but it never returns the result, not sure if it's some socketio issue or browser issue or why it's not working
 
8:36 AM
Hi, i am trying to install sklearn to my raspberry pi, as I already have a program based on it. I always get this when installing with pip3:

    Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-        install-mpx8pkea/scipy/setup.py';f=getattr(tokenize, 'open',     open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-_x6zslus/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-mpx8pkea/scipy/
 
9:33 AM
Hmmm
I know the last line of this fails with NoneType has no attribute app_context
def execute(app_import, worker_class, worker_id):
    # push the app context
    sys.path.insert(0, os.getcwd())
    app = locate(app_import)
    sys.path.pop(0)
    app.app_context().push()
locate seems terrible:
def locate(path, forceload=0):
    """Locate an object by name or dotted path, importing as necessary."""
    parts = [part for part in path.split('.') if part]
    module, n = None, 0
    while n < len(parts):
        nextmodule = safeimport('.'.join(parts[:n+1]), forceload)
        if nextmodule: module, n = nextmodule, n + 1
        else: break
    if module:
        object = module
    else:
        object = builtins
    print("wow")
    for part in parts[n:]:
        try:
            object = getattr(object, part)
but I can't get to interact with it, the breakpoints do nothing and I don't see the prints. I guess that is because the worker runs in it's own process
 
 
2 hours later…
user11702787
11:20 AM
question about flask, is there a way to update the website when I made change to the code, without quit and restart flask ?
 
11:33 AM
flask does that automatically if running with the debug werkzeug server. In production you don't want that
 
208
Q: Auto reloading python Flask app upon code changes

PassidayI'm investigating how to develop a decent web app with Python. Since I don't want some high-order structures to get in my way, my choice fell on the lightweight Flask framework. Time will tell if this was the right choice. So, now I've set up an Apache server with mod_wsgi, and my test site is r...

 
 
3 hours later…
2:37 PM
Just had a tricky little dependency dance to remove security vulnerabilities - updated some libraries, then had to update pillow. Then found that recent releases of pillow don't support Python, so fell back to the latest version that did, then had to update reportlab. Life shouldn't be this complicated.
 
Yeah, some days be like that. I'm making an issue because the tutorial doesn't work of flask-worker, even though the last commit is 2months old. So that kinda throws me really of.
 
If recent releases of pillow don't support Python, what do they support? Did they pivot to a new language?
Ah, thumbing through the release notes I see they dropped 2.7 support for their 7.0.0 release.
 
Yup. Too late to edit, but it should have said Python 2 as you correctly Kevined (to rhyme with "divined")
 
I suspect most readers figured out your real meaning instantly, but I do like to jump way past the most obvious explanation into the realm of pure fancy
Wouldn't it be more interesting to live in a world where a popular and long-lived library abruptly says one day, "we're a java lib now, deal with it"
 
2:53 PM
@Kevin lol
 
3:05 PM
@Kevin As long as it's got a reasonably efficient Python binding, am I bothered?
(and as long as it works, naturally)
 
In that world there is a flourishing ecosystem of "fourth party libraries" that exist exclusively to provide bindings for fickle third party libs
The average team takes a few days after the "deal with it" announcement to publish something useful, but if you've got deep pockets and aren't too good for a little gray hat corporate espionage, then you can commission a zero day binding
 
why is it called third party anyways? It's me and them so who is the third party?
 
My guess is that you and your user(s) are the first two parties
Today's koan is: is your user the first party, or the second?
 
I just read about paradox logic and it's lovely. I never thought that something which I thought of so fundamental to the world is in the end just an axiom, which as it seems a whole world region doesn't use
 
3:21 PM
Which axiom are we talking about here
 
"give 'em an inch and they take a mile"?
 
Discarding the parallel postulate is a fun way to run things off the rails
Noneuclidean geometry comes in handy if you happen to live on the surface of a sphere
 
@Kevin The law of excluded middle
That took me way to long to find out, thanks for the question :)
 
@Hakaishin So which region of the world doesn't exclude the middle?
 
according to erich from eastern philosophy and religions.
 
3:33 PM
Hmm, reminds me of Godel's 1st Incompleteness Theorem, i.e. "there are true propositions that can't be proven, and false propositions that can't be disproven"
 
@Kevin If I had a wish, it would be to understand gödel. Every time I get into it. It feels like climbing a mountain and you get a little bit of clarity slowly, but then you don't engage with it for a few weeks/months and it's all gone :D
 
(I'm probably discarding a lot of important detail when I summarize it that way, but oh well)
 
@Kevin also I feel like there are so many misunderstandings about him. I really would like to understand him, because as I understand it he says some pretty upsetting things. And it's hard to argue against upsetting things, if you don't understand them :P
@Kevin because I feel like that is a simplification that is not correct. But I'm not sure either :P
 
Perhaps a better summary is "in an incomplete system, there are propositions that can't be proven or disproven. In an inconsistent system, there are propositions that can be proven and disproven simultaneously. All systems are either incomplete or insonsistent"
My first summary missed the mark because a system that uses "1 = 0" as an axiom can trivially prove or disprove any proposition it likes
The fact that the system is also trivially inconsistent is neither here nor there ;-)
@Hakaishin I found Hofstadter's Godel Escher Bach to be a pretty accessible introduction to the idea.
 
Yes, that sounds more correct. Also very upsetting. This reminds me of bells theorem.
 
3:41 PM
Not exactly a breezy afternoon read but I don't remember any particularly challenging cliffs I had to climb.
 
Yeah, a few people recommended me this book, I really should give it a go some time. But i just started the culture series so not sure when I will have time again :D
 
The incompleteness theorem is indeed a little upsetting. Suffice to say, I am less than pleased that the foundations of logic will never be able to succinctly explain all possible phenomena
Even if the unexplainable is quietly squirreled away in little curled up corners of mathspace where we can safely ignore them
 
3:58 PM
> What is powerful about Bell's theorem is that it doesn't refer to any particular theory of local hidden variables. It shows that nature violates the most general assumptions behind classical pictures, not just details of some particular models. No combination of local deterministic and local random hidden variables can reproduce the phenomena predicted by quantum mechanics and repeatedly observed in experiments.
I can definitely see a similarity between Bell's theorem and the incompleteness theorem. They both prove something about a wide category of systems, and you can't wriggle out of them without discarding the thing that makes that category useful to begin with
Bell's theorem doesn't give me the willies the way that the incompleteness theorem does, though. Only the latter gives me the feeling that I'm living in a simulated reality that was thrown together in one Red-Bull-fueled weekend
I can tolerate living in a nondeterministic universe, so I won't cry about the impotence of classical physics. Arguably, that's desirable, since then you can convince yourself that free will exists.
 
4:15 PM
Even if you proved the existence of free will one could argue that the desire to prove said existence thereby bound you to to a deterministic fate
 
Honestly I'm not really sure what free will is. I haven't found a definition that clearly includes "Kevin decides what to have for lunch today" and excludes "Kevin's program examines the value of x so it can evaluate a if x else b"
It sure seems like my program is choosing between different possible courses of action unimpeded, for certain definitions of "choose", "possible", "action", and "unimpeded"
 
@Kevin "Accessible?" Perhaps so, but tendentiously long. I think I got tired of its pedantry about three quarters of the way through.
 
Despite the lack of clarity in the problem, in my day to day life I act as if free will exists, because that's more convenient. Either I'm right and don't need to go through the trouble of proving it, or I'm wrong and you can't blame me for my lack of rigor, because you can't blame anybody for anything in a deterministic system.
@holdenweb You're not wrong :-)
I like pedantry though, so that's less of an impediment for me
Even so, I admit that my enthusiasm flagged at points. Overall I still liked it though.
 
My opinion is that if you had enough insight into the mechanics of the universe and a big enough data set you could precisely model everything from start to finish. So deterministic for a being with a big enough brain.
 
4:33 PM
As far as I know, the Three Body Problem can't be solved with perfect precision, even for simple cases such as "here are the particles' states at time T=0, now find their exact positions at T=1 second". So even simulating classical Newtonian physics would be beyond our current capabilities, even if we had a very fast computer with a very large hard drive.
Unless you wish to handwave about planck times and distances, which could be fun
1 + (10 ^ negative googolplex) is basically equal to one, right?
 
Looks like it's free will for the foreseeable future
 
It's not out of the question that a big brain could discover an exact numerical solution to the N-body problem that represents a classical simulation of our universe, but it would be a very big equation indeed
You definitely wouldn't be able to write it down even if you harvested every atom in the universe for ink and paper, so constructing the big brain ourselves is a no-go. We'll need to take some DMT and petition the eleven dimensional machine elves to use some cycles on their servers instead.
 
user13415013
Hi guys,
What do you think which would be fast?
 
user13415013
1) ' '.join( [w for w in input.split() if len(w)>1] )
2) re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '', text)

from removing single character stackoverflow.com/questions/32705962/…
 
When regexing, raw literals are your friend. Gets rid of the need for all those annoying double-backslashes.
 
4:46 PM
My wild guess is that #2 is faster. Especially if you re.compile the pattern beforehand and don't count that as part of the runtime
 
user13415013
oh thanks, thanks
 
If you want an objective answer, the only thing you can do is measure it yourself. timeit comes in handy here.
 
user13415013
yes
 
Can I do something like a, b = dict[c] if dict[c] key is same for both cases but only the value differs?
 
If dict[c] evaluates to an iterable object with two elements, then your code will run
 
5:00 PM
a, b = d[c] will only work if d[c] evaluates to an iterable that produces precisely two values. a and b will afterwards reference the individual values.
 
>>> dict = {23: (42, 99)}
>>> c = 23
>>> a,b = dict[c]
>>> a
42
>>> b
99
I'm not sure I understand what "the same for both but differs" means
 
Bu I suspect we may be answering the wrong question and cinfusing the questioner into the bargain.
 
Maybe you're asking something like "I have a dict {1: 2, 3:4}. Is there any way I can do something like a, b = dict[1,3], with the outcome being that a equals 2 and b equals 4?"
In which case i don't think there's a solution that's more concise than a,b = dict[1], dict[3]
 
Okay so to be more precise , I have dict1 and dict2 both previously referred as dict[c] due to same key that have both the same keys but different values , so instead of doing a = dict[c] and b=dict[c] is there a way I can do a,b = dict[c], and no it is not iterable, both have only one value.
@Kevin Oh wait, I can store two values for one key you mean?
 
user13415013
@kevin but it seems there is no way to use timeit given in time it doc.
that python interface is quite confusing to use my thing,
timeit.timeit('"-".join(str(n) for n in range(100))', number=10000
 
5:04 PM
Need not say I am new to python lol
 
@AshwinPhadke Strictly speaking no, but effectively yes. Each key in a dict can have only one value, but that value is allowed to be a tuple*, which itself contains multiple values.
(*or literally any other kind of object)
@nerd I, too, find timeit's interface confusing. I always have to read through the examples a couple times before I get the syntax right.
 
user13415013
Oh i found solution given and it is quite lengthy but works
timeit.Timer('char in text', setup='text = "sample string"; char = "g"')
 
@Kevin a, b = d[x] for x in (1, 3), if you don't like obviousness. (I do, this was just for fun).
 
@Kevin Ah sneaky one, so you mean this is possible
@Kevin Nah both have same values to edit your example it is like dict1= {1:2 } dict2={1:3} .
 
Ok, I understand the question now. My conclusion is mostly the same: you can't do much better than a,b = dict1[c], dict2[c]
You could take a page out of holden's book and do a,b = d[c] for d in (dict1, dict2) :-)
 
5:10 PM
Kevin'd again!
 
I do think there is a time and place for generator expressions being used for multiple assignment, but if you only have two values, it's probably overkill
 
@Kevin This works
@Kevin That's just a lot of steps to write what you meant by ` a,b = dict1[c], dict2[c]`
 
@Kevin As I said, just for fun!
 
👍
 
@AshwinPhadke we like to play with Python, but your intuition is correct: the most obvious code is usually the best. Readability counts hugely in Python.
 
5:14 PM
Exploring fun options is valuable just in case the conditions of the problem change in the future
 
And of course now you've simplified it as far as you have, the obvious next step is to simply use two statements, making the meaning crystal clear!
 
@holdenweb Yeah the latter method doesn't look easily readable if you are a first time reviewer, or maybe inexperienced like me.
@Kevin This is so true, I mean you could go for simplicity, but with increasing complex architecture you might as well do the complex way.
 
I'm a great believer in code being as obvious as possible. This could be because I have no subtlety ...
 
Keep things simple, unless you have a really good reason to do otherwise
 
But I'm also a great believer in play.
 
5:16 PM
@Kevin +1 , also make sure your methods have docstrings lol
 
user13415013
It is difficult to pass re module inside timeit
g = timeit.Timer("re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '', word)",  setup='word="sfasfasdf1324sv)*ER"')
 
("Making things obnoxiously complicated is a fun game" is a valid reason btw. As long as you're not inflicting it on others)
 
^that
I 100% support doing things that you should never do in a production environment
 
I wrote Hello, World using a thousand lambda expressions that built church encoding from the ground up, so I'm no stranger to fun. For certain definitions of "fun".
 
New commit message after this : " - changed the way paths are handled , learning curve with dicts too."
 
5:20 PM
@Kevin youtube.com/watch?v=6avJHaC3C2U I feel that he shares your definition of "fun"
that is, he literally created the Rockstar language, and then (at the end of the talk) sang FizzBuzz.
 
Kevin presents his new project dot png
 
user13415013
I tried to use another library datetime, and looping failing re module. is it true
other: Duration: 0:00:00.002000
re: Duration: 0:00:00.006993
 
As in, you're using datetime instead of timeit because it's a pain in the butt to get the latter working? timeit is typically the most accurate measurement because it takes steps to compensate for slowdowns caused by things outside the process' control. So datetime's measurement is somewhat less reliable.
 
@Kevin lol
 
Whether it's so unreliable that you should just ignore your current results, I'm not sure.
Let me see if I can finagle timeit to work on your examples...
 
user13415013
5:28 PM
I also tired to use time, It is also showing other performance better. but anyway i will use re.
other: --- 0.05758857727050781 seconds ---
re: --- 0.09551548957824707 seconds ---
 
user13415013
ok thanks.
 
import timeit

a = "' '.join( [w for w in text.split() if len(w)>1] )"
b = "re.sub('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)', '', text)"
c = "pattern.sub('', text)"

setup = """import re; pattern = re.compile('(\\b[A-Za-z] \\b|\\b [A-Za-z]\\b)'); text = "I have a million oranges" """

print("string join:", timeit.timeit(a, setup=setup))
print("re.sub:", timeit.timeit(b, setup=setup))
print("pattern.sub:", timeit.timeit(c, setup=setup))

#output:
#string join: 4.888432971
#re.sub: 3.993395522
#pattern.sub: 0.9547343389999998
 
user13415013
whoa, great thanks
 
user13415013
pattern is more good.
 
@nerd I believe the recommended technique is to import the namespace of your current module into the code you are timing. I even used to know how to do it, but I'm old now.
 
5:35 PM
@nerd Yes, but I'm cheating a little bit by compiling the pattern in the setup, which isn't timed.
 
Something like from __main__ import *, IIRC
 
Generally, compiling a pattern is only useful if you're going to use it more than once. Otherwise, it doesn't save you any time.
 
user13415013
@holdenweb, why so, to import that module while timing, any reason? thanks
 
importing __main__ saves you some aggravation with string formatting, but I always flub it and cause an infinite loop on the first try, so I use the string approach if the code is sufficiently small
I'm pushing it here because I had to use my nemesis, semicolons
 
And anyway the re module caches the last 25 (?) patterns used, so you don't lose a lot unless you're using humungous numbers of re's.
 
user13415013
5:38 PM
@Kevin that means all previous, time, datetime are not reliable.
 
@nerd For access to its names! If your program has imported re already, it's now available for you to use in your code. And the import is fast, since no new code needs to be read.
Plus if you do it in everything you time you can assume it's constant time.
 
user13415013
oh , yes
 
No instrument can measure time with perfect accuracy, and even if you could, knowing how fast your computer is doesn't tell you much about everyone else's computer. In that sense, everything is unreliable.
 
user13415013
oh
 
user13415013
heisenberg uncertainity principle
 
5:42 PM
Hi
 
That's why people usually use Big O analysis as the first step in assessing the efficiency of their code. What is O(N^2) on my computer will be O(N^2) on yours, regardless of how many petaflops you've got
 
user13415013
thanks for info, how can i apply big O principle in my case here.
 
You can't, because the big O complexity of re.sub is not documented :-P
 
user13415013
@kevin what would you recommend to use parallelising the python code?
 
A perfect example of why people usually use Big O analysis, but not always. Sometimes you can't ¯\_(ツ)_/¯
@nerd I've heard good things about multiprocessing.
 
user13415013
5:51 PM
thanks. but when i first look docs, it looks like jibrish code,
 
That's what it looks like to me too :-)
 
user13415013
guys, If you are working on gpu, i heard python numba or something for nvidia. Did you have some experience in it,
 
user13415013
I think nobody.
 
Too newfangled for me.
I do want to learn how shaders work one day, but I think I need a better groundwork of C knowledge first
 
user13415013
C programming would break eye. Its so messy.
 
user13415013
6:01 PM
as i previouly talked, why not python like c programming ???
 
user13415013
@Kevin, Give me name of one programming language,preformance exactly comparable to C or more, and syntax like Python and follows zen.
 
user13415013
any
 
I've heard good things about golang that may or may not be hot air
 
user13415013
yeah it has cool syntax but uses braces for loops . dont follows our zen principal
 
user13415013
Like even today c++ is used in most preformance related task, because of fastness and oop.
 
user13415013
6:17 PM
Why not developer used c++ code to look like python syntaxes, It will develope more productivity.
What would you compare language to c++ as powerful greater or equal and syntax and idea looks like python?
 
user13415013
ok guys, good night its 12 pm, but please think, It is serious problem to solve
 
6:32 PM
Aug 9 at 10:25, by nerd
but why java dont use python like syntax, Is there any problem with java. or even c++, Why should not change syntax of c++ or even java
Can you please stop making this pointless point?
Aug 9 at 10:28, by Andras Deak
you seem to have some peculiar notions of programming languages and I don't think we can have a reasonable conversation based on that
 
6:52 PM
@smci don't even bother looking at that code, I'll have to rewrite that whole module. thanks anyhow
 
7:13 PM
@PedroSpinola Sure, no worries. Try to vectorize those calculations, avoid for-loops and list-comprehensions everywhere possible, and familiarize yourself with the many different ways to declare matrices in numpy and the many different special-purpose matrices and associated methods numpy has. Best is to look at real numpy code other users have written for iterative DE solving.
 
7:35 PM
@PedroSpinola: hey, strongly recommend you look at some of the videos of talks from SciPy conference on how to rewrite a computation in a matrix formulation and improve its performance. That's top-quality stuff taught by experts who write and use scientific libraries as a daily thing... come back let us know which if any specific talks you find useful.
 
 
2 hours later…
9:58 PM
not even a question, needs more focus stackoverflow.com/questions/63477103/…
 
10:30 PM
@smci: thanks for those tips and links! I won't have enough time now to look properly into this and implement it, but they're saved for later! (I have realized I'll need to rework the abstraction)
@smci: As a beginner, I have never searched for other people's code to learn, apart from help/tutorial links of course. What manner you'd recommend to find some real DE solving code of high quality and accessible to a beginner? Seems like a great idea. Should I just go to github and search differential_ev ?
 
11:01 PM
@holdenweb Here's a handy table of which Pillow versions are compatible with which Python versions: pillow.readthedocs.io/en/stable/installation.html#notes
 

« first day (3594 days earlier)      last day (1369 days later) »