« first day (3246 days earlier)      last day (1717 days later) » 
00:00 - 17:0017:00 - 00:00

12:44 AM
@AndrasDeak Nor do i, you will just get shot in the face
cbg
 
1:02 AM
rbrb
 
 
2 hours later…
3:25 AM
@AndrasDeak Nor to be confused with curling, a game from Scotland.
 
 
1 hour later…
4:36 AM
cabbage
typing should have an annotation to indicate sorted data structures
 
5:23 AM
being sorted has nothing to do with type though
 
correct
I found myself wishing for an indication as I moved the sorting step between objects in a processing sequence
 
IMO the whole idea of typing variables is half-baked - I'd much prefer being able to specify arbitrary preconditions/invariants for variables (things like "sorted", "non-negative", "length > 0", etc)
 
I like typing, I find it tremendously helpful to have a type indication.
The precondition/invariants you mention are also desirable IMO
 
6:04 AM
@ReblochonMasque I know almost nothing about typing, so feel free to ignore me, but I agree with Aran-Fey. There are virtually unlimited ways that a collection could be sorted, depending on the type(s) of its items; OTOH, I guess a collection type could indicate whether or not it's ordered.
 
yes, thank you @PM2Ring
BTW, yesterday I re-pinged New Mexico Tech as I had not received an answer.
 
I really love the freedom of Python, and I'm not a fan of type hints, etc. But I do understand why some people like them, and I agree they can be useful in complicated or large programs.
 
Yes - as you get older, you might be happy to have them... :D
 
I've been coding for almost 50 years. Most of the languages I used before Python require you to declare the types of variables, so I know how to use that stuff. OTOH, coming to Python was very liberating.
@ReblochonMasque Thanks. It's disturbing that they still haven't replied to you.
 
A little bit... although it was summer holidays, so maybe it was hard to get the people able to decide together to make a decision, IDK.
 
6:13 AM
One neat thing in Pascal is that you can declare the range of legal values of an integer variable. That can be used for automatic bounds checking of array indices, and similar things.
@ReblochonMasque Good point.
 
What is a bit annoying is that it seems like the web archive is also gone - I cannot connect to:
 
@ReblochonMasque I had problems too, but I refreshed & it loaded! Try web.archive.org/web/20190221165157id_/http://infohost.nmt.edu/…
 
ok, that's good to know, thanks
 
It's definitely flakey, though.
 
6:59 AM
Is there a recommended way to activate a venv, install all dependencies and run a python script in one go?
 
7:29 AM
@nobism I'm shilling it a lot already, but you can use poetry. poetry run my_script.py would work if you have a pyproject.toml file that defines the dependencies.
 
7:54 AM
@ReblochonMasque you can create a NewType to indicate "feature-having thing": docs.python.org/3/library/typing.html#newtype
e.g. SortedList = NewType('SortedList', List)
or just use an actual sorted list: grantjenks.com/docs/sortedcontainers/index.html
 
ohhh, many thanks @MisterMiyagi :)
Yes, I thought of using Grant Jenkins sorted containers, but the collections I am manipulating are rather smallish, and do not justify it.
...but maybe I'll take a new look.
 
I'm having a read back through the transcript but it's loading slowly... what is it you're trying to do @Reblochon ?
 
from 20ish posts above: typing should have an annotation to indicate sorted data structures
 
@ReblochonMasque if you just want to sort it once, then sortedcontainers is overkill indeed. I am rather happy about the performance when repeated sorting is needed, though
 
Yes, @MisterMiyagi, the whole idea is to sort it once.
I kept pushing the sorting downstream as I discovered further needs to modify the collection, and I got to a point where I was confused as to whether it was sorted already or not.
 
8:04 AM
then NewType seems like what you want
 
yes, that looks quite promissing.
many thanks.
 
8:17 AM
ahh Pascal - don't think I've touched that in goodness knows how long... was a big user of
the first Borland Delphi's at the time... so must have been mid 90's I guess?
 
8:53 AM
hey guys. This is not the ideal place to ask but I've a question on stackoverflow that was marked as too broad and was put on hold. I've edited my question but it's still on hold. I've also set a flag for moderator attention, and nothing yet. Any suggestions?
 
What is the link to your question @John?
 
@John you got a response yesterday... see stackoverflow.com/users/flag-summary/10371713
 
I suspect the response was something along the lines of "this is not what mod flags are for"?
 
It must be a mod only link @JonClements
 
Sorry I just saw that they voted to remain closed
I don't see why though
Should I create the same issue again?
 
9:00 AM
nope
personally, I don't think your question is too broad - I think it's unclear, though. Do you want to write a python program that spits out a link, or do you want a python program that visits said link and downloads the csv?
 
I want the link because then I can download the file with a simple python request
 
@ReblochonMasque kind of... flags are private... so yours are stackoverflow.com/users/flag-summary/2875563 - only you and mods can access that page
 
:)
@John sure... but the question (in of itself) is still either too broad or unclear... even after the edit... have you tried stuff not using selenium, such as actually requesting the page using requests for instance and then parsing out the link somehow? Or...
 
9:16 AM
there's no link, it's all JS
 
which makes it extremely fun then?
 
yes, fun, exactly :P
 
I haven't looked at the site... but sometimes it's a possibility to just find the appropriate ajax endpoint and issue an appropriate request to that
 
 
1 hour later…
10:40 AM
hi
i want to create course online and i want to run a function that checks if students are watching the courses or not.
how can i do that ?
i wanted to be sync with the server side.
 
So, uh, which part of this whole system are you programming? The server software? The client software? Both?
 
both
but front end can be alterd
iam trying to count on server side as much as possible
 
And the course is what, a video stream?
 
@diamond watching as in "streaming the video", "having the video tab open", "looking at the video tab"?
 
@Aran-Fey @MisterMiyagi yes and yes (watching as in "streaming the video", "having the video tab open", "looking at the video tab"?)
at some point i want to check if he scrolled down to certain content as well.
 
10:50 AM
Miyagi's wasn't a "yes or no" question lol
anyway, I don't know webdev - no clue how you'd check if someone's still connected to a video stream
 
ohh ok
i was thinking something similar to online web games
where everything is server sided
(miyagi's )oh if that is not a yes or no question then what it would be ? i dont know how to answer that then
 
choose one of the 3 options
 
i want the max security so all of them ?
 
well, that's gonna be a little difficult
 
i think it's worth it
i also want to track every seconds that passes while each user watches the video to determine the percentage of watched part.
grades will be based on that.
 
11:05 AM
oh my
 
._.
 
I'm sure students will be thrilled to get an A in "(pretending to be) watching a video stream"
(source: am student)
 
Sam
Hello again ;-;
I have python 3.7.3
and windows
Which are the requirements as far as I know for tensorflow-1.9.0-cp37-cp37m-win_amd64.whl but still it returns ERROR: tensorflow-1.9.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform. Does anybody know why?
I think it has to do with my combination of windows and processor, since pip search tensorflow returns tensorflow but pip install tensorflow says there is no tensorflow
I already tried the --ignore-installed option
And ofcourse, I tried to download 3 different wheels, which all didnt work.
1: `tensorflow-1.9.0-cp37-cp37m-win_amd64.whl`
2: `tensorflow-1.14.0-cp37-cp37m-win_amd64.whl`
3: `tensorflow_gpu-1.14.0-cp37-cp37m-win_amd64.whl`
 
you sure you have amd64?
 
Sam
Is that different from x86? The downloads don't show any x86 so I downloaded amd64
Okay nevermind, just found out they are not even close to being the same
So I happen to have x86, does that mean tensorflow is just 'not a thing' for me? Can I compile them myself?
As github.com/tensorflow/tensorflow/issues/19998 mentions, the solution does not work for me.
 
11:14 AM
don't know, sorry
 
Sam
I hope this works, but I will try to git clone and pip install . or python setup.py
 
@Sam please omit the expletives here
Any chance that something like conda would help your install problems?
 
cbg. just found out the hard way that getattr doesn't shortcircuit. Surprised because i just expected it to work that way for some reason
 
short-circuit how?
 
it requires the 2nd argument to "exist" so to speak as well, even if the first does. say..
getattr(str, 'zfill', i_dont_exist) #error
I just assumed that would have worked just fine.
 
11:27 AM
you mean NameError on i_dont_exist?
 
I don't think that has anything to do with getattr :)
 
cool, i'll have to try to figure out what exactly getattr does then!
 
I mean, the function never gets a chance to be called if one of the arguments is a NameError
 
oooh
derp, that makes sense
i blame the ide for highlighting getattr and making me think it was something special :P
 
11:33 AM
it is... it's a built-in ;)
 
:D dang it, let me blame my ide shhh :P
 
@JonClements The problem is that the link is dynamically generated. Meaning I cannot know it before loading the page. And that's the issue. I don't want to use Selenium and I don't know any other option.
 
Hi any room for QT C++ ?
 
11:48 AM
@ParitoshSingh Because that's not what 'short-circuiting' means. Think about things from the point of view of the compiler for a minute. How do you expect it to handle nonexistent identifiers? Delay trying to interpret them until it's sure they're needed in the code, somehow?
 
that would be so lazy
 
Cbg
 
@diamond that basically means you have to do image recognition on their local camera to track eye movement
 
^ edit: I should have said 'interpreter'. (Compilers have more ability to go try to resolve things, such as forward declarations in C/C++)
 
@AndrasDeak do you know if it exists à python library as redux js equivalent
 
11:56 AM
Anyway, picking up an old question of yours, @Paritosh, from August (as mentioned I have been mostly offline since July)
 
since Python guarantees evaluation order, it has to throw an error if a function argument does not exist
 
@Ctrl why are you even asking me that?
 
@ParitoshSingh My point was you didn't ask the general-sounding "How does a for-loop work?" Originally you asked "How does a for-loop work, IN PYTHON?". As if looking for comparison to for-loops in other languages. Anyway the question you cited specifically for party in feed.entry: print party.location.address.text It's asking "How do iterables work in Python?"
...to which answer is in Glossary or stackoverflow.com/a/9884259/202229 An iterable is an object which returns an iterator.
...then the next question on understanding for party in feed.entry: print party.location.address.text would be "What type of an object is party? What are its members party.location.address.text, what do they do?" (No idea, whatever that object hierarchy documentations bloody well says they're supposed to do) But really all that has nothing to do with ""How does a for-loop work?".
...but the OP in that question never said what type of object feed.entry was, so none of that is answerable. I suppose they were simply asking "What is an iterable and how does it work in for x in <iterable>?" If so that needs retitling away from the misleadingly general "How does a for-loop work?"
 
12:23 PM
If I never see another question like "why does this crash? for x in seq: print(seq[x])", it will be too soon
 
@MisterMiyagi you are joking right ?
@Aran-Fey lol
 
@Kevin good news is there's a dupe for that
 
I suspect it's extra good at confusing newbies because if they try out the code with seq = [0,1,2,3,4], then it produces apparently sensible output that reinforces their incorrect mental model
A rarer specimen is one that asks "why is my output all scrambled?" and posts code like:
>>> seq = [2,4,3,0,1]
>>> for x in seq: print(seq[x])
...
3
1
0
2
4
 
cbg
 
cbg
 
12:37 PM
Today I am annoyed that ThreadPool.imap_unordered(target_fn, generator_yielding_zillions_of_entries) consumes the generator into an internal deque before starting up the workers.
 
@Aran-Fey Good job. I tried to make some edits to improve clarity - your points were getting buried under the code and outputs. Also, let's not get in the habit that newbies have of writing long essays "My code is like <this> and the output is like <this>, what am I doing wrong?". Simply adding "...when iterating with a for loop, and referencing lst[i]" seems fine to me
 
Is there any alternative of fromisoformat api which is present in datetime.date which come as part of Python3.7 ? I need some API something similar which will work for below version
 
strptime?
 
12:45 PM
Thanks; Let me try
 
@smci hehe aye, makes sense. It was a derp moment.
 
@Aran-Fey worked! watermelon
 
@ParitoshSingh That's cool. It just confused the **** out of me that you kept saying you wanted to know how for-loops work, but referencing the confused question of an OP who was really asking about iterables. (I just edited OP's title to not be misleadingly generic-sounding, but can you do that in future or let us know? Otherwise SO gets littered with low-quality generic-sounding questions, and the search box ceases to be useful)
 
@smci Hm, i can see how one can read what i said that way (as in: * As if looking for comparison to for-loops in other languages* ). But i suppose then let me ask you this: "What, according to you, is the correct question for which the answer is the block of code that explains a lower level version of a for loop (using a while and a try except)?
# What is the question where the following is the answer

for i in some_iter:
    #work
#This is equivalent to the following:
iterator = iter(some_iter)
try:
    while True:
        i = next(iterator)
        #Work
except StopIteration:
    pass
 
Q: write code to demonstrate how iter generates the subsequent element in an iterable and raises StopIteration
 
12:58 PM
"How does Python implement for/while-loops internally/under-the-hood?", or in this specific case "How does Python implement for x in <iterable>?" . Which is clearly asking about implementation details. Not "How does <feature> work?" which is asking about its functional spec, its contract with the rest of the code about how it behaves.
 
Noted @JonClements, thanks
 
Also, the thing being asked about is called an iterable. Call it by its name... "say iterable in the title, say it ten times if needs be... don't just keep saying "a for-loop".
 
I wouldn't call that an implementation detail. that is pretty much the contract of for and its iterable
 
@MisterMiyagi It's about iterables. Not about for-loops.
 
the Q was "How does Python for loop work?" which is very clearly about for-loops
 
1:02 PM
pointless argument
 
@MisterMiyagi It was very clearly not about for-loops, but about iterables. Really, about "How do iterables work in a for-loop?" I just spent 10 min above pulling the citations for Paritosh's benefit. It's generally better to be precise, avoids to-and-fro.
 
"How do iterables work in a for-loop" is about the contract of for and its iterable
 
@anon609 Noone's having an argument, and yes there is a point. Paritosh was citing someone else's question with misleadingly generic title; I just now figured out the root of their misunderstanding and fixed the title
 
for the record, I find the edited title much less useful - it implies that a for-loop could work without an iterable
 
@MisterMiyagi But the original question never said that, it said "How do Python for-loops work?" . It didn't ask about iterables.
 
1:09 PM
the question is now really hard to understand for someone new to python who knows about for loop but not about iterables
 
@smci which is why I have no idea why you insist it is/was about iterables instead of for loops
at most, it is about both. it is definitely about for
 
@Arne Thank you for stating this, essentially that was my bottom line. Spot on.
Though i don't particularly want to continue this discussion, And it's okay to disagree on interpretations if it comes to that at the end of the day.
 
@MisterMiyagi Because a for-loop can work without using an iterable (it could just be a list, or a range, or a tuple, or a set), and for 40+ years in most older 2GL languages that's exactly what happened. In C++, iterables were added as a kludge that needs tons of boilerplate; the OP got confused because they were looking at an iterator without the boilerplate; this answer to the OP's question illustrates(stackoverflow.com/a/1292205/202229)
 
"a list, or a range, or a tuple, or a set" all are iterables
 
@smci honestly, your edits to Aran's Q also make it harder to understand, and the changes to the A feel like someone is shouting into my face.
 
1:15 PM
@MisterMiyagi No, Python defined them to be, but they don't necessarily need to be. How do you think Fortran or C for-loops work? for x in range(6) does not need the language to have a concept of an iterable.
 
@smci that is exactly why it is relevant that the question asked about Python for loops
a Python for loop works on an iterable
 
@MisterMiyagi You're going in circles now. For-loops in 2GLs were quite primitive. Python elegantly generalized the concept of a for-loop using an iterable. But we can make for x in range(6) work without needing the language to have a concept of an iterable. Remember teh distinction between range() vs xrange() in Python 2? range() returned a Python list object - we could have gotten away with implementing it without an iterable
 
range() and xrange() in Python2 both are iterables!
we can make any string of characters mean whatever we like, that is entirely irrelevant for what that string does mean in Python
so no, we could not have gotten away without implementing lists as iterables because that is how for loops work in Python
 
@MisterMiyagi range() returned a Python list object - we could have gotten away with implementing it without an iterable. The language designer could just have chosen to have the interpreter itself expand range(6) directly into [0,1,2,3,4,5]. The language does not necessarily need to have a concept of iterable in order to handle for-loops with ranges Fortran and C managed to implement for i=0; i<n; i=i+1 for many many decades, without needing iterables.
 
I seriously have no idea what you are talking about. Sure Python could have been defined differently but it is not.
and range(6) did expand to [0,1,2,3,4,5] in Py2
 
1:27 PM
@Arne whoa, I just looked at the answer without my usual dark theme and I have to agree the bold text is seriously excessive
 
@MisterMiyagi range(6) can be expanded to [0,1,2,3,4,5] without using an iterable, and without the supporting language even needing the concept of an iterable. Agreed? This is exactly how languages worked for decades before Python (strictly, it was C++ that popularized iterators).
 
this is such a strange discussion
 
are you asking me whether I question the equivalence of turing machines? oO
 
@MisterMiyagi Arguing that "Python can't implement X without needing an iterable, because the Python spec insists that X must be implemented using an iterable" is pure circular. I cited you the obvious example that Python 2 could well have chosen to implement range() without needing an iterable.
 
that is not what I argued
either way, that is not circular that is called self-consistency
 
1:34 PM
@MisterMiyagi "range() and xrange() in Python2 both are iterables!" you said. So what? range() didn't need to be implemented as an iterable - range() is merely a function that returns a list. That was a language choice. It might have made things more elegant. But it didn't need to be an iterable.
 
let's leave it at "Mister Miyagi has no interest discussing every theoretically possible alternative implementation of Python"
 
I'll leave it at "Many simple loops in Python (e.g. for i in range(6)) do not require an iterable, and could be implemented without the language having a concept of one. Hence, the question "How do for-loops work in Python?" is not synonymous with "How do for-loops with iterables work in Python?". Arguing "well even the simple loops in Python use iterables, because the language designer generalized it so" is circular.
 
@MisterMiyagi wise
 
argh... must... resist...
 
I suggest a mug of coffee ;)
 
1:38 PM
I've already had three :/
 
You'll never win
 
I think smci is confounding iterators with iterables. There are iterables that are not iterators. Examples of these include lists, sets, ranges, etc. Does this help clarify things?
 
cries
 
@MisterMiyagi I have no interest either, but I'm not dissing you. (And it was't "every theoretically possible alternative", just "the way all the other languages implemented things for 40 years"). A long time ago Paritosh asked me something, and I was following up on getting him an answer. In the course of it, I fixed the question cited, so it's less misleading. None of you guys did. It had been around for a month.
 
1:41 PM
suffice to say, I disaggree
 
@inspectorG4dget I don't believe I confused them anywhere. I did say they were not necessarily needed for simple constructs.
 
you might be interested in coming up with a better title for my much older question as well, then
 
I'm going to comment on the communication and NOT the subject matter. @smci you keep saying the same thing and expecting that we all follow what you mean that "the concept of iterable is not needed". I'm stuck there mentally and I need extra information to understand how that is possible. However, you aren't providing that extra bit of information and I have higher priorities than to go look for it.
Truth is, I don't know if you're right/wrong/mistaken because I'm not quite sure what you mean.
 
is there some concise way to type hint optionals? it seems excessive to write Tuple[Optional[BaseException], Optional[Exception]] when the code is just a, None or None, b or None, None
that's an English or, not a Python or
 
not AFAIK. I've been looking for something like this for a while, myself. The hack I've been able to come up with is to write a function that takes shorthnd input and returns the more verbose typehint
 
1:55 PM
something like ?Exception would be nice for type annotation
I wonder if it was proposed already, it's pretty common syntax in other languages I think
 
IIRC it was, but they wanted type hints to stay Python expressions
 
'?Exception', then
 
@inspectorG4dget Your question "What happens when you call if key in dict" is about implementation of in operator/__contains__ method, but you could optionally add "under-the-hood" if you want. But the body of your question is less general than the title suggests: for performance reasons, you're wondering whether it can use __eq__ and only fall back on __hash__ when __eq__ compares non-equal, and whether that's safe.
 
I changed my mind. I accepted that annotated function signatures will break lines anyway, so verbosity is fine for me.
 
@piRSquared I've stopped discussing, and no I didn't keep saying the same thing. I pointed out that a) programming languages managed without iterables for decades and b) even in Python simple loop constructs could be implemented without the language design necessarily shoe-horning everything into an iterable. Python 2 range() being the obvious example - it was merely a function that returns a list..
 
2:06 PM
(I hate this keyboard)
 
@smci Thank you for explaining my question to me, but I'm pretty sure I knew what my question was.
 
@piRSquared I already provided all the information you need. Just reread it if you didn't understand it. I never said "the concept of iterable is not needed". I did say "It's not needed for simple loops, such as using range objects". MM disagreed and said "But it is needed even for simple loops, because the language design arbitrarily decided that" which is circular; the language design didn't need to generalize even simple loops and ranges into iterables, along with "proper" iterables.
There's your entire redux. No "extra information" needed.
@inspectorG4dget Don't be rude; everyone else (including me) can't tell what your question was about from the title until they read through the body. Some of that is performance is implementation details, so the cPython implementation might do it one way, PyPy another. (PS you could title it "Luxury Yacht" and equally say " I'm pretty sure I knew what my question was.", but again the title wouldn't throw light on what you were asking)
 
I really want to ask about Boaty McBoatface, now
 
@Arne I am somewhat undecided on that. I've repeatedly overlooked important features of type hints because they were drowned by wrappers.
 
I sense tension. I hope everyone is playing nicely today.
 
2:17 PM
FWIW, I'm actually gonna take a break from SOchat and do my erm... job... for a while
 
@inspectorG4dget Again, same theme as with the question Paritosh cited: misleading falsely-general-sounding title on a question that's much more specific and less reusable resource to everyone.
 
@MisterMiyagi with wrappers, do you mean stuff like Optional, Union and the like?
 
yepp
tuple as well for functions
it would be nice to have "regular syntax" like (int, int) -> (int | None, bool). then again, with large types one would likely miss the tiny punctuation.
 
@smci I don't want to start an argument but I believe this will be for the greater good. (I might be wrong) You are obviously (to me) an intelligent and well spoken individual. However, your tone is harsh and off putting. So much so that if I see walls of your text, I'll stop reading. I don't know if that is your intention but I thought it better to say it rather than let it linger. I apologize if that offends you, I don't mean to.
 
I've lugged around a lot of AsyncContextManager[AsyncIterable[T]] lately, it would be easy to miss some trailing symbol there
 
2:21 PM
@Kevin I dug up an old question Paritosh asked me some weeks ago (I've been offline), followed up, researched it, and fixed the underlying question's title (which anyone else could have done, but didn't). Then people jump on me. No good deed goes unpunished. "StackOverflow: a suboptimal morass of general-sounding question titles that turn out to be highly dependent on question details, and/or not so reusable to the rest of us"
 
@Kevin definitely
 
@MisterMiyagi that might be a good idea for a package, typs or something. After importing it you can write shorter types
 
import typetools
^ suggested name
 
@Arne don't tempt me, I was this close to writing a LISP-like import hook preprocessor already
there are some decisions about typing that make me less enthusiastic to contribute, though
 
@smci an understandable frustration.
 
2:27 PM
@piRSquared But hang on. There was no "wall of text" until people kept demanding a justification (that iterables are not strictly needed for simple loops) Then you came to the discussion after it finished, didn't really follow it, misparaphrased me (which makes me look really stupid) and essentially demanded a personalized summary, which I wrote for you, but then you complain again. Can't win.
 
one could probably write a mypy plugin though
 
@piRSquared import stenotype? after the typewriters stenographers use
 
(It would be far better if you had said "I can't follow the above discussion but you seem to be saying X, is that the case?". I try to take care not to misparaphrase people)
 
@inspectorG4dget sweet name
 
I'd like everybody to keep in mind that technical arguments are best viewed as "us against the problem" rather than "me against you"; and that "I disagree", "I don't understand", "let me explain this in great detail", and "I think this could be improved" don't necessarily carry any subtext regarding the recipient's character or skill
Without the aid of body language, 90% of nuance is lost in text-only communication. Where ambiguity exists, try to read messages in the most charitable way
 
2:30 PM
one problem would be that a ton of tools rely on type hints, and they might break in unfixable ways
 
it is even free on pypi =D
 
go ahead and grab it, then ;)
 
my measly attempts to help with mypy showed me that I'm still a couple years early writing something like that
 
I am unfortunately hung up on a few other projects - can't start stenotype just yet. If one of you fine pyple would like to get it rolling, I'd be happy to contribute when I'm waiting on experiments at work
 
2:36 PM
but if noone else wants to, I still might an try to grow with it
cool, I'll ping you in particular when I get around to it
also a reason to try out github actions, so many things to do out there!
 
heck yes! If you ping my email from my profile, I'd be happy to collaborate
 
you guys' productivity is contagious... I'm suddenly all motivated to finish all the stuff I've not been working on for months
 
recbg
 
2:54 PM
So, I'm wondering: could PyCharm slow down a Neural Network's training ?
Somehow, when I test with a small dataset from within PyCharm, the GPU is used and things go fast. But now that I've integrated a generator to throw the whole dataset at the network, the GPU is barely used, and PyCharm is saying `Low Memeory : The IDE is running low on memory and this might affect performance. Please consider increasing available heap.`
Should I be running my training python script from within a terminal?
 
your memory has to host both pycharm and the NN. if the NN grows larger, pycharm has less memory available
consider shutting down process unrelated to your training (that does include PyCharm)
 
how do I kill PyCharm's process without killing the training script it has launched?
 
don't start the training from PyCharm in the first place
launch it as a regular python process
 
also, a NN usually doesn't grow in size (and I believe it is within the GPU's memory)
 
I mean "grow" as in "you put more initial data into it"
 
2:57 PM
Launching the script from within PyCharm reduces the available heap for the training to the effective IDE's allocated Heap, or the process is independant and can take as much space as is available generally speaking (thus, it can grow bigger than PyCharm's allocated memory)?
 
PyCharm launches scripts as subprocess. The heap is not shared.
 
Can people please stop flagging half of a prolonged discussion that led nowhere? Thank you.
 
This means that if I was launching the script from the terminal, but left PyCharm open for whatever reason, I would see 0 performance gain?
 
notwithstanding impressions of measuring 0 performance gain, yes
 
as it is, it barely looks as if the thing was doing anything
on smaller dataset, the Memory would grow to 95% usage, and the CPU would keep oscillating at a much higher usage %... and now the GPU is at 0%...
 
3:01 PM
Cbg
 
@payne memory at 95% sounds like thrashing. Any high disk IO to go with it?
 
I am having trouble with some indexes, wonder if someone can give me a hand pastebin.com/nH1GPrid
 
which subprocess is my script, anyways?
@AndrasDeak yes, I have to take the images from a folder through a generator
and every single image is split in subregions and data-augmented
 
I mean even without your own IO if you use up all the memory then excessive swap use can slow your system to a crawl. I could imagine GPU not getting a chance to work when that happens; I have zero experience with GPU.
 
and all those images are used as the target of the NN, but also I need to downsize them by a 4x factor because that is the input of the NN
(I work on SuperResolution, increasing an image's resolution by 4x the amount of pixels)
 
3:04 PM
Enhance! :P
 
@AndrasDeak "My own IO"? What do you mean?
 
> I have to take the images from a folder through a generator
That's your own IO.
 
also, when thrashing occurs, wouldn't the CPU be highly occupied?
 
the OS shuffling data to and from disk to use instead of RAM that's not available is not your own IO
 
3:06 PM
it means it is mostly waiting for data
 
CPU does "iowait" in unix lingo, no idea if that's a metric on windows
twiddling its thumbs for data
 
how could I see the amount of page faults and swaps?
(i.e. how to diagnose thrashing?)
 
95% RAM use might be a smoking gun, depending on swapiness of your OS
 
hey
is concurrent.futures real multithreading?
 
it still has the GIL
 
3:09 PM
in the sense that multithreading doesnt provide C# like mulithreading
@MisterMiyagi grim
thanks
 
python.exe isn't even listed in the Disc section... Moreover, the Disc usage is at 0%. I highly doubt that thrashing is the problem?
 
Former not relevant I think. Latter is. Probably not thrashing then.
 
3:33 PM
come to think of it, if there is 0 reads on the disc, that's definitely concerning :p
 
3:51 PM
@payne You said you're getting the files from a generator. Any chance you end up with what's effectively an infinite loop?
 
I tested the stuff with a smaller dataset and everything worked fine :/
See my project's repo if you're curious.
I g2g for a job interview. I'll be back in a few hours.
 
going from smaller to full might exactly change something that involves the generator running in an infinite loop that never yields
 
it's the first time I work with generators, too. Might be worth mentioning :D
 
@payne too high-level. If you can pin-point where the generator is and what the difference is between small and large someone might be able to help. No rush though, good luck with the interview
 
smalls = 12 images
large = >1k images
this is the generator
 
3:56 PM
I see while True with no break or yield on the top level, which is promising
 
this is where the training using the generator happens
 
I don't do neural networks
 
isn't the while True a basic necessity of the generators in Python?
 
no, just for infinite generators
 
anyways, really g2g. Check it out if you wish. I'll be back eventually :)
 
3:58 PM
rhubarb
If pattern_path = self.__get_pattern_match() can return a path that doesn't have any files in it then you wouldn't yield anything in that iteration, possibly ending up in a no-op infinite loop. If you only put the while True there because you thought that was necessary for generators you probably don't actually need it. Loop over the files, handle what's missed in the modulo (or something like that), and let the generator finish.
Unless there's a chance that subsequent iterations yield subsequent files, but I'm pretty sure that whatever setup you have you don't have an infinite number of files, so an infinite loop with no break is uncalled for.
 
Sam
4:57 PM
Good evening
 
cabbage
 
Sam
face_len = len(face_images) - 1
face_images_encodings.append(face_recognition.face_encodings(face_images[face_len])[0])
 
1. there's no return anywhere, 2. no
 
Sam
Sorry, I was confused. I mean, this shouldn't return an error
Or should it
Because it does
 
How about we don't play "pin the tail on the donkey" and you tell us what your problem really is?
 
4:58 PM
Exceptions are usually raised, not returned
 
Sam
Yes, sorry, was thinking about something else
 
Some functions do return Exceptions instead of raise them, but they're weird. Anyway, words are important.
 
Sam
IndexError: list index out of range
 
@Sam there we go
That's coming from [0]. What's before it is an empty string or tuple.
 
Sam
I shall post a more complete picture:
 
00:00 - 17:0017:00 - 00:00

« first day (3246 days earlier)      last day (1717 days later) »