« first day (3379 days earlier)      last day (1792 days later) » 
06:00 - 18:0018:00 - 00:00

06:07
cbg guys o/
@PaulMcG Any better blog/site for understanding Flask streaming than this: flask.palletsprojects.com/en/1.1.x/patterns/streaming ?
06:54
cbg
 
2 hours later…
09:36
Pandas is often over-suggested for CSV problems, and that's one thing...but this is egregious: stackoverflow.com/a/59766561/5087436
is that discouraged?
or are you saying using pandas for that purpose is not required
Well its definitely not required
it's just like "hey whats the best way to hit this nail?"
gotcha
and someone suggests "Well you could hire a 500-person construction crew"
09:46
Enough people use pandas that it can be a useful answer, so I don't think its particularly worth downvoting. But while it gives a solution its also not really an answer either (it doesnt actually address the underlying problem OP has)
user10984358
I feel like I am not following "DRY", just a regex snippet http://dpaste.com/1MFRK3B

can anyone see this code and tell me how I can avoid repeating my self
goodness
user10984358
should i instead write one mega regex using named capturing groups and do match_obj.groupdict() ?
please no
lol
you may be better off using string methods.
user10984358
09:49
what can i do here? i have so many blocks like this, final aim is to json them, there are so many tabs and white space
basically you need to .split(':') to separate keys and values, .strip() away whitespace, and do a conversion of the key to lowercase.
String methods are better if possible. But regardless, use string interpolation to fill in the search term.
"asdf{text_here}qwerty".format(text_here="Cylinders") turns into "asdfCylindersqwerty"
user10984358
i did try that conversion of keys, but for other "blocks" the names are not exactly lower cased versions
if you're on Python < 3.6
if you have 3.6, use f-strings
use a dict to map the text to find to the dict key they should go to
@alkasm yes, it's not worth downvoting because it works. may be a comment from the downvoter would have helped.
09:58
assumes that the value you care about always comes after a : and all whitespace otherwise can be ignored.
user10984358
that is what i need, i was trying something with enumerate :/ but yours is peachy!
i try not to use regex in Python. Regex isn't all that bad in general but the alternative in python is just so readable!
cheers, happy programming
10:17
I am using Flask. I am writing to the logger using: app.logger.exception("error message") - I have configured handler as file. In the command promt screen I can see the error and traceback. But in the file I can only see the error message and no traceback.
Any ideas on why traceback is not getting into the file?
10:35
I need some help in converting an xml file to a csv file. I want to do this process automatically, which way do you recommend?
did you take a look at Python's standard XML and CSV modules?
yes I have, I initially tried to pare through it and create lists of the variables I need but it did not work because of the nature of the xml file I have
I want in python some code that would iterate over each id and collect the data I need for each id and save them under that id.
id | measurementTimeDefault | index | FlowRate | Speed
id1 | 2020-01-15T13:11:00Z | 1 | 720 | n/a
id2 | 2020-01-15T13:11:00Z | 2 | 120 | n/a
id3 | 2020-01-15T13:11:00Z | 3 | 55 | n/a
That doesn't look like an XML file to me
this is the example output I want
Ok, so what part of parsing the input are you struggling with? You've not given a specific problem for us to help with
10:48
In many examples that I have seen they start iterating from the root, I want to start iterating from a child of that root. How do I do that?
Essentially, to find a specific tag "Tag1" and start iterating within that tag
Presumably it's parsed into a nested structure of lists and dictionaries. It's really not possible to say unless you give us some example
@Vasilis please don't invite me to another room for this. There is no reason that it can't be discussed in the main python room (there is no guarantee that I can solve anything, but others might be able to help instead). You should make a minimal but complete example. As per the room rules you should post off-site and link here if it's a lot of code/XML
<messages>
  <note id="501">
    <to>Tove</to>
    <from>Jani</from>
    <heading>Reminder</heading>
    <body>Don't forget me this weekend!</body>
  </note>
  <note id="502">
    <to>Jani</to>
    <from>Tove</from>
    <heading>Re: Reminder</heading>
    <body>I will not</body>
  </note>
</messages>
This is a very good example I think, I want to output what I showed above for each id
Ok, so you've parsed this into a structure, right?
I guess so
doc = xmlread.parse(output_xml)
Well, have you looked at doc?
10:59
the documentation? I have looked for both mindom and elementree, and I still struggle with this part
they do not create a table like I showed above in the examples I ve seen
No, your variable named doc. If you apply your parser to the example you linked here, that will give you some idea of what you're dealing with without being overwhelmed with data
when I print doc I only get that : <xml.dom.minidom.Document object at 0x00000179C1430A68>
Oki doki, I guess this is the part that I have to actually replicate on my end because I didn't assume it would give a custom document object, I assumed it would just spit out lists and dicts :)
thank you, any help would be useful at this point!
Further to my post from 15:47, if I use the following code then the traceback gets printed to the file:
import traceback
app.logger.warning(traceback.format_exc())
user10984358
11:13
@variable just curious if the time stamp you sent is from your end or it changes according to my timezone? sorry for asking as off topic question
My end
Its what I can see on the chat history
Huh, I didn't think I'd be struggling with XML parsing. I hate the fact that children should be accessed by index and not by name. It could take me a bit of time to figure this out @Vasilis
Guys - technically are both these exceptions: raise AttributeError, raise Exception('some msg'). Or only the 2nd one is an exception and the 1st one raises an exception.
11:33
@roganjosh I know it is not as easy as it looks, thanks for any input you give!
I've sussed it, I just now need to work out how to get a pipe character on this keyboard :P
Oooh, that's a bit annoying. I did have the pipe character but Spyder 4 italicises it to look like a forward slash when it's in a string by default. I'll need to fix that. Very much a TIL morning :)
One sec, I should demo getting the ID of the note
@Vasilis This works for your example:
import csv
import xml.etree.ElementTree as ET

tree = ET.parse('example.xml')
root = tree.getroot()

rebuilt = []

for note in root.iter('note'):
    rebuilt.append([note.attrib['id'],
                    note.find('to').text,
                    note.find('from').text,
                    note.find('heading').text,
                    note.find('body').text])

with open('example_out.csv', 'w', newline='', ) as outfile:
    writer = csv.writer(outfile, delimiter='|')
    writer.writerow(['id', 'to', 'from', 'heading', 'body'])
I'm uing Flask to deploy a dominant color finding application and developed the main application in CPP, once the user uploads the image I process it in C++ , but when I pass the file parameter to it it says invalid file however it works in python. Any suggestions?
` if file and allowed_file(file.filename):
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
p = Popen(['/blah_blah/dominant_colors', 'file', '3'])
flash('File successfully uploaded')
return redirect('/') `
@AshwinPhadke triple quotes won't work in chat. See the formatting guide
Change Popen(['/blah_blah/dominant_colors', 'file', '3']) to Popen(['/blah_blah/dominant_colors', os.path.join(app.config['UPLOAD_FOLDER'], filename), '3']) ?
@Aran-Fey will try it out
11:50
In Flask, we have the python root logger, flask logger (flask.app), werkzeug logger (if using the flask development server), and if we use production server then we have for example gunicorn's logger. I am looking for any documentation that will help understand what gets logged into each of these loggers. Any help will be appreciated
@roganjosh thanks, didn''t knew it doesn't work
@variable they both raise Exceptions.
Do you guys log whenever a method (that isn't a private helper method) is called?

Logger.debug(f"[className:methodName({arguments...})")
Is that good practice? Or is it just clutter?
Is it just me or is the XML parser just incredibly brittle? Like root[0][1].text seems awful for accessing child objects, and rank = country.find('rank').text immediately fails badly if 'rank' isn't found, but there's no way given to elegantly handle the missing key; just go straight for .text
@MisterMiyagi ok thanks
12:18
How can I remove .0 from column called YEAR type float
By converting to an integer using astype, assuming this is pandas/numpy
@roganjosh yes it's in pandas. It doesn't let me convert as I have some NaN values that want to keep
@Aran-Fey that method worked but now C++ program isn't doing anything not even returning error, gotta look at that , thanks for the help.
@ruben.lfdz Then, unless you're in the latest versions of pandas, you can't have NaN in an int column. What version of pandas?
@roganjosh column is type float
@ruben.lfdz That's not what I asked. I asked you what version of pandas you are using
@ruben.lfdz import pandas as pd; print(pd.__version__)
@roganjosh Yes totally. Just don't use XML parser, and have to write code that mimics the structure of the tags. Use xpath syntax (with lxml, ElementTree or whatever).
@roganjosh '0.25.3'
Thanks. I actually rarely ever do HTML/XML parsing so I was quite shocked that I couldn't find an elegant way to handle it. I think it's a combination of no real experience and an... interesting... library
@ruben.lfdz pandas will not downcoerce if an integer column contains any NaNs. This is a well-known issue. Does your column contain any NaNs? see (.value_counts(..., dropna=False). If yes, is it ok to replace them at read-time with some sentinel value like -1, 9999 etc.?
12:39
@ruben.lfdz In that case you can use the nullable integer type
@roganjosh Hmm, didn't know: "It is not the default dtype for integers, and will not be inferred; you must explicitly pass dtype = 'I' or dtype = pd.Int64Dtype()". Does it work well with operations like aggregation, apply, pickling etc? without having to redo astype() everywhere?
@smci I can't answer any of those questions because I've only just upgraded myself but I'd wager that pickling is the only real concern there because the other 2 are handled internally. I'd imagine that pickling was pretty much broken
It's also survived 0.24 --> 0.25 in an experimental state so I guess they're having issues
@roganjosh Ok cool then you might like to update the existing questions on how to read/handle integer columns containing NaNs, almost noone is aware of this. Also there's this person using 0.24 who wanted to infer float or Int64, else string: stackoverflow.com/questions/55479542/…
:S I'm only aware of its existence, not actual practical usage, and it's still "experimental" so it's perhaps best to hang-fire
@roganjosh Right but I also wonder if you do an aggregate/apply/transform/assign/etc if it preserves Int64Dtype or not... don't have time to test it and I suspect it has corner-cases and limitations which aren't well-documented, even in github... beware fresh new barely-tested pandas functionality...
@ruben.lfdz Did that fix it?
12:50
Sure. I only suggested because they seemed pretty adamant that they didn't want a float type. In reality, I expect the requirement to remove .0 is an XY problem
@ThiefMaster - in this (github.com/pallets/flask/issues/2704#issuecomment-381543277) post you have mentioned that - "General advice: Don't use app.logger for more complex apps. Instead, just use Python's logging module directly.". So what is the purpose of the flask logger (flask.app)?
And will I have to turn on the propagate on the flask.app so taht the logs from flask all come into the python loggong module also if I wish to capture them at one palce (that is on teh looging module's logger)
@smci I'm not sure how to apply it. Another solution would be turning 0 values to NULL in int64 column. How can I do that?
That's the opposite of what you've been requesting and directly conflicts with what we're saying
@ruben.lfdz You can't have NaN and a dtype of int (of any kind). I gave you a link to an experimental implementation in pandas where this is possible
@roganjosh to explain myself better I have to present a column that represents values and 0 has to be Null so what I did is turned the column into np.nan although values now show with .0, instead of doing that another solution would be turning 0 values to NULL in int64 column. Did I explain myself better?
@roganjosh thanks a lot anyways. I'm not sure how to apply that
13:08
@roganjosh thank you very much I will work on it, have a good day!!!
what you say looks like astype
@ruben.lfdz Like this:
import pandas as pd
import numpy as np


df = pd.DataFrame({'a': [1, 2, np.nan]})
print(df.dtypes)
print(df.head())
print('........................................')

df['a'] = df['a'].astype(pd.Int64Dtype())
print(df.dtypes)
print(df.head())
you can do the same as @roganjosh said earlier
but with astype
@roganjosh exactly similarly i guess .astype('Int64') should also work :)
@anky_91 I'm kinda inclined to say I'd keep it as obvious as possible that this is a distinct, pandas-only, dtype that isn't gonna work elsewhere.
hmm, I understand
13:12
But, hey, that's just personal preference :)
No i get your point which is correct
:)
@variable it's just something convenient for very small apps.. in a bigger app where you want structured loggers it's not usable anyway
13:26
@ruben.lfdz Look, don't keep the NaNs unless there's a valid reason (e.g. you're doing aggregation, summary statistics (e.g. mean), tabulation, value-counts, histogram etc. and you don't want them included). Can you convert them to a sentinel value like -1 or 9999, yes or no? If you can, do. If you can't, please explain what information would you actually lose if you converted them to a sentinel value? We don't know your data so we can't answer for you. You have to tell us.
@roganjosh It worked ! thanks a lot
@ruben.lfdz You're welcome. Keep in mind that it's possible that you'll get wonky behaviour because it's something they are experimenting with; you should read the docs that I linked earlier
13:46
@ruben.lfdz Like I said, the Int64Dtype will work if you don't do much postprocessing on the data, but if you do the things I mentioned, type coercion or some other functionality will probably get broken, and in surprising and untested ways (since this is new functionality). Whereas people have been replacing NAs with sentinel values for 10+ years, as a workaround.
14:13
morning cabbages
Current status: trying to answer a question about nested data structures while not looking directly at the flame war in the comments
People have very strong opinions about whether a dict of lists of dicts of strings is "a JSON" or not
it's obviously a dict of lists of jsons smh
My opinion, which I will not express within the question, lest I be sucked in: there's no such thing as "a JSON", but you can tell what people mean when they say that, so just go with it
In the event that the question asker is vague about whether their JSON is a (string in memory that needs to be run through json.loads | a file containing said string that needs to be run through json.load | an actual useful dict), you may then scold the asker. But this particular question has a functional MCVE so that's not an issue here
It's often not clear whether an OP has actual JSON or a dict/list, so I do wish people would use the correct labels instead of calling literally anything under the sun "a JSON"
how can I create a formula that extracts the info within the () in a column with data like: 28 of november 2012(London), 2012(Amsterdam)
14:25
>>> import re
>>> pattern = re.compile(r"\((.*?)\)")
>>> pattern.search("28 of november 2012(London)").group(1)
'London'
>>> pattern.search("2012(Amsterdam)").group(1)
'Amsterdam'
>>> pattern.search("what (happens) if there are (mulitiple) (parens)?").group(1)
'happens'
>>> pattern.search("what (((((happens))))) if there are nested parens?").group(1)
'((((happens'
>>> pattern.search("what happens if there aren't any parens?").group(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
@Kevin since JSON is a notation, the only corresponding thing in Python is a string adhering to said notation. Just as a URL is a string adhering to URL format.
everything else is lies, damned lies, and fluffy kittens. But not JSON.
I know, but the steady tide of neophytes don't know that, and it will always ever be thus
The only viable way forward is to teach these neophytes about The True Meaning of JSON. Then, they will teach others The True Meaning of JSON.
@Aran-Fey I suggest that's worth posting on Meta, trying to agree what the different possible meanings are.
Until we all live happily under The True Meaning of JSON. Then we'll realise that perhaps, just perhaps, it's fine that a JSON is a dict sometimes.
14:30
@smci Feel free to post it on meta, but I'm not gonna bother
A daycare employee may say "in my fifty years in this business, not one of my charges has known multivariate calculus", but he should not despair at the mathematical illiteracy of the last five generations.
@MisterMiyagi More like, it's a subset of possible dicts. There are restrictions on both keys and values. See chat.stackoverflow.com/transcript/message/48233650 passim. (The confusion/definition about this keeps coming up on SO...)
the valid JSON runtime representation is Union[str, int, float, bool, None, Mapping[str, JSON], List[JSON]]
Recently I noticed that the website we use at my workplace makes firefox emit a warning that "this TLS version is outdated and will be disabled in March 2020". I wonder if they're going to fix this in time or if our company is gonna implode in March
I hope they release JSON 1.1 soon, with support for trailing commas
14:37
possibility #3: We'll switch to Internet Explorer
netscape for the win!
All: we don't seem to have a findable canonical target for "Why shouldn't you remove items from a list you're iterating over?", stated in prescriptive language like that. I only see How to remove items from a list while iterating? (locked, 2009) which indirectly implies "You shouldn't do that, if you need that then you probably want a list-comprehension instead" . In comments someone recommends...
I'm decently sure removing items from a list during iteration is plain undefined. Just because it yields some deterministic behaviour on all implementations doesn't change that.
So we get confused questions like this: stackoverflow.com/questions/30731114/… not being closed as dupe, just as 'Needs more focus'.
@MisterMiyagi Sure. I mean a high-level "Don't do this in Python, because it's not supported, what precisely happens will be implementation-dependent, so your code won't be well-defined, testable or portable". Not like excruciating detailed analysis on what all the diffrent implementations and versions do.
Related, do any of you ever encounter legit use-cases for using .copy() or [:]?
The behavior of removal-and-iteration is somewhat defined. docs.python.org/3/reference/… contains a "Note:" box describing the interaction. But one might argue that this is a description of current behavior and not a prescription of required behavior
14:45
well, they all do the same thing -- pointer increments without consistency check.
The builtin data types are actually documented quite poorly... I can't even find documentation that says "lists are iterable"
The behavior of for loops in general are well defined: get an iterator, and call next() on it until it stops giving you values. So we kick the can down the road and ask whether the behavior of a list's iterator object is well defined during mutation
@Kevin Decently sure that's not a rule, since it talks about sequences, and these don't have any obligation like that,
Mm hmm, I'm inclined to say that "Note:" boxes have dubious canonicity
14:47
@Kevin It's impenetrable to newbies, and it doesn't hint at "Don't try this at home, kids". (And even saying 'iterable' instead of 'list' is impenetrable to newbies and Google, which is how people find stuff...)
indeed, the docs are pretty bad at making it clear that some things you can do but really should not
cbg all
@smci I've used [:] to make a quick copy of a list, without the reference to the original.
How do I pass a image from a python function to a c++ code except using popen and executing the cpp binary like a command?
ah, here it says "Lists implement all of the common and mutable sequence operations."
@smci yes, whenever I pass an owned list to something expecting ownership as well.
I'd prefer .copy() over [:] though. The latter is rather obscure.
14:50
@MisterMiyagi Can you give me an example? (Isn't that a code smell you should probably be using an object or mixin class, with some accessor method?)
I fail to see how objects or mixin classes are related to list mutation.
I fail to understand your reasoning then. I asked can you show me an actual example of two things (objects?) expecting joint ownership of a list, so they can both mutate it? Any time I've needed to thtat, I'd use an object.
a list is an object oO
I do copy lists sometimes, but I can't think of an actual example. More often than not I use list(some_iterable) to create a "copy" because I can't be sure that my input is a list
I wonder how hard it would be to create a list subclass whose __iter__ doesn't seemingly jump around during mutation-and-iteration.
For starters you'd need to override every mutating method so they can inform any living iterators that the size of the list is about to change
14:56
I imagine it's probably more tedious than hard
@Kevin Override delete as well.
guys, stupid question...python is running through the apache webserver with a wsgi module...how do I run a script that works ok on local machine ? i have and endpoint something like : 10.5.5.55/start
@MisterMiyagi You think? I don't think I've ever used .copy() = I've used deepcopy, but from memory 'most'(?) of the beginner tutorials showed you [:] even if it's just part of showing how slicing works.
@MisterMiyagi We know that. But you seemed to be talking about two (bodies of code?/functions?/methods?/other objects?) containing/wrapping/referencing a list. It would be clearer if you showed a specific example.
I think the problem gets fuzzy at the edges, in particular when some operation both changes the size of the list and changes the value of the element that was most recently yielded by the iterator. Let me compose an example...
14:59
@Kevin You quite likely have to consider the 'add'/'insert'/'append'/'delete'/'mutate'/etc. cases differently.
I feel like this would be weird... So you want an object A that keeps track of what is the next objects. You would need to have __next__ keep track of the "current" object say i. So, suppose you call del on and object. Say that you call del B on an object in A. But del B then calls A.i=0 to reset counter...
def __awake_all__(self) -> List[Tuple[Coroutine, Interrupt]]:
    """Awake all waiters"""
    awoken = self._waiting.copy()
    self._waiting.clear()
    for waiter, interrupt in awoken:
        __LOOP_STATE__.LOOP.schedule(waiter, signal=interrupt)
    return awoken
x = [0,1,2,3,4,5,6]
for item in x:
    print(item)
    if item == 3:
        x[2:5] = [3, 23, 3, 42, 3]
        #at this point, x is now `[0, 1, 3, 23, 3, 42, 3, 5, 6]`.
        #what is the next item that should be returned by the generator?
@Kevin ...use a dictionary under the hood? Use indexes as keys? Maybe using 1.1, 1.2 etc for elements added between index 1 and 2...although frequently composing an ordered list of indexes and iterating over them might get slow.
Somehow I think you can't do this because of self-referential paradoxes.
15:02
It's easy enough to answer this question in terms of what the existing generator does, but it's not clear to me what the hyptothetical "smart position tracking" generator would do
@smci basically whenever I use lists between concurrent actors, if that is any help to you. The use-case doesn't lend itself to copy/pasting examples that show more than the call to .copy.
@MisterMiyagi Right. But isn't that semi-manually implementing concurrency?
@toonarmycaptain obscure as in "people who don't know what it doesn't won't guess correctly"
You could separate element index and 'id' so to speak. You might lose relevance of the actual value of index, but you keep track of which element is current and next, etc even if the precise location in the list changes.
@smci Perhaps you could clarify what you mean by "semi-manually implementing concurrency" and how that makes it not a viable use-case?
15:07
cbg all
any ideas, guys ?
For non-concurrent use-cases, I've almost exclusively switched to avoiding mutations like the plague that it is.
@MisterMiyagi Fair enough. I'd assume they wouldn't know what list[3:] or list[::4] do either though.
@MisterMiyagi I didn't say it wasn't a viable use-case. But it's pretty rare and not something new users would do. (Personally, I've only ever implemented concurrency in C++ and Java not Python.)
@Kevin you still have that gist in your repo of the check_temp threading you sent me?
15:11
@smci Iteration by itself is concurrency. It's exactly why list iter+del is bad - we are concurrently iterating and deleting.
Had a question on it, I can sanitize mine and pop into a gist if necessary. Seem to be having an issue with it not updating quickly
@AnotherUser31 It's (IMHO) unanswerable. There's no detail given
@biggi_ Hmm, if I do, I can't find it
I don't often delete gists, but Github's search function often fails to find things it should
@roganjosh i just simply have no idea how to run a script from using the setup described above...
Nor do we. Launch it as a subprocess?
15:14
@roganjosh it's a flask app...some image processing...
that works ok when run from my local machine...
Ok, well then you definitely can launch a subprocess but it'll tie up one of your threads/processes until the image processing is done
How are you expecting it to work? That might be a better starting point
Can an RO please clear my latest stars please? I think newer visitors have mistaken it as an upvote
I'm not sure whether "that works ok when run from my local machine" is prescriptive (as in, "when my project is complete, it must work ok when run from my local machine. Unfortunately, right now it doesn't") or descriptive ("currently, it works ok when run from my local machine")
Thanks :)
@MisterMiyagi Huh? Concurrency implies 2+ actors. Iterating over a list doesn't. (We might choose a concurrent implementation for a for-loop, but it isn't necessary or implied). If that's wrong, tell me how.
@Kevin no issue :) let me see if I can get mine down to a bare min example and I'll post it. It seems like my queue is piling up and it's only updating like 10-15 seconds after the data has changed. I've even got the message queue set to not send another message to the queue unless it's empty now and it's still not quite working
15:20
@roganjosh I expect to see a form html interface...I can input a code, and the script will start it's job...
how the subproccess would work ? sorry if questions are too stupid...
so what's considered after_idle for a tkinter window? is it if there's nothing running in the background or just the window?
@smci In the iter/del scenario, are two separate tasks/stacks: The iterator body and the loop body. With the exception of 0/1 length iterators, the iterator state persists while body runs and vice versa. While they don't run in parallel, they run concurrently.
@biggi_ Nothing in the window, as far as I can tell. Tkinter doesn't care if you have other threads doing stuff.
This is text book cooperative concurrency.
Hmmm weird. I'm not doing anything else on the window and it's super laggy
15:27
Note that async/await is just syntactic sugar for iterators, and a for loop is a very primitive variant of an event loop.
@AnotherUser31 the questions aren't "too stupid". But are you ok with the browser hanging while your do the processing or do you want something else?
@MisterMiyagi Ok I think you're saying the iterator itself, and whatever the loop body is doing, are two separate concurrent actors. (At least, they are when the loop body does something that alters the state. If it doesn't, wouldn't we consider they conceptually just have one (shared) state? which is the usual case)
yes, loop body and iterator each are separate actors.
They are even if none of them alter state. The problem with iter+del is that they share state, and one expects said state to be static whereas the other mutates said state.
It's fine to share state, and it's fine to mutate state. But as soon as one shares and mutates state, the usual concurrency problems manifest.
@roganjosh totally ok with browser hanging ! for now...later maybe I can learn how to do the asynchronous stuff. All I have now is that endpoint...when I open the page it's just a Hello World message. When I try to upload my script, say, image.py and open the page, I still see that Hello World message.
@AnotherUser31 for example, at work, my model takes ~90 secs to run. I decoupled it completely from the flask app; the user submits the model, the body of which is dumped to json, and then an entry is written in my db. I have a completely different script that queries the db to find the next model to solve. Once a job is submitted, the user's broswer is freed up and they can do other things
@AnotherUser31 This seems more fundamental than what you first asked. image.py isn't going to change the template that you're rendering
15:39
@roganjosh I don't get it..how I can run another script rather than that is is now...
def application(environ, start_response):
status = '200 OK'
output = b'Hello World!'

response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)

return [output]
The conventional way of calling one script from another is to import it, and then execute whichever function you need
So you'd do import image at the top, and image.do_whatever() inside the application function
@Kevin mind helping me how to call the script here in my script above ? say I want to call the image.py
We're coming at this from different angles. Why aren't you using render_template if you want something displayed in a browser?
In fact, I'm really lost. You want to use a URL pattern in the browser to start a script off, but it looks like you're designing a programming API with no front-end
How do you intend to submit 10.5.5.55/start to your server?
Is that a GET request from some other program?
@roganjosh it's not an API...
Ok, so then you want to be using render_template I think. You're trying to render HTML and also start a process?
15:47
again...i just want to run a script that works on my machine...from apache webserver with a wsgi module...
@roganjosh yes ! I do use it inside my app, but I don't need to display any text back or something...
for now...i can't even start it !
that's my problem
So I guess image.do_whatever() didn't work for you? Is it still displaying "hello world", or are you getting an error message, or what?
kevin: lemme look at this more this morning. may do a gist this afternoon if i'm still having issues. Learned a lot the past week ripping my hair out...maybe my brain can breathe better so I can do more XD
@TheLittleNaruto Not that I know of. Note this only describes the server side of the issue also. You'll need to also learn the client side, either requests or httpx. We use requests at work; in which you set stream=True in the get call, and then use iter_content on the response object, specifying a chunk_size - for completeness, I'll repeat that I recommend 32 or greater for this. The default of 1 will make things horrendously slow.
@Kevin...i imported the way you told me...but I don't really get it how to do this part image.do_whatever()
It's referring to the entry point of whatever your image code is supposed to do. How do you actually trigger the code without the web server?
16:01
"I don't have a method named do_whatever()", you hypothetically say, "or any other single function that makes for a convenient entry point. When I usually run image.py, I do it by invoking it straight from the command line. I have a lot of logic at the global scope, and I need it all to run when my 10.5.5.55/start page loads".
If that's the case, there are two possible solutions: 1) refactor image.py so that it does have a single function that makes for a convenient entry point. In the best case scenario, this is as easy as indenting all of your global code and adding a def main(): to the top.
2) leave image.py exactly the way it is, and instead of using import, execute it using subprocess.check_output or similar.
may have figured it out? I had a print statement in an exception block on my on_idle task and it seems like removing that fixed everything?
str(subprocess.Popen([cmd, *args], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0], 'utf-8') has been my go-to way of dealing with this sort of thing, of late
@Kevin yes, you're so right !!! I would like the second option, with subprocess. @inspectorG4dget where I should add that part ?!
put that subprocess stuff in the python file that executes image.py, wherever you would like to call image.py
C:\Users\Kevin\Desktop>type image.py
import time
print("Frobnicating the gumpy...")
time.sleep(3)
print("Frobnicated.")

C:\Users\Kevin\Desktop>type main.py
import subprocess
print("About to execute image.py...")
subprocess.run(["python", "image.py"])
print("Executed.")

C:\Users\Kevin\Desktop>main.py
About to execute image.py...
Frobnicating the gumpy...
Frobnicated.
Executed.
In production-quality code, there is almost never a good reason to use subprocess to execute another python script that you have control over. You should almost always rewrite the script so it can be imported normally. But if you just need a dirty hack to get things off the ground, there it is.
16:10
@MisterMiyagi More like: it's not fine to share mutable state without some concurrency construct (semaphore, mutex etc.) with something that might change it, is the version I was taught.
@biggi_ "I removed a print and it started working" is a classic symptom of a race condition that hasn't been solved, and has merely gone dormant
Sometimes, it stays dormant forever. Sometimes, it crashes your demo during the big important presentation you're giving to your investors
@smci "it's bad to mutate shared state without protection" and "it's not bad to mutate shared state with protection" seem to amount to the same thing.
Exactly why the question mark was there :) I'm still confused on how it could be a race condition on idle though
going to put a small wait in my while loop too
give it time to catch up instead of running full bore
@MisterMiyagi It actually depends on how many training-wheels the language builds in. Cheerfully providing lots of different copy commands and syntaxes, without any argument for an optional builtin concurrency construct, seems dangerous.
Maybe you already know this, but many threading-related objects have a "wait until something important happens" method, which is usually preferable to use instead of time.sleep. For example, queue.get will wait until the queue contains a value.
16:17
I did not know that. I'm just curious as to how after_idle could have a race condition
I just got something to work. Did it a little too elegantly ("and he sticks the landing!"). My ego is soaring out of control. Quick, please list a few python things that I am not yet familiar with and take me down a peg. Numpy and Pandas, sure. What else?
All I'm doing threading wise is running my logic in background and displaying a value of it on a front panel
And yes; this is still that god forsaken lidar thing
They keep wanting more :)
thank you
If you happen to find the link to my threading code in the transcript, let me know, and I'll double check it for race conditions and/or slowdown issues
16:19
@smci Well, if we started telling newcomers "for loops are stackless cooperative concurrency", no one would ever get done training their neural networks.
yup! that did it. Much thanks
@PaulMcG Could you be able to share a sample(Please!)? The documentation seems not complete.
Hi can I use txt file for range? When I try that I get 'str' object cannot be interpreted as an integer. I need
for dec in range(1, 100):
user(dec)
I tried

f = open("demofile.txt", "r")
for dec in range(f.readline()):
and

f = open("demofile.txt", "r")
for x in f:

for dec in range(x):

private(dec)
Is this possible?
@Pijes, I'm not totally sure I understand what you want, but perhaps try for dec in range(int(x)):
Coupled with Python's strong stance on non-concurrency being more important than concurrency, it makes sense not to have training wheels for concurrency. They're really costly.
16:20
@Pijes I have a pretty strong feeling that you've been directed to the code formatting guide before
@Kevin found it. It wasn't a gist pastebin.com/2u6M5F91
for dec in range(len(f.readlines()))
@Pijes Is the content of demofile.txt something like 1, 100 by an chance?
@MisterMiyagi Many of my Python and non-Python friends simply say the language is broken for concurrency.
@biggi_ Thanks.
16:23
@smci I would not disagree.
Though I'd say that about most languages, actually.
@MisterMiyagi Do you ever use Go for that reason, then? or C++?
Go is a total mess RE:concurrency.
Isn't that basically its primary purpose?
Python is a bit of a concurrency mess. Concurrently, Go is a mess
Its basic algebra of goroutines, channels and other synchronisation does not compose.
C++ and any similar language built directly on mutating memory is a nightmare for concurrency.
I vastly prefer Rust, though incidentally iter+del would break with its borrow model in the same way as in Python.
16:26
@Kevin Yes you are right. Thank you. How I can get rid of \n ValueError: invalid literal for int() with base 10: '1, 100\n'
I'm guessing that your file has more than one number per line
@Kevin yes
Parsing multiple arguments for range() from a line is not so easy as parsing one
actually, map(int, line.split(',')) works well
Yes, for dec in range(*map(int, line.split(','))) would do it, but I maintain my position that this is not as easy as the previous code.
16:29
kev: headed to lunch with a supplier. you can ping me if you find an issue, but no rush ;)
I'm going to keep looking at it this afternoon
"Solvable by us in three seconds" is still easier than "solvable by us in fifteen seconds", even if both look like magic to the observer ;-)
Ey boys, getting an indexerror with a script I am trying to run, anyone any clue? stackoverflow.com/questions/59771391/…
@Kevin On second thought, you might be onto something ^^
In any case, the "right" way to do this would probably be to use a well-established serialization format, like json, instead of trying to parse the numbers yourself
@Kevin...when I try to run this simple piece : ``` from datetime import datetime, timedelta


def application(environ, start_response):
status = '200 OK'
output = b'Hello World!'

response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)

now = datetime.now()
Day = (now.strftime("%A"))
print (Day)
if 'Friday' in Day:
due_date = datetime.utcnow().strftime("%Y-%m-%d")
due_date = datetime.now() + timedelta(days=3)
output = due_date.strftime('%Y-%m-%d')
16:32
@AnotherUser31. Hmm. Other than the missing indentation, I don't see anything obviously wrong. Check the server error log to see if there's a stack trace. If you don't have a server error log, you may need to create one.
Wild guess: maybe [output] needs to be a list of bytes objects, but your output is a string. Try return [output.encode()] and see if that does anything.
due_date = datetime.utcnow().strftime("%Y-%m-%d")
due_date = datetime.now() + timedelta(days=3)
This is completely superfluous because you just overwrite the name. In both cases
@Kevin for dec in range(*map(int(x), line.split(','))):
I get the same error
Sorry I put line instead of x and now is OK. Thank you
for line in f:
for dec in range(*map(int, line.split(','))):
Also, there is a weekday method in datetime so there's no need to have Day = (now.strftime("%A")); if 'Friday' in Day:
@MisterMiyagi Thank you
@Kevin @roganjosh I'm lost...I'm givin' up...didn't thought it would be so damn complicated :(
16:46
@roganjosh That's for sure. Thank you
"I didn't know this would be so darn complicated" -- I say that about three times a day
@AnotherUser31 Sleep on it :) Don't give up!
@AnotherUser31 Please don't give up. Someone will help you. Just use some time to rest. Python is future
@Kevin :)))) @roganjosh @Pijes you guys are fantastic !!!
Software development is the art of reducing a giant mountain of complicatedness into many small molehills of complicatedness
16:50
it is definitely an art !!!
@TheLittleNaruto Something like this:
response = requests.get(url, data=params, auth=self.auth, stream=True)
response.raise_for_status()
text = []
for line in response.iter_content(chunk_size=1024):
    text.append(line)
return json.loads(b''.join(text).decode())
Maybe post as a self-answered question so this can be more generally accessible instead of just in chat.
17:07
hi guys
I wanted to implement websocket into my existing flask app
there are so many different ways to do that..
my flask app looks like this:
    app = mesito.app.produce(session_factory=session_factory)
@Kevin Personal question: where do you buy your dynamite?
in the mesito.app.produce, I just wireup the routes and return the app:
app = flask.Flask(__name__)
CORS(app, resources={r"/*": {"origins": "*"}})


app.route(
'/api/put_machine', methods=['POST'], endpoint='put_machine')(
lambda: put_machine(session_factory=session_factory))

return app;
and now I want to add websocket like this:
http_server = WSGIServer(('', 8000), app, handler_class=WebSocketHandler)
    mesito.app.register_server(server=http_server, app=app)

    http_server.serve_forever()
So I use these two libraries:
from geventwebsocket.handler import WebSocketHandler
from gevent.pywsgi import WSGIServer
it looks like it could work... BUT now I just wanted to add cross origin * to the websocket
how do I do that?
@toonarmycaptain The most potent demolition equipment comes from within
because my server is setup with WSGIServer, I can't just use the Socket.io which is so much nicer
I'm batting 0.000 for WSGI questions today ;_;
17:13
0.000 whuat? o_o
In other words, of the two WSGI questions asked today, I have 0 satisfactory solutions
the other question is.. what is this WSGIServer at all? hmm ok I should go read first by myself
:( oh Kevin, oh Kevin
@kevin you worked with gevent-websocket
why is this page so unfinished :(
where is the github page of that hehe
downloaded the project, and looked for "cross" in the whole project
nothing found
so its not possible with that library?
17:34
@PaulMcG yeah please do that. I'll award a bounty
:-)
"you worked with gevent-websocket" -- I did? Doesn't ring a bell, I'm afraid
I see yamming timeline links on posts! D:
@AndrasDeak I actually like them :/
me too, it's been many years overdue
Oh, I mistook your shock for disgust :P
17:36
it's too prominent though; it would make more sense as a named link under the post
you usually don't want to look at the timeline
@TheLittleNaruto Sorry, I meant you could do this. I don't post very many questions generally, and you would do a better job at describing what your original question actually was.
@PaulMcG Cool! I'll do the question part. You do the answer part lol
but then it won't be self-answered 🤔
06:00 - 18:0018:00 - 00:00

« first day (3379 days earlier)      last day (1792 days later) »