« first day (4216 days earlier)      last day (955 days later) » 

05:11
Hello all. I have a quick query. I have some code that I want it to keep executing, like keep looking for new data, or anything like that. Is there some tech word that I cannot think of or find? Or should I just use the while loop.
Like, if i wanted to make a project on scrapping the data from this chat, and I can get all the existing chat, but to find new messages, I should have some code that will find the messages, and the code will keep scanning the site.
05:51
Just use a loop
 
3 hours later…
08:34
Why would pandas need to include both isnull() and isna() when they are both exactly the same thing?
Wes
Wes
my program after writing python the way i program in php:
my program after i program in php:
tbh python grew on me. it's just the imports thing that is insane
09:01
@Pherdindy there isn't any reason other than probably just confusion when designing the API. Given the fundamental nature of the check, it's not inconceivable that there could have been multiple people working on the implementation in the early days and then they just ended up aliasing one of them for backward compatibility or something. But, I don't know
Yeah was just hoping they linked each other in the docs to show they are equal. Read through both pages to try find the difference lol
 
4 hours later…
13:00
Today I am trying to execute tk/tcl commands from Python without using tkinter classes. The docs mention a module that might be useful: "_tkinter: A binary module that contains the low-level interface to Tcl/Tk. It is automatically imported by the main tkinter module, and should never be used directly by application programmers."
13:38
from tkinter import Tk
root = Tk()
root.tk.call("button", ".b", "-text", '"Hello, World!"', "-command", "exit")
root.tk.call("pack", ".b")
root.mainloop()
I guess I don't mind using the Tk class, just for basic setup
I would have preferred to write each command as a single string, rather than breaking them up, but AFAICT that's not allowed.
I think tkinter talks to tcl via tcl_evalobjv, which only understands commands in array-of-arguments form. Tcl_Eval understands single strings, but I don't know if it's accessible from the Python environment.
Oh good, it is.
from tkinter import Tk
root = Tk()
root.tk.eval('button .b -text "Hello, World!" -command exit')
root.tk.eval("pack .b")
root.mainloop()
14:01
#one line guis, for all your code golfs that need both concision and style
(lambda root: root.tk.eval("""set keysym "Press any key"; pack [label .l -textvariable keysym -padx 2m -pady 1m]; bind . <Key> {set keysym "You pressed %K"}""") or root.mainloop())(__import__("tkinter").Tk())
(code golfs? code golves?)
14:13
I have four variables in my csv: ID, From (start of interval), To (end of interval) and Assay (from an analytical machine). I want to build a new dataset from this that looks at the ID and if the Assay is the same value it will take the first From and the last To of the repeated value on the Assay. Does that make sense? Trying to figure out where to start, what is best to use to say if IDs are equal and assay is the same, do I use a counter?
How are you reading the csv?
"I haven't figured that part out yet" is an acceptable answer
pandas
file = pd.read_csv('REN.csv')
if there is a different way to go about this or idea, please let me know
I'm just testing things.
Ok. I bet you could do something with groupby and agg and first and last. It would be similar to pandas.pydata.org/docs/user_guide/…
k
let me try it, thank you!
14:32
I would experiment with it myself, but somehow my pandas install has broken. It complains of AttributeError: module 'tokenize' has no attribute 'Name'.
as IT would say "did you restart your computer" HAHAHA
Ah, I had a name collision in my module search path. Now it's working.
import pandas as pd
df = pd.DataFrame({
    "ID":    [ 1, 1, 1, 1, 2, 2],
    "Assay": [ 3, 3, 4, 4, 5, 6],
    "From":  [ 7, 8, 9,10,11,12],
    "To":    [13,14,15,16,17,18]
})
result = df.groupby(["ID", "Assay"]).agg({"From":['first'], "To":['last']})
print(result)
#          From   To
#         first last
#ID Assay
#1  3         7   14
#   4         9   16
#2  5        11   17
#   6        12   18
Not sure how to get it into a less weird shape
@Jas_99 I'll take a look.
18 hours ago, by Andras Deak -- Слава Україні
@Jas_99 please stop linking that
I need to find all the nodes that needs to be completed before I can do start to work on a certain node. It's a directed acyclic graph what method does this?
user18313765
I posted the link 2 days ago in the chat. Then I published it yesterday, but rightfully I was canceled. It has now been 2 days since the last publication. I don't think it's spam after 2 days
14:48
@Pherdindy Regular recursion ought to work, since there's no risk of an infinite loop
@Pherdindy Invert the graph, then walk from the target node to all reachable leafs.
It's not clear to me that the graph needs to be inverted, since the problem description is vague. "a graph where each node points to the things it depends on" is just as valid as "a graph where each node points to the things that depend on it", IMO
Yeah I have to think of a way to do that manually I guess. graphlib doesn't really have much functionality like a method that gets a node as an argument then it returns the path from start to end
def iter_descendants(node):
    for child in node.children:
        yield child
        yield from iter_descendants(child)
I don't think I've seen graphlib before... Looks like it's useful for verifying that a directed graph is acyclic, but that's about it
15:07
Ohh right makes sense should be able to create a graph only containing all children
nodes that are related to the node of interest
Then that's the one i'm going to use in graphlib
Whoops, I just noticed my code has exponential slowdown if the graph has a lot of "diamond" connections. For example if A points to B and C, and both B and C point to D
Then D and all its descendants are iterated twice. And if any of those descendants have diamond connections, then they get iterated four times. And so forth.
I've seen a book using networkx but wanted to try this one first
I was hoping to find something that already has built in methods
But making graph algorithms is fun :-)
graphlib seems to be new like 2-3 years old
15:10
(sincerely)
Yeah although i'm gonna be putting lots of time and it'll end up becoming impractical lol
Just wanting to make a to-do-list
@Kevin it's new, only has toposort for now. It was added for the new parser or something like that.
def iter_descendants(node, seen=None):
    if seen is None:
        seen = set()
    for child in node.children:
        if child not in seen:
            yield child
            seen.add(child)
            yield from iter_descendants(child, seen)
Something like this would prevent exponential slowdown. And it won't loop forever if there's a cycle.
Or some other CPython feature. Unless I'm completely off the mark.
I'd usually expect to see a "new in Python X.yz" label somewhere on the page
There's a "New in version 3.9" but strictly speaking it doesn't tell us when the module came to exist, only the class inside the module. Occam's razor suggests they sprang into being at the same time, I suppose.
15:15
Feb 25, 2020 at 17:14, by MisterMiyagi
Is there anything similar to Py3.9 functools.TopologicalSorter available for earlier python versions?
I tend to believe that guy
wait, functools?
3.9 confirmed as sprang-into-being time: docs.python.org/3/whatsnew/3.9.html#graphlib
OK, so much for believing that guy
Never trust that guy!
In the movies, mister Miyagi was most wise, but his English was not perfectly fluent. Gotta give him some slack.
I never saw that movie without dubbing
15:18
If his Hungarian is perfectly fluent, then you're missing a facet of his character
it probably isn't, but I don't remember
Ok Kevin, got this to work and got it to spit out the ID on every row. Thank you! What if I have extra columns coming along for the ride. So if I have another column called Method and just want whatever the majority of the interval run is for the repeated assay. So if in your example I have another column:
"Method": [RC,RC,RC,Core,RC,Core].
Maybe there's an aggregate function for that. Wild guess that I have not checked: "mode"
>>> import struct
>>> data = b"\x01\x02\x03\x04"
>>> struct.unpack("<4s", data)
(b'\x01\x02\x03\x04',)
>>> struct.unpack(">4s", data)
(b'\x01\x02\x03\x04',)
Why does > or < not change the output? I would have expected one of them to be (b'\x04\x03\x02\x01',)
15:24
IIRC, ">" and "<" only change the outcome of numeric types larger than 1 byte. s doesn't fit the bill.
Why only for numeric types?
> and < say whether the last or first byte denotes the "largest digit".
Only numbers have digits...
docs.python.org/3/library/struct.html says "Alternatively, the first character of the format string can be used to indicate the byte order, size and alignment of the packed data, according to the following table"
No mention of digits or numeric types. Just says "byte order"
Never mind on the numeric part, I only imagined that. But s still doesn't fit the bill because it's 1 byte
15:27
Kevin's hint was that 4s yeah that ^
if there were a non-numeric character code with size larger than 1 byte it could be byte-reversed
But for s the number means the length. 4s is not ssss
Also from the docs: "For the 's' format character, the count is interpreted as the length of the bytes, not a repeat count like for the other format characters; for example, '10s' means a single 10-byte string, while '10c' means 10 characters."
It's still just an array of individal char, though.
I do find it curious that s has no value specified in the "Standard size" column in the docs. I feel enough doubt to reduce my confidence to 97%.
Hmm okay, that makes sense
@MisterMiyagi getting ontological, are we?
15:30
So which format character should I use if I want 4 bytes?
Instead of an array of 4x 1 byte
@rattlesnake wouldn't that depend on the semantics, and how the bytes were packed?
if you want to reverse the bytes in a bytes, do that
data[::-1]
@AndrasDeak--СлаваУкраїні struct did it first!
My mental model is that c and s work identically under the hood, and it's only after the initial unpacking is completed that it looks for s values and converts them from arrays into strings
@rattlesnake 4c should do.
@AndrasDeak--СлаваУкраїні Can't do that. data is longer than in the example above. I just reduced it to a minimal example.
15:32
@rattlesnake so you want to reverse each 4-byte subsequence and put them back together into a bytestring? Or what?
Unpacking as uint32 with one endianness and packing as the other (or something like that) would practically do that, right? Intentional numerical mojibake.
@MisterMiyagi No, that would turn it into 4 individual items
Okay, give me a second, I'll make a better example :)
Hello everyone. I have used the following command to install discord. But it doesn't work. I am getting the error no module named discord found. But when I run it in the terminal python, it works. Not in vscode.
py -3 -m pip install -U discord.py
>>> data
b'\x01\x02\x03\x04\x05\x06\x07\x08'

>>> struct.pack('>2i', *struct.unpack('<2i', data))
b'\x04\x03\x02\x01\x08\x07\x06\x05'
@DeepakVerma why not pip3 install -U discord.py ?
tried that as well. Also pip install -U discord.py
pip install discord
nothing seems to be working
15:35
Also what do you mean by "it works" , that's pretty vague
when I am using import python, it doesn't give any errors
but it's only in terminal, with python command running
If it's the official discord repo, shouldn't the install script be named as "setup.py"
@DeepakVerma that is still vague, show the traceback or the error message
Traceback (most recent call last):
  File "c:/Users/dpkvr/Desktop/discort-bot/script.py", line 10, in <module>
    import discord
ModuleNotFoundError: No module named 'discord'
15:37
:54476050 Found it, thanks Kevin.  ended up doing this
lambda x:stats.mode(x)[0]
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import discord
>>>
this one doesn't give any errors
Looks like a local/global environment error to me. mostly a typical PATH variables issue that isn't importing environments/PATH variables in VSCode .
Structure of 4 bytes: a = 3 bytes, b = 1 byte. Data is saved as little-endian. Example data = b"zyx1". So I want a, b = struct.unpack(..., data) = b"xyz", b"1". What is the format string that I should use?
@AshwinPhadke But in that case, shouldn't I be getting ModuleNotFoundError with all other libraries. All the modules, pandas, scrapy, pymysql, tweepy, ....... etc etc, I am able to work with all, just not discord
@rattlesnake 3ss should work.
that's a 3-byte plus a 1-byte sequence.
15:43
(env) C:\Users\dpkvr\Desktop\discort-bot>python
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import discord
>>>
this one's working from env
@MisterMiyagi No, it ignores the endianess
>>> struct.unpack("3ss", b"zyx1")
(b'zyx', b'1')
there is no endianness.
unpack it, then invert whichever you want inverted.
@DeepakVerma well, there you go.
I don't think there's any way to get b"xyz" out of that data just by using unpack. In my experience it's pretty atypical for string-like data to get saved so that its final character appears first in the file.
Saving "Hello World!" typically does not write "!" first, for example
If you don't want to reverse the output manually because it would increase memory usage, consider the reversed iterator, which is quite memory efficient.
@MisterMiyagi Obviously I could do that... but that doesn't make sense to me. Why shouldn't I be able to < or > for a string, too? :(
15:47
Because strings don't have endianness.
At least not the standard, garden gnome variety C strings that struct understands.
@MisterMiyagi "In computing, endianness is the order or sequence of bytes of a word of digital data in computer memory." 🤷‍♂️
Are you aware what they mean by word in that quote?
@MisterMiyagi Does word imply a numeric value? I thought word is just for any data unit.
The important part is that "word" refers to a machine word, not the things in a sentence.
15:51
Yes, I'm aware of that. ;)
Anyway, the "digital data" you are working with here are chars.
Each is one byte so, there is no sequence of bytes in a char.
Do you have access to the source code that is saving this string data? Maybe you can look at its string serialization algorithm, and write it in reverse to get a deserializer.
Yes, so it comes down to the "array of chars" interpretation. And with < and > I set the endianess of a single char, so they stay unchanged
For all we know, it's using a whizbang third party module that has an interesting idea of what "endianness" is, but otherwise works fabulously. In that case it might be sensible for you to use that module too.
No, I only have the data.
15:55
Or possibly the code has a comment like #ugh, why doesn't struct.pack let me write little-endian strings? I have to reverse it myself, in which case you shouldn't feel too bad about also reversing it
I think I'm okay with this explanation above. But just so I get it right: with struct.unpack there's just no way to read a word of n bytes length.
Maybe I'm just going to turn this into a question...
And post it
Okay, another issue: so if I'm switching to short and long (because those words have a length > 1), are those guaranteed to stay 2 and 4 bytes and are not dependent on the platform?
They're guaranteed to be 2 and 4, unless you explicitly request native sizing using the @ flag, or specify no flags
TBH I don't think "read a word of n bytes" and struct.unpack belong together. struct is for C style data representation; n-byte words are either a fixed increment stream or some wholly custom format.
cbge
@rattlesnake maybe you look for pythonhosted.org/protlib
16:13
Yeah, it seems like I'd have to create a custom CType. I think that's exactly what @MisterMiyagi said as well. A 3 byte word (or whatever length word longer than 1) is just not part of C by default.
Thanks for the help everyone. :) I learned a lot
Wes
Wes
i require advice about multiprocessing because i am clueless. i tried using multiprocessing.Queue but ended up being super slow
basically i have a child process (if that's the right name) that should continuously update some array. this array must be pulled by the parent process at its own slower rate
in other words the parent process must not slow down the child process
the child process will perform several changes to the array, before the parent process tries to read said array
any idea which managers i should be using?
16:36
"the child process will perform several changes to the array, before the parent process tries to read said array" this doesn't sound like a job for a queue, then.
Can you share it directly?
Using a synchronized Array might reduce congestion considerably.
If you are fine with having partial updates, having the array in unprotected shared memory would be the fastest.
Multithreading could be a solution here, I guess.
Wes
Wes
@MisterMiyagi i was passing the entries one by one with a queue and updating the array on the main process
that was a bad idea clearly :B
when i perform a change to multiprocessing.Array from the child process, will the parent process try to read the change immediately?
in other words, is the message... whatever it's called... sent out to the parent process even if the parent process didn't ask it?
Basically i have a child process that must run as fast as possible, and the parent process is a GUI that displays telemetry from the child process. Of course it can't render stuff as fast as the child process is doing its stuff. Instead, it should ask the telemetry like once every 500ms or even 1s and do its thing without slowing down the child process
lemme have a look at the sync array. thanks in advance tho for any further input
16:53
is there a nice built-in way to format datetime.time(hour=15) as '3 PM' and datetime.time(hour=15, minute=37) as '3:37 PM' (meaning that if minute is 0 than the :00 part should not be added)?
Nah, I doubt you can do that without an if statement
Good, thank you
Disclaimer: I arrived at that conclusion without even opening the datetime docs
Feel free to retract your thanks, I won't hold it against you
That "thank you" was for validating my thoughts. You could be wrong, but I would still be thankful
I can't figure out how to sort by a multi-index that uses lambda
df_multi.sort_values(by=[('Depth', 'mean'), ('Depth', 'sum')], ascending=False)
this is the example I found but it doesn't show me what to do if I used lambda
.sort_values(by=[('Level 1', 'Level 2')], ascending=False)
generic
but I don't know what to put cause the column spits out <lambda>
and that doesn't work
17:10
Ji
Hi*
17:27
nvm, figured it out. I didn't need to sort by the lambda indexes. Just the other ones.
g1.sort_values(by=[('HOLEID',''),('SAMPFROM','first')])
17:44
now how to export a multi-index header but I only want my original header
I could remove the header, and export out two times into csv
Got it, just used this:
g1=g1.droplevel(level=1, axis=1)
then exported to csv
18:16
@Kevin I don't think it worked quite right. in a run where the data has [0.001,0.001,0.002,0.002,0.001,0,0,0.001] it lumped it all together.
@Wes shared memory doesn't work via message passing, so you don't have to worry about stalling or backlog with too slow/fast messaging. Both processes will operate literally on the same memory and directly read/write it. There is merely a lock to avoid interleaving read/write giving inconsistent data;for some use cases, you won't even need that.
 
1 hour later…
19:25
oh I ssee what it is doing. It's looking for all occurrences of the same value and then putting the first and last. This is spatial data though. I want it to start going down and when it sees the repeats in order, lump them into one interval.
Wes
Wes
what happens if e.g. i add an element to the shared array from the child process, while the parent process is iterating over the array?
shared memory seems dodgy
19:44
@CelesteWilson please don't ask here and on the main site in parallel
Sorry, I guess that's what desperation does. Won't happen again. Thank you.
no worries
but especially with pandas you're bound to get an answer very soon if you ask on the main site
20:32
with done, pending = asyncio.wait(futs) is it possible to get any pending tasks? or is there pending tasks only if this returned sooner (e.g. FIRST_COMPLETED, FIRST_EXCEPTION) or if there's a timeout?
The latter AFAIK
20:57
i guess it makes sense to still log or something in case the unexpected happens, but yeah I wish the docs actually mentioned this
21:26
Good day hooligans. Please can someone who has experience with machine learning and classifiers point me to the direction to obtain help with a simple iris dataset that I am doing on Jupiter notebook? Or if I can ask here.
@FrancescaC you can ask here if your question is not what you asked on the main site 5 minutes ago
No actually, it is basically that I have created this convoluted notebook on this iris dataset, and I feel that it could be simplified. I was wondering if there is anyone who would like to look at the notebook & tell me what I have done that is really redundant or overly done...also I am having difficulty creating a way to get a bunch of accuracies of multiple classifiers... its quite trivial
Sure, you can try posting here. If anyone has expertise and feels like checking it out, they can get back to you.
Okay!
It doesn't seem to allow file upload actually, only image. argh
21:41
indeed
google colab might work, and I think github renders ipynb files to some extent
import matplotlib.pyplot as plt
from matplotlib import pyplot
I'm not a ML person but at least that is clearly redundant
with so many imports make sure you actually use them
You've got plt, pyplot, sns and a bunch of plotly imports are you sure you need all of them? Pandas seems to be imported twice too.
import plotly.offline as py
from plotly.offline import init_notebook_mode, iplot
# but then later you use py.iplot instead of iplot
mm yes, I still need to sift through my redundant imports okay thank you for that!
Why should stratify = y in dataset splits?
would this be accurate? it is not so clear on the internet. "We stratify by the y variable which is a binary array of 0s and 1s, this means that whatever the percent turns out to be, it will ensure that the split occurs with the same percentage of the current ratio beween the binary values. it works best for highly unbalanced datasets or small datasets"
22:21
Thanks everyone for all the help today, got what I needed. We'll see what tomorrow's problems are hahahahahhaha

« first day (4216 days earlier)      last day (955 days later) »