« first day (4428 days earlier)      last day (741 days later) » 

01:08
hey python peeps
If anyone could help me with a simple file content deleting issue on my page that would be great
I want to know how to delete some lines in a .txt file with my script
01:47
Usually it's easier to write a new file with only the lines you want, then rename the new file to the old name
 
1 hour later…
02:55
Examples on how to take the cube root in python all basically show the same thing
x ** (1./3.)
Are the dots in 1. and 3. just to make sure that they are cast to floats?
I'm assuming that's just legacy from when dividing ints would result in an int? Or is there any other underlaying reason that would still be applicable for modern python versions?
 
1 hour later…
04:01
@MitchellvanZuylen Right. The dots aren't needed in Py 3, or in Py 2 with the __future__ division import. But some people like to be explicit that it's a float division, especially if they're coming from C. Of course, the division is performed at compile time, not run time, so however you write it has virtually no impact on performance.
04:12
Thanks for the insight!
 
4 hours later…
07:53
"You've got to be kidding me" of the day:
>>> datetime.datetime.now().strftime("%Y年%m月%d日")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'locale' codec can't encode character '\u5e74' in position 2: encoding error
had a similar error too once, although I wasn't using the datetime module
08:32
... yeah that's pretty egregious
though the 'locale' codec is rather sus
 
3 hours later…
11:14
@Aran-Fey Works fine for me. What happens if you try just '\u5e74' at the prompt? smells like a terminal config issue
(and we should have a canonical that covers such configuration)
It's not the prompt that's the problem, it's strftime
>>> '\u5e74'
'年'
11:30
that seems worth a question, although it might be a python bug tracker question.
@Aran-Fey Same exact input as the post: stackoverflow.com/q/16034060/12349101 :P
Yeah, with my own format string the problem is not quite as obvious...
>>> _ = datetime.datetime.now().strftime('%Y-%m-%d %H꞉%M')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'locale' codec can't encode character '\ua789' in position 11: encoding error
did you try the answers there? one of them might work
@KarlKnechtel From what I've gathered, it seems to be a known issue... in Windows
Oh yeah, easy workaround, just annoying
 
1 hour later…
12:51
Hello, I asked this question in another room and it was recommended to me to ask here. I have the feeling this question was reopened for incorrect reasons (see comments to the question), I'd like to have some feedback. Here is the link: stackoverflow.com/questions/74625434/get-top-3-max-values
jezrael's objection seems weirdly pedantic to me, but the proposed target doesn't seem super high quality either.
I don't think I know enough Pandas to judge
@KarlKnechtel I think this target is really the right duplicate: stackoverflow.com/questions/20069009/…
I just started to add more than one because jezrael keeps reopening questions with weird reasons… I actually had closed before they answered
Perhaps you could compile a list of jezrael's weird reopens and put them in the "flag for moderator attention" box
I think I'll start to keep track (I hadn't), the issue is that it's not always easy to notice as I don't always come back to older questions
13:03
Reasonable
@Karl so you think it's better left reopened?
I'm leaning towards "leave open" on this one. It's been reopened once, there isn't a super high quality dupe target, and it has an accepted answer. By themselves, none of these factors would strongly convince me of the question's merit. However, taken all together, they tip the scales in favor of "leave open"
Devil's advocate: if you did get it closed again, I expect jezrael wouldn't start a big argument over it, since he already got his points.
13:26
@Kevin the issue (to me) is not much about this particular answer, more about the repeated pattern, I'll keep track now I guess
14:05
I got this: gist.github.com/secemp9/3999e73f5537739d140f6a05acee9179 I want to basically merge functions called inside a specified function inside of itself (so I can send it to a separate python process, and don't need to manually redefine them, etc)
any way to either improve this or add indentation support? For the later, I got some ways to do it, but it might trip over when there mixed tabs/space in functions...
this is to be used with subprocess btw, otherwise I wouldn't be asking that if this was for multiprocess :)
inspect.getsourcelines(eval(j)) makes my eyes bleed
ah yes, I know using eval is a bad idea, but using ast.literal_eval doesn't work in this specific case...
I'm sure there a better way, so I'm just testing things out here
I'm also aware this can horribly break since if the content of one of the functions contains something that has the same name as one of the local functions, it will incorrectly detect it. But again, this is just me playing around
What's the end goal there? What are you going to do with that string once you have it?
Because this looks like an XY problem to me
@Aran-Fey well yes, I did say this is to then send to a subprocess that execute a python script, namely, because of another thing I'm doing that I talked about here (LibreOffice/OpenOffice thing I mentioned last time)
I just thought of doing this, so I don't need to manually redefine everything when I send a function to it
But why merge the source code for all the functions? Why not send the source code for the whole file?
You have already written working code, why do you need to change it?
14:19
@Aran-Fey because the other python executable is a different version, and I don't necessarily want to do if 2.X and what not everywhere :)
I don't know if there a better way than what I did above anyway, so that's why I'm asking too
Why do the functions have to be nested inside of each other though?
It's probably better to just do if 2.X: everywhere, rather than do fancy code rewriting logic
because I'm using this on the output before sending it to a py2 subprocess: stackoverflow.com/a/56468576/12349101 which is used on this (experimental abomination ahead): gist.github.com/secemp9/9654bf1d6ff91ad623fa3246525ed918
It may be possible to consolidate much of the version-specific logic into one location, so it's not sprinkled all around the code
@Kevin it's mostly so I can quickly test features of both API (since they're slightly different as you know, because of their different python version, among other reasons).
btw, the above works provided you have installed openoffice/libreoffice with the optional requirement...I'm getting around to make a python/powershell command to install it that way, but it's taking longer than I liked
14:26
for example, rather than doing x = input() if using_python_3 else raw_input() in a dozen different places, you could do input = input if using_python_3 else raw_input once at the top of the file, and just use input everywhere else
I know this kind of thing is very common in JS. They call them "polyfills". Useful for supporting older browsers without losing access to cool shiny features.
So the goal is to pickle a function? In that case dill might help
If the APIs are slightly different, you'd probably need a polyfill-like approach whether they use the same Python version or not
for example here it use this but I want to also send the whole function inside, which is this.
@Aran-Fey yes, but I don't want to support third-party stuff, at least for now. It's already a pain with two slightly different API on different python versions, might be even more when dealing with dill. And I recall I stumbled on some bugs too, although I guess I could report it, it would take too long to fix (if it's even fixed upstream that is)
@Kevin hmm, but I don't know how this would look like. Do you have any example, maybe with a small snippet?
@Aran-Fey lol daily psa
I'm still glad I got this far with this implementation though. It's horrible but it took me more than a week to make (probably would take less time for most here, but that's beside the point)
it also work as it is, except for the sort_selection_human function (since it call the other function as I mentioned above, which need to be redefined on the other side)
14:39
I sometimes feel like you'd have easier time just porting the API to newer Python version
I wouldn't be so sure. There is a lot of Java stuff springled inside, so I can already feel the incoming hurdle from here.
For some people here, it might be easier to do that, but for me, it will be like scalping a montain
This looks like a really bad idea to me. Sending a function from one process to another is ok as long as the function is self-contained, but automatically including all of its dependencies is bound to go wrong. What do you do if function A depends on function B from class C in module D?
hmm, I do know this can break horribly, but this is just for small function anyway, this isn't a full package/made for everyone use yet
Have you checked the pypi.org/project/unotools for Python 3.3+? I'm nearly certain there has to be an easier way
Or any of the 2 alternatives listed
@matszwecja seems like this mostly list Linux and MacOS...I wanted to support Windows too, but I guess I'll try this and see if I can make it work with Windows. Otherwise it's back to the horrible implementation I made
@Aran-Fey also, FYI in case I forget: github.com/uqfoundation/dill/issues/254 doesn't work across different python version (even if you use the same pickle protocol, since the code will be different, etc)
so that's also why I don't use dill
14:58
I never said this was fine, I did say this was horrible, but temporary.
if it's for testing stuff fast, it's not that bad, especially if I'm the only one who use this (at least in it's current state)
Something about your project and all the requirements emanates this vibe @NordineLotfi My condolence (:
I don't know what you mean by that. Anyway, if you don't believe me, it's fine.
Nono I believe, keep doing you I'm sure you are learning a lot trying to do what you are trying to do :D
@matszwecja I discovered some 4-6 year old bugs in the last few months, yeah it's never temporary :D
15:02
@Hakaishin I never said I was doing this solely for learning, but I guess it's fine if you see it that way
Anyway, I am working with an "API" that doesn't have any way of checking the ~3 day duration task's progress, and I'm afraid I'll have to do that by reading the partially incremented result file. What approach do you suggest for checking the file for updates and streaming new data?
What's the file format?
We have a CSV file at work with 22 mil rows that was given to us in cp1252 encoding. We convert this to utf-8 using some command line tool (I think it's through AWS console) before trying to either a) read into pandas or b) do a direct upload into Redshift. Of all those rows, there's a single row that has one too many columns when it gets parsed back in, so both readers then fail. Encoding is not my game but is there a way to break this following example or am I off-course?
import codecs
import pandas as pd

df = pd.DataFrame({'a': ['p', 'q'], 'b': ['r', 'something_here']})
df.to_csv('test.csv', encoding='cp1252', sep='|', index=False)

with codecs.open('test.csv', 'r', 'cp1252') as infile:
    with codecs.open('test_out.csv', 'w', 'utf-8') as outfile:
        while True:
            contents = infile.read(1048576)
            if not contents:
                break
            outfile.write(contents)

df = pd.read_csv('test_out.csv', delimiter='|')
print(df.head())
Such that, when you try to parse it back in, 'something_here' has been lost in translation and it now thinks that there's a "ghost" column 'c' because something wasn't escaped properly and it ends up having an extra pipe delimiter when parsed back in as utf-8
@Aran-Fey log-like
@roganjosh Whoa, that chunk size is 1000 times larger than what I usually use O.o
15:11
Shamelessly cargo-culted. That isn't how we do the conversion, I just wanted something quick that was presumably analogous
That explains the use of codecs
@roganjosh I thought pd.read_csv takes an encoding parameter have you tried just using that?
It does. I think the conversion is done outside of our control. Other files we can just straight-up read in cp1252 directly and it's fine. But this file is part of a pipeline that's abstracted from us and just dumps it into S3. It's when we try to parse the thing in S3 that it blows up
I'm just curious about whether, buy some horrible coincidence, you could break what I understood to be the CSV protocol and get it to read a certain substring and somehow throw a delimiter in the middle of it
Because, in theory, there shouldn't be an upstream process that exports from a DB (on the customer side) and throws one row with an extra column in it... for one lonely row. So where does it come from when we try to read the utf-8 converted version? I guess I'm sorta setting it as a challenge for anyone interested in encodings
@matszwecja If your program is actively running throughout this whole process, then all you need is a for line in file: print(line), I think?
@Aran-Fey won't this just read what is currently there and finish?
15:17
Not entirely sure, but I don't think so. Might depend on your OS?
If you need to find out where you stopped reading, there's file.tell()
file.tell() only work if you use "rb", not "r" or "r+" AFAIK
Depends on your definition of "work" (:
@roganjosh I don't suppose you know the precise contents of the row that's failing?
@Aran-Fey so you're telling me this works?
I tried it once, that's why I'm asking
It returns a number
15:22
If it contains sensitive info, we can negotiate an NDA :-P
I know it returns a number, I used it before
If that's your definition of "working", then yes it works
but that only does if you use "rb", not "r" right?
Not as far as I know
15:23
@Kevin I'm trying to get my colleague to isolate the exact row verbatim but they're also in a meeting. In the meantime, I only have what redshift reports as an error (if that gives any clues, but obviously it's not the raw CSV format)
@roganjosh Are you sure the file wasn't already broken before re-encoding? Since cp1252 is a single-byte encoding, it seems quite impossible to mess this up
Line is too long for chat as it is - dpaste
I wonder if those 793 whitespace characters at the end have something to do with it
That's why I don't trust that. It's what comes out of the redshift stv_errors table and it throws its own formatting in. I think we'd only be interested in the combinations of non-whitespace chars
But, we're looking at something post-potentially-impossible-issue because it'd already now be a | in this representation :/
The trailing spaces makes the whole string 1024 characters long. An exact power of 2, hmmmm
File under "harmless quirk?"
15:28
@Aran-Fey I'm not. I just set out the circumstances of why I think it wouldn't be given that it was done in a single DB pull on their side
Maybe stv_errors defined that column as 1024 bytes wide, and always adds whitespace if the string is shorter. In which case I don't expect it to be a direct cause of the problem
@Kevin it's not even a valid CSV format. It's missing quotation marks. I just gave the closest representation I have atm. I can give you the raw line in the file shortly
Does the command line tool you use for re-encoding take input/output via stdin/stdout or via files?
@Aran-Fey see comments on here: gist.github.com/secemp9/53dada081e5b5279d0b19d198a8ab88d tested on Win10
Well, define "valid CSV format". If I understand correctly, there is no single global standard.
15:31
I used it a lot a couple months back, so that's why I said that earlier, but I guess you managed to give me enough self-doubt that I tried again
I will accept "Whatever format pandas is using" as our definition, or whatever else you like
@NordineLotfi Oh, well that sucks
yeah, it wasn't like that on 2.X, but there was problem then too because "rb" wasn't used, so they removed it (probably not the main reason since I didn't look it up, but it's how I see it at least)
there some "hack" or weird way to get around it, at least on SO, but those are probably slower than just using "rb"
I half-expected the error output to have some mojibake near a pipe. That would have indicated that a multi-byte character was incorrectly split into 2+ single byte characters, one of which is 0x7C aka "|". But there's no mojibake here, so I guess that's not the problem.
Mojibake is the word I was looking for, not "akin to a hash collision". I'll get back to you if I can get the very original file
15:35
I wonder if some kind of unicode normalization could be happening... Some exotic unicode character which looks very much like a pipe, getting converted to an actual pipe
I did not know this was a thing, especially for those cases that I noticed too (the word Mojibake that is)
@Aran-Fey I don't think a for loop helps me in any way. I'm looking for a way to watch the file for a new data and trigger something based on that event.
Suppose the original file was foo!bar|baz, where "!" is some unicode char that looks like a pipe, but is not treated like a pipe by a typical CSV parser. The parser would consider "foo!bar" to be the first entry, and "baz" to be the second. But if the pipe-like character got turned into a pipe somehow, then the entries would be "foo" and "bar" and "baz"
@matszwecja is the file where that content is getting incremented in size or? is it like a DB where there entry that get removed/replaced, or is it just appending new data on it?
if it's always appending, you could just get the total amount of bytes and check against that
if it's line-based, you could check the number of lines
I see some pipe-ish characters here.
15:39
if you don't care about third-party module, you can use a watchdog or other polling implementation for file modification
watchdog does seem like something I was looking for
@Kevin well, this in itself was an eye-opener for me. I thought there was a constraint that strings had to be enclosed in "" such that you could differentiate between delimiters and characters in text, but looking at the raw file of my example that doesn't seem to hold; somehow a choice was made to remove them
Yeah, I usually only expect to see quote marks if the string specifically contains a delimiter character or some other kind of special character. If the string does not have any tricky contents, then the string may not be enclosed in quote marks.
As in my foo!bar|baz example
15:48
Mmm, I've just spotted that
import csv
import pandas as pd

data = [["a,", "b"], ['c', 'd']]

with open('csv_mod.csv', 'w') as outfile:
    writer = csv.writer(outfile)
    writer.writerows(data)

df = pd.DataFrame(data)
df.to_csv('pandas_csv.csv', index=False)
So perhaps the problem isn't with encoding at all. Maybe whoever is creating the original file is writing all of his strings without quote marks, and it has worked fine up until now because no string has ever contained a delimiter character.
They're both (basically) consistent but my thoughts that you could get a mojibake scenario has gone up considerably with this knowledge
@matszwecja Hmm, yeah, files don't seem to work that way when the reader and the writer are two different processes. I've experimented with os.stat() and file.tell(), but somehow none of that works until the writer closes the file
yeah, because file.tell only give the position for the current open() call (I think).
I know that on Linux, you can open the same file with a different process and see that file.tell does indeed do that. Didn't test yet on Win10 though
in before you make me self-doubt and test that again :P
Pretty sure that's gonna be the same as tell just tells you a position of the cursor
15:56
@Kevin this is the row in the original cp1252 CSV (before we do a conversion to utf-8). This is the line immediately before. I should be able to investigate this one myself now; wasn't sure whether we would get the un-altered raw data
@matszwecja maybe, but I was thinking more on a permissions/error level here. I know you can't rename files/move them when they're open on Windows, but you can do that (in most cases) on Linux. On Windows, you get errors (which some people use internally for their libraries)
again, I didn't test what I said above, so I'm just speculating.
@roganjosh you could use (as Aran mentioned earlier) the position when the error occurs, but you need to use "rb" instead of "r" as I mentioned.
then you can just take a chunk based on that position and see what's hiding there using an hex viewer or something
1.00|EA|.0 vs 1.00|EA||.0
I'm more suspicious of the ".000000" element... It stands out when I try to make the rows line up dpaste.com/CY99ACVA3
Or, hmm. There are multiple ways to line things up and the problem could be anywhere between "EA" and ".000000"
16:12
mmmm. Maybe this issue isn't as exciting as I first thought. It looks like the raw data is corrupted for a single row
Someone fat-fingered and put "EA|" into the data submission form, didn't they
feel free to use an hex viewer in case an invisible byte is hiding there (because of the encoding, etc)
Must be a considerably fat finger, since "A" and "|" are about 12 inches apart on a US keyboard
and "|" requires shift to be pressed down
maybe they switched the keyboard layout by accident. I know it's possible since I do it all the time on Win10 (with the <> key I think? not sure if those are the culprit yet)
or maybe their pet walked all over the keyboard
16:43
I just remembered we talked about mixed tab/space here before. I guess that just gave me an idea :D
17:22
this works, at least for one function inside another (only). Better than nothing I guess: gist.github.com/secemp9/3999e73f5537739d140f6a05acee9179
 
1 hour later…
18:49
@Kevin looks to me more like either a specified data format or an input form's design changed silently, such that the .0 was put into a different column of |-delimited data
19:02
mod closed it without understanding the question
 
2 hours later…
21:08
I don't follow. Mod appears to have attempted to clarify the problem, and then closed it... as a duplicate, of what you personally suggested?
(Or perhaps all of that is resolution that happened after you posted here?)
at any rate, as the question currently stands, the duplication seems clear to me.
21:39
@Aran-Fey mine for yesterday was "sys.platform != 'linux' on linux on pythoff"
Wait, what? What was it then? O.o
and this was the high road, apparently, because the other option was "'linux2' or 'linux3'"
Could also do .startswith('linux')
fortunately we only have a handful of modules that must be 2-3 compatible, but every time I touch them I feel like I'm trying to cross a minefield with a pogo stick
@Aran-Fey yeah, that's the workaround, once you know that it's crazy
Looks like Windows is always win32. I always write .startswith('win') just in case
21:48
@0x263A Christine McVie née Perfect, singer / songwriter / keyboard player from Fleetwood Mac, has passed away. :( The first song I heard from her was a cover of Etta James's I'd Rather Go Blind.
@AndrasDeak--СлаваУкраїні I noticed that too once. I don't yet know the real reason they're doing that, but it's weird
I see. I meant for pythoff though
(or do you that's it's mostly enforced by package under pythoff instead?)
I that it's in the distant past
got you
I see what you did there btw (I forgot to use "mean")
21:57
I have no idea what you're talking about :P
:D
also, since you're working with pythoff, do you have an advice for someone who would want to send something to a subprocess, running pythoff for a weird API? I mean that for this mostly: gist.github.com/secemp9/9654bf1d6ff91ad623fa3246525ed918
I'm using queue + pickle and a while loop, but I feel like there might be a better way
I forget, why don't you use stdin/stdout again?
my recommendation is to do something better, such as tazing your nethers
Fortunately I don't have to get shoulder-deep into the muck, it's enough if I don't break what's already working
can we just port libreoffice to python 3.11? :P
Yes, but it will take 15 years :P
Aren't there also stuff like sockets for interprocess communication? Assuming stdin/-out is a no-go.
I haven't really read the linked gist, mind you
22:08
@Aran-Fey hmm, I honestly don't know. I thought at some point that it might be weird to use stdin/stdout (if you look at the comment on here you'll see what I mean). I guess I'll make another version using just stdin/stdout to make extra sure It's actually not working (or working, depending on what happens, because I didn't take notes :/)
@AndrasDeak--СлаваУкраїні haha, yeah, working with both pythoff and latest python is not fun...
@AndrasDeak--СлаваУкраїні libreoffice is already at 3.8 IIRC, it's just for OpenOffice that I have to do that :/ mostly because of internal struggle and people leaving the organization, so they don't always update or plan on ever updating it to 3.X
@AndrasDeak--СлаваУкраїні I planned on making a socket version, but right trying to make the first abomination version of this work first
@AndrasDeak--СлаваУкраїні :/ alright
@NordineLotfi And can you just let go of ooffice?
And whose organization?
@AndrasDeak--СлаваУкраїні I could, but I think I got an obsession with making it work with both now. I initially wanted to do it for just LibreOffice, but after it took me like 4 days, I thought "why not support OO too, since some people are stuck on there for automation", or something like that
@AndrasDeak--СлаваУкраїні sorry, meant Apache OpenOffice (the name of the organization). This is only what I read on their forum, at least for the reasons as to why it's not updated
it might not even be the real reason, but yeah
@NordineLotfi ah
@NordineLotfi OK. I wasn't sure if you meant your organization, in which case I wouldn't have understood the situation.
Yeah, I mean you know I usually work on weird stuff anyway. Would have mentioned if this was for a job :P
I don't really follow that, sorry
22:16
alright
22:27
@Aran-Fey btw, I think I understood in part what you meant earlier. I don't need to indent the functions I need to grab as dependencies. I can grab them and keep their indentation and it'll work fine.
Yeah. Although that presents new challenges, like knowing which one of the 5 functions you wanted to call
yep, indeed
Huh, apparently mypy doesn't support Concatenate yet? Wild
22:45
yeah, it's weird indeed. It seems like they set it for this issue but didn't do it (yet). They made a new one here. related comment: github.com/python/mypy/issues/8645#issuecomment-742744754
23:41
@mozway Various people have complained on MSO about Jezrael's dubious dupe-hammer usage for years. Eg, meta.stackoverflow.com/q/359059/4014959 He even posted a (self-deleted) answer there claiming he won't do it again.

« first day (4428 days earlier)      last day (741 days later) »