« first day (4484 days earlier)      last day (469 days later) » 

8:34 AM
Something I'm wondering about now: how does round actually work internally? I mean, how does it know how to pick a value, such that the string-formatted result will have the right number of decimal digits? e.g.
round(0.1 + 0.2, 1) will report 0.3, differently from the raw addition (the classic float imprecision example).
but it seems like it must be fairly complex, internally, to subtract just enough from the 0.1+0.2 result to make it display as 0.3 when shown as a string.
 
I was wondering about that too once. I remember trying to get a specific number of precision (eg: 1 or 2 for a float) but didn't like the result of round, so ended up just converting it to string and back
 
@KarlKnechtel hmm, I read something about that on my quest to understand floats
maybe I can find it
yeah, here. Can't link the exact section but it's pretty close to the top: docs.python.org/3/tutorial/floatingpoint.html
> Starting with Python 3.1, Python (on most systems) is now able to choose the shortest of these and simply display 0.1.
 
the naive implementation I guess is to subtract epsilons until it finds something that works.
I can't think of a more efficient way than that, nor can I think of a more efficient "does it work" test than actually string-formatting the number. but.
having round as a built-in seems like a design mistake, though. it encourages wrong thinking simply by being there
 
8:53 AM
what I found to be decent answers: stackoverflow.com/a/15398691/12349101 for rounding and for truncating: stackoverflow.com/a/783927/12349101
technically truncating (without rounding) it can be done in a single line using math.floor or math.trunc, but at least the answer above does go a bit more in depth I think
using math.ceil for rounding works too I guess: stackoverflow.com/a/60294994/12349101
still didn't find an in-depth explanation for why round on python is doing this though.
 
are (0.2 + 0.1) and 0.3 actually the same numbers when it comes to the binary representation? If that's the case, what additional data Python stores with floats to tell them apart?
 
they are not, which is the motivating case for the canonical - stackoverflow.com/questions/588004/…
@Arne looks like there's quite a bit of code "underneath" that. browsing large C projects without an IDE is painful.
 
@KarlKnechtel couldn't have said it better
 
 
4 hours later…
1:29 PM
"I'm sure the math for round() can't be all that difficult."

ds = (dval(&d2)-1.5)*0.289529654602168 + 0.1760912590558 +
i*0.301029995663981;

"... Never mind"
 
 
1 hour later…
2:54 PM
is that in the C code somewhere? I hope those numeric constants are explained in the comments
 
3:06 PM
TLDR: ds is the number of digits to the left of the decimal point in the value d2. The expression is a fast and slightly imprecise approximation of log10(d2). The numbers are specially chosen so the approximation is never an underestimate.
I reckon that much of the hard math in this function is for the purpose of 1) calculating the size of the resulting char array; and/or 2) taking simple math and making it 1% faster by making it 1000% scarier looking
 
Hi, wondering about design ideas for loading data from csv files to database.

In current implementation, I iterate by file, split each file in number_of_lines / batch_size files and load the batch.
 
Consider using the csv module.
 
I would like to play a bit with multiprocessing (not sure if even possible) but it feels there shall be some sort of mechanism which starts reading a new file once the remaning number of batches from current file is below certain limit (number of cores?).
Does this instinctively make sense to you?
 
3:21 PM
unrelated, but I might end up going after a C# adventure for the first time. Does anyone have any recommendations/tips for someone from a python background?
 
you're not likely to get any advantage to multiprocessing as the bottleneck is going to be the db storing transactions/table locking
@aeiou what DB is it?
 
Oracle
 
is the CSV file available to the DB?
 
Yes
 
then use Oracle's bulk load facility and let it sort it out
 
3:24 PM
@NordineLotfi Take care with most assumptions that things work like they do in Python
 
@JonClements You think the standard functionality will be faster than anything written in python?
With python, it is easier to schedule things and do some transformations etc.
I have used both and the speed looks about same..
@matszwecja Could you elaborate? Just curious
 
@aeiou what kind of transformations are you doing?
 
@NordineLotfi C# has equivalents of f-strings and list comprehensions. learn.microsoft.com/en-us/dotnet/csharp/tutorials/… learn.microsoft.com/en-us/dotnet/csharp/tutorials/…
 
@JonClements At the moment none. Just creating sort of a general python based loader. When csv file is loaded in dataframe, it is trivial to adjust data (presumably easier than in sql loading scripts).
 
so you're using pandas?
 
3:33 PM
@aeiou Well, not that I'm very experienced with C#, but from what I can tell it's much more rigid than Python. Static typing, namespaces are more explicit, you've got keywords like private, static etc.
 
For csv read / modification yes
 
@aeiou so can you not use DataFrame.to_sql with a suitable sqlalchemy connection object?
 
For someone with only Python experience having to be explicit about this kind of things might be a bit of a culture shock
 
@JonClements Inserting with cx_Oracle. Maybe it uses same stuff as pandas for execution, dunno.
 
How are you currently inserting/upserting?
 
3:37 PM
The shock of static typing can be lessened somewhat by using var, where reasonable. I don't have too much trouble with namespaces, as Visual Studio often recommends what using statements I should have.
 
@JonClements converting df to list of tuples, as required by oracle.connect(..).cursor.executemany(). I would expect sqlalchemy does same conversion to flatten the dataframe.
 
private and static and so forth... I won't say they're trivial. Python is pretty ambivalent about what design paradigm you use, but dotnet wants you to use OOP. If you don't know OOP, this will be challenging.
 
@aeiou possibly but you can let it work that out... and df.to_sql('some_table', your_connection, if_exists='append') is just clear as to what you're doing...
 
@JonClements is df.to_sql attempting to parallelize?
I will try to use to_sql as well, to get some performance comparison
 
@aeiou probably not... but as mentioned before... depending on locking/connection modes etc... you're unlikely to get a massive (if not penalised) for trying that sort of thing
 
3:54 PM
@JonClements Came across stackoverflow.com/questions/55585858/… and insert /*+ append parallel.
 
cool
 
4:06 PM
@matszwecja yeah, I know what you mean. Thanks for reminding me :)
@Kevin Nice :D Thanks for the docs links.
hopefully, there are similar instances, so I don't suffer too much. I know OOP is really prevalent in C#, which already gives me shivers, but I guess we'll see...
 
The different SQL dialects sometimes get frustrating. With postgres and redshift, COPY would be the be the best method to do bulk inserts, via very different mechanisms, but apparently in Oracle that's just for copying data between tables :/
@MuhammadAnasAtiq please see our room rules in regards to waiting 48 hours before linking questions from main
You also have loads of answers already and you haven't responded to any of them to say what you think still remains to be answered
 
sure, I already checked these solutions.
 
4:22 PM
But you haven't fed back on any of them, so what was the need to ask here? If you have issues with the answers given, it would be polite to actually feed back to the authors rather than fishing to get more people to pile in
 
4:48 PM
When getting pandas error tokenizing data on pd.read_csv() how do you determine which part of the csv is causing the error?
 
I thought it would actually throw with the position it was upset with. Otherwise you'll probably need something akin to binary search
You could use a combination of skiprows and nrows to batch the file down and start trying to pinpoint the row
 
@Pherdindy as @roganjosh says... I'm sure it does - I had one today: ParserError: Error tokenizing data. C error: EOF inside string starting at row 8828
 
Indeed. I don't think I've ever had a completely blind start on this kind of problem, though I don't have the error handy
 
I get this error and don't really know what to make out of the error pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 224, saw 10
 
Well, it's just told you the row?
 
4:56 PM
It was an error in line 224 of the c_parser_wrapper.py which is chunks = self._reader.read_low_memory(nrows)
 
@Pherdindy you might also want to look at the on_bad_lines argument...
 
I did something like that df_merged_quotes = pd.concat([pd.read_csv(f, header=None, error_bad_lines=False) for f in quotes_list])
It works but I was thinking it removed valuable rows
 
I'm getting confused now. In one part you appear to be suggesting that it's line 224 of the code being reported but then suggesting it could be because you're already batching the file. pandas can be wonky but Expected 9 fields in line 224, saw 10 makes no sense if it's reporting the line of code in its own internals
 
what about trying on_bad_lines='warn' instead of the deprecated error_bad_lines and see how many warnings you get?
 
Sorry for the confusion I haven't touched python in more than a year. Literally my first time using it again
 
5:05 PM
"It was an error in line 224 of the c_parser_wrapper.py which is <...>" I strongly suspect it's not. It should be referring to the line of the file
 
I just thought it was the line of the package since it was also on 224 lol maybe a coincidence
But I do have an idea now thanks for the help will try troubleshooting this by trying a few things
 
Expected 9 fields in line 224, saw 10 sounds like there is a comma in a data field.
 
I did try adding the pd.read_csv(f, sep=",") and it didn't quite do it
 
What will that do if there is a comma inside a data field? For instance, what if one of your entries in column 0 is "ALICE,BOB", and not enclosed in quotes? This will look like a row with an extra column.
 
It would try expand it into an extra column because that's malformed but I had this debate with Kevin a couple of months ago IIRC and I was surprised to find that the csv format isn't standardised
 
5:13 PM
I see thanks i'll inspect how the data is structured and see how to work from there. Maybe there is an issue with the raw files since they are manually uploaded in a FB page by people
I would guess some guy didn't upload the file with the correct format
 
Ick, yes. Scrubbing input data is the first task in analysis, especially when obtained from random sources such as social media.
@roganjosh there is only so much mind-reading that can be done with data collected from the wild
 
I was genuinely shocked about the general encoding of strings in CSV; I've probably mis-advised multiple people over the years on how delimiters would be handled in the file
 
The csv module supports various "dialects" for handling unfortunate CSV data, defaulting the the one used by Excel (where strings are generally unquoted, unless quotes are necessary due to embedded quotes or separators).
 
which is a sensible default...
 
5:22 PM
I like my own dialect of csv: fields = line.split(","). If you want commas inside a field, too bad.
 
I hear this is a good book for data scrubbing - amazon.com/Cleaning-Data-Effective-Science-command-line/dp/…
 
@JonClements is it? I would have thought the opposite - quote all the strings and escape as needed
 
David Mertz - "Cleaning Data for Effective Data Science"
Quoting strings unnecessarily makes the file much larger
 
@roganjosh file size and err... sometimes things won't interpret "'s as a string and treat it as actual text data for the column anyway... shrugs
 
Thanks will take a look at that book I thought it was just easy as glob the csv's and then read to a dataframe hah
 
5:23 PM
"42" should be parsed back in as a string, 42 is an int. Without it, you surely just invite ambiguity when reading back in
 
Half serious opinion: human-readable file formats should not strive to have a small file size
 
In any case, I would strip pandas out of this investigation and just iterate the rows of a normal file object, using enumerate to start printing the rows above and below the problem row
 
also... over the years... I've always been surprised by enterprise level systems that can't import "A",""",""","B"
 
A string I guess, because it could start with 0?
 
yep... that and postal codes aren't always numeric :p
 
5:32 PM
But, I'm not responsible for writing it to a CSV. That's always on the client, but I mean the CSV format itself should be unambiguous in the way the client saves it
 
Yes - sorry for the delete, I can't really delve into this just now
I should be paying attention to the day job - sorry
 
It's fine. Now I think I'll start a new hobby of just entering " into every form that will accept it and see if I can create chaos
 
I saw a good post the other day - be sure to insert a comma in every password, to mess up the CSV file in the inevitable security breach
 
nice
 
@PaulMcG thanks for the book reference though I do think it will help a lot
 
5:36 PM
nice
This second "nice" of mine (above) was sent by mistake, it was giving a "timeout" error when sending my first "nice" and ended up being sent more than once, it was certainly a problem with my internet.
 
@Pherdindy does it mention csvkit.readthedocs.io/en/latest - that's quite a handy set of command line utilities...
 
@roganjosh Will consider this thanks
@JonClements nope doesn't seem to have any indicator of that package but it does look very useful will keep note thanks
 
5:56 PM
I'll also put in a little plug for my littletable package - very lightweight, low-complexity package for tabular data of various types, easy CSV import, which gives a Table that is just a wrapper around a list of Python objects.
 
@PaulMcG how does that compare to rich ?
'cos textual is looking darn sexy these days
 
6:21 PM
littletable uses rich in its Table.present() method.
I don't think the rich Table class has any behavior other than display.
So you have to populate it with whatever means you have.
Yes, I want to do some textual dabbling soon
 
6:52 PM
Rust is so goddamn complicated
 
not really?
 
7:12 PM
I find it pretty complicated. I can read it pretty well but writing it is... not simple for me
I definitely found Java easier but then I was just extending libraries and I imagine some people would be horrified in how I did it... rust just won't let me do it
 
I find Java to be a nightmare, mainly because of dependencies and the different versions and changes between libraries. If I want to do X, it's harder to make it compile for me, etc. Maybe I'm biased since I didn't try hard enough. C looks easier to me, but still relatively hard.
I don't want to say Shells or even Python is easy, but compared to the above, there is definitely a difference.
 
The thing with java (and certainly python) is that I can see a clear path to disassembling the problem. I can see beauty in how it's done in rust but I've just not got my brain into the right frame to do it myself. I think a lot of complexity just comes from that tbh
In any case, I think I'll be hedging a lot on it very soon, so hopefully it becomes mainstream and I can pick it up faster :P
 
Part of me wants to try Rust, but another part of me still wants to continue on the not-too-popular path of going for C and whatnot. I even wanted to try Go when it was popular but didn't do it.
@roganjosh I think the main problem for me (based on introspection) is probably just the syntax itself. Like how I map the understanding I have vs the syntax and the missing stuff in the docs.
 
I don't see anything wrong with sticking with C. Tried and tested path, but rust does seem to be coming up
The docs are very good
 
ah yeah, the docs part was for Java. But yeah, I do recall I heard good things about the Rust docs and even some books
 
7:26 PM
Oh, java, I just faff and hope. Maybe cast a few things around and see if the method names make more sense. Doesn't help that the library I was working with is almost entirely undocumented
 
coming from a C background... I like Rust
 
8:08 PM
My impression of Rust is that it's a novel way to have memory safety without garbage collection. But I do not care if my programs have garbage collection, so I am not eager to try Rust.
Maybe the next time I play with my Arduino... Can't put Python on that.
Or maybe you can, I forget
 
@Kevin I recall you mentioned using Arduino before. I think you even mentioned IronPython as a possible alternative to usual Python, but don't know if it's a good enough equivalent.
there are other fork/version for embedded devices though.
 
All I recall from my last jaunt is that the Arduino's native language is rather C-like
... It might literally be C
 
hmm, I don't really know either, but I guess it might be C.
 
I dontthink you can use rust on these devices. IIRC there was an article tentatively trying to get it running on rasp pi recently
 
8:28 PM
Glancing through blog.logrocket.com/complete-guide-running-rust-arduino it seems it's possible, albeit circuitous
I want my setup instructions to be "launch the arduino IDE. Choose 'rust' from the dropdown. Type your code, and press the green triangle button".
I do not see a green triangle button in these instructions.
 
since we're talking about buttons, what IDE/setup do you use for C#? I imagine VScode or visual studio?
 
Visual Studio. Just because my employer already had a license for the Fancy Edition.
 
got you
 
I'm not sure I have ever used any Fancy-exclusive features though. They're probably ten menus deep.
I am usually a luddite with my personal project dev environment, but I admit a certain fondness for VS' "go to definition" and "find all references" features.
 
I'm similar. I don't really care if it's Pycharm, Geany, etc. If I had to choose, it would be whatever suit my mood or need.
 
9:31 PM
maybe unclear is better
 
10:09 PM
cbg
I've used this on other microcontrollers. I imagine you can flash it to an arduino, too.
 
duplicate; see comment discussion stackoverflow.com/questions/75240162
stackoverflow.com/questions/75240404 needs debugging details (maybe one of you has a clearer idea of what debugging details are needed, outside of a complete error of course)
 
10:43 PM
36 messages moved to MetaPython
 
@CodyGray can I ask for an interdependent review of ^^^ please?
 
Thanks, Jon.
I just want to know a moderator's opinion on this, I don't want to cause any confusion.
I believe I have not disrespected anyone here.
And I don't even intend to.
I'm just really upset.
 
oh, don't get me wrong @Marco, I think you're wrong and I trust the RO's here... and I stepped down as a moderator in the new year after 7 years
 
Yes, I know you are a moderator. And ok, if you say I'm wrong, I'm already satisfied.
I would like this to just be recorded. Thanks.
Got anything else to say?
I'm already satisfied (although I don't agree, but I respect that decision).*
 
nope - hoping a current mod can get review and let everyone know
 
10:53 PM
Ok, alright. But aren't you already a current moderator?
Ah, I'm sorry, I get it now, you are no longer a moderator.
I got confused, read too fast.
 
I'm a room owner, have been since 2012 or something, so there's COI when it comes to that and being a mod, but no longer a mod...
 
Perfect, and I know you own the room too for quite some time.
Thanks.
 
so just RO... up to you... I've asked a mod to put their input in... they'll review it when they have time?
 
I didn't understand what you said, sorry. Could you clarify please?
 
A moderator will review what you've said and get back to you at some point - try to work out if it's a R6 issue or you kind of thing
forgot to eat...bbl
 
11:02 PM
Okay, got it, thanks.
 
11:17 PM
Is there any library that make replicate the production development environment with the local development environment apart from docker. Like, working with M1 chip in local and production environment having Linux based, there is always conflict of package version require in these different kind of environment. Any solution for this apart from docker?
 
None that I am aware of
It's interesting that you raise the issue of M1 because all of those died away on my hassle list at work, so I thought most libraries were stable against it now
 
Hello!
What's a good name for a image that will be used to compare against another reference image?
Can't figure out if there's a name for something like that, haha.
 

« first day (4484 days earlier)      last day (469 days later) »