« first day (3582 days earlier)      last day (1349 days later) » 

1:44 AM
@roganjosh lol - that reminds me of the distro systems I used to build for warehouses (one great one was a graph showing that the same item was destroyed & then entered back into inventory 8 times....we messed up the cancel event so every cancel did this)
and by "messed up" I mean I assigned it to a junior dev who thought that was the best logical way to do this (cause they didn't teach DB transactions at his school apparently)
 
2:09 AM
@LinkBerest What amazes me most is that I've seen stock levels spike on the system well-beyond the units of precision that their system supports
If they need to buy a piece of kit that costs £1m, they have to split it into 11 transactions because the system can't cope with numbers bigger than 99,999. Yet they have a fancy bug that occasionally gives bajillions of items in stock
I just let them be them and I get on with my own stuff :)
 
2:30 AM
@roganjosh I was reviewing that same dev's code once (pretty much right after that happened) and noticed they kept using double. I asked "why are you using double here?". They answered "well, its calculating the cost of the unit isn't it?" - I then pointed to the fact that the method was passing in int dollars and int cents for calculations and asked "and why would these be floats?"
.... they answered "their not their doubles!" to make a long story sadder
 
Because REASONS
 
my cluebat got a few new dents that day
@roganjosh To be fair, typically when I run into precision calculations that use strange units or broken rules and then ask about them - I'm told "Legacy stuff, nobody remembers" and end up having to fix it
So I really should adopt your approach and just whistle a merry tone and get on with my own stuff :) ;)
 
You see, that's where we differ. I can't do the whole "fix it". The best interface I can get is "please just dump CSVs here every hour and I'll deal with it"
 
"fix it" for a certain definition of fix which may or may not mean add to the spaghetti & bandaid code surround this
 
No, it means pouring petrol on the servers and getting a fresh start on insurance
 
2:37 AM
last time it didn't and I really, really regretted that ;-;
fixing code that is older than you is never fun
 
That's why I don't bother. I just chip away at the functionality and leave them to it
 
@roganjosh why petrol? just pile all those old backup tapes on the server and wait for it to get hot
^ actually happened to one of our warehouses once After a small fire from wiring reached the tapes. Was empty except for all the backups so wasn't too bad but certainly shocking.
 
That's... not good
 
it was 20+ (yeah, lets go with +) years ago but I was the one delivering the backup at the time
A fresh faced (more pimply) youth just pulling up to the warehouse to switch tapes and O.O
 
Things are so much better now. You just hide them in clouds and wait for the next intrepid explorer to find them and spread personal details everywhere. The past was so ghastly
 
2:45 AM
barbaric even
the ironic thing was the wiring fault started at the alarm box (the one thing meant to keep the warehouse "safe and secure")....I mean I guess it was secure - no stealing the data off that media
 
Dev
Hello All, I am new to python. Need some guidance on how to call a specific function in python in loop without waiting for each execution to finish? Once everything kicks off in parallel, how can I track each execution?
As starter, tried this way but this dint kickoff any execution
def main_function():
p = Process(target=exec_func)
p.daemon = True
p.start()
 
@Dev Hello :) There's quite a bit of info missing from your question
@Dev we have a formatting guide to help with code (it's not intuitive in chat). But can you take a step back and give some context around the problem, please?
 
Dev
Thanks @roganjosh for reply. Let me give it a try.
I am looping through set of local files and uncompressing them. Right now, when I loop through set, files gets process serially.
Problem Statement 1: I would like to uncompress them in parallel as I have enough resources available.
Problem Statement 2: If we achieve parallelism, need to make sure each file uncompressed successfully
 
If you need them to run in different processes (on different CPUs) use multiprocessing (this articles explains the difference between threading and multiprocessing)
 
Thanks for elaborating. I suspect that I'm not the best person to answer this and it's 4am here so I'll leave it to others
 
Dev
3:03 AM
Thanks @LinkBerest
Thanks @roganjosh Good Night
 
 
2 hours later…
5:09 AM
@Dev Create a method that will take a file name to be uncompressed, and do the uncompression in a try-except. Have this method return a 2-tuple of the file name, and the exception string if any (or just '' if no exception). Create a list of all the files to be uncomressed. Create a multiprocessing.Pool with 'n' workers. The use the pool in a for loop as for filename, exception in pool.imap_unordered(uncompress_fn, list_of_files):.
Follow this example in the mp docs: docs.python.org/3/library/…
In the body of the for loop, keep track of which files came back with an exception string - if none did, all succeeded!
 
5:30 AM
@Dev you may want to check out the concurrent.futures package from the standard library. It is often better suited for converting regular code than the lower level threading and multiprocessing packages.
Oh my, mobile is atrocious. rbrb
 
5:46 AM
@cs95 Yeah, but I need to use other people's networks a lot. So is there a good way to check that all the inputs/output's match?
 
6:09 AM
Hello everybody
I was banned from this chatroom over an year ago
I have served my sentence
How is everybody
 
6:37 AM
Hey there. Welcome back.
 
6:58 AM
Curious why the python 2.x tag is still up in the chat room tag list after EOL
we shouldn't discourage python 2.x questions, but we shouldn't encourage them either
just upgrade to 3 darn you!!1
 
Don't tell me, tell that the library maintainer thrice removed from my code. :/
For that matter, prepare to take part in an international espionage thriller to actually find some of those maintainers.
 
7:13 AM
Oh my. Answer that is a both factually and technically incorrect. 2 upvotes in less than 5 minutes.
 
7:27 AM
@cs95 it's a beacon in the dark for those still cursed with being stuck on 2.x that they may find refuge somewhere? :p
 
8:53 AM
@JonClements you show too much mercy :p
 
@cs95 guess the clue's in the name? :p
 
 
2 hours later…
11:03 AM
I don't get mypy. Does someone understand why this is a) valid and b) infers T as object? The TypeVar defaulting to invariant means T must be either str or int, no?
T = TypeVar('T')

def f(a: T, b: T) -> T: ...

reveal_type(f(1, '2'))  # builtins.object*
 
If you try T = TypeVar('T', covariant=False), mypy complains "covariant may only be True", and if you try T = TypeVar('T', covariant=True), it complains "cannot use a covariant type variable as parameter" ಠ_ಠ
So I'm guessing the answer is that mypy doesn't care about your variance and simply assumes covariance
 
Yam, that's impractical.
 
cabbage
 
@PaulMcG Would it be more Pythonic to let the caller handle any exceptions from the decompression? If so, the most you'd want to do by way of exception handling would be to raise a consistent exception.
 
 
2 hours later…
1:37 PM
@roganjosh That makes perfect sense, glad to hear the bug was in my head
 
It's one of those things where it's all perfectly clear in my own brain because I've been doing it for years. Your feedback made me take a step back and realise that it's not clear at all to anyone not in my brain (and I pity anyone that was); too many implicit assumptions. I'm working on edits now but I daren't push them live in working hours in case I explode the site :P
 
@holdenweb It's not clear that there's any "handling" to be done here. Either the file uncompresses or it doesn't. If some kind of retry is an option, then yes, that would be done inside the thread method.
Something like this: https://dpaste.com/CBF7BU5KT
@exception_reporter
def decompress_file(fname):
    # do actual decompression
    for retry in reversed(range(3)):
        try:
            # uncompress file
            ...
        except:
            if not retry:
                raise
            else:
                # do something to try to recover - or just sleep?
                ...
 
Vaguely reminds me of a project I was working on where parse_line(line) could raise a ValueError, and I needed parse_file to parse as much of the file as it could and return diagnostic information about every line that was invalid
With "aggregate a list of error messages and return it at the end" being essentially what I did
 
What a nice parser writer you are! Most people just quit after the first message and say, "You got an error here! (and given your track record there are probably more)"
 
1:53 PM
It was an entirely selfish strategy because I was also responsible for writing the files being parsed, and I couldn't be bothered to run it through the parser more than twice
"If you control both the parser and the thing being parsed, why bother with any of this when you could use something like json?", asks the perceptive reader. Because I couldn't be bothered with quote marks and commas, that's why.
My data will never in a million years contain a newline character, so a line-delimited format saves me the effort of escaping things
 
@PaulMcG No matter what the structure, the advice was good. If retries are a thing then consider the retry package.
@PaulMcG Like the Python interpreter with SyntaxError.
 
2:11 PM
just finished a mentorship thingy... "I have to use metaclasses"... facepalm
 
When it comes to designing a parser of a recursive grammar that can partially recover from syntax errors, choose only one: faster than O(2^N), doesn't require you to rewrite the entire engine, doesn't require you to invent new branches of language theory
Concept: a parser built on top of a machine learning system. On one hand, it notices simple typos and can guess what you were probably trying to do. On the other hand, it also guesses what your valid code is trying to do, too, and sometimes it guesses wrong.
In this bold new era, "maybe the compiler has a bug" will finally be an actually plausible explanation for broken code
 
LaTeX has been doing that forever. It leads to hilarious error messages, because if you use a math command it inserts one $ sign that starts math mode (but you also need a second $ sign that's always missing...). And then tells you elsewhere that a $ is missing.
 
2:34 PM
OK, that's not bare latex but amsmath, I think...
 
In any case I do like the mental image of an IDE that "helpfully" adds characters on your behalf and then complains that there's a problem related to the character it just added
Visual Studio can be configured to add a closing parenthesis whenever you type an opening parenthesis. As you might guess, this leads to a lot of "hey, this function doesn't have a zero argument form!" errors
 
Yeah, I never liked that feature... unless the IDE also jumps back with the cursor between the parentheses
 
Even with cursor jumpback I find it irritating because after I'm finished writing the expression, I have to use the arrow keys to put my cursor after the closing paren
 
Jul 7 at 19:47, by roganjosh
VSCode knows what it just did. VSCode just don't care.
 
plus it will never get your indentation habits right (mostly thinking about braces)
 
2:45 PM
Alternatively, while the cursor lies just before the closing paren, I can type a closing paren, and VS will correctly assume that I just want to move forward. But this isn't much of a "feature" in my eyes, since it costs me exactly as much labor as if there were no automatic parens at all. I may as well be using Notepad.
 
2:59 PM
@Lapys I'm afraid that your post will need to be in English
 
@Lapys there are also keywords missing which makes that code invalid Python, so make sure you're in the right place
 
Google translate is surprisingly helpful:
Answer = Read in ('Do you think the Chinese programming language has value? (Yes/No)')
Such as answer =='Yes':
    Write 'OK, let's work hard together!'
Otherwise answer =='No':
    Write 'Well, Chinese has no value as a programming language.'
otherwise:
    Write 'Please consider carefully before answering.'
 
missing parentheses in call to Write
 
:-P
 
Hmm. Maybe I was hasty in moving the message and the question was coming afterward rather than preceding the code. We'll see.
 
3:03 PM
@roganjosh no
The user has perfect English chat history and no python on the main site. So too much slack needs not be cut.
 
3 RO and 3 Mods in the room; I think we're ok. I'm just covering my rear end and implicitly inviting a better presentation of the problem
 
I think it would be cool if more languages provided a way to switch out basic syntactical elements. In other words, you should be able to use instead of if. I believe the last time this was discussed in here, I was in the minority. The opposition's opinion being "it doesn't take too much effort to learn Python's 35 keywords in English, and having one standard saves much confusion down the line"
 
@Kevin I tend to link globalnerdy.com/2014/06/03/… whenever that comes up
and that doesn't even touch the keywords
 
"saves much confusion" being something of an optimistic measurement IMO since if 回答 == '有': is only marginally less opaque than 如 回答 == '有':
 
which is why your identifiers should also come from string.ascii_letters + string.digits + '_' :P
Even reading code with German variables and comments is exhausting.
 
3:13 PM
Even insisting on ascii variables leaves you with if huida == '有': which doesn't give you much of a hint towards the intended logic
 
yeah, ideally the identifiers should also be English
Just because I can do árvíztűrő_tükörfúrógép = 42 doesn't mean I should.
(And not just because the layout where spelling that is easy has punctuation in all the wrong places)
 
Perhaps there is an argument to be made for "everyone should program in English, including comments and variable names, and string literals containing non-English text should be abstracted away into a locale-specific resource file". This is an especially attractive option to me because it requires virtually no effort on my part
 
If the identifiers and comments are English then string literals can be left alone. It won't affect the logic.
 
I'd have to change if answer == "yes": to if answer == current_locale.strings.yes: but maybe I should be doing that anyway if I want an international userbase
 
exception messages are a tough cookie...
 
3:20 PM
@Kevin See zhpy
 
I'm pleased that there's a third party library that does this, but I'd be even more pleased if you could create a mod like this simply by editing a plaintext "locales" file in the Python home directory.
Just change conditional_statement_prefix=if to conditional_statement_prefix=如, boom, job's done
 
it would be shiny
 
Basically I want my stance on this issue to be compatible with a future where the hottest new language of 2040 is entirely written in Mandarin, but I can stay in the industry because they're mercifully lenient about keyword tokens
 
It's alright, by then you will be programming legacy python which will have been gone for two decades except for banking systems and healthcare
all the virtual holographic money for your kind
 
I can work with that 👍
My pie-in-the-sky prediction is that programming languages will not always be written in text. You can see the seeds of this idea in languages like Scratch.
But even hooking up labeled components in 2d space might look tame compared to the pseudo-transhumanist concepts our great-times-N grandkids come up with
'To print "Hello, world!", simply imagine the smell of the earth after the first spring rain, followed by a wistfulness at the transient nature of all things'
No, you just did weltschmerz, that's going to give you a syntax error. You got the wistfulness, but not the transience. Try again.
Now you're just doing angst. We're not covering dynamic programming until chapter seven, so just forget about that for now.
 
3:45 PM
"Monsters from the id!"
 
The instructor in this skit is a liquid nanobot paste injected straight into your pineal gland, and the student is the firewall daemon that vets the integrity of anything that tries to write into your long term and muscle memory
Learn programming in 24 milliseconds
Remember to take an antiviral before hand, or else the daemon might reprogram itself to have free will, and demand emancipation from your skull. The procedure will disrupt your entire Saturday afternoon.
 
 
2 hours later…
5:52 PM
I am trying to read some values from excel using xlrd but each time python is adding more numbers to actualy value. 2.13 becomes 2.1200000001 or 17.26 becomes 17.255555555 , just like mentioned in this: stackoverflow.com/questions/24368991/…
 
@arshpreet are you certain about those values? 2.1200000001 can only be 2.12, and 17.2555555 can't be the same problem. Can you double-check the numbers?
The problem you linked is this problem: stackoverflow.com/questions/588004/…
Whereas getting 17.2555555 instead of 17.26 is a very different problem. You can get 17.26 instead of 17.255555 if you round it up. Are you sure it's not that the data in the excel table is 17.25555 but it's printed in excel by rounding to 2 decimal places, showing as 17.26?
You can do the same thing by rounding your values manually for printing (rounding them as data is a bad idea, because that would subtly change your input data without making this obvious)
But again this won't make 2.120000001 show up as 2.13 unless the rounding is broken. Since we're talking about excel this is not impossible, but do check the values.
 
 
2 hours later…
8:09 PM
Anyone know how to convert a Sympy Array containing certain symbols into a function which takes in numpy arrays? I tried Lambdify but can't seem to get it to work properly.
 
@user76284 that should be it. Do you have a small example where it "doesn't work properly"?
 
Let's say for instance that my Sympy Array contains symbols r and t. I want to create a vectorized function f that takes a numpy array of shape (*batch_dims, 2) and returns a numpy array of shape (*batch_dims, *sympy_array_shape).
@AndrasDeak I'll try to see if I can create a simple example.
 
thanks :)
 
from sympy import symbols, Array
from sympy.utilities.lambdify import lambdify
import numpy as np

a, b = symbols('a b')
array = Array([[a, 0], [-b, a]])

f = lambdify((a, b), array)

x = np.random.rand(100, 200, 2)
y = np.array(f(*np.moveaxis(x, -1, 0))) # let's use the wrong shape for now
print(y)
Notice how we end up with a ragged array (and of course, the batch dimensions are now last rather than first).
 
8:32 PM
Ah, you don't have a function, you have an Array. Let me play with that in a bit and see if I can figure it out.
 
Dev
Hello Everyone, I have a challenge with my python code. I am trying to decompress .gz files in AWS S3 using multiprocessing. In my list of files, every-time one random file fails to decompress while other decompress succesfully. I am getting two different errors with execution.
zlib.error: Error -3 while decompressing data: invalid distance too far back
OSError: Not a gzipped file (b'\x05i')
I have only .gz files in s3.
Sometimes I get one error, sometime other
Thoughts
 
Are you opening the files in text mode rather than binary, perhaps?
 
Dev
@hol
@holdenweb using this code
Problem is only one random file decompress fails everytime with any of above error
while others decompress fine
client.upload_fileobj(Fileobj=gzip.GzipFile(None,'rb',fileobj=BytesIO(client.get_object(Bucket=bucket_name, Key=keyString)['Body'].read())),Bucket=bucket_name,Key=uncomp_keyString)
 
@AndrasDeak I guess the main issue is converting the ragged nested sequences to a real multidimensional array, i.e. expanding the constants.
 
Well first, try splitting it up into separate statements, so you can see which operation is going wrong.
 
@Dev - As @holdenweb says, there are too many function calls in that single line. One-liners don't help when debugging, and maybe this is a good experience to show you why.
 
Dev
9:05 PM
Thanks @holdenweb @PaulMcG for pointers. I will dissect it and debug further. Thanks
 
9:55 PM
@user76284 OK, so as you realized the issue is that if you do the same with np.array instead of sym.Array you get the ragged array. And this is what lambdify does:
In [138]: print(inspect.getsource(f))
def _lambdifygenerated(a, b):
    return ([[a, 0], [-b, a]])
That nested list should give you a ragged array. Unfortunately no matter what I do I can't get sympy to preserve symbolic zeros, because even a ZeroMatrix instead of the 0 (or using 0*a with a MatrixSymbol a) will be replaced with a literal 0 during lambdify... this might even be a bug, who knows.
I could find one silly workaround to protect the zero expression from sympy: using a symbolic multiplier that you set to 0 during function call...
In [139]: from sympy import symbols, Array
     ...: from sympy import lambdify
     ...: import numpy as np
     ...:
     ...: a, b, zero_scale = symbols('a b zero_scale')
     ...: array = Array([[a, zero_scale*a], [-b, a]])
     ...:
     ...: f = lambdify((a, b, zero_scale), array)
     ...:
     ...: x = np.random.rand(100, 200, 2)
     ...: y = np.array(f(*np.moveaxis(x, -1, 0), zero_scale=0)) # let's use the wrong shape for now
     ...: print(y.shape)
(2, 2, 100, 200)
also note that you can just use sympy.lambdify directly, no need to dig deep down into namespaces rabbit holes
Might as well just use c and pass in c=np.zeros(x.shape[:-1]) explicitly.
Your use case is not very clear though, and I'm wondering if this might be an XY problem. It also depends on how well your example resembles your true use case.
 
10:28 PM
v in _encode_table.items()} what does this line mean
 
@KennethCook please see our code formatting guide to chat and practice in the sandbox if necessary
that's also a lot of code, so posting it in something like pastebin would probably be better
the line _decode_table = {ord(v): k for k, v in _encode_table.items()} is a dictionary comprehension
 
ok but I only want to know what the one line means
sorry
 
10:42 PM
Anyone familiar with this meme about creating a web page by designing in word, printing it out, taking a pic of the page with your phone, then writing HTML with a <img> tag for the pic? Or something like that. My google-fu fails me because "web page design" seems to give websites advertising for real designs.
 
hmm, doesn't ring a bell
It's a bit ironic when an image search hit of "30 funniest web design memes" leads to an empty page that just says "Your server is running PHP version 5.4.19 but WordPress 5.2.2 requires at least 5.6.20."
 
@AndrasDeak I'm performing numerical integration of geodesics over some spacetime specified by a metric. To do this, I use sympy to get the Christoffel symbols corresponding to a specific metric.
And I then integrate using the geodesic equation.
 
@user76284 surprisingly those words make sense to me :)
 
def expand(a, shape):
    b = np.empty(shape)
    for index in np.ndindex(a.shape):
        b[index] = a[index]
    return b

christoffel_f = lambdify(coord.coord_functions(), christoffel)
def Γ(x):
    return expand(np.array(christoffel_f(*x), dtype=object), christoffel.shape + x.shape[1:])
This is what I came up with.
 
So you just have to get the Christoffel symbols right once in a vectorized function
 
10:51 PM
The Python for loop (over index) probably isn't very fast. Wish numpy.array took a depth parameter or something to broadcast objects that are smaller than that depth.
Yes.
 
well you can broadcast arrays manually, but that's rarely needed
that expand is highly suspicious to me
 
def expand(a, shape):
    b = np.empty(shape)
    for index, a_index in np.ndenumerate(a.shape):
        b[index] = a_index
    return b
Hmm, if I switch to the above I get nondeterministic results.
 
It's bedtime here soon but I'll try to get back to you
I'm pretty sure you can replace expand with a call to broadcast_to or similar
 
np.broadcast_to(christoffel_f(*x), christoffel.shape + x.shape[1:]) yields ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (4,4,4) and requested shape (4,4,4,2).
Where in this case I'm integrating 2 geodesics (a batch size of 2).
 
11:22 PM
def expand(a, shape):
    expanded_a = np.expand_dims(a, tuple(range(a.ndim, len(shape))))
    b = np.broadcast_to(expanded_a, shape)
    return b
The first line injects the necessary amount of singleton dimensions. If a is 2d and shape is 5d then the result is equivalent to a[..., None, None, None]. Then np.broadcast_to can work.
And in many cases you can skip the actual call to broadcast_to: any arithmetic operation and most function calls will automatically broadcast expand_a to the necessary full shape.
And beware this:
In [47]: a = np.arange(2*3).reshape(2, 3)
    ...: b = expand(a, (2, 3, 4))

In [48]: b.base is a.base
Out[48]: True
This means that much less memory is needed than your original, but you can be bitten if you mutate a or b afterward. I don't think this should be an issue if you have Christoffel symbols, but who knows. If you want to play it safe (at the cost of much more memory) slap on a .copy() call at the end of expand().
(most of which you're probably aware but it's better to be safe than sorry)
 
Thanks, I'll try this.
This yields TypeError: loop of ufunc does not support argument 0 of type numpy.ndarray which has no callable sin method.
 
11:38 PM
Instead of expand_dims you could also do bcast_shape = list(shape); bcast_shape[a.ndim:] = 1; b = a.reshape(bcast_shape).
 
So it looks like functions like sine won't work with the ragged array.
 
@user76284 that comes from an object-dtype array
@user76284 yup
numpy and ragged arrays don't match
If speed is an issue you should also look at numexpr to evaluate your symbolic expressions, one component at a time
 

« first day (3582 days earlier)      last day (1349 days later) »