« first day (3816 days earlier)      last day (1357 days later) » 

01:16
needless dupe with ugh answers stackoverflow.com/questions/66804336/…
 
10 hours later…
11:24
Is it possible to use if statement in a function and integrate it using scipy.integrate.quad ? or I want to integrate them as two ranges ?
@Janith why would it not be?
But it might not behave well. You're probably better off integrating on each domain.
If you know your function has a discontinuity, don't make scipy find it
EX: is it possible to integrate this using scipy.integrate.quad
def square_wave(a):
A=2
T=2
return (-1)*A*((a%T)<T/2)+(A)*((a%T)>=T/2)
You can also pass a set of problematic points to quad
@Janith try and see? You can compare with the integral by hand. Square waves are easy to integrate.
11:30
I did .I used it for calculation continues fourier transformation .
Oookay
Hi, is it possible to stop python from compiling modules before bringing them to the main program?
I want to have parts of my code separated in modules and bring text one after another.
11:47
@enthu I don't really understand, but that sounds like an XY problem
It really shouldn't matter to you when python compiles your modules. That's a sign that you're doing something wrong, but I have no idea what
@Aran-Fey The problem raise from where I want to use API codes of a software...
I can not call features I have defined in a main program in the modules
For instance, I make A feature in the main program
if I want to call it in the module, I should call all permeable in the module
and when I do it, the software goes to create a new feature A
I afraid I am not describing my problem correctly again :((
This is too abstract for me, can you create a small MCVE?
no sure...
@enthu you need to put your code in functions
Code at the global scope gets executed on import. Don't put code there you don't want run on import.
Could also be a circular import though
11:55
@Aran-Fey "software goes to create a new feature A" sounds different
Obligatory advice: Learn by writing smaller programs first.
Taking a look at some of the libraries you are using, including the standard library, might help too.
 
2 hours later…
13:39
Hi.

I have such 2d numpy array:
[[1. 2. 4. 6.]
[1. 2. 4. 6.]]
How can I check if 1d numpy array [1, 2, 4, 6] is in that one above?
I was thinking this will give me what I want np.isin(lines, np.array([1, 2, 4, 6])), but it made smth like this:
array([[ True, True, True, True],
[ True, True, True, True]])
ok. Got it.: [1, 2, 9, 6] in lines.tolist(). Is it efficient?
@entithat probably not
@AndrasDeak what about such case: np.array([1, 2, 9, 6]) in lines.tolist()? Im getting an error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
when I want to check if numpy array exists in 2d array
Compare (np.array([1, 2, 4, 6]) == your_array).all(-1).any(). It will have different CPU and memory need. Also watch out for comparing floats exactly.
and if the 2d array is large and you're looking for many rows in it it might be better to convert the 2d array to a set of tuples
but again that would only work for exact comparisons, and would probably need large arrays to be better
@AndrasDeak what does any(-1) mean?
anyway, thanks a lot
@entithat all(-1)
It's same as all(axis=-1). Read the docs and/or play with it.
13:55
@AndrasDeak oh ok. Thanks
hey , is it safe to save files in it's own format instead of data on the database that could be converted back to its original formate on request?
safe as in what? safe against attacks? safe to not lose data?
Yes and also if its safe against performance ?
yeah, you want to protect against performance ;)
it feels like 5000 files on the database ( saved as ACID ) is more likely pressure the server on the long run.
@AndrasDeak lol, I meant if it's a safe decision to make
i expect to have anywhere from 500-5000 a month in files
basically what I know so far

A: files in its own format ( less secure , better performance , backing up db doesn't include the files , files directory could be modified in case of being hacked )
B: files on Database ( ACID - Bolb ) ( more secure , less performance , backup db would include the bolb data along with it , harder to hack)
I feel like keeping the files in it's own format would also be less likely to get corrupted by the system itself. ( ex: if the files needed more than 256 character it may not be able to store the whole bolb correctly. )
is there is anyway to scan the files before dealing with them ?
14:24
I don't think I understand any of that...
What is "its own format", especially in light of it being truncated to 256 characters?
And if I was all up in your server, what is your database going to help you with vs. flat files? You'd need an encryption layer and I'd already have the key to that, wouldn't I?
its own format ( like .pdf , .xlsx , .jpg , .png ,... )

about the **256 character** , correct me if i'm wrong , to convert any file to **bolb** it would create a big string and store it in one of the database column of that row
A string that you're putting a character limit on, making the entire format invalid if it dared cross that limit
Separately, it's "blob" not "bolb" btw
ok that is better, I thought some files would go along with the character limit and others wont because they are bigger, those I should be more careful there.

" You'd need an encryption layer and I'd already have the key to that, wouldn't I?"

what do you mean by encryption layer ? ssl ?
also I didn't understand how would you have access to the key of it ?
@roganjosh typo sorry.
@LoopingDev I'd have the encryption key because I've just hacked your server, so I can see exactly where all your code loads the key from, and I could just read that file myself. And no, I don't mean SSL, I mean encryption of the data within the database
Once the server is compromised, the attacker can see the encryption key
I only use passlib but I only encrypt the passwords
14:36
No, you hash the passwords
oh, yeah that is right.
Well, at least I hope you do. There is an important difference between encryption and hashing
How could I encrypt the database?
Well, are you clear on the purpose of why you're doing it in the first place? I get the sense that you're not
Yeah, I'm new, I'm only doing it because you said if the server is compromised and I don't have any encryption on the database, it would be less likely for the database to be secured for much longer
14:40
I didn't say that at all. I said that your encryption would be useless if the server is compromised because I'd just find the key
oh!, I see now. so even If i encrypted the database, It would be all for nothing..
so I'd be better off preventing the server from compromising, is that right ?
That should always be your main priority!!
Heh, let's not start with disaster scenarios before you think you've done all you can on that front :P
Yup, It scares me though that wherever we host the system online, could do any shady deals and sell the data and claim that he was hacked.
that is a risk we kinda have to take anyway unless we host on our own servers.
Now you know why everybody is building datacenters like crazy
You typically don't find cloud service providers down back alleys....
14:45
Another concert is how to prevent a virus from being uploaded as a jpg for example.
Or, at least, shouldn't
@roganjosh Meh, my host just vanished one day, took everything he had. I could claim the last few payments back, since it was over paypal, but still. There are some shady hosters out there
@Hakaishin But this would be a Joe Bloggs host and not something like DigitalOcean?
Haha yes
yup, I have plans on checking on with google cloud and Amazon as first priorities , those guys already have all our data, so we wont be risking any new data to begain with lol
14:47
@LoopingDev you might want to revisit your requirements. .pdf, .xlsx, .jpg, .png are just file suffixes, but they stand for data formats. When you put the data from a .png file into a database, it will still be PNG data.
It worked perfect for a year with low cost, then I guess he said ** it, because I couldn't even find the company anymore. Luckily it was just my moms website read only nothing important
You might want to read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) and then ponder the fact that, in a way, every kind of data format is just some encoding on top of bits and bytes.
@MisterMiyagi Yes, I understand that, but i'm yet to make a decision about which way I should store the files? .. as data or as files?
also how to scan the files before being uploaded to the server ( the server would be a linux based server, so its less likely to get the virus but it still can distribute it to the windows users as jpg or pdf for example)
Ohgoddonoteverthinklinuxserversaresafenonononono
its not , it's just safer
14:53
@LoopingDev That's a bit like asking whether you should store your suitcase as a suitcase or clothes.
its because it doesn't give option for users to write .. as far as i know
while windows system , you could find a way around that.
Are we talking about the database server getting a virus? How would that happen? As long as all it does is store and retrieve data in the database, there's literally 0 risk of a file causing damage, even if it's a virus or trojan or whatever
@Aran-Fey yeah but you could also be distributing it as a jpg file to all the connected users.
Ah, I see
@MisterMiyagi it feels like storing it as blob might risk performance, or at least that is what I read over the internet.
at the moment , I'm leaning toward storing them as directory files but I just want to make sure that they aren't infected before I do so.
14:59
In general, it is better to separate data from metadata. Often enough, you want to work just with metadata to then find out which subset of data you need. When you conflate the two, you are working with much more data as needed.
For example, when you want to find "the newest 5 files", you only need path+age. You do not want to read the content of all files.
It's still possible to separate data and metadata in a Database, but many DBs do not offer an advantage over files for storing huge blobs of data.
@LoopingDev What do you mean by "infected"?
yeah, I think blobs is a mess and could be causing a headache on the long run.
@MisterMiyagi like if the jpg files is malware or a trojan for example but in jpg format
or pdf
And do you actually know how to do that? Check all of these different data formats mentioned for malware?
no I dont, I'm asking if anyone knows how to do so
flask upload for example has secure_filenam()
I'm still not really sure why you think it's your responsibility to detect malware. Even if a virus is sent through your database to someone's PC, that PC will have a virus scanner anyway.
@LoopingDev That does just what it says on the tin: create a filename that is secure. For example, ; rm -rf would not be a safe filename.
The file content can still be pure malicious evil.
15:10
It's sad that we live in a world where file names are sometimes executed as code :(
no very likely, I expect the users that would be using this system would more likely to have a virus on their own computer and they don't know how to get it out.
they just keep the slow pc forever.

in other words, I expect the users of this system to be very new to technology in general, and unfortunately, that is very likely
Consider this: If your service claims to protect against viruses and it does not do so thoroughly, you expose users to more of a threat because they will assume to trust your data.
I wont claim but I just want to do my best
I though there would be a flask package that I'm not aware off that could scan the uploaded file.

for now I guess I could just move on and keep looking for a way to scan the files in the future.

thanks a lot guys :)
yes, the famous Universal Malware Filter function
in the next version it will have a keyword argument erase=True that removes the detected malware from every computer connected to the internet
15:29
Plot Twist: Universal Malware Filter detects its own malware deletion engine as malware and goes rogue!
Odd question here: is anyone in a position to provide one-to-one tutoring on Python topics? [Asking for friend ...]
15:49
haha, as crazy as that may sound I was thinking of some sort of scanner that scans the files and make sure it's legit before storing it.

Maybe even execute it in a secure environment and see if it behaves any differently like trying to access something that normal jpg files won't need.
16:00
Ahh, the good ol' sandbox approach
@AndrasDeak can those functions be in a separate python file?
@roganjosh lol, yeah I think that would be nice.
It would indeed. Shame that it's horribly complex to get right
(without using the Universal Malware Filter, obviously. In that case, it's trivial)
@MisterMiyagi thank you... but this seems to be a tough problem in APIs written to link commercial softwares to python packages. Abaqus python API, CSI SAP2000 python API link, AutoLISP functions can not be called in separate programs for AutoCAD, etc...
@roganjosh I also found this stackoverflow.com/questions/42156349/…
which is suggesting the use of that : clamav.net/downloads
16:05
Softwares APIs are not accessible outside the softwares running, their are not also available through pip. So, they are in python, but can hardly be accessible outside the software. this is the problem.
Similar to the email scanner approach
@enthu If the problem is tough, that should be motivation to invest more in its foundation rather than less.
@MisterMiyagi yes, but I am not a developer of those commercial softwares. I should find a way to write my code to get the results and pass to another project... so it is tough going through fundamentals of closed API packages...
honestly it kinda odd that this is not old news.
How does anyone secure their applications from such attacks using python?
there has to be a way, if there is not a way I think there would have been no system that would last long without distributing masses of viruses to its clients
i thought they would be a service for free or paid that checks the uploaded files by now to say the least.
@LoopingDev because softwares links. APIs in commercial engineering softwares act as link to give the user freedom to do parametric modeling. So they are not main software.
16:12
@enthu Look, the way Python executes packages is easy. If you have trouble to grok this, and your messages suggest as much, you miss the groundwork to venture towards more complex tasks.
@LoopingDev for instance, enginering packages like openfem are all in python. commercial software like Abaqus is not in python. API for abaqus is an added feature to give user freedom to do systematic coding and automation. that is why the python api does not come with pip. it comes when the software is installed.
@MisterMiyagi all right, thank you.
@enthu I got what you mean, so in short we might be able to scan the files using api that connects the python application to the anti-virus scanner that might be installed on the same server as well, am I correct?
Oh my. This has become very confused indeed
@LoopingDev it is not a virus and anti virus story... or I am missing your point of view...
@enthu this is a specific situation that you are facing, and it's totally non-standard. It has nothing to do with what @LoopingDev has been asking.
16:18
oh dear, lol.
I got it we are on different pages.
@enthu is trying to interface with some software that happened to drop a bunch of .py files on Windows, @LoopingDev is talking about file uploads on a website and them potentially being executed. There is almost zero cross-over between your discussions
goes to show how complicated concurrency can be
speaking of the devil > i found this pypi.org/project/eyeflask
thank you guys, sorry btw
"The Eye-Fi card can connect to it and will upload images to the folder specified in the config. EyeFlask attempts to verify the file integrity using the same security protocols used by Eye-Fi Server."
16:23
erm, "integrity" usually means something else than "security".
Basically, it checks whether all the bits and pieces are there as claimed by the sender. It doesn't actually check whether any of these pieces are gremlins trying to eat your hardrive.
@MisterMiyagi as long as you don't feed the process some bits after midnight, you'd have all the protection you needed, though? :P
ok that was a bust but this one looks legit!
this pdf clams that is made to protect python webapps from virus
pyClamd
pip install pyclamd / python setup.py install
Hello, is there any way of using a timer in an app which won't be affected by the numbers of prcesses which run on the computer? For ex if I use a timer for 1.234 secs, having more or less apps opened on my pc, sometimes the timer ends earlier or later
last page of the pdf , that is scary lol
@roganjosh Plot Twist: You don't know the timezone in which midnight is measured!
16:31
@CătălinaSîrbu Not really, no. How much precision do you need and why?
NooOOOOoo. It's always about timezone-naive issues. <angry fist-shaking>
@CătălinaSîrbu You can never avoid the time taking a bit longer than expected, since that hinges on how your OS schedules the process.
I would like miliseconds precision
Under which conditions does it end earlier, though?
Important question: Are you running Python2 or an old version of Python3?
python 2.7. I have a block and I want to move it along a "L" shape. Some times it follows the line ok, some times it turn earlier or later. The code is go forward, wait X ms, turn, wait Y ms, stop
16:35
Before Python3.5, various system calls including time.sleep may finish earlier than expected; see PEP 475. You must take care to manually check whether the sleep duration was long enough and if not sleep again.
@CătălinaSîrbu have we discussed yet what your good reason is to use python 2?
@CătălinaSîrbu That sounds like you should do it backwards: Don't sleep until you have to do something; sleep for however long you want, and then do everything that should've happened in the meantime at once. (I.e. when you wake up, figure out where the L should be right now and move it there)
Can you say again please ?
I didn't get it
Say you have an animation like this:
1) Start at (0, 0)
2) Over 5 seconds, move downwards toward (0, 100)
3) Over 2 seconds, move rightwards toward (30, 100)
What you're doing is equivalent to sleep(5); block.position = (0, 100); sleep(2); block.position = (30, 100). What you should do instead is elapsed_time = now() - animation_start_time; block.position = calculate_position_for_block_after_x_seconds(elapsed_time)
16:57
ok, thanks I will try
17:09
Is there a post explaing XY problems? I need to link it to someone.
@CătălinaSîrbu Expecting millisecond precision in a garbage-collected language is asking for trouble.
17:48
I have an array of ordered coordiantes that make up a polygon. Is there any numpy method that will retrieve all the outer points, that when connected, the lines won't intersect the existing lines of the polygon?
Another example:
@AnnZen without actually using it, this might help ConvexHull
@piRSquared Thank you!
18:36
For Completeness
from scipy.spatial import ConvexHull

x = np.random.randint(10, size=20)
y = np.random.randint(10, size=20)
h = ConvexHull([*zip(x, y)])
v = np.concatenate([h.vertices, [h.vertices[0]]])

plt.scatter(x, y)
plt.plot(x[v], y[v], 'r--')
18:50
hey, does anyone uses flask for uploading ?
I it works well with me but I think it overwrites the file if it has the same name
I kept looking over the document for what I can do about it, I didnt find any option for it.
I could right a function to loop through the entire folder and see if there is a file with the same name but I think there might be a less complicated solution for that.
just to clarify , I'm using https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/
not flask_uploads
 
3 hours later…
21:46
@smci about the same, tbh. Trying to pull ahead at work :)
22:43
Does python have some way to compose modules paths from a string, or at least programmatically? I think we can use get_attr but I wonder if there is a better way?
str = ".foo.bar"
import big_module.str
getattr won't work for imports, you need importlib.import_module
get getattr seems to work...
only if it's already imported
Yeah, it is..
Basically I need to rewrite or "monkey patch" some imported classes with others. So this works fine but I had to fix a bug with module name resolution...
Monkeypatching imported modules sounds too straightforward compared to your usual needs. I'd expect at least rewriting module files on the fly and recompiling them programmatically :P
22:55
I mean, I'm technically compiling a lot of performance critical code on the fly because the underlying modules invokes GLSL which the runtime graphics driver converts into its native format before running. ALthough before seriously shipping, I'll need to probably precompile them as a prebuild step.
much better
How many levels of abstraction are you you on today?
Really the thing that keeps me up at night is if cythonizing my code will cause the data types to change in a way that won't be compatible with pybind11. Or possibly even worse. They will be compatible with pybind11 BUT I'll have to write more code to support them :-(
Shouldn't your cythonization follow your original types? With the possible exception of integers I assume
Well things like py::object are wrappers around python's reference counted things. But with cython? IDK.
Custom objects with cython; does that even work well?
I'm not a cython user, whenever I rarely need acceleration I use numba or more recently pythran
23:01
Doesn't cython replace some of the reference counted python objects with its own stuff? Including integer types, for example.
but I only do that with numerical code anyway, where everything has a native type
@Mikhail no idea
I also have no idea but that won't stop me from talking :-P
I only have a vague impression which is heavily biased by my numerical use cases
"heavily biased" doesn't sound IEEE compliant

« first day (3816 days earlier)      last day (1357 days later) »