I have such 2d numpy array: [[1. 2. 4. 6.] [1. 2. 4. 6.]] How can I check if 1d numpy array [1, 2, 4, 6] is in that one above?
I was thinking this will give me what I want np.isin(lines, np.array([1, 2, 4, 6])), but it made smth like this: array([[ True, True, True, True], [ True, True, True, True]])
ok. Got it.: [1, 2, 9, 6] in lines.tolist(). Is it efficient?
@AndrasDeak what about such case: np.array([1, 2, 9, 6]) in lines.tolist()? Im getting an error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
when I want to check if numpy array exists in 2d array
it feels like 5000 files on the database ( saved as ACID ) is more likely pressure the server on the long run.
@AndrasDeak lol, I meant if it's a safe decision to make
i expect to have anywhere from 500-5000 a month in files
basically what I know so far
A: files in its own format ( less secure , better performance , backing up db doesn't include the files , files directory could be modified in case of being hacked ) B: files on Database ( ACID - Bolb ) ( more secure , less performance , backup db would include the bolb data along with it , harder to hack)
I feel like keeping the files in it's own format would also be less likely to get corrupted by the system itself. ( ex: if the files needed more than 256 character it may not be able to store the whole bolb correctly. )
is there is anyway to scan the files before dealing with them ?
What is "its own format", especially in light of it being truncated to 256 characters?
And if I was all up in your server, what is your database going to help you with vs. flat files? You'd need an encryption layer and I'd already have the key to that, wouldn't I?
its own format ( like .pdf , .xlsx , .jpg , .png ,... )
about the **256 character** , correct me if i'm wrong , to convert any file to **bolb** it would create a big string and store it in one of the database column of that row
ok that is better, I thought some files would go along with the character limit and others wont because they are bigger, those I should be more careful there.
" You'd need an encryption layer and I'd already have the key to that, wouldn't I?"
what do you mean by encryption layer ? ssl ? also I didn't understand how would you have access to the key of it ?
@LoopingDev I'd have the encryption key because I've just hacked your server, so I can see exactly where all your code loads the key from, and I could just read that file myself. And no, I don't mean SSL, I mean encryption of the data within the database
Once the server is compromised, the attacker can see the encryption key
Yeah, I'm new, I'm only doing it because you said if the server is compromised and I don't have any encryption on the database, it would be less likely for the database to be secured for much longer
oh!, I see now. so even If i encrypted the database, It would be all for nothing.. so I'd be better off preventing the server from compromising, is that right ?
@roganjosh Meh, my host just vanished one day, took everything he had. I could claim the last few payments back, since it was over paypal, but still. There are some shady hosters out there
yup, I have plans on checking on with google cloud and Amazon as first priorities , those guys already have all our data, so we wont be risking any new data to begain with lol
@LoopingDev you might want to revisit your requirements. .pdf, .xlsx, .jpg, .png are just file suffixes, but they stand for data formats. When you put the data from a .png file into a database, it will still be PNG data.
It worked perfect for a year with low cost, then I guess he said ** it, because I couldn't even find the company anymore. Luckily it was just my moms website read only nothing important
@MisterMiyagi Yes, I understand that, but i'm yet to make a decision about which way I should store the files? .. as data or as files?
also how to scan the files before being uploaded to the server ( the server would be a linux based server, so its less likely to get the virus but it still can distribute it to the windows users as jpg or pdf for example)
Are we talking about the database server getting a virus? How would that happen? As long as all it does is store and retrieve data in the database, there's literally 0 risk of a file causing damage, even if it's a virus or trojan or whatever
In general, it is better to separate data from metadata. Often enough, you want to work just with metadata to then find out which subset of data you need. When you conflate the two, you are working with much more data as needed.
For example, when you want to find "the newest 5 files", you only need path+age. You do not want to read the content of all files.
It's still possible to separate data and metadata in a Database, but many DBs do not offer an advantage over files for storing huge blobs of data.
I'm still not really sure why you think it's your responsibility to detect malware. Even if a virus is sent through your database to someone's PC, that PC will have a virus scanner anyway.
no very likely, I expect the users that would be using this system would more likely to have a virus on their own computer and they don't know how to get it out. they just keep the slow pc forever.
in other words, I expect the users of this system to be very new to technology in general, and unfortunately, that is very likely
Consider this: If your service claims to protect against viruses and it does not do so thoroughly, you expose users to more of a threat because they will assume to trust your data.
@MisterMiyagi thank you... but this seems to be a tough problem in APIs written to link commercial softwares to python packages. Abaqus python API, CSI SAP2000 python API link, AutoLISP functions can not be called in separate programs for AutoCAD, etc...
Softwares APIs are not accessible outside the softwares running, their are not also available through pip. So, they are in python, but can hardly be accessible outside the software. this is the problem.
@MisterMiyagi yes, but I am not a developer of those commercial softwares. I should find a way to write my code to get the results and pass to another project... so it is tough going through fundamentals of closed API packages...
honestly it kinda odd that this is not old news. How does anyone secure their applications from such attacks using python? there has to be a way, if there is not a way I think there would have been no system that would last long without distributing masses of viruses to its clients
i thought they would be a service for free or paid that checks the uploaded files by now to say the least.
@LoopingDev because softwares links. APIs in commercial engineering softwares act as link to give the user freedom to do parametric modeling. So they are not main software.
@enthu Look, the way Python executes packages is easy. If you have trouble to grok this, and your messages suggest as much, you miss the groundwork to venture towards more complex tasks.
@LoopingDev for instance, enginering packages like openfem are all in python. commercial software like Abaqus is not in python. API for abaqus is an added feature to give user freedom to do systematic coding and automation. that is why the python api does not come with pip. it comes when the software is installed.
@enthu I got what you mean, so in short we might be able to scan the files using api that connects the python application to the anti-virus scanner that might be installed on the same server as well, am I correct?
@enthu is trying to interface with some software that happened to drop a bunch of .py files on Windows, @LoopingDev is talking about file uploads on a website and them potentially being executed. There is almost zero cross-over between your discussions
"The Eye-Fi card can connect to it and will upload images to the folder specified in the config. EyeFlask attempts to verify the file integrity using the same security protocols used by Eye-Fi Server."
erm, "integrity" usually means something else than "security".
Basically, it checks whether all the bits and pieces are there as claimed by the sender. It doesn't actually check whether any of these pieces are gremlins trying to eat your hardrive.
Hello, is there any way of using a timer in an app which won't be affected by the numbers of prcesses which run on the computer? For ex if I use a timer for 1.234 secs, having more or less apps opened on my pc, sometimes the timer ends earlier or later
python 2.7. I have a block and I want to move it along a "L" shape. Some times it follows the line ok, some times it turn earlier or later. The code is go forward, wait X ms, turn, wait Y ms, stop
Before Python3.5, various system calls including time.sleep may finish earlier than expected; see PEP 475. You must take care to manually check whether the sleep duration was long enough and if not sleep again.
@CătălinaSîrbu That sounds like you should do it backwards: Don't sleep until you have to do something; sleep for however long you want, and then do everything that should've happened in the meantime at once. (I.e. when you wake up, figure out where the L should be right now and move it there)
Say you have an animation like this: 1) Start at (0, 0) 2) Over 5 seconds, move downwards toward (0, 100) 3) Over 2 seconds, move rightwards toward (30, 100)
What you're doing is equivalent to sleep(5); block.position = (0, 100); sleep(2); block.position = (30, 100). What you should do instead is elapsed_time = now() - animation_start_time; block.position = calculate_position_for_block_after_x_seconds(elapsed_time)
I have an array of ordered coordiantes that make up a polygon. Is there any numpy method that will retrieve all the outer points, that when connected, the lines won't intersect the existing lines of the polygon?
from scipy.spatial import ConvexHull
x = np.random.randint(10, size=20)
y = np.random.randint(10, size=20)
h = ConvexHull([*zip(x, y)])
v = np.concatenate([h.vertices, [h.vertices[0]]])
plt.scatter(x, y)
plt.plot(x[v], y[v], 'r--')
hey, does anyone uses flask for uploading ? I it works well with me but I think it overwrites the file if it has the same name I kept looking over the document for what I can do about it, I didnt find any option for it.
I could right a function to loop through the entire folder and see if there is a file with the same name but I think there might be a less complicated solution for that.
just to clarify , I'm using https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/ not flask_uploads
Does python have some way to compose modules paths from a string, or at least programmatically? I think we can use get_attr but I wonder if there is a better way?
Basically I need to rewrite or "monkey patch" some imported classes with others. So this works fine but I had to fix a bug with module name resolution...
Monkeypatching imported modules sounds too straightforward compared to your usual needs. I'd expect at least rewriting module files on the fly and recompiling them programmatically :P
I mean, I'm technically compiling a lot of performance critical code on the fly because the underlying modules invokes GLSL which the runtime graphics driver converts into its native format before running. ALthough before seriously shipping, I'll need to probably precompile them as a prebuild step.
How many levels of abstraction are you you on today?
Really the thing that keeps me up at night is if cythonizing my code will cause the data types to change in a way that won't be compatible with pybind11. Or possibly even worse. They will be compatible with pybind11 BUT I'll have to write more code to support them :-(