I'm thinking about db testing. Kinda caught between using one or two databases (eg empty/populated) for many tests, or spinning up a clean database for each test...
What I'm really caught up with is how do I test elements as a condensed unit, providing both the database data and the expected result, in a concise manner. Like if I create a default populated db fixture, the expected data that gets pulled out of it, defined in the test, is a long way from the test/returned data. Also, how to do yet not rely on direct sql statements to construct data in the db dynamically (kinda fragile), or rely on the db write code to operate properly (not ideal).
@Sam Not sure how your example goes from splitting decisions to the final terminal node (to me this seems like it could end up with multiple terminal nodes though some could end up converged). Either way, you might want to look at Classification and Regression Trees (CARTS) as they can be a good way of performing these type of decision trees (ones where splits can vary).
They basically consist of using a dataset with for loops to build k folds and cross-validated splits, calculating the Gini purity (index; again for loop), determining the best split point (this is the hard part and usually nested for loops), and just determining if a node is the terminal or not (few if/else statements)
I managed to realize that if the prediction is 1 if it's value is above 0.5, and 0 if it is below that (relying on: "If the prediction is >= 0.5, it is positive else it is negative." of tensorflow.org/tutorials/text/text_classification_rnn). I didn't manage to get probabilities, but what I have is good enough for now
my goal is multiclass classification, not just positive or negative, so that's what I am working on now
(I did the binary classification to have something which I can work on for multiclass)
will send update about multiclass in a short while hopefully :)
I am working on code for the Charcoal project and need help with a regex. How can I detect what is after watch in the following string: user12345678 watch test How can I extract test? Note that the username will change, watch can be watch or blacklist and there could be a comment after test in this form (?#this is a comment) which needs to be excuded.
test could have spaces but there will always be a space between user and test
There will be no space between test and the comment and the comment will always be given in (?#form)
@LinkBerest I don't see any reason not to have a "paid resources" section if there are books that we want to suggest. You're better-placed than me to judge what might be useful suggestions
I know I saw a comment from Martijn somewhere on an answer with "you can't go wrong with..." and then I forget the book title. :P I'll go hunt it down shortly
Both now closed but I don't think "typo" was fair for the second one, I think it was mostly unclear what you were asking. Anyway, it doesn't really matter :)
Thanks. :) Yeah, I came back once to the second one, and didn't even understand what I'd asked, myself. I made an attempt at clarifying it with an edit, after going through the question multiple times, and looking at the files in which I was learning Python at the time, but the issue itself still wasn't appropriate for SO.
No worries. Don't forget to clear up your comments. Particularly for the one that has 4k views - it seems people are landing on it so no reason to have any extra noise
I'm pretty sure it's the error message attracting people, which, doesn't really have anything to do with multiline comments, hence why it should just be deleted.
I really wouldn't be too concerned about it, honestly. It has upvoted answers, and the majority of people who find value in a thread don't upvote, so we can't really be sure whether people are actually finding it useful :)
But please do clear up the comments. There's no need for them now @Andreas
No, we know at least one person found it helpful. That's what I'm saying - most people won't upvote things they find useful. We can't really ever have a definitive idea on what helps others
I commited something but it isn't showing anywhere
dmanokhin@MacBook-Air SD % git push
To github.com/Daniil-M-beep/SmokeDetector.git
! [rejected] master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/Daniil-M-beep/SmokeDetector.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
@Andreas Eh, that's up to your judgement I guess. I've just noticed that you're one of the leaders for the charge against reactions. Perhaps that'll give you some metadata on who finds the answer useful :P <ducks>
Can somone tell me all of the paths that I need to add to add a virtual enviroment to path. I have "env\Lib\site-packages" so far which seems to normally work but seems to not be working for from lxml.etree import Element?
I'm making an effort at consistency because we occasionally have some users that are somewhat problematic, and I can't really be enforcing rules on some people and not others, even though your code was indeed well-formatted
That's fair enough. Genuinly. I'm sorry I keep breaking the rules. It's not intentional, I'm dylexic and so reading rulebooks is generally not easy for me.
def Run(ImportName):
import sys
sys.path.append(ImportName + "Source")
sys.path.append(ImportName + "Source/env/Lib/site-packages")
# import os #Currently not working (this bit of code is designed to bring working directory into running project)
# CL = os.getcwd()
# os.chdir(ImportName + "Source")
# sys.path.append("")
# sys.path.append("env/Lib/site-packages")
__import__(ImportName)
# os.chdir(CL)
I can see it, but stars should really be reserved for things that are interesting for the whole room (notice the star board on the right hand side) - my comment is now at the top, but it's not really useful for other people
Anyway, why are you trying to activate a virtual env inside a function?
because I've learned the hard way that it's a bad idea to install all modules globally.
I work in automation and so I need to create queues of different automated processes. I've therefore created virtual enviroments for each project and need to use the vertual enviroment's modules in each project/process.
Hence the Run function which I can stack in a "queue"
I appreciate you trying though @roganjosh the above function has seemed to work for me for everything else (before this mail merge). Is it possible that I just need to add one more path to the above to get 'from lxml.etree import Element' to import?
I'm just not sure where all of the places you need to import from are.
Part of me feels that the imports will be cached. You may be able to get away with it being in the function scope but I'm not confident enough to make assertions without being able to test
i don't use virtualenvs all that much but this feels like a roundabout way to run scripts in individual virtualenvs. Shouldn't the venv itself be responsible for making sure all the import statements point to the right path? Ideally, you would never append to sys.path yourself.
I don't, and I was about to say "how hard could it possibly be?" but whenever I say that, it turns out the answer is "very difficult indeed", so I won't say it.
docs.python.org/3/library/venv.html tells me that Windows venvs don't have a "bin" folder, they have a "scripts" folder. Do you have a "scripts" folder?
I still have a niggling question about why we're doing this at all? Why do these processes need to be in a dynamic queue and not just scheduled tasks? Are they really so CPU-intensive <or, insert some other restriction here> that they need queuing?
I've just created a venv directory with python3 -m venv path_goes_here, and inside the Scripts directory that is created, I see a file python.exe. I suspect that you can use this to run scripts within the venv
To do it within another python script, I suspect something along the lines of subprocess.run(["path_goes_here/scripts/python.exe", "script_name_goes_here.py"]) would do it
Usually when a subprocess call fails, it will print an explanation why. You could also check the returncode attribute of the object returned by run, and perhaps also its stdout/stderr
And as always you should double check that your try-excepts aren't catching errors without logging this fact somewhere
You would not believe how many questions there are on SO like "my program is failing, how do I get more diagnostic information?" when their code is like try: whatever(); except: pass
My little test script is working the way I expect it to, so it must be something that you're doing and I'm not doing, or vice versa
I don't have any guesses as to what that might be, so it may be time for you to compose an MCVE. In other words, a self-contained file or two that demonstrates how subprocess.call isn't working the way it should be
For example, here are the contents of my scripts where one calls the other successfully.
If subprocess was somehow invoking the wrong environment, I wouldn't expect it to run quickly and silently, I'd expect it to loudly crash with ImportError: can't find module thing_that_exists_only_in_the_venv
Ok, I just tested it out. I installed toolz in the venv and verified that it doesn't exist in my primary Python environment. I added import toolz to wait_five_seconds.py. Running wait_five_seconds.py by itself crashes with ModuleNotFoundError: No module named 'toolz'. Running it via subprocess.run([r"C:\Programming\Python 3.8\sample_venv\Scripts\python.exe", "wait_five_seconds.py"]), it executes with no problem.
This indicates that modules installed in the venv will be accessible to the script being run in that venv.
@Starter I'm not quite sure what you mean about not overriding but to compair two columns in two different files I would load them both into a list each and then create a third empty list (list3 = [])
i = 0
for line in list1:
if line == list2[i]:
line3.append('match')
else:
line3.append('does not match')
i = i + 1
@JamesMcIntyre you'll probably want to look into enumerate to get rid of that pesky counter, but don't big me up too much on the Python side; I'll defer to several other members here for the serious stuff :P
@JamesMcIntyre sorry for my lack of clarity. file1 has 4 columns, file2 has 6 columns, when I compare columns 1 and 2 in file1 to columns 3 and 5 in file2, if there is a match, I add column 3 in file1 to the row in file2 else I add NA. whats happens is that only NA's gets appended to the row even if there is a match. So I feel there is an override going on
Do you remember I asked about a cuda related problem? It seems that CUDA was installed by default (have a NVIDIA GPU which supports CUDA). However the version was old and I didn't know it mattered until I came across Pytorch. Once I updated the driver I was able to use CUDA.
Sorry that I was so clueless while asking the question.
@roganjosh lol. If I get some time after the start of the semester I'll look at adding my suggested reading list (after editing it as it's also based on what we have in our library)
@JamesMcIntyre @roganjosh, is it okay if I post just part of my code for you guys to evaluate my logic. I am thinking @JamesMcIntyre earlier comment may be helpful
I still think a CARTs approach would work on that earlier problem (I reach for that when db filters are not enough - usually dynamic enough if it just takes too many it's time to clean my data)
It's not quite what I imagined, but I think it's in the same category
Line 11 looks problematic. File objects can only be iterated once, after which they will only yield an empty list. Try moving f1_data = list(csv.reader((f1), delimiter="\t")) up a few lines so it's outside the loop.
what I was thinking was what if I added @JamesMcIntyre suggestion of using line count so no one line gets overridden or read after it has been processed. @kevin sure that would work if the number of comment lines remain constant and is known like you mentioned.
Ok, here is a quick prototype of a dict-based file merger. It won't work perfectly with your data since the formatting is different, but it should be possible to fix
My data contains no comments, so I don't have to pop anything. My data contains a unique primary key in the first column in both files, so it's easy to match up one against the other.
I see you doing a bunch of rsplits and stuff to determine whether two rows match, so I guess you don't have a simple primary key. But the approach should still work even for more complicated matching logic. It could be a column, or a portion of a column, or a combination of columns, whatever.
If you can't guarantee uniqueness at all, you need to think about what the output should be if a row in file2 matches more than one row in file1, and vice versa
@Kevin, I see exactly what you are trying to do. I think I can modify that. I can guarantee that uniqueness based on the column combinations am manipulating to find a match
@JamesMcIntyre This kind of "walking a copy of a list and modifying the original list" code can hide many bugs. For instance, when you pop the i'th element in a list, the indexes of your copy and original no longer match up relative to the list start, so your next pop will be off by one. You can fix by putting the i = i + 1 in an else. But cleaner is to use a list comprehension with a filtering condition in place of your copy.deepcopy.
no_comments_list = [line for line in original_list if not line[3].startswith('#')]
Also, using startswith is safer than indexing [0], in case of an empty string (which would raise an IndexError).
Rule of thumb: you almost never need to do i = i + 1 when iterating a list. zip and enumerate and list comprehensions eliminate 95% of common use cases.
For example you might do the_list = [row for row in the_list if line[3][0] != "#"]
Oops, PaulMcG beat me to it. Oh well, independent verification is good.
you make a good point @PaulMcG I've actaully come accross this before and was trying to get around it by creating this copy but you're right that it still has this same problem.
I do have some code which does this job correctly but it was a bit long to just copy and past
@PaulMcG @Starter if this helps at all, this is how I removed prevous matching entries in a list if it helps?
for line in reversed(Vdata):
if row[0] == line[15]:
if (datetime.datetime.strptime(line[6], "%Y-%m-%d").date() + timedelta(days=31) > datetime.date.today()) or (datetime.datetime.strptime(line[2], "%d/%m/%Y").date() + timedelta(days=31) > datetime.date.today()):
r = True
break
else:
a = a + 1
else:
a = a + 1
@Kevin
This run fucntion not working is doing my head in. Any chance I can share my screen with you to show you what's going on because it just looks weird to me :/
Say I'm looking at a question and see that an edit is pending. I go check it out, make a change or two, and hit Improve Edit so that it posts immediately. Does the person who first suggested the edit get any rep, or is it lost?
all I am doing is just compare multiple columns in 2 files. i.e file1 and file2, if there is a match take a field from another column of the matching line in file1 and append it to the matching line in file2 if not append NA. I am not sure why my tiny brain can't get a handle on this. I feel dumb sometimes
@Starter trust me we all do somtimes. With programming you either feel like your a god or a worm's pet. Right now I defeinetly feel like the latter considering I can't even get the thing that I've created and works greate on my own computer to run on a virtual machine
I got dotdotatdotatdotdot.at at a rather low point in my life, and I would start laughing every time I'd log in the the server control panel, so it's been a big help. I've used it for throwaway addresses online, but I'm just waiting for the opportunity when there's no one in line behind me and the salesperson asks for my email address.
I was about to suggest that. But it might be something else, because I still get 403 when I apply the user agent in the manner suggested by stackoverflow.com/questions/24226781/…
@JohnnyApplesauce have you tried my code with requests? It's also possible that if you've pinged the server with bad requests too many time, they may have temporarily blacklisted you. You might need to wait 5-15 minutes...
@JamesMcIntyre Yep. After tinkering with it, I got ModuleNotFoundError: No module named 'mailmerge'. This indicates that subprocess is successfully running CommercialDDConfirmation.py.
I was getting no output at all when I used the subprocess. It just looked like it had ran but actually did nothing.
you can see that subprocess is currently disabled but there are other bits that are enabled which are allowing the running of the file and the import of cx_Oracle for instance
Right. I commend out the __import__ bit and uncommented the subprocess bit, because that's the approach I'm interested in. so subprocess does indeed produce output on my machine.
For the record, exactly what I'm running is:
import os
def Run(ImportName):
import sys
sys.path.append(ImportName + "Source")
sys.path.append(ImportName + "Source/env/Lib/site-packages")
os.chdir(ImportName + "Source")
sys.path.append("")
sys.path.append("env/Lib/site-packages")
import subprocess #This is a suggestions from the chatroom
print(os.listdir())
subprocess.run(["env/Scripts/python.exe", ImportName + ".py"])
I changed some of the paths because I was getting "file not found" otherwise
I decided to create a fresh env because I didn't think the python executable from your computer would run on mine. Differing architectures, that kind of thing.
My guess is that it's probably not an architecture incompatibility, since the OS or Python would probably say "I can't run this!"
Now that I've pip installed mailmerge, I'm getting ImportError: cannot import name 'MailMerge' from 'mailmerge'
Oh, it's probably docx-mailmerge I'm supposed to install. Now I'm getting ModuleNotFoundError: No module named 'cx_Oracle'. This implies to me that the mailmerge import succeeded, and now it's import cx_Oracle in Functions.py that's failing.
Perhaps this is a silly question, but how can you tell the difference between "the program executed successfully", and "the program exited without doing anything"? As far as I can tell, you don't call print() at any point, so wouldn't those two outcomes look the same?
There's no shame in liberally sprinkling print("got this far") throughout one's code base to see how many show up on the console
@MattDMo yes, "improve edit" == "approve edit single-handedly and apply your own", "reject and edit" == "reject edit single-handedly and apply your own". There's nothing special about the approve/reject in this situation
If I return early from GetVDDs and RemoveWorked, then MakePDFs crashes with NameError: name 'MailMerge' is not defined. Not too surprising, since importing MailMerge in CommercialDDConfirmation.py won't make it visible in Functions.py
When you say "it fails on trying to import the mail merge" I assume you mean it fails when you are using the __import__ based approach rather than the subprocess approach. That makes sense to me. Modules that are installed as packages have different import logic than simple directories containing .py files. __import__ can't find lxml.etree because it has no reason to think that lxml is a package.
Getting nested directory imports to work has always been a tremendous headache for me, which is why I've been campaigning so hard for the subprocess approach. The venv handles the packaging for you.
I'm quite happy with using the subprocess method but when I do that as you can see above it just produces nothing. I even just put in a print('reached here') at the end of the CommercialDDConfirmation.py file but it didn't print anything.
create a bool Valid (for example) and after the line triggers, use break or return, alternatively if you wanted it to run a set amount of times create an int X and when it triggers X += 1, and then use an if statement, if (X == 8): break/return
it usually is bad code tbh, if it runs with no obvious flaws try an accept it but if theres glaring mistakes then get them out of there chair, sit down and just rework that s**t