« first day (3561 days earlier)      last day (1600 days later) » 
00:00 - 19:0019:00 - 00:00

I'm thinking about db testing. Kinda caught between using one or two databases (eg empty/populated) for many tests, or spinning up a clean database for each test...
What I'm really caught up with is how do I test elements as a condensed unit, providing both the database data and the expected result, in a concise manner. Like if I create a default populated db fixture, the expected data that gets pulled out of it, defined in the test, is a long way from the test/returned data. Also, how to do yet not rely on direct sql statements to construct data in the db dynamically (kinda fragile), or rely on the db write code to operate properly (not ideal).
1:21 AM
@roganjosh It's okay (all the books are Python 3 and still good) we could add some to it though
@anafitch Any book on this list of undergrad computer science topics - several are purely Python focused and free
err....several were free (seems that sale ended)
@Ffisegydd cbg Fizzy - good to see you after so long :)
1:54 AM
@Sam Not sure how your example goes from splitting decisions to the final terminal node (to me this seems like it could end up with multiple terminal nodes though some could end up converged). Either way, you might want to look at Classification and Regression Trees (CARTS) as they can be a good way of performing these type of decision trees (ones where splits can vary).
They basically consist of using a dataset with for loops to build k folds and cross-validated splits, calculating the Gini purity (index; again for loop), determining the best split point (this is the hard part and usually nested for loops), and just determining if a node is the terminal or not (few if/else statements)
 
2 hours later…
4:22 AM
Yet another dupe of f-p arithmetic inexact, this time using decimal package. What's the best dupe target? stackoverflow.com/questions/62927128/…
4:44 AM
Hi Everyone
 
3 hours later…
7:44 AM
@Sam yes, that's correct, I was looking for some probabilities
@LinkBerest thanks! Now I have a more complex issue, I' am phrasing it to a question.
Sam
Sam
@zabop did you manage to fix it?
I managed to realize that if the prediction is 1 if it's value is above 0.5, and 0 if it is below that (relying on: "If the prediction is >= 0.5, it is positive else it is negative." of tensorflow.org/tutorials/text/text_classification_rnn). I didn't manage to get probabilities, but what I have is good enough for now
my goal is multiclass classification, not just positive or negative, so that's what I am working on now
(I did the binary classification to have something which I can work on for multiclass)
will send update about multiclass in a short while hopefully :)
Sam
Sam
8:00 AM
Did you try adding the activation to the final dense layer like I said?
yeah
it didn't push it to between 0 and 1
(to my surprise as well)
Sam
Sam
something doesn't feel right at all there.
yeah I know...
working on it
Sam
Sam
ping me if you need any assistance :)
thanks! :)
8:16 AM
question about numpy
import numpy as np

inputs = [ 1 , 2 , 3 , 2.5 ]
weights = [[ 0.2 , 0.8 , -0.5 , 1 ],[ 0.5 , -0.91 , 0.26 , -0.5 ], [ -0.26 , -0.27 , 0.17 , 0.87 ]]
biases = [ 2 , 3 , 0.5 ]



dot_product = np.dot(weights, inputs) + biases

dot_product
why causes dot_product = np.dot(inputs, weights) + biases this error ?
ValueError: shapes (4,) and (3,4) not aligned: 4 (dim 0) != 3 (dim 0)
just for understanding
got it the number of columns in the first matrix must be equal to the number of rows in the second matrix
Sam
Sam
8:30 AM
works for me
Oh sorry, looking at the wrong part
 
1 hour later…
user12867493
9:35 AM
I am working on code for the Charcoal project and need help with a regex. How can I detect what is after watch in the following string: user12345678 watch test How can I extract test? Note that the username will change, watch can be watch or blacklist and there could be a comment after test in this form (?#this is a comment) which needs to be excuded.
user12867493
test could have spaces but there will always be a space between user and test
user12867493
There will be no space between test and the comment and the comment will always be given in (?#form)
user12867493
Thanks for the help
Does it have to be regex? Can't you just split on space?
user12867493
Possibly, let me check
user12867493
9:39 AM
But test could have spaces in it
user12867493
And there's no space between test and the comment
user12867493
And the username could have spaces
user12867493
Example: User 123: watch test string(?#comment)
user12867493
I want test string
user12867493
I should also note there's always a colon after the user
9:40 AM
If the user name can contain spaces and watch isn't a fixed text either, how the heck is this supposed to work?
ah
user12867493
I assume you can do a lookbehind to test the text for watch/blackist and exclude the comment after?
user12867493
It's almost guaranteed the username won't contain watch/blacklist but it's good to check for a colon just in case
something like .*?: \S+ (.*?)(?:\Z|\(?#) I guess
user12867493
And then what do I do, a match()?
yeah
and then a .group(1)
user12867493
9:48 AM
👍 Thanks
actually, change \(?# to \s*\(?#
user12867493
Then it doesn't work properly
user12867493
user 123: blacklist test sds(?#test)
user12867493
Group 1 is test sds(?
oops, forgot to escape the ?
\s*\(\?#
10:05 AM
@LinkBerest I don't see any reason not to have a "paid resources" section if there are books that we want to suggest. You're better-placed than me to judge what might be useful suggestions
I know I saw a comment from Martijn somewhere on an answer with "you can't go wrong with..." and then I forget the book title. :P I'll go hunt it down shortly
user12867493
10:23 AM
@Aran-Fey Thanks
user12867493
Does anyone have experience with git/GitHub here?
who doesn't?
user12867493
Me :)
user12867493
I created some changes locally but don't know how to PR them
I have to google that every time myself: docs.github.com/en/github/…
10:30 AM
self.pos.pop(val, 0)
anyone seen that before? whats the 0 for?
its not in the docs
because you're looking at the wrong docs. Try looking at dict.pop
user12867493
@Aran-Fey I can't create a branch
sorry
Then create a fork
@Permian See this
user12867493
10:33 AM
I already have another fork of the same repo
@roganjosh thanks
@toonarmycaptain I've been stuck on your last sentence for some time now. I don't think I get what you're asking
11:00 AM
Hello. Am I allowed to ask for some assistance closing two old Python questions of mine, that are off-topic, and have answers?
Preferably, I'd want to have them deleted, but closing them at least gets the message out that these questions are not appropriate for SO.
sure
@Aran-Fey Thanks. These two
That last one still surprises me every time I see it; I don't understand how I failed to understand such a simple thing.
11:15 AM
Both now closed but I don't think "typo" was fair for the second one, I think it was mostly unclear what you were asking. Anyway, it doesn't really matter :)
Thanks. :) Yeah, I came back once to the second one, and didn't even understand what I'd asked, myself. I made an attempt at clarifying it with an edit, after going through the question multiple times, and looking at the files in which I was learning Python at the time, but the issue itself still wasn't appropriate for SO.
No worries. Don't forget to clear up your comments. Particularly for the one that has 4k views - it seems people are landing on it so no reason to have any extra noise
I'm pretty sure it's the error message attracting people, which, doesn't really have anything to do with multiline comments, hence why it should just be deleted.
Anyway. Thanks for helping.
I really wouldn't be too concerned about it, honestly. It has upvoted answers, and the majority of people who find value in a thread don't upvote, so we can't really be sure whether people are actually finding it useful :)
But please do clear up the comments. There's no need for them now @Andreas
Well; I gave an upvote on both of them quite some time ago, so we know there's only 1 other person that may have found them helpful.
11:30 AM
No, we know at least one person found it helpful. That's what I'm saying - most people won't upvote things they find useful. We can't really ever have a definitive idea on what helps others
@roganjosh Already did on the other one. Shouldn't I leave the one with links to two other questions?
user12867493
I need help with git
user12867493
I commited something but it isn't showing anywhere
user12867493
dmanokhin@MacBook-Air SD % git push
To github.com/Daniil-M-beep/SmokeDetector.git
 ! [rejected]            master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/Daniil-M-beep/SmokeDetector.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
@Andreas Eh, that's up to your judgement I guess. I've just noticed that you're one of the leaders for the charge against reactions. Perhaps that'll give you some metadata on who finds the answer useful :P <ducks>
11:41 AM
@roganjosh "One of the leaders" :P :D
Well, you now have 600 and 1 rallied behind you :P
@roganjosh All support is greatly appreciated. :P It'll be interesting to see SE's decisions and data after ending the feature test tomorrow.
I think it's safe to say I've changed my mindset/approach on this site.
Eh, there's little enclaves of sanity. This room, hopefully, one of them :)
Can somone tell me all of the paths that I need to add to add a virtual enviroment to path. I have "env\Lib\site-packages" so far which seems to normally work but seems to not be working for from lxml.etree import Element?
You need to activate the environment
11:57 AM
what would the Python code be for that?

for example, my run function is currently this:
Can you please put that in a gist/dpaste/other and link here, as per the room rules?
I'm confused. I thought you said before that I should put it in here with ctrl+k. I just reminded myself of that rule by going to that place
"If your code is longer than about 12 lines, use an external paste tool such as dpaste.com."
Sorry, I can make it shorter than 12 lines by removing the try
I'm making an effort at consistency because we occasionally have some users that are somewhat problematic, and I can't really be enforcing rules on some people and not others, even though your code was indeed well-formatted
12:06 PM
That's fair enough. Genuinly. I'm sorry I keep breaking the rules. It's not intentional, I'm dylexic and so reading rulebooks is generally not easy for me.
def Run(ImportName):
    import sys
    sys.path.append(ImportName + "Source")
    sys.path.append(ImportName + "Source/env/Lib/site-packages")
#    import os #Currently not working (this bit of code is designed to bring working directory into running project)
#    CL = os.getcwd()
#    os.chdir(ImportName + "Source")
#    sys.path.append("")
#    sys.path.append("env/Lib/site-packages")
    __import__(ImportName)
#        os.chdir(CL)
I hope the tone of my comment shows that there's no reason to apologise - it happens :)
^ I tried to star that as a "thumbs up" but I'm not sure if you could see that @roganjosh
I can see it, but stars should really be reserved for things that are interesting for the whole room (notice the star board on the right hand side) - my comment is now at the top, but it's not really useful for other people
Anyway, why are you trying to activate a virtual env inside a function?
because I've learned the hard way that it's a bad idea to install all modules globally.

I work in automation and so I need to create queues of different automated processes. I've therefore created virtual enviroments for each project and need to use the vertual enviroment's modules in each project/process.

Hence the Run function which I can stack in a "queue"
Interesting. So you have some global queue for completely disparate processes?
12:16 PM
yes
Is any of this helpful? I can't say that I've been in this situation
It could be but I'm confused. I don't have a 'bin' folder in my env. just an 'include' 'lib' and 'scripts'
Are you on Windows?
Unfortunatly yes :(
12:36 PM
Grr, I don't have the bandwidth (in the literal, internet sense. I'm tethered to my phone) to start playing around with environments to test :/
I appreciate you trying though @roganjosh the above function has seemed to work for me for everything else (before this mail merge). Is it possible that I just need to add one more path to the above to get 'from lxml.etree import Element' to import?

I'm just not sure where all of the places you need to import from are.
Part of me feels that the imports will be cached. You may be able to get away with it being in the function scope but I'm not confident enough to make assertions without being able to test
12:51 PM
i don't use virtualenvs all that much but this feels like a roundabout way to run scripts in individual virtualenvs. Shouldn't the venv itself be responsible for making sure all the import statements point to the right path? Ideally, you would never append to sys.path yourself.
Aye that's fair enough @Kevin but do you know how to run (with Python) a .py file with a specific python environment without a 'bin' folder?
I don't, and I was about to say "how hard could it possibly be?" but whenever I say that, it turns out the answer is "very difficult indeed", so I won't say it.
docs.python.org/3/library/venv.html tells me that Windows venvs don't have a "bin" folder, they have a "scripts" folder. Do you have a "scripts" folder?
I still have a niggling question about why we're doing this at all? Why do these processes need to be in a dynamic queue and not just scheduled tasks? Are they really so CPU-intensive <or, insert some other restriction here> that they need queuing?
becuase it you shedule them then you don't know when you would schedule the next one for as we don't know when the last one would have finished
(the idea is that eventually this machine will spend basically all of it's time doing one automation or anouther automatically)
So, you have a cascade of processes that have to be run in order, but each with their own individual venv?
1:06 PM
I've just created a venv directory with python3 -m venv path_goes_here, and inside the Scripts directory that is created, I see a file python.exe. I suspect that you can use this to run scripts within the venv
To do it within another python script, I suspect something along the lines of subprocess.run(["path_goes_here/scripts/python.exe", "script_name_goes_here.py"]) would do it
@roganjosh yes. doesn't nesserily need to have their own enviroment but not doing that can casue conflics unless work around anouther way

I'll give that a try @Kevin, thanks
Would doing it that way wait util it had finished running before moving on though? (like an import does)
Yeah. subprocess.run waits until the process exits, so script_name_goes_here.py will execute completely before moving on
when I tried to run it like that it said it have finished running very quicky but didn't actually do anything
1:28 PM
Usually when a subprocess call fails, it will print an explanation why. You could also check the returncode attribute of the object returned by run, and perhaps also its stdout/stderr
And as always you should double check that your try-excepts aren't catching errors without logging this fact somewhere
You would not believe how many questions there are on SO like "my program is failing, how do I get more diagnostic information?" when their code is like try: whatever(); except: pass
aye, I've removed my try: except: for debugging but it still just ran to completion
My little test script is working the way I expect it to, so it must be something that you're doing and I'm not doing, or vice versa
I don't have any guesses as to what that might be, so it may be time for you to compose an MCVE. In other words, a self-contained file or two that demonstrates how subprocess.call isn't working the way it should be
For example, here are the contents of my scripts where one calls the other successfully.
did you have modules in your enviroment which the scrip your calling needs and that are only in that enviroment?
1:43 PM
No, I didn't think it was relevant
If subprocess was somehow invoking the wrong environment, I wouldn't expect it to run quickly and silently, I'd expect it to loudly crash with ImportError: can't find module thing_that_exists_only_in_the_venv
Ok, I just tested it out. I installed toolz in the venv and verified that it doesn't exist in my primary Python environment. I added import toolz to wait_five_seconds.py. Running wait_five_seconds.py by itself crashes with ModuleNotFoundError: No module named 'toolz'. Running it via subprocess.run([r"C:\Programming\Python 3.8\sample_venv\Scripts\python.exe", "wait_five_seconds.py"]), it executes with no problem.
This indicates that modules installed in the venv will be accessible to the script being run in that venv.
hmm... thanks @Kevin! let me poke around with this some more and see if I can get that working as well
Any idea what I'm doing wrong?:
2:00 PM
I have a question about comparing columns in 2 files. How do you avoid overriding a comparison made already in a for loop.
def Run(ImportName):
    import sys
    sys.path.append(ImportName + "Source")
    sys.path.append(ImportName + "Source/env/Lib/site-packages")
    import subprocess
    subprocess.run([ImportName + r"Source/env/Scripts/python.exe", ImportName + ".py"])
@Starter You're going to need to be clearer than that, I'm afraid. Please give a MCVE and post it off-site to link here if it's quite long
@roganjosh okay. I am still learning the ropes. Is there a specific off-site place I can use?
@Starter dpaste isn't a bad shout, but you might already have a github account where you could post a gist
@Starter I'm not quite sure what you mean about not overriding but to compair two columns in two different files I would load them both into a list each and then create a third empty list (list3 = [])
i = 0
for line in list1:
    if line == list2[i]:
        line3.append('match')
    else:
        line3.append('does not match')
    i = i + 1
2:07 PM
I wanted to say "horses for courses" there in terms of pasing but I'm not sure how colloquial that is
we have that saying in Scotland :)

@roganjosh is much more experienced than me in Python so if he says my way is not a good way of doing it then he is probably right :P
@Kelvin any idea what I'm doing wrong as each time I use the above function it runs with no error but does nothing?
Did the above answer your question @Starter?
@JamesMcIntyre you'll probably want to look into enumerate to get rid of that pesky counter, but don't big me up too much on the Python side; I'll defer to several other members here for the serious stuff :P
@JamesMcIntyre sorry for my lack of clarity. file1 has 4 columns, file2 has 6 columns, when I compare columns 1 and 2 in file1 to columns 3 and 5 in file2, if there is a match, I add column 3 in file1 to the row in file2 else I add NA. whats happens is that only NA's gets appended to the row even if there is a match. So I feel there is an override going on
@roganjosh, I am still looking at the logic
Am I right in suspecting that this is using Pandas?
nope I am avoiding pandas my file is humongous
and they are not sorted
well about 400, 000 lines in one and 200,000 in the other
2:19 PM
mm, so many questions. You're going to need to give us the MCVE that I suggested
400K lines is not humongous btw
@JamesMcIntyre Hmm, I can't tell from just that code. You're going to need to give us the MCVE that I suggested
c-c-c-combo
I'm a little fuzzy on how exactly Run is being run, i.e. from where and with what arguments
yes I am creating the MCVE as we speak
@Kevin ... Did I Kevin Kevin (albeit on a different "thread")? :P
I would move your message to the Ouroboros to dispose of the evidence, if I didn't think it would spark a war of all against all
@Kevin it's just being run as:
Run('CommercialDDConfirmation')
2:27 PM
Ok. I was worried you were doing, like, Run('math')
* my project folder is called CommercialDDConfirmationSource this is where CommercialDDConfirmation.py is
@Kevin au contraire. We shall erect a maypole to rejoice :P
Hello @roganjosh
And we can get a good discount on the maypole, since it's the off season.
Only suckers buy maypoles in May, you have to be either early or late for the best deals
@roganjosh those books were being offered free because covid but that ended on the 1st and I didn't realize it :)
2:31 PM
Do you remember I asked about a cuda related problem? It seems that CUDA was installed by default (have a NVIDIA GPU which supports CUDA). However the version was old and I didn't know it mattered until I came across Pytorch. Once I updated the driver I was able to use CUDA.
Sorry that I was so clueless while asking the question.
Becoming less clueless over time is a good thing :-)
^ tis our continuous battle
@LinkBerest still, I think you might have some good suggestions that we could put into the guide
@Kevin Thanks, I hope one day I will be as good as you all to help a lot of ppl
@roganjosh lol. If I get some time after the start of the semester I'll look at adding my suggested reading list (after editing it as it's also based on what we have in our library)
@JamesMcIntyre @roganjosh, is it okay if I post just part of my code for you guys to evaluate my logic. I am thinking @JamesMcIntyre earlier comment may be helpful
2:43 PM
I'm happy to have a look
@Starter Go for it. Remember to use pastebin or similar if the code is long (say, more than 12 lines)
I still think a CARTs approach would work on that earlier problem (I reach for that when db filters are not enough - usually dynamic enough if it just takes too many it's time to clean my data)
I'm curious how you're matching up the rows of the two files, considering they have a different number of lines
I wager 0.25 quatloos on "double for loop"
@Kevin if I was needing to see if there were any matches at all rather than than one line matching then I've always done a double for loop too :P
It's a fine approach, although in this scenario it would take 80,000,000,000 iterations, which is a bit much
2:51 PM
here is the partial code for merging pastebin.com/gjuNBTEd
@Kevin you are right its a double for loop
It's not quite what I imagined, but I think it's in the same category
Line 11 looks problematic. File objects can only be iterated once, after which they will only yield an empty list. Try moving f1_data = list(csv.reader((f1), delimiter="\t")) up a few lines so it's outside the loop.
Oh true
The drawback of this solution is that it will take 80 billion iterations to complete, rather than the ~600,000 you currently have
Perhaps a nice dict would help correlate rows more quickly
@kevin thats a lot of iterations. you mean using dicts will be quicker?
I am not sure how to cast it as a dict though
would the alternative be to load the files into lists as lists can be double iterated?
2:59 PM
Not your fault. I'm being vague in my explanation because I'm not fully sure what the program is supposed to be doing.
@JamesMcIntyre Essentially yes. That's what f1_data = list(csv.reader((f1), delimiter="\t")) is effectively doing, but it needs to be done only once.
ahh yes, so it's actually the list that's being iterated in the first place then?

It's all got a bit complicated for my understading. I normally just do the below to get a lists and then work with the lists:
with open(Source, newline='') as csvfile: #Open latest full export file
    data = list(csv.reader(csvfile))
    headers = data.pop(0)
*Source = "Example.csv"
@JamesMcIntyre what do you do if you have multiple lines containing > or # at the start of your file and you want to skip them?
pop(0) just skips the first line I thought
in this case, I am reading lines after those containing that particular char
It skips the first line and also removes it.

If you did data.pop(0) again it would remove the then top line (as the previous top line has already been removed)
i.e. in the above example, the headers are no longer in data but they are in headers
okay but that is if it is only one line header. Am I correct?
mine is more like multiple comment lines that I do not need in the computation
If you know exactly how many comment lines there are, for _ in range(number_of_comment_lines): data.pop(0)
3:14 PM
what I was thinking was what if I added @JamesMcIntyre suggestion of using line count so no one line gets overridden or read after it has been processed. @kevin sure that would work if the number of comment lines remain constant and is known like you mentioned.
If you were looking at the 4th column:
Ok, here is a quick prototype of a dict-based file merger. It won't work perfectly with your data since the formatting is different, but it should be possible to fix
import copy

ListCopy = copy.deepcopy(List)

i = 0
for line in ListCopy:
    if line[3][0] = '#':
        List.pop(i)
    i = i + 1
My data contains no comments, so I don't have to pop anything. My data contains a unique primary key in the first column in both files, so it's easy to match up one against the other.
I see you doing a bunch of rsplits and stuff to determine whether two rows match, so I guess you don't have a simple primary key. But the approach should still work even for more complicated matching logic. It could be a column, or a portion of a column, or a combination of columns, whatever.
If you can't guarantee uniqueness at all, you need to think about what the output should be if a row in file2 matches more than one row in file1, and vice versa
@Kevin, I see exactly what you are trying to do. I think I can modify that. I can guarantee that uniqueness based on the column combinations am manipulating to find a match
3:28 PM
Oh good, because many-to-many relations are a whole can of worms
I know thats what makes this so challenging
I am trying to be as efficient as possible but it seem like that maybe tougher than I thought
Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
@JamesMcIntyre This kind of "walking a copy of a list and modifying the original list" code can hide many bugs. For instance, when you pop the i'th element in a list, the indexes of your copy and original no longer match up relative to the list start, so your next pop will be off by one. You can fix by putting the i = i + 1 in an else. But cleaner is to use a list comprehension with a filtering condition in place of your copy.deepcopy.
no_comments_list = [line for line in original_list if not line[3].startswith('#')]
Also, using startswith is safer than indexing [0], in case of an empty string (which would raise an IndexError).
Rule of thumb: you almost never need to do i = i + 1 when iterating a list. zip and enumerate and list comprehensions eliminate 95% of common use cases.
For example you might do the_list = [row for row in the_list if line[3][0] != "#"]
Oops, PaulMcG beat me to it. Oh well, independent verification is good.
you make a good point @PaulMcG I've actaully come accross this before and was trying to get around it by creating this copy but you're right that it still has this same problem.

I do have some code which does this job correctly but it was a bit long to just copy and past
3:42 PM
The word "Pythonic" has been overused to death but I think avoiding manual calculation of indices really applies
@PaulMcG @Starter if this helps at all, this is how I removed prevous matching entries in a list if it helps?
        for line in reversed(Vdata):
            if row[0] == line[15]:
                if (datetime.datetime.strptime(line[6], "%Y-%m-%d").date() + timedelta(days=31) > datetime.date.today()) or (datetime.datetime.strptime(line[2], "%d/%m/%Y").date() + timedelta(days=31) > datetime.date.today()):
                    r = True
                    break
                else:
                    a = a + 1
            else:
                a = a + 1
@Kevin

This run fucntion not working is doing my head in. Any chance I can share my screen with you to show you what's going on because it just looks weird to me :/
@PaulMcG, I am looking into it. All valid points. I actually like list comprehensions a lot
and your code gives me some more ideas
@JamesMcIntyre Hmm, I don't think that would do much to give me new ideas. I'd really like something I could play around with on my machine.
I'll try... one sec
how would I send you a zip file @Kevin?
4:01 PM
Say I'm looking at a question and see that an edit is pending. I go check it out, make a change or two, and hit Improve Edit so that it posts immediately. Does the person who first suggested the edit get any rep, or is it lost?
all I am doing is just compare multiple columns in 2 files. i.e file1 and file2, if there is a match take a field from another column of the matching line in file1 and append it to the matching line in file2 if not append NA. I am not sure why my tiny brain can't get a handle on this. I feel dumb sometimes
@mat
@MattDMo Sounds worthy of meta
r = session.get("view-source:https://[redacted].com/[redacted]") isn't working for requests_html. How do I get a view-source: url so I can parse it?
@Starter trust me we all do somtimes. With programming you either feel like your a god or a worm's pet. Right now I defeinetly feel like the latter considering I can't even get the thing that I've created and works greate on my own computer to run on a virtual machine
@JamesMcIntyre Hmm, tricky. Usually pastebin is sufficient for sharing code, but it only accepts plaintext.
@MattDMo meta.stackoverflow.com/a/374309/953482 makes it sound like the first suggester gets the same credit whether their edit is approved or improved
"The user who made the original edit suggestion gets an Approval in the process."
meta.stackoverflow.com/a/368611/953482 likewise implies that an Improve is equivalent to two Approve votes
How about a temporary dropbox link? That's what I do. I also sometimes make dummy email addresses at one of my domains
4:08 PM
Neither explicitly says that they get the reputation, but Occam's razor
@Kevin thanks, that's what I was looking for. I thought I'd remembered seeing that before, but I wasn't sure
I really ought to get a domain one of these days.
I have a feeling that kevin dot com is taken
> The user who made the original edit suggestion gets an Approval in the process.

That says "they get rep" to me.
If you can figure out how to make a block quote without quoting every line in the message, let me know. It's my white whale.
Well, eggshell whale. My white whale is backticks in multiline messages.
I own mattdmo.com and my personal favorite domain of all time: dotdotatdotatdotdot.at
3
> If you can figure out how to make a block quote without quoting every line in the message, let me know.
I guess you just need to separate them
4:13 PM
Yeah, that works most of the time, but sometimes someone posts in the middle and messes up my aesthetic
Kevin: > You miss 100% of the shots you don't take.
John Q Neophyte: anybody here good with numpy?
Kevin: -- Abraham Lincoln
Sometimes I'll write out the whole message in a text editor, then do copy/paste as fast as I can
Abraham Lincoln never asked if anyone was good with numpy! John Q Neophyte has made a fool out of me!
How do you know?
Numpy hadn't been invented yet because it was incompatible with the prevailing vigesimal counting system. You can't optimize "four score and seven".
Remember, if you have a base-infinity counting system, you will never have float rounding errors.
4:22 PM
@Kevin I've created a copy of my project with the sensative data removed.

You can download it here:
https://drive.google.com/file/d/1mJq5uIy1ZiqhDyJLfCuATO1v3Ohn1JP0/view?usp=sharing
@MattDMo Reminds me of RRRRThats5Rs.com, a site for quirky flash games with an intentionally confusing url.
(it's 'Q1.py' that I'm trying and failing to run on the VM)
r = urllib.request.urlopen(relev)
Is just getting me a 403.
Here's the full MVCE
@JamesMcIntyre Cool, I'll peruse through it.
Thanks @Kevin, appreciate it
4:25 PM
I got dotdotatdotatdotdot.at at a rather low point in my life, and I would start laughing every time I'd log in the the server control panel, so it's been a big help. I've used it for throwaway addresses online, but I'm just waiting for the opportunity when there's no one in line behind me and the salesperson asks for my email address.
import urllib.request
relev = "https://www.discounttire.com/tires/winter-catalog"
r = urllib.request.urlopen(relev)
src = r.read()
print(src)
What I want to do is access view-source:discounttire.com/tires/winter-catalog
@JohnnyApplesauce Try changing your user-agent to a real value. The site is 403ing me, too, using requests, but it works fine in my browser.
I was about to suggest that. But it might be something else, because I still get 403 when I apply the user agent in the manner suggested by stackoverflow.com/questions/24226781/…
Yes still forbidden
@Permian what are you doing?
@JohnnyApplesauce i had a question but i solved it as i was typing out the question
4:36 PM
Ok
saves everyone the time
>>> import requests
>>> headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"}
>>> r = requests.get("https://www.discounttire.com/tires/winter-catalog", headers=headers)
>>> r.status_code
200
>>> source = r.text should do the trick
dpate instead:
Odd, I wonder why requests would work when urllib.requests doesn't. More sophisticated headers, maybe?
4:44 PM
@Kevin were you able to download that?
Relevant part: <h1 id="title" style="">Forbidden</h1> <div id="description" style=""> <h2 style="">Error Code: <em>403 (Forbidden )</em></h2><div class="detail" style=""> <strong>Server stopped processing your request(ID:1426879762). </strong></div></div> </div></div></body></html>
@JohnnyApplesauce have you tried my code with requests? It's also possible that if you've pinged the server with bad requests too many time, they may have temporarily blacklisted you. You might need to wait 5-15 minutes...
@JamesMcIntyre Yep. After tinkering with it, I got ModuleNotFoundError: No module named 'mailmerge'. This indicates that subprocess is successfully running CommercialDDConfirmation.py.
IT"S ALIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIVE
Thanks Matt
@Kevin

Yes! that's what I get! well actually it's within from mailmerge import MailMerge it fails to import from lxml.etree import Element
but you'll see that mailmerge is installed in that env
4:51 PM
Oh? I was under the impression you were getting no output at all. I must have missed a message.
I was getting no output at all when I used the subprocess. It just looked like it had ran but actually did nothing.
you can see that subprocess is currently disabled but there are other bits that are enabled which are allowing the running of the file and the import of cx_Oracle for instance
Right. I commend out the __import__ bit and uncommented the subprocess bit, because that's the approach I'm interested in. so subprocess does indeed produce output on my machine.
For the record, exactly what I'm running is:
import os

def Run(ImportName):
    import sys
    sys.path.append(ImportName + "Source")
    sys.path.append(ImportName + "Source/env/Lib/site-packages")
    os.chdir(ImportName + "Source")
    sys.path.append("")
    sys.path.append("env/Lib/site-packages")
    import subprocess #This is a suggestions from the chatroom
    print(os.listdir())
    subprocess.run(["env/Scripts/python.exe", ImportName + ".py"])
I changed some of the paths because I was getting "file not found" otherwise
ooo you did a os.chdir

Do you know why it's not working for you though (not importing mailmerge)?
I decided to create a fresh env because I didn't think the python executable from your computer would run on mine. Differing architectures, that kind of thing.
is that maybe what's happening for me?????

I created that env on my own machine but I'm trying to run it on a VM!
4:59 PM
Hmm, I'm not sure. You could test it by going into the VM and activating the venv and seeing if you can run a simple Hello World
will do, one sec
but the mail merge still isn't importing on your machine right?
My guess is that it's probably not an architecture incompatibility, since the OS or Python would probably say "I can't run this!"
Now that I've pip installed mailmerge, I'm getting ImportError: cannot import name 'MailMerge' from 'mailmerge'
Oh, it's probably docx-mailmerge I'm supposed to install. Now I'm getting ModuleNotFoundError: No module named 'cx_Oracle'. This implies to me that the mailmerge import succeeded, and now it's import cx_Oracle in Functions.py that's failing.
cx_oracle has always worked for me (with the previous method) getting that from the local env
(I activated the env through the command line on the VM and print('hello wolrd') worked)
Ok, so the python executable is most likely fine, then
If you think you needed to create a brand new env for your machine... do I need to create a new one on the VM?

It does seem odd that the subprocess method is working better for you than me :/
I'm gonna give that a go
5:10 PM
It's worth a shot.
Perhaps this is a silly question, but how can you tell the difference between "the program executed successfully", and "the program exited without doing anything"? As far as I can tell, you don't call print() at any point, so wouldn't those two outcomes look the same?
There's no shame in liberally sprinkling print("got this far") throughout one's code base to see how many show up on the console
beucase it proceduces PDFs if it's run correctly

(it does this on my mahcine)
5:28 PM
@MattDMo yes, "improve edit" == "approve edit single-handedly and apply your own", "reject and edit" == "reject edit single-handedly and apply your own". There's nothing special about the approve/reject in this situation
I wish "approve edit single-handedly" was an explicit power that came with edit privileges
((You can submit an empty edit))
Yeah, but it's the principle of the thing
Yeah. And the no-op edit will show up in the revision history anyway.
I think one of those questions I linked even says "try not to submit an empty edit". I want enthusiastic consent.
5:33 PM
@AndrasDeak Can you? I seem to recall that something has to change before you can click the Improve Edit button
@MattDMo I thought so too but I saw someone try it on unix.SE
unless it was a mod with special mod powers...
or a userscript
no
I'm trying to dig it up
meta.stackoverflow.com/questions/327397/… says there was a bug, and it's fixed now
I saw this a few weeks ago
5:36 PM
@Kevin just reran it with the natively compiled env and still got the same thing:

File "CommercialDDConfirmationSource/env/Lib/site-packages\mailmerge.py", line 4, in <module>
from lxml.etree import Element

ImportError: DLL load failed: The specified module could not be found.

Any luck your end?
Mine runs up until it tries to connect to a database. I don't have a database, so it crashes. Not too surprising.
non-mod editing three weeks ago
Ooooh it was a reject edit
¯\_(ツ)_/¯
sneaky
@Kevin what if you put a pass in instead of any of the database stuff? does it sucussfully import the mailmerge and attmept it?
(the database stuff works fine for me, it fails on trying to import the mail merge which comes after the import of cx_Oracle)
5:44 PM
If I return early from GetVDDs and RemoveWorked, then MakePDFs crashes with NameError: name 'MailMerge' is not defined. Not too surprising, since importing MailMerge in CommercialDDConfirmation.py won't make it visible in Functions.py
ahh yes, I forgot that I had disabled the import in the function as part of my testing. in theworking version it is enabled
def MakePDFs(data, Variable):
    import sys
    from mailmerge import MailMerge
When you say "it fails on trying to import the mail merge" I assume you mean it fails when you are using the __import__ based approach rather than the subprocess approach. That makes sense to me. Modules that are installed as packages have different import logic than simple directories containing .py files. __import__ can't find lxml.etree because it has no reason to think that lxml is a package.
Getting nested directory imports to work has always been a tremendous headache for me, which is why I've been campaigning so hard for the subprocess approach. The venv handles the packaging for you.
when I use the subprocess method to proceduces the following:

ln [4]:runfile('C:/Users/svc_automations01/Documents/BotNeoni/Q1.py', wdir='C:/Users/svc_automations01/Documents/BotNeoni')
Reloaded modules: ZeroFunctions, Functions, cx_Oracle, lxml

ln[5]:
@Kevin

I'm quite happy with using the subprocess method but when I do that as you can see above it just produces nothing. I even just put in a print('reached here') at the end of the CommercialDDConfirmation.py file but it didn't print anything.

Any more ideas before I give up for the night?
Not at the moment.
5:59 PM
@Kevin ok, I really appreicate your time and effort btw. thank you very much.

thanks again and have a good night/day depending on which part of the world you are
👍
hello leonard?
How would you check if a line has already been seen or processed in python?
put a line after that saying print("Hello world") if it triggers your good
i want to skip the line from further processing thats why I asked
6:13 PM
create a bool Valid (for example) and after the line triggers, use break or return, alternatively if you wanted it to run a set amount of times create an int X and when it triggers X += 1, and then use an if statement, if (X == 8): break/return
how to deal with coworkers writing garbage code
it's often too much responsibility pointing every mistake out and sometimes just fixing it yourself
and even then sometimes people aren't convinved that they are writing bad code
xD
cant help there
it usually is bad code tbh, if it runs with no obvious flaws try an accept it but if theres glaring mistakes then get them out of there chair, sit down and just rework that s**t
@RE60K Does your company have code reviews?
6:36 PM
@Starter Do you want to skip a line (one, well-known content) or some line (one, unknown content) or some lines (many, unknown content)?
no code reviews
main focus is on product delivery
@MisterMiyagi I am reading a file line by line, process each line and move to the next.
00:00 - 19:0019:00 - 00:00

« first day (3561 days earlier)      last day (1600 days later) »