« first day (3126 days earlier)      last day (1823 days later) » 

12:00 AM
Do it a third time just to make sure :P
 
lol
also FYI, you can ALWAYS run pip with your current python via python3 -m pip ... instead of pip ...
and then it will install into the same environment that python3 is currently linked to.
 
But more importantly, you want to get a solid understanding of some sort of virtualenv
 
^
 
all good now
thanks a lot !! ☺
 
go learn you some python :)
 
12:06 AM
i have done it before a couple of times.. just most tutorials don't discuss much about versions and symbolic links there as well as virtualenv.. tutorials just keep showing how to write if statement in python
but will do anyways
i heard but not sure that the py2 is coming to end of its lifetime by the end of this year?
 
@JoeSaad yea, this stuff is more unix-oriented things, like the binaries on your PATH and standard locations for executables and stuff.
 
thanks @alkasm
 
cbg
 
12:37 AM
cbg
 
 
1 hour later…
2:07 AM
@BenjaminGruenbaum very belated response, but did you find your answer on CrossValidated? Please tell us what it was!? Your question is mainly a stats question, not programming. But FYI the invite-only R room here is GMTs, there is an R Public but it gets much fewer eyeballs, but the GMTs membership might see a post there and respond.
... But you have to reformulate it; the assumptions behind A/B testing are that all trials are (reasonably) independent, else methodology goes out the window. You need to clarify what "the users are cooperating" means: all running instances on the same underlying node (hence seeing 'shared' performance results), or cheating on a crowdsourced test (e.g. CrowdFlower)? So, do whatever binning makes the trials independent (e.g. "users that started in a certain hour", and/or "instance name/host IP")
If it's still an issue, please ask it on CrossValidated, post the link here, and I can share it with R guys. And if you solved it, I'd love to see the solution, and I encourage you to post and self-answer on CV.
 
 
1 hour later…
3:36 AM
@wim I dunno, it's a cruel old world... maybe they were on mobile and their finger accidentally slipped to downvote instead of upvote. Anyway I upvoted. For posterity you might like to add a one-liner clarifying whether shortcircuiting is part of the spec or not (with citation), the 2 and 3 docs still don't address whether it is. Seems worthy of a docbug on both 2 and 3 doc.
 
cbg
 
3:54 AM
cbg
 
4:04 AM
cbg
 
4:20 AM
@0x45 so I decided to edit the 7-years-out-date top answer on What is __init__.py for?. EAFP...
"It used to be a required part of a package (old, pre-3.3 "regular package", not newer 3.3+ "namespace package")"
Honestly we each have to start doing basic housekeeping like that, or SO will turn into a museum of old, stale, broken, incorrect answers... also arguably the question statement should be updated to "What is _init_.py for in a pre-3.3 Python source directory?" . And I tagged it . Now should it also be tagged or ?
I award myself a beer.
 
4:47 AM
Lol
rbrb
 
5:35 AM
pet peeve of the last 4 years: pathlib.Path.absolute still not officially existing
 
Hi there
 
@U9-Forward Lol @ which post in particular...?
 
I need to know that, can we modify metadata of word document which has info such as author, title, date modification. If has please reply which python library to use.
 
@Rink16 MS-Word, OpenOffice/LibreOffice, or what format?
 
yes MS-word
PDF
which already has some information but i need to modify or want to add more attribute in metadata
 
5:42 AM
@Rink16 what: you want one package that does both MS-Word and PDF? It'll presumably be two separate packages.
 
yes no problem i start first for word document MS-word
 
@Rink16 You didn't make any effort to search did you...? for example Automate the Boring Stuff: Chapter 13 – Working with PDF and Word Documents and SO Python: How to Modify metadata of Microsoft Office files?, 2016 question has one answer for .xml. There may well be other questions. If you find out better answers please post an answer
 
i actually get confused becoz i have MS_word
and PDF
 
@Rink16 Did you even read what I just wrote you above? "you want one package that does both MS-Word and PDF? It'll presumably be two separate packages."
 
yes thanks
 
5:59 AM
do you understand that these are two different file formats? code which works for one will generally not work for the other without modifications
Word is .docx (or way back just .doc) created by Microsoft Word or a compatible program; .pdf is Adobe's Portable Document Format which can be created by Adobe Acrobat but also many, many other programs, including Word
 
@tripleee I guess the OP wasn't aware. But anyway, as to this different question (which is not what above OP's looking for) Open document with default OS application in Python, both in Windows and Mac OS, the accepted answer from 2009, using os.system is terrible for various different reasons, and a decade out of date.
 
@smci and you bring it up because ...?
 
@tripleee Because this is a very common basic question, one which will attract lots of new-users and inbound users from Google, and our top-voted answer is crap and ancient history from way back 2.x, and urgently needs to be fixed. I'm editing it right now as we speak. But it needs much more work. Can you look at it?
 
@smci I'm certainly looking
 
6:14 AM
@tripleee Ok cool. I'm making a quick edit now on the top-voted answer to at least decipher it into something intelligible. Wait a min...
 
@smci the previous edit is by a user whose name translates to "chewing gum coder" (-:
 
@tripleee The tragedy of the commons...
 
Nick's answer from 2009 gets my upvote
 
6:27 AM
cbg
I got a question about flask's jinja2(stackoverflow.com/questions/56033452/…)
Hope anyone can give me a hand :)
 
room rules, no self recently asked questions until 48 hours please. link top right or clicky
give it time, you'll most likely get a response in its own time
 
Okay, I see.
 
this is when most folks over at usa are probably asleep, so responses are a bit slower for niche problems at this time.
 
@tripleee I don't think the responsible thing is to just down/upvote old answers that are many years out of date. One or both need updating. I did a major revision #6 on Charlie Martin's answer. Now it is legible, much clearer, also breaks out the discussion of subprocess vs os.system() into clear points, so we can keep that discussion neutral and fact-based. I'm not saying it's a good answer, just less bad. Also, no mention of shutil.open
 
it's not really true that you can't use os.system() for file names with spaces etc; you just have to know how to quote them properly for whichever shell ends up parsing the command
I understand this is completely voodoo black magic on Windows but then why use Windows in the first place if you don't like that sort of thing
also "os.system() is the simplest way" isn't really true at all
should the bullet points be indented differently?
negative return codes are probably also completely specific to Windows
 
6:39 AM
@tripleee Then edit Charlie Martin's answer further... it needs tons more work. subprocess.call is old, since 3.5 we're supposed to use Popen. Other considerations are wildcarding, blocking vs non-blocking, arg-passing, any changes since 2.4..3.8 ...
 
@smci ugh, no! you should use subprocess.run() if you can, and only fall back to subprocess.Popen() when the existing wrappers don't solve your problem
 
@tripleee because I want my future son to be like Bill Gates and grow up to marry the daughter of the Central Bank of India... (as the joke goes)
 
I didn't want to override your edit without coordinating, need to run to a meeting soon but I'll take another look when I get back
 
cbg
 
@tripleee I know, I know, I only did cosmetic edits and major reformatting on his very obsolete answer, and tried to dial down the opinion on os.system vs subprocess.call (esp. as they're both wrong). It's still sorely missing subprocess.Popen (3.5+) and shutil.open. I'm tired. Tired of wading through crap.
 
6:42 AM
@smci sure, thanks for the ping and for your efforts so far, I'll see what I can do
not going into shutil.open though; maybe post a separate answer about that?
 
@smci to your drinking beer comment :-)
 
@tripleee Well, this is our ongoing existential question in [python] tag on SO: there are so many obsolete, bad, stale, broken, cruft questions and answers out there, we are sagging under the gravity of them; even if good new Q&A get written, Google (and SO search) will still misdirect eyeballs to the wrong bleeding answers. Perhaps we need a establish a "monthly community hitlist" of really bad, old, highly-trafficked answers that need an urgent date with machine tools or a flamethrower,
... i.e. major surgery and/or massive downvoting on existing text of answers, the question itself, tagging, version tagging, calling out version-specific assumptions...
 
not sure what the proper process should be, do we have any draft or something?
I mean how exactly to fix badly aged Q&A
maybe create a modern-day canonical and mark the old one as a duplicate?
 
@U9-Forward Sure, what's your poison? Currently mine is Negra Modelo Especial, I'm not crazy about it much myself but bought it for a friend's birthday, he likes it.
 
^ i think this sounds best
oh, rip
marking old ones as dupe
or perhaps, we could argue for a "outdated" redirect?
add a new feature, similar to dupe redirects, but with a different message that states why one should not use it anymore
"exists for historical significance, but has since been made redundant to a point editing the answers is not enough, and a more updated question answer pair is needed"
 
6:52 AM
@tripleee This is a major ongoing issue, wait until the day-shift crew are back to discuss. It can start major fist-fights if you close highly-trafficked ancient Q&A (even ones that beg to closed) into new canonical Q where you authored an answer, people will accuse you of rephounding (unless you mark the new question CW/'community wiki', which is also a non-solution because other people start mutilating the answers)
 
>>> def pairwise(iterable):
... "s -> (s0, s1), (s2, s3), (s4, s5), ..."
... a = iter(iterable)
... return izip(a, a)
This function gives me pairs from the list , but I need s0,s1 then s1,s2 and not s0,s1 then s2,s3
 
@ParitoshSingh There are 215 questions already on Meta.SO about 'outdated'...... ! I think that alone warns you that this is a highly controversial, opinionated topic, and no matter what you do/don't do, some set of people will complain and attack you. Give those Meta questions a read and post the best links here, if you want. But, the bottom line is, there is no consensus on this topic...
 
@pythonRcpp couple different methods. one here. Or you can use zip: assuming python 3. are you using python 2? if yes, why D:
 
Both trying to achieve (language-agnostic) consensus on Meta in general, and attempting to discuss solutions to -specific issues with people who have zero understanding of the language and won't lift a finger to do any homework before chiming in with a sweeping but utterly ignorant opinion, based on their experience in (language Y)... I'm beginning to think Meta is a read-only resource for any non-trivial language-specific question. Either way, don't waste time and energy on it.
 
aah. whelp. thanks for the heads up
 
7:00 AM
@ParitoshSingh I mean, my instinct is to phrase that in very blunt monosyllabic Anglo-Saxon words about the group of people who hang out on Meta and loudly pass opinions all day on things they don't even understand... thank God they don't certify airplanes or something.
 
def pairwise(iterable):
    i1 = iter(iterable)
    i2 = iter(iterable)
    next(i2, None)
    return zip(i1, i2)
@pythonRcpp ^
 
@pythonRcpp Eeek! then why not 👇
def pairwise(iterable):
    return list(zip(iterable,iterable[1:]))
to run it:
print(pairwise([1, 2, 3, 4, 5]))
output:
[(1, 2), (2, 3), (3, 4), (4, 5)]
 
recbg
@Aran-Fey @U9-Forward both are wrong
 
@AnttiHaapala Eeek, why?....
 
@U9-Forward @Aran-Fey it is in our canon: sopython.com/canon/16/…
 
7:12 AM
ah
 
@pythonRcpp Here is your answer: sopython.com/canon/16/…, listen to antti haapala, he's an expert
@AnttiHaapala Gah...
 
@U9-Forward yours does not work with non-indexables, and @Aran-Fey yours does not work with non-reiterables
 
@pythonRcpp He is also a RO (room owner)...
@AnttiHaapala ohh
@AnttiHaapala Now it does:
def pairwise(iterable):
    return list(zip(iterable,list(iterable)[1:]))
 
umm no. Well it works with non-indexables, but not with non-reiterables :D
 
@AnttiHaapala Do you know how tee gets around non-reiterables? Something with views?
 
7:16 AM
@Arne it has an internal buffer of what has been seen but not yet consumed by the other branch
 
@AnttiHaapala What you mean by non-reiteratables :-)
 
i just realised i need 3 items at a time sopython.com/canon/16/… gives only 2 like 1,2,3 then 234 345 and so on
 
squares = (x*x for x in range(10))
def pairwise(iterable):
      return list(zip(iterable,list(iterable)[1:]))
pairwise(squares)
=> []
 
@AnttiHaapala hmm, means advancing the initial object invalidates the tee
so that needs to be kept in mind
 
>>> from itertools import tee
>>> def pairwise(a):
...     b, c = tee(a)
...     next(c)
...     return zip(b, c)
...
>>> squares = (x*x for x in range(10))
>>> print(list(pairwise(squares)))
[(0, 1), (1, 4), (4, 9), (9, 16), (16, 25), (25, 36), (36, 49), (49, 64), (64, 81)]
 
7:20 AM
@AnttiHaapala Eeek, i need to remember too get everything right :-)
 
every iteration can be written using iterators.
 
@AnttiHaapala Now?
def pairwise(iterable):
    iterable = list(iterable)
    return list(zip(iterable,iterable[1:]))
 
but now you copy the whole thing
 
@Arne At least it works lol :-)
 
@U9-Forward ok try with this:
 
7:24 AM
so does tee, without the copying :p
 
squares = (x*x for x in range(10000000000))
and take the 10 first elements :P
 
there we go =D
 
@AnttiHaapala Ugh, no....
@AnttiHaapala It gives a:
MemoryError
 
>>> from itertools import tee, islice
>>> def pairwise(a):
...     b, c = tee(a)
...     next(c)
...     return zip(b, c)
...
>>> squares = (x*x for x in range(10_000_000_000_000))
>>> print(*islice(pairwise(squares), 10))
(0, 1) (1, 4) (4, 9) (9, 16) (16, 25) (25, 36) (36, 49) (49, 64) (64, 81) (81, 100)
 
Wow, let me forget that one...
 
7:29 AM
@AnttiHaapala Neat, could you augment the canonical with discussion of the 'wrong' ways to do it, viz. non-indexables, and non-reiterables?
 
@AnttiHaapala Tried it with [*iterable], even worse, my PC says low memory, in a popup...
 
@smci you could
 
@AnttiHaapala My understanding is nothing compared to yours. If I could have edited it myself, I already would have and not asked you.
 
@U9-Forward even more fun with infinite iterables
 
Ugh, i don't even want to think about that...
 
7:38 AM
@smci you should add another answer
like Aaron does...
@smci you get understanding by answering
 
you can give tee a second parameter that controls the amount of returned branches, two is just the default
 
@AnttiHaapala Sure but as you may have seen, I already spent enough time editing answers today, and researching stuff. Personally my #1 priority is still finding a decent Pythonic solution to yesterday's question about how best to do a range_exclude(start, stop, step, exclude) where exclude can be an integer/list/set/iterable(/chain of iterables?)
(PM2Ring did give me a ton of useful links on prime-sieving and generators)
 
Is anyone programming with VS Code, is there a way to automatically run my pytests on save?
or even check changes on method under test and run that specific test?
 
@AnttiHaapala Here is something else Python I invested time in, which disintegrated into a food-fight: Meta: What on earth is [step] tag, it's not synonymous with [increment], should it be redefined?. Any guidance, or is tag janitorial not our dept, or is Meta the wrong place? Why is Meta so full of opinionated people yet mysteriously noone will put their name to an answer?
@AnttiHaapala Anyway I'd appreciate any one-line advice on what to do with that one.
 
8:06 AM
@smci people are slow
 
How can I do something like this?
float(rowI[6].value | 0)
I want to set 0 in case the left expression is none
float(rowI[6].value or 0) ?
 
8:22 AM
just write an if else. either normally, or using ternary operator float(rowI[6].value) if rowI[6].value else 0 or something like that
 
It's safe to write something to a file before entering a context, right? I need to do something like this
fileobj = path.open('wb')

if password is not None:
    fileobj.write(b'encrypted=True\n')
    fileobj = AESEncryptor(fileobj)
else:
    fileobj.write(b'encrypted=False\n')

with fileobj:
    fileobj.write(data)
 
does the first line not open fileobj in an unsafe manner? if theres an error right after, it wont self cleanup.
atleast that is my understanding
 
true
 
if you need to open it, use the context manager.
 
So I need to do it like this?
with path.open('wb') as fileobj:
    if password is not None:
        fileobj.write(b'encrypted=True\n')
        fileobj = AESEncryptor(fileobj)
    else:
        fileobj.write(b'encrypted=False\n')
        fileobj = contextlib.nullcontext(fileobj)

    with fileobj:
        fileobj.write(data)
 
8:33 AM
i think it is acceptable here to have some logic inside the context manager, because what you're writing depends on it. However, at the same time, i dont quite understand the fileobj = AESEncryptor(fileobj) part, which means you're modifying the handler or getting a new handler out, and that complicates things.
 
@AnttiHaapala I don't understand: 'slow' meaning 'not smart', or 'must allow 365+ days to reach a consensus', or what? Am I wasting my time asking that Meta question? If not, how could I ask it better?
 
AESEncryptor is just a file-like object that writes encrypted data to the underlying file object
 
So, i guess the reason you probably asked is because this version "looks" weird. I wonder if there's a better way..
but yes, this is atleast safer than earlier
tbh, i dont even know if this works, does it? can you have two handles to same file at once?
 
I don't really have two handles to the same file... I'm just assigning a different file-like object to my fileobj variable
 
and that's basically the "open" thingy right? uh.. sec
 
8:39 AM
cbg
can I execute tensorflow in cpu if I have the tensorflow-gpu installed?
 
does this work?
if password is not None:
    header = b'encrypted=True\n'
    func = AESEncryptor
else:
    header = b'encrypted=False\n'
    func = contextlib.nullcontext

with func(path.open('wb')) as fileobj:
    fileobj.write(header)
    fileobj.write(data)
 
unfortunately no, because that would encrypt the encrypted=True. I need to write that part before I turn the file object into an AESEncryptor
 
i see
yep, sorry i couldn't help, out of ideas then. unless you wish to open the file, write the header, then open it in append binary
though tbh, now that i think about it, i dont think that's a bad thing here
 
I'd rather not re-open the file tbh
 
with path.open('wb') as fileobj:
    if password is not None:
        fileobj.write(b'encrypted=True\n')
        fileobj = AESEncryptor(fileobj)
    else:
        fileobj.write(b'encrypted=False\n')
        fileobj = contextlib.nullcontext(fileobj)

    with fileobj: #do we need this line??
        fileobj.write(data)
 
8:47 AM
that's a good question
 
with the handler being passed to get a diff handler, im just not clear on how the whole thing is behaving. Are we accidentally opening an "unsafe" or hanging handler at fileobj = AESEncryptor(fileobj) this step? and, if not, then will the cleanup still be handled correctly?
 
We can assume that AESEncryptor will automatically propagate the __enter__() and __exit__() calls to the underlying file if necessary. What I don't know is whether AESEncryptor works without a context manager. It might do some setup in its __enter__(), I'm not sure
 
@smci this is the day shift over here, I guess you are talking about US time zones?
 
@tripleee Yes indeed, I'm in California. Hello in Estonia. PS your webpage is down.
 
9:06 AM
Estonia? That's just...a punny URL?
 
@smci (yeah, I'm not actually in Estonia, and it's not down, it's nonexistent)
 
@tripleee Eh? your homepage is 404? or not even 404 :D
 
@smci well it's parked somewhere but basically I just registered the domain and planned to populate it with something at a later date which has apparently not yet arrived
 
@tripleee Alrighty. Was just letting you know in case your host had accidentally killed your webserver process or something.
 
nope, not yet, but thanks (-:
 
9:15 AM
What a fun morning I'm having, destroying someone's day :'( For some reason, people don't seem to be able to distinguish between '.ca' and '.com' on forms. I haven't checked this account for a while but there's missed interviews and all sorts of invoices that I now have to forward on.
The take-home message is that, if your email doesn't end in .com, you need a new email address. I have 5 alter egos, last I counted
 
@roganjosh a colleague of mine was receiving namesakes' bank statements and very sensitive medical information ... and his name wasn't even particularly common
 
I was so happy when I found the email address of my name was free.
@tripleee I had a failed mortgage application that included their financial score that I had to forward on
 
it's not just .com vs the world, it's also people assuming firstname.lastname@gmail.com will be owned by the person they are imagining
 
Hey who maintains the search engine on sopython.com/canon? I get a 500 Internal Server Error is I search on "Why is it a mistake to assign the result of list.append()?" If you drop the () and ? punctuation search succeeds. Is that a known issue or some new breakage?
 
9:21 AM
I've managed to track 2 of my alter egos down. One in Canada, one in Wigan which is just a few miles away. The rest I can't do too much about
 
It's almost as if people were idiots
 
@AndrasDeak it's almost as if engineers designed things only fools will like
 
I've already had a science journal account automerged with that of someone else.
 
oh don't get me started on heuristically merging records...!
 
Some of them are fun. I got invited to a stag party in Vegas. I also got invited to help with a reading club somewhere in America (ironic, considering the sender clearly couldn't read the email address) and also got asked to beat a church cross with a chain to give it a "weathered" look
The not-so-fun side is all the order receipts, mailing lists for dogs (I did quite enjoy dog-of-the-week), a failed mortgage application and like 7 or 8 job interviews
 
9:27 AM
a mailing list for dogs? Isn't there one for puppies? Bloomin' ageism at work I tell you! :p
 
Well, about dogs. Poor phrasing :P
 
Is it normal to take some minutest to read 24k rows from an excel file and insert them into a mongodb? Or is it too much?
 
Sounds an awful lot of time
 
@smci not really, no
 
so, how can I paralellize it?
 
9:33 AM
(I work for a different company now though)
 
@roganjosh Unless it's a sheet with 24k rows along with 16k columns and mongo is busy otherwise...
@BenjaminGruenbaum you're about again Ben boy... how you doing mate?
 
@JonClements no, it has like 12 columns
 
@QuicoLlinaresLlorens that's not the solution. You should first be looking at the complexity of the current code
 
@roganjosh it is an n complexity
for i in range(2, workSheet.max_row + 1):
                checkAndShowPercentageCompleted(workSheet, i)

                rowI = workSheet[i]
                nameInRowI = rowI[1].value
                descriptionInRowI = rowI[3].value
                unitInRowI = rowI[5].value.title()
                priceInRowI = float(rowI[6].value or 0)
                ITARInRowI = bool(rowI[12].value)
                manufacturerInRowI = rowI[4].value
                articleCategoryInRowI = rowI[8].value

                collectionsToAppend.append({
 
@QuicoLlinaresLlorens the problem (without even reading your code) is that you are reading each line and inserting each object, one at a time without any sort of buffering
ah, there's your code.... so, yeah, it's what I said
hehehe also...
 
9:36 AM
how can I buffer it?
 
that check and show % complete... dude... make that only run once every 500 rows or so
this alone will probably increase your performance significantly... excel is VERY slow at updating the screen within macro code
 
@JonClements hey :] What's up? Doing great, traveling a bunch - mostly life's good
How about you? What's up?
 
@QuicoLlinaresLlorens I don't know how mongo works within VBA, so I don't know; but you probably want to look into a function that would let you insert several objects at once into your collection and then buffer them manually in your loop...
(to buffer them manually; one way is to insert into an array (instead of mongo directly) while counting and once your array reached (ex. 50), insert into mongo, clear array, zero counter, continue)
 
24k rows is peanuts. just load it all in memory instead of reading through worksheets. pandas or something?
 
9:40 AM
Yes, I am doing that. I am inserting all them at once
@ParitoshSingh but I need to read them before, no?
 
yeah, but i am thinking that you can use something that handles reading tables well, like pandas
and get it all in a dataframe, and only then worry about insertion to mongodb. just time how long it takes to read
 
@LFLFM mmm, sounds well
 
i wonder if its the worksheet iteration that's slowing you down, and you can essentially stop working with excel as soon as your data is loaded in memory.
 
worksheet iteration is slow; but not "minutes" slow
in cpu-only VBA (as in, no IO; no file writing, no database interaction, etc), the "delays" are caused by UI updating, so we disable UI updates
when IO is involved, then... well, it's always IO right
bu this actually reminds me
 
So, I try to run like this:
for i in range(2, workSheet.max_row + 1):
                # checkAndShowPercentageCompleted(workSheet, i)

                rowI = workSheet[i]
                nameInRowI = rowI[1].value
                descriptionInRowI = rowI[3].value
                unitInRowI = rowI[5].value.title()
                priceInRowI = float(rowI[6].value or 0)
                ITARInRowI = bool(rowI[12].value)
                manufacturerInRowI = rowI[4].value
                articleCategoryInRowI = rowI[8].value
 
9:45 AM
why are we talking vba, this is python right?
 
just the read and assign the values
 
Mm.. oh yeah, sorry; they just look alike (and actually perform not too far alike either, so it's valid :D)
@QuicoLlinaresLlorens that alone will improve things a bit, probably not enough though
 
"and actually perform not too far alike either" This part i dont buy. atleast my personal experience, if python has to interface with excel, its a nightmare the longer you deal with excel. read the data, get rid of the excel step asap when working with python
vba will significantly beat out python if the actual work is reading and writing in excel.
 
@ParitoshSingh yeah, but that's generally not excel's problem, it's just normal IO (and maybe the module's lack of io optimization)... VBA itself is not far slower than python
 
mm yeah, that i can agree with.
regardless though, i am willing to bet on good speedups if you use pandas to read, and insert into mongoDB as a 2nd step
 
9:50 AM
@QuicoLlinaresLlorens check how you could insert an array as multiple objects into your colletion and do some buffering... even 20 rows of buffer would likely grant you massive performance improvement
@ParitoshSingh I don't know pandas, but you're definitely right
 
Yes, I think my array gets slower as it has more elements, so maybe inserting a piece and deleting them, would be a good idea.
 
ah, pandas is just python's version of tabular datastructure. pretty nifty library, and built on numpy, so what im saying is a generalization, but it deals with tables well.
Lots of things about it may irk me, but it has solid merits
 
basically; remember this: IO is damn slow.... but.... people tend to forget that IO is not slow because it's slow, it's slow to start and stop... each TIME that you have an instruction that uses IO, you get a huge overhead; so avoid having loops with IO inside - buffer as much as you can and then write bigger blocks at once
 
@BenjaminGruenbaum ssdd as always - but ultimately nothing to complain about :p
 
9:55 AM
I think the slow thing is the adding to the collectionsToAppend array
 
That will force a copy of the whole DF
 
@smci now that I am reading the question, the thing is: is increment a good tag. What does it have to do with step. Why are you asking for the actual use of STEP as a taggable thing to be *removed from that tag*
 
@QuicoLlinaresLlorens yes, that is definitely the slowest part of your code and your current bottleneck, which is why I'm asking you to buffer and insert "en mass"
 
@smci and this could be discussed in SO Close Vote Reviewers
 
If you're appending to a DF in a loop then you'll have real problems. I'm extrapolating a bit from what you're saying, but it sounds like that's what you're doing
 
9:56 AM
@BenjaminGruenbaum or not... as a Brit having something to complain about is pretty much the purpose of existence :p
 
I am running the assertion part to see how much it takes
 
@JonClements how's the weather, Jon? :P
 
it's peeing it down.. garbled mumbles about the bloomin' weather...
 
How's England? Is everything politicians doing exactly the way you would prefer or are certain things somewhat inconvenient?
 
@Paritosh to the rescue! :p
 
9:57 AM
anytime, heh ;)
 
@AnttiHaapala directed replies save lives :P
 
good job guys
I feel better now. One shall celebrate with a cup of tea... bbiab
 
jolly good
 
why thank you ol' chum :)
 
@LFLFM It is also taking a lot to just read and assign the values
 
10:04 AM
@Quico is your collectionsToAppend.append({ actually inside the for-loop?
 
@BenjaminGruenbaum Dissolved kingdom of Lesser Britain and.
 
@Quico and what's that function do at the start? I'm afraid that's not a really useful MCVE that can be used to help you here...
 
@BenjaminGruenbaum I guess for one to feel happy with current state of things there your alignment would have to be "chaotic evil" :D
 
@smci and others I made another edit to the accepted answer, pls review stackoverflow.com/questions/434597/…
 
@JonClements yes
def migrateArticleData():
        workBook = load_workbook(constant.ARTICLES_WORKBOOK, read_only=True)
        workSheet = workBook[constant.ARTIKEL_WORKSHEET]
        print("Started migrating article's data")
        start = time.time()
        appendRowToCollections(workSheet)
        print(time.time() - start)
        print("Finished migrating article's data")
        collection.insert_many(collectionsToAppend)

def appendRowToCollections(workSheet):
        for i in range(2, workSheet.max_row + 1):
This is what I am running right now. The other function which call migrateArticleData is nothing just some ifs
 
10:11 AM
Mm.. if this is taking long, then the worksheet module is doing something stupid, like maybe reading the HD directly for each row... check if the module is capable of buffering the file
 
10:33 AM
Cbg
I mentioned pyenv briefly yesterday and two people said they didn't like it - are there any reasons for that?
 
10:48 AM
cbg
 
cbg
 
hi
 
Hello
 
hey guys
a simple question:
I am using an `iglob` to walk through a directory containing images to obtain a Generator.
I want to be able to skip a file every now and then (aka NOT YIELD IT) based on an input percentage value.
How should I do that?
 
Put an if statement around the yield?
 
10:56 AM
ah, I was complicating the thing, just realized I can just skip the first part of the directory directly
 
if random.random() < threshold: next(generator)?
 
I was trying to skip every 1/10, for example
yeah, thanks guys
I could use a seed to ensure same results from random.random() right?
 
yeah
 
(so my training set and my validation set could end up being perfect partitions of the directory)
 

« first day (3126 days earlier)      last day (1823 days later) »