« first day (3300 days earlier)      last day (1641 days later) » 
00:00 - 20:0020:00 - 00:00

12:03 AM
in order to build python from source I had to module unload python 2...
 
 
1 hour later…
1:11 AM
hello!
Is it possible to iterate this function to an excel workbook? (stackoverflow.com/questions/58511787/…) Any help will be appreciated.
 
 
4 hours later…
4:53 AM
cbg
 
 
2 hours later…
user10984358
7:10 AM
Heya guys, quick question. If I do the following command “someCommand > output.txt” in command prompt and the output.txt file is 500 mb. Will doing a subprocess.run with capture_output=True and getting the stdout in my python code still consume the same (500 mb) memory?
 
7:27 AM
no, it'll hang because python can't buffer 500MB
 
user10984358
So how to deal with such a case if my code depends on a subprocess call?
 
hmm, actually, I'm not sure if that's true. If you set stdout or stderr to PIPE, there's a limit to how much can be stored before the subprocess hangs. But it's possible that capture_output takes the output out of the PIPE and stores it somewhere else, in which case it wouldn't actually hang
 
subprocess.run will properly communicate with the process
so yes, you get 500MB in memory
 
@TheNamesAlc you process the output while the process is still running
 
user10984358
How do I do that?
 
7:31 AM
if you cannot stream-process the output, you might want to consider dumping the output to a temporary file.
 
@TheNamesAlc Good question. Reading subprocess output while it's still active is always a bit of a tricky thing to do. If I remember correctly, the right way to do it is for line in process.stdout:
 
user10984358
The output is of xml format so I have to parse that using element tree
 
In that case it'll definitely be easier to process it all at once. If 500MB is too much to keep in memory, write it to a temporary file
 
user10984358
Actually it’s 740 something mb
 
user10984358
I call the subprocess and writeline stdout to file using for line in process.stdout?
 
7:41 AM
no, just set stdout=your_file
 
user10984358
Thanks a bunch both of you.
 
user10984358
8:20 AM
Without redirecting I got MemoryError, with redirecting it was written to a file in just a little over 550 seconds.
 
user10984358
When I read the file I get a memory error now, how can I parse it?
 
user10984358
My goal was to use ET.fromstring(file.read()) so I can assign it as root element
 
9:49 AM
@TheNamesAlc if you don't have the memory to hold your all data, yet you need all data -- there isn't much you can do. Best bet is to reconsider why you need all data, and change that.
 
Writing bug reports takes surprisingly much effort. Apparently I just spent one and a half hours on it.
 
Only if you care to write good bug reports i assume. Same as SO questions :P
 
I'm sure I probably wasted a good twenty minutes trying to convince myself that I'm not just being needy and "I want better error messages" is really worth creating an issue for
 
Ha! i can easily visualize you doing that
 
Me: *writes `background: contrast(#111, #222, #333)`*
CSS precompiler: *ignores extraneous 3rd argument and incorrect return type and happily outputs `background: {"ratio":"1.2","error":"0","min":"1.2","max":"1.2"};`*
Browser addon: *never shows me the compiled CSS and throws a cryptic error*
Me: *spends an hour debugging trivial problem*
So yeah, in the end I decided I'm not just being needy
 
10:04 AM
frontend 😱
I will never leave the coziness of my command line
 
I like my dark themes, and since nobody else seems to be capable of writing good ones I have to do it myself :(
 
well, that is a valid reason for sure
 
cabbage
 
10:23 AM
cbg
 
10:38 AM
cbg
 
10:53 AM
When writing code what is the easiest way to know if the way you wrote is the best way to do it?
umm.. I know experience plays a huge role in knowing that
on the web people show the difference by using stuff like timeit to measure performance
 
Please define "the best way to do it"
 
Umm.. Big(O)
but generally, sometimes when I write a piece of code my senior walks in and makes some changes which still does the same thing but in lesser line of codes.
He says its better to do how he is suggesting it... I am like how does HE KNOW? which is the most optimized way to write something..
Is there a package which can analyze my code and point out possble pitfalls.
 
there are a bunch that try, but no way around just getting experience and a nose for what is good code and what isn't
 
You need to decide what your target is...
Faster code, less lines of code, more readable...
Maybe ask your senior what his main target is?
You'll learn a lot about this by reading code from the code base, if you have access to it.
 
11:14 AM
Right!
 
To a point, less code also means less room for bugs. For instance, copy/paste/modify coding will create lots of code, but then if a bug is found in the original, many copies must be tracked down and fixed. I rely on DRY (Don't Repeat Yourself) to avoid this, it generally makes the code tighter also. (Note leading To a point qualifier, though.)
 
11:36 AM
Wow... Monica's fundraiser has already exceeded target...
 
what's she raising funds for?
 
A replacement for stack?
 
11:56 AM
@JonClements that's news to me too :)
> The fundraising goal will rise over time. That is the nature of litigation. This amount covers the first round of costs.
so I guess there's always room for more :P
 
12:27 PM
if there was a staightforward objective way to determine whether an algorithm was optimal, and/or determine the optimal design, we'd all be out of a job
 
@JonClements It was over $4k in 4 hours.
 
to be fair there was a $1k and a $500 boost
 
yeah... been some quite big numbers - not your average everyone puts in $10 or something thingy...
 
New closed / on hold messages now undergoing A/B testing on SO: meta.stackexchange.com/q/337013/334566
 
hey, I'm in the new group
wonder what this should solve i.stack.imgur.com/twhcr.png
it still shows who closed it to the asker, which I thought was one problem for many people
 
12:40 PM
My default assumption for all GUI redesigns is that it's not intended to address any problem other than "people are complaining that we haven't updated the interface in a while"
 
In other news, compiling python with custom ssl is kicking my butt bugs.python.org/issue34028
@Kevin nah, this is at the heart of the welcome wagon
people feeling hurt by 5 bullies closing their questions
 
@AndrasDeak Ah, right. I didn't look at the details. Some people who want to donate haven't been able to do so yet because it doesn't accept PayPal.
 
user11867329
@Kevin Thanks again, buddy.
 
What is she suing for?
 
@AndrasDeak Me too.
@ReblochonMasque Well, she doesn't really want to sue. And to sue for defamation / libel would be expensive. And risky. As far as I understand, at this stage she just wants a lawyer to talk to SE on her behalf because she's not getting satisfactory communication with them.
 
12:48 PM
Did I miss the change from "Too broad" to "Needs more focus"?
 
@Arne we probably all did
 
oh huh, you're right, it's changed now
 
there may have been a meta asking about how these new blurbs should be phrased
 
Thanks @PM2Ring
 
"Needs more focus" should be used exclusively for questions where the code is posted as an image, taken by a blurry camera
Please upgrade to iPhone 11 Pro and edit your post
 
user11867329
12:52 PM
>:|
 
Good morning fellow humans
 
guess that's my cue. rbrb
 
Greetings. I hope you charged slept well, fellow human
 
Eh, we'll see. Had a little too much bourbon while doing some LabView last night at home XD
 
the "off topic" reasons also look different, non-repro doesn't include the word typo any more. Now I need to reread them all again instead blindly clicking at the tried and tested three options that I like =(
 
12:57 PM
wty
 
wty?
 
If the new design is that nobody can see the names of close voters, then just choose a reason at random, since there won't be any accountability for your actions
 
@biggi_ what the yam
 
user11867329
I need to create a system to unlock/secure office doors.
 
user11867329
8 doors
 
user11867329
12:59 PM
Needs to:
1. Not be costly
2. Easy to maintain
3. Must'nt be a security joke
 
user11867329
I thought of making an intranet-accessible server/webapp and an according APK/ios
 
I bet 99% of the work will be verifying that you're adhering to the fire code
 
so far we know nothing about your project except that you need to interface with some hardware
 
user11867329
Maybe not 99
 
If the fire alarm goes off and all the doors stay locked because you forgot a semicolon, that's a problem
 
user11867329
1:00 PM
but I mean, it's only one way, doors should always be openable towards exit
 
This looks like it was probably closed as No MCVE, but it just links to the Ask page, not the MCVE page. stackoverflow.com/q/58606652/4014959
 
user11867329
Big bar in the middle, manual door opener
 
user11867329
Just in case
 
user11867329
But I just don't want to have to buy a prebuilt system
 
user11867329
with the freaking manuals online everywhere
 
user11867329
1:01 PM
or pay a million to the guys who know what they are doing
 
user11867329
Middle ground, need to make it.
 
user11867329
So trying to fetch me some ideas if anyone had any
 
Find guys that know what they're doing that don't charge a million? :p
 
@PM2Ring the old style still does
 
I rank this project at a difficulty level of 8/10. 9/10 if you're going to be installing the doors and running the wire etc etc all by yourself
 
1:04 PM
@OakDev "secure office doors" is a rather broad requirement.
just to avoid being helpful: "intranet" is not the same as "secure"
consider using PKI to authenticate the requests.
 
@Arne Hmmm. Supposedly, the close reasons themselves haven't changed, only the wording in the close menu & banners. But that message doesn't really apply to typos. That could lead to confusion...
 
user11867329
@MisterMiyagi Thanks for PKI lead. No worries, I know intranet is not necessarily secure. Then again, I'm not "head of security"
 
@PM2Ring the new notifications look nifty, but it seems dubious to have --two systems-- one system with two different wording at the same time.
 
user11867329
just an IT guy doing his best.
 
when you say "I need to" do you mean "my boss ordered me to while making threatening faces" or just "always wanted to build my own super fortress of doom"?
 
1:11 PM
@PM2Ring crap, you're right
this is a mess
 
@AndrasDeak "If there are things about the new notices that you dislike, please let us know in a respectful and constructive way. We are happy to receive and take all feedback into consideration."
@Kevin The names are visible to people with close vote privileges, and (presumably) the post author.
 
Oh, disappointing. I'll put away my balaclava
 
I guess a typo is non-reproducible once the typo is edited. ;)
@OakDev I think this is on-topic on Information Security security.stackexchange.com/help/on-topic "physically securing the office, datacentre, information assets etc."
 
1:44 PM
Is it guaranteed that pickle.load will leave the file pointer exactly at the end of the pickle data, or is it possible that it might read more? In other words, is this code guaranteed to work?
import pickle

with open("output.txt", "wb") as file:
    pickle.dump(1, file)
    pickle.dump(2, file)

with open("output.txt", "rb") as file:
    print(pickle.load(file))
    print(pickle.load(file))
My concern is that the first pickle.load call will successfully extract the value 1, but in the process partially read the pickle data for 2. Then the second load call will fail because the file pointer isn't in the right place.
 
user11867329
@MisterMiyagi No mean faces. He asked me to do it, and I wouldn't mind being in charge.
 
Imagine a hypothetical pickle protocol that, as an efficiency measure, reads from the file in blocks of 32 bytes. If the pickle data for 1 is 27 bytes, then that's a problem
 
user11867329
@PM2Ring I'm currently chatting with them also, but and android app might be implied, figured why not, interesting topic.
 
But it's only a problem if the file contains more than just the pickle data of a single object. Calling read(32) on a file that's only 27 bytes long works just fine.
 
I'm 99% sure that's a documented feature of the pickle protocol, but I can't seem to find that documentation
 
1:48 PM
@Kevin yes
 
Certainly the code works on my machine now, and it worked when I tried something similar last year, so at the very least I'm confident that CPython's implementation of pickle protocol 4 plays nicely with file pointers, and most likely has done so since its introduction
But who's to say that IronPython's implementation of pickle protocol 2 might do something else
 
> Bytes past the pickled representation of the object are ignored.
Close, but not quite good enough
 
Yeah, I'm paranoid that an implementation can "ignore" bytes by reading and discarding them
 
@Kevin AFAIK IronPython does all sorts of things differently anyways.
 
Currently digging through the PEPs to see if the protocol specs make any guarantee about file pointer position
The real reason I'm asking is because I occasionally see people doing multiple consecutive loads on the same file, and I always tell them "I think that's not guaranteed to work", and I want to know if I can upgrade to "that's not guaranteed to work". Without the "I think" weasel phrase, I'm more vulnerable to Well Actuallies, but since the three of us couldn't find a citation in thirty minutes, I think I'm mostly safe.
 
2:03 PM
I built a python 3.7 from source. It doesn't have pip, but if I create a venv with it I have pip inside. Is this normal?
 
hello!
 
user10984358
Heya guys, is opening a file in w+ mode using a context manager for writing to it then using file.seek(0) and doing operations on it preferable or just use another context manager and open the same file in “r” mode?
 
Either one seems fine to me.
 
user10984358
Thanks, today I’ve run into so many MemoryError’s
 
@TheNamesAlc that sounds completely unrelated to file handling
 
user10984358
2:11 PM
Earlier I was asking about a subprocess problem and I want the final output to be in a file
 
user10984358
I was able to read the subprocess output lazily but I ran into errors when writing a file so I was wondering if that context manager had something to do with that
 
MemoryError is usually about "not enough memory", not contention for specific objects or addresses in memory
 
@TheNamesAlc I find file.seek rather obscure. two contexts' would win in the readability contest for me.
 
@TheNamesAlc I'm not sure I want to ask what you were trying to do with subprocess but will do so anyway?
 
Hello guys! I will really appreciate if anyone can help, suggest or give me some guidance of how to make this function iterates in an excel file(stackoverflow.com/questions/58511787/…) . I have tried in multiple ways, unfortunately no success so far.
 
user10984358
2:14 PM
It’s a command line tool that returns a list of devices among others in an xml format. I wanted to change that xml schema to a different schema and write that to a file.
 
user10984358
The original xml had 21 mil lines and so the MemoryError’s began
 
@Verbamore "it doesn't work" is not a very good issue description. What doesn't work? Which part throws the error?
 
@MisterMiyagi thank you for replying! the function works fine in a single dataset but I am getting an error when I iterate it through multiple sheets. ValueError: Invalid length for values or for binner
 
let me rephrase my earlier query: At which line does the error occur?
 
@MisterMiyagi, last line, when I call the function
 
2:20 PM
but inside that function, which line throws that error?
 
It might be useful to add the complete stack trace to the question
Everything from Traceback (most recent call last): to the ValueError
 
I just added a screenshot
 
please post it as text
 
So the error is occurring on the line df.KW.resample('-30T').sum(), and based on the "resample.py" filename in the third stack trace element, it's probably resample's fault.
 
@Kevin The Python 2 docs say it's legal to make multiple Pickler.dump calls to the same file, using the same Pickler object. I assume that also applies to the dump convenience function. But I can't see the same info in the Py 3 docs.
 
2:29 PM
It seems likely to me that this problem isn't specifically related to reading multiple Excel sheets. Rather, the problem is that one of the sheets contains data that resample doesn't understand. You would have gotten the same error when you were running your code on a single sheet, if that sheet had bad data too.
 
From docs.python.org/2/library/pickle.html#pickle.Pickler It is possible to make multiple calls to the dump() method of the same Pickler instance. These must then be matched to the same number of calls to the load() method of the corresponding Unpickler instance.
 
@PM2Ring Hmm, interesting. I see that the guarantee is present in docs.python.org/3.0/library/pickle.html#pickle.Pickler too, but it's gone by 3.1.
 
@Kevin Ah. My guess is that although it works, they want to discourage the practice because it's messy.
 
come to think of it, there's really no reason to pickle multiple things into the same file
 
github.com/python/cpython/commit/… is the specific commit that removed the guarantee
 
2:40 PM
repeated dump/load is vitally important when pickling to a stream that is connected to some peer
 
I'm not too familiar with memo so it's unclear to me whether multiple consecutive dump calls depends on it, or what
 
it also means that you can stream-process the content and preserve identity of nested duplicates
 
If you want multiple things, just put them in a tuple & pickle that. If you had 5 different pickles in the 1 file, and you only want to unpickle the last one, you'd still have to unpickle the other 4 anyway.
 
 Undocument the existence of the memo attribute and its main use-case.

Apparently, Guido don't like this behavior. [1]  I also agree that the memo
should be left as an implementation detail.

[1]: mail.python.org/pipermail/python-dev/2009-March/086809.html
 
@MisterMiyagi Yeah, ok.
 
2:43 PM
GVR mentions sending pickles over a stream, and how he doesn't much care for the idea anymore
 
@JonClements The earlier conversation is here: chat.stackoverflow.com/transcript/6?m=47698363#47698363
 
The most common justification I see for multiple dump is: suppose you have a program that adds a single new element to a pickle file per execution. If pickle objects may be concatenated, then each execution can run in O(1) time. If the program has to open the file, unpickle the object, add an element to it, and pickle it again, then each execution runs in O(N) time.
 
@Kevin Hey Kevin, I am so sorry, I have sent the wrong script. I am so sorry. Instead of #df.KW.resample('-30T').sum() it shoud be df= df.set_index('dtime').asfreq('-30T') > can send you an example where this function works fine in a single data set
 
@Kevin isn't that what multiprocessing does?
 
I have never fully understood how processes talk to one another. My assumption: OS-level magic
I guess it's plausible that pickle-over-streams could be built on top of the magic.
@Verbamore An example would be good. But I'm more interested in an example that causes the error, rather than one that works fine.
 
3:05 PM
docs.python.org/3/library/multiprocessing.html#pipes-and-queues confirms that multiprocessing queues use pickle.
 
3:17 PM
@Kevin Kevin, I just added the dataset(excel file) to the question
 
@AndrasDeak I couldn't find proof, but I bet the venv executable vendors its own pip that it uses internally to bootstrap everything, and it uses that as well to put pip inside the new env. after all, you can use pip to install pip.
 
Thanks. I still find it weird that it doesn't come vendored with python itself, since pypi says "pip is already installed if you are using Python 2 >=2.7.9 or Python 3 >=3.4 downloaded from python.org"
 
3:34 PM
it isn't? doesn't python -m pip --version do anything?
It does for me in the pythons I compiled from source
$ python3 -m pip --version
pip 19.2.3 from ~/pythons/python_3.8/lib/python3.8/site-packages/pip (python 3.8)
 
@MisterMiyagi i have added the dataset to the question too.
 
hello, do you know what this notation is called?

{w: i + 2 for i, w in enumerate(words)}
I think its list comprehension? but not sure of the colon after w
 
Close. It's a dict comprehension.
 
dict comprehension
GAH!!!
 
3:40 PM
thanks
 
the colon sets up the key:value pairs within the dictionary
cbg folks!
also... could I get a hug (or a high-five)? I'm not having the best day
https://images.app.goo.gl/Zoh9bbGZ4FvyySr58
 
There, there.
Even if you don't land the big contract, you'll still have my unconditional support
 
@Kevin Kevin, did you have any chance to run the script?
 
I'm looking at the data now, going to run it in a bit
 
many thanks, K
 
3:46 PM
@Kevin Thank you!
 
Google tells me that this error can occur if your data isn't sorted. Dunno if that's relevant here
 
@Kevin I don't think so! let me send an example where the function works fine. just give me a sec
@Kevin here I have an example. one dataset. dropbox.com/sh/8yscqnwvqfi6dzg/AACVg3RloSFCSu76L_JCNJ8Ja?dl=0
 
I suspect that the code is failing because your data has two rows with identical dates and times. In particular, the third and fourth row both have a dtime of 8/27/2019 22:30.
I think those are the only duplicate rows on the first sheet. Try deleting one of them and see if that helps.
 
Arbitrary code execution with pickle, round 2: Set the winner variable to True. Difficulty: Epic
import io, pickle

class RestrictedUnpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module not in {'functools', 'operator', 'types', 'typing'}:
            raise pickle.UnpicklingError("forbidden: {} {}".format(module, name))
        return super().find_class(module, name)

data = b'YOUR DATA HERE'
RestrictedUnpickler(io.BytesIO(data)).load()

if winner: print('You win!')
Though I'm thinking there's probably an easier solution that only needs access to operator
Actually, I overcomplicated the everliving heck out of it
 
4:01 PM
Oh, the other sheet has the same duplicated row. If I delete both duplicated rows, then the program runs without crashing.
I wonder if there's some whizbang dataframe method that removes duplicates, so you don't have to remove them yourself in excel
 
@Kevin yup
 
@Kevin Kevin, i think it is working. thank you so much for your help!
 
 
@Kevin ahaha
 
4:18 PM
It takes all of my wiles and cunning to answer a pandas question without actually learning anything about pandas myself
 
4:37 PM
I'm getting close to the yak's skin. I have to exclude the test_socket test which hangs on the cluster, but I need to run the tests in order to optimize the interpreter. When this make test runs without hanging I'll have to figure out how to pass "skip this test" to an optimized compile. My first guess will be to use the exact same command line arguments...
 
speaking of... anyone have an good pandas resources to get me started?
 
I've been muddling through with just the official docs and while it ostensibly explains the "how" of anything you'd care to do, it hasn't answered many of my "why"s
I have a decent chance of being able to understand somebody else's code, if I'm willing to look up every function. But it's very rare for me to be writing code and say "you know what would improve this? pandas."
 
4:54 PM
I bet that's already true for numpy, which has broader use cases
if your problems are not numerical or data-y then you'll see little use of these libraries
 
Yeah. My projects tend not to be data-y so it's not necessarily a mark against pandas that I don't often find a use for it
 
My problems are data-y but I don't want to use pandas. Then again my problems are data-y enough that vanilla numpy is perfect.
 
looks like SO pushed an update this morning
 
yup
 
the MCVE close reason has changed as have the look for closed and deleted questions.
 
5:04 PM
also typo is no longer a reason
 
oic...it's just "no longer can be reproduced"
 
run away votes on an adequate answer but far from the best. I do my best to comment but alas, I fear my comments lead to rebellion votes or rebellion non-votes /grumble
 
baaah, I thought my python was working, and it's borked :(
0:00:00 load avg: 0.14 [1/1] test_ctypes
test_ctypes skipped -- No module named '_ctypes'
test_ctypes skipped

== Tests result: SUCCESS ==
hilarious
I hate it that it's so yamming difficult to figure out the actual dependencies of cpython cpython-devguide.readthedocs.io/setup/#build-dependencies
every suggestion is fine and dandy as long as I have root privileges and can run apt :|
 
wim
5:23 PM
@AndrasDeak yes
 
wim
and you should probably ensurepip when you compile python
it's a configure flag
 
ah, thanks
although I'm happy with a venv-only pip too, that's how I'm going to use it anyway
 
wim
it's still useful to have a system pip for stuff like pip download and pip list
 
but I'm just a user, I can't system install anyway
guess I have to procure libffi...
 
wim
5:30 PM
you never use pip install --user ?
 
that's what I always do on my home computer, but on clusters I stick everything inside a venv
 
wim
some stuff that I use in every project, I prefer to have in the user site than per-venv (e.g. black, reorder-python-imports)
 
seems cleaner and safer to make the workers activate that before running my jobs
and I only use python for one thing...
 
wim
ah ok I didn't realise this was for a cluster - yeah I can't think of any reason to need pip there
 
Number of times I've used assexps for a serious purpose += 1
if x := a_filename_set-b_filename_set: print(f"files exclusive to {a}: {x}")
 
5:34 PM
I find your lack of parentheses disturbing
 
hi, I am trying to get this github.com/ETCBC/bhsa to work I tried this, colab.research.google.com/drive/… But I get an error "ModuleNotFoundError: No module named 'tf'" as you can see
oh I see !pip3 install text-fabric
as noted, it turned out ^ worked
thanks
 
5:52 PM
sorry didn't see you had added to the comment
 
boss: "We need to make changes to this file, whose format is proprietary and undocumented."
me: "you're asking for the moon. It could be literally anything. I'll need ten years, five postdocs, and twenty million dollars. And even then, we could turn up noth -- oh, it's just a zip archive containing xml files. Never mind."
me: "ok, I can do it with Python in five minutes"
boss: "it needs to run on the high-security server, which isn't allowed to have any scripting languages"
me: "ok, I can do it with Bash in ten years, five postdocs, and twenty million dollars"
 
6:09 PM
bash doesn't count as a scripting language?
 
Shh, don't tell the high security server administrators
 
All I need is a Python-to-bash transpiler and I'm good
 
6:22 PM
Reading Fluent Python yesterday about += augmented addition really cleared up a few things, but russled some other jimmies (if I may borrow your expression, Kevin). I'm pretty sure I recall from C++ days that += was an in-place mutator, so that in Python list_a += list_b was equivalent to list_a.extend(list_b). One side effect for me is that I often forget to return self when implementing __iadd__.
But += turns out to be a mutator and assigner. The pivotal example was, what does this do?: t = (1, 2, 3, [4, 100]); t[-1] += [200]. It turns out it both mutates the last element and raises TypeError for trying to mutate a tuple, by implicitly assigning t[-1] = t[-1].
 
Hmm, surprising
 
So t[-1].extend([200]) succeeds, since it is an in-place mutator.
 
Yep, dis.dis("a[-1] += [2]") has both INPLACE_ADD and STORE_SUBSCR all right
 
hello, anyone know how to iterate over the items in a pandas dataframe group (result of a groupby)
 
Is it anything like itertools.groupby?
 
6:25 PM
I don't know what itertools is
 
@erotavlas SO main is full of groupby, just looking at coldspeed's or piRsquared's post should give you hints
 
:? I've been searching on google for the past 15 min, I haven't found an example yet of iterating over the individual items of a group
for each group i can get the head(), count() show the entire group etc
 
This ties in to my conspiracy theory that pandas makes it intentionally hard to iterate by anything other than columns
 
Following modules built successfully but were removed because they could not be imported:
_ctypes
AAAAARGH
cpython hates me (and it's becoming mutual)
 
6:36 PM
i got it :D
sentencegroups = data.groupby("Sentence #")
for name, sentencegroup in sentencegroups:
    print(name)
    print(sentencegroup)
    print(type(sentencegroup))
    for index, row in sentencegroup.iterrows():
        print(row['Word'], row['Tag'])
    break
 
The break will cause you to exit after the first group, but I often do that also to early exit when first developing/debugging so I don't dump out more than I need to figure out the iteration bits.
 
Hmm, just unzipping and rezipping this proprietary file format is enough to make the proprietary file reader reject it as corrupted. So much for a five minute solution.
 
@Kevin start recruiting those phd students
during their downtime they can research how to build python without root privileges and missing dependencies...
 
@erotavlas groupby will return a tuple of the grouped key and the subset of the dataframe in that group. In the outer loop, separate the key and the data, and in the inner loop, loop through the group data as you would any normal df
 
Hmm, if the problem really is metadata, that might explain it. I'm unzipping and rezipping using 7zip, from the file explorer. It's possible that this doesn't persist the metadata. Perhaps if I zipped and unzipped from within Python, the metadata would be preserved.
 
6:44 PM
I'm on my phone so going from memory, but it's not anything esoteric to do
 
@Kevin Also zips have timestamps, right? I had an asker whose testing/checking was foiled by timestamps in zips
 
Anyway, time to drive home so I won't be able to help for like 45 mins.
 
I'm optimistic that the proprietary file reader doesn't care about timestamps. The user is allowed to modify the file, so the reader probably isn't bending over backwards to verify things for security purposes. I'm just trying to figure out how to change the file without the provided toolset, which is annoying to use.
Nevertheless, I'll keep it in mind, in case no other explanation is forthcoming for why my changes don't work
 
guess you can also vimdiff the zips...
 
The old zip and the new zip differ by 39 KB, so something is definitely going on
 
6:51 PM
yeah...
 
At this stage I don't need direct tech support, but I ask each of you to join me for a ten second lamentation at the imperfection of the universe
Wailing and gnashing of teeth optional. And... Go.
 
good thing I got a good 4 hours headstart
 
[lamentation intensifies]
That's my secret, cap... I'm always lamenting
"Lamentation? In my emotional landscape?" It's more common than you might think
 
7:09 PM
Maybe just with and without compression? Since there is a size difference? Or you are using a more recent compression mode that postdates the prop software?
Or aliens?
 
7:22 PM
Close reason has been changed to: It's caused by a typo or problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers. — Yaakov Ellis ♦ 3 mins ago
 
@Kevin You're falling behind on your quota a little bit. You should give more priority to your pulls glasses assignment.
@AndrasDeak you're on a scientific HPC cluster? Intel compiled python3? IIRC ours once had the proper dependencies with a module load for python3.
 
This isn't going to affect my performance review, is it? If I get high marks I can turn it in for bonus tokens at Chuck E Cheese.
My skeeball skills are sub-par and I need the handicap
 
@MisterMiyagi if I had system python I wouldn't want to build it from source
the fun thing is this:
 
ah, I'm just used to python3 being horribly outdated. non-existence is shocking indeed.
 
$ module load python/3.6.1
$ python3
Python 3.6.1 (default, Mar 14 2018, 10:39:45)
Traceback (most recent call last):
  File "/etc/pythonstart", line 7, in <module>
    import readline
ImportError: libreadline.so.6: cannot open shared object file: No such file or directory
I have working 2.7.5 (...) and broken 3.6.1
so I figured I'd build 3.7 myself
 
7:32 PM
@Kevin you got some bonus point for trying. Plus damages for pain and suffering.
 
@AndrasDeak looks like an environment loader. What server are you working with? I have a feeling I've worked with something similar in the past
 
@AndrasDeak do you have the 3.6 loaded when trying to build?
 
@MisterMiyagi no, but that's a really good idea, thanks
@inspectorG4dget beats me :)
RHEL according to python's startup blurb
 
if it's any help, you probably have way more than 5 postdocs sharing your pain.
 
module load is something I've seen in professionally managed clusters. But I digress
 
7:35 PM
here's hoping that whatever is breaking the system 3.6 won't break my 3.7
@inspectorG4dget yes, it's an hpc cluster
> checking for X509_VERIFY_PARAM_set1_host in libssl... no
 
@AndrasDeak AHA! Wait, you're in Canada. Any chance you're using ComputeCanada?
 
@MisterMiyagi it was worth a try ^
@inspectorG4dget missed by 7k kilometers ;)
 
@AndrasDeak whoops!
 
user11867329
@Kevin Quick question about the thing you did yesterday. Does it follow this stucture: ...
 
You were only off by like 1.5 CanadaWidths
 
user11867329
7:39 PM
3 paragraphs, 1 line of 1rst paragraph is element1, last line of para1 is desc (element2), in between is element3 (shortdesc). Then para 2 is eleemnt 4 and para3 is element 5?
 
I'll still try building it, because I have ssl built and functioning locally...
 
user11867329
Cause I checked the content file and some of them are not properly forumlated, I am sifting throught the 8000 lines
 
odds are the system doesn't have libffi development headers to begin with :(
 
@OakDev Yeah.
 
user11867329
Just need to know the proper way to do it so I don't work for nothing... like you did for me
 
user11867329
7:40 PM
Is it, or you're getting rid of me?
 
No, that's how it's structured.
 
user11867329
You're a genius. Thanks again.
 
nope, still no ctypes :( Thanks for the tip anyway, @MisterMiyagi
 
user11867329
7:52 PM
@Kevin and I grant you my authorization to kick/ban me out of emotions, twice.
 
there's a couple of pros in this room right now. I'd like to ask you guys for some career/professionalism advice
a while ago, I was put on this project (let's call it wine). It took me a while to get my bearings with the data, etc, because I'm new in g4dgetCorp. But I eventually came up with a design and started to implement it
there's another guy in g4dgetCorp who has worked on a similar project in the past, and he got on board to help out. We were already against a super-tight deadline, so I was happy to take the help
soon, he started to change experimental parameters and completely changed the research question we were asking, which lead to a LOT of time loss; to the point where Mr. Manager had to put two other ninjas on the project to get it across the finish line in time
I took it hard. It was a gut-punch that the thing that I had set up to work so beautifully got replaced (Mr. Otherguy ended up using his model from the last project, instead of what we were developing here). (I still think) we could have maged to finish it if Otherguy hadn't kept dismissing my decisions at every turn
 
@AndrasDeak then my professional advice is "complain to the admins. loudly."
 
00:00 - 20:0020:00 - 00:00

« first day (3300 days earlier)      last day (1641 days later) »