« first day (3460 days earlier)      last day (1505 days later) » 
00:00 - 17:0017:00 - 00:00

5:24 PM
@roganjosh Brython looks amazing
Is DOM stuff possible with Brython?
 
Honestly, I don't know. It's not something I'm really interested in tbh, just something I'm aware exists
 
Oh okay. I'll look into it then and find out
 
user11585758
Hi guys whats up
 
Not much my dude how about you?
 
user11585758
fine :)
 
5:34 PM
Me too ;)
I'm going to try to learn how to use Brython
 
I'd be interested in what you report back when you have. I don't think I really understand what underpins its inception rather just learning some JS, wonky as it can be
 
strong typing ;)
 
TypeScript. Just sayin' :)
 
5:49 PM
if youre at all comfortable with JS or one of it's superscripts, everything else that's not-js is an unnecessary abstraction for you.
 
Oh, neat. Looks like this also answers your question about DOM too, @NoahBroyles
 
What's a concise way to do this:
for i in pos_word_soup:
    if i[1] == 'UH':
        return choice(uhh)
If you only are concerned with the second part of a tuple, for example, you're not going to just use in.
 
@JohnnyApplesauce that's alright. You could use next with a default but that's not necessarily more readable to everyone
 
(I'm using NLTK's pos_tag(), which returns a list of two-tuples consisting of a word and a part of speech like "VB" or "UH")
 
(and probably marginally slower)
Does the return value depend on the tuple? In your example it's unclear...
 
5:52 PM
i find this format pretty readable. if you wanted, you could use any with a generator, assuming you don't really care to use the actual information given by the tuple itself
d'oh! kevin'd
 
Wait, how would you do that with any?
 
@JohnnyApplesauce if any(i[1] == 'UH' for i in pos_word_soup): return choice(uhh)
 
what I meant was hit = next((i for i in pos_word_soup if i[1] == 'UH'), None) where None is what you get if there's no hit
 
5:56 PM
cbg
 
cbg?
cabbage
 
I was guessing that that might have been a filter thing
 
What might have been a filter what?
 
It's part of our optional salad language.
 
6:01 PM
Last semester two things happened whose coincidence I noticed
1st: I was in a computer room on campus and I heard one guy cutting up. He mentioned someone who had graduated from this uni (in CS) and, at the end of it, didn't know what a for loop was.
One person lamented about how he cheated himself into a degree and out of an education, and another joked about how he must be a Haskell master.
2nd: I was grading an assignment (I was UG TA) for an intro course (not even for CS, for IS, so less rigor), and I noticed that the Python submission wouldn't run.
That student came to me earlier (this was the last or second to last assignment of the semester) and was like "how do you make this instruction repeat?" and I told them "put it in a loop."
It now wouldn't run, unlike earlier, because that line now ended .mainloop()
The traceback was impressive
 
Wow lol
 
@Permian a database is a kind of storage. a cache is a use-case for storage. a link is only useful if complete.
 
6:16 PM
fellow pythonistas can you review and open this post if you deem it salvagable - i think it is : stackoverflow.com/questions/61060291/…
 
@Datanovice I'm not gonna vote to reopen that because they don't even have the original data. "current output" and "expected output" are not helpful without an input. Is there a reason you want it re-opened?
 
I generally find textual examples of dataframes acceptable - sure it's best practice to generate them as code but I dont think that question was any different from the standard daily ones under the pandas tag @roganjosh - infact with a sample input and output it's already ahead of 60-70% of the tag
 
@JohnnyApplesauce, they are destined for management
 
@Datanovice Am I missing something? I don't see any attempt or technical problem description by the OP.
 
6:22 PM
@Datanovice That's just a reflection on the tag, though, right? It doesn't mean that we should reverse a positive outcome
It's unfortunate that crap keeps getting answered on the tag, but if we normalise for the stuff that gets through, then the problem just gets worse. Do you think it will really help to have this answer reopened?
 
@roganjosh I guess you're right - I just thought it was a tad unfair noted @MisterMiyagi
 
check this answer . looks weird
 
@αԋɱҽԃαмєяιcαη what's wrong about replacing the escape with its unescaped value?
 
how about if there's multiple chars?
 
I'd assume that there is some ready-made code for such a translation, but it seems to work.
Then replace/translate them all. oO
 
6:35 PM
html5lib is already handle the AMP
 
Besides, there aren't.
 
well. i found it's not elegant to replace it manually if there's a parser which care about that actually
 
I often avoid additional dependencies for solving minor, isolated problems. Good to have both answers, though.
 
Yes, i were in the same logic before. but while working on big scraping project where i noticed multiple data is missing. then i noticed that some of the pages used different escape chars which i was need to check what those is. at the end i found that i can escape all of them without care about difference between each page.
 
Understandable, but that doesn't make the accepted answer invalid.
 
6:48 PM
well it's works for it's current state then.
but don't tell me it's looks like the site has added to avoid scraping
 
Yeah, that was weird. It's gone now, though.
By the way, @αԋɱҽԃαмєяιcαη, are you aware that you are using code formatting instead of emphasis/bold? E.g. in your recent comment, "OP", "AMP", "Technically", and "looks like the site has added to avoid scraping" are formatted in code monospace.
 
Yes I'm fully aware of that.
 
user10984358
7:15 PM
replace(" u'", "u'").replace("u'", "").replace("'", "") if that is done to "Some text editing to remove unicode characters (commented line)" is it badly written python code?
 
so you want to escape the unicode within string?
 
user10984358
I am going through an old code some one else wrote, I came across that line
 
user10984358
it felt weird to me and I cant ask that person cuz he isnt working here anymore
 
user10984358
I am in the process of converting that to python3 so I wanted to know what that did
 
i can't confirm if this is unicode even. it's looks like a wired output where he were looking to remove the single quotation badly.
 
user10984358
7:24 PM
that is what it seemed like to me
 
he was can just strip and [:-1]
 
user10984358
that whole code is making me uneasy, who has a logger where the first line is their name followed by a debug comment? log.DEBUG('name:HERE1')
 
something like he were having a text in that shape bla 'n where he wanted to have it like bla'n
 
user10984358
removing space?
 
user10984358
I guess I must understand what unicode is
 
7:27 PM
Oh no. i think not.
he just removed space and single quotation
 
@TheNamesAlc what is this used on? Python source code?
 
looks like somebody who doesn't understand what u"..." means trying to convert it to "..." strings; we see that in beginner questions a lot
 
user10984358
@MisterMiyagi python source code that was used in the org
 
but why replace the single quote as well? it would change u'Hello World' to a bare Hello World.
 
user10984358
i thought this was similar to remove b in byte strings without decode
 
7:30 PM
@MisterMiyagi if that's the case, so he just remove the first part?
 
@TheNamesAlc That's like saying "remove the . in floats". Sure one can do it, but it totally changes the meaning.
@αԋɱҽԃαмєяιcαη pardon?
 
I mean if he badly try to convert it to string so he just remove (u') which is the first part
 
user10984358
there is also this comment # Looks ugly, but works
 
@αԋɱҽԃαмєяιcαη the replace("u'", "") will remove the leading u and single quote, and .replace("'", "") will remove all single quotes. If the input was a unicode string literal, the output won't be a string literal.
Unless it was a double-quote string literal inside a single-quote unicode literal...
 
Oh yes, you correct. just noticed the second replace
 
user10984358
7:37 PM
In cases where you cant run the whole code but you are expected to what do you say? I am the only person working on this
 
user10984358
Pretty sure I can debug this if i trace the input that is passed, which is the problem since I cant get the code running without a python2 setup
 
without knowing any context of this line, it's very hard to say what to do.
 
user10984358
was list comprehensions a thing in 2013?
 
yes
Mar. 21, 2013 that article been written :D
 
user10984358
ok so that does it, I am going to tell my mentor this is not pythonic I have other instances where the code is not pythonic
 
user10984358
7:44 PM
I am better off writing my own, thanks for the assist everyone :)
 
am not expecting a good code after unicode replace, so you have to do it from scratch
 
so i spent almost a day and couldn't find the reason that why i can't install mysql-python for my django 1 project :(
 
@HabibRehman I suggested earlier that you need to install mysql on the system
 
user10984358
they dont take interns seriously here :/ so I have to go above and beyond to prove a point
 
The python libraries that link to it are useless if you don't actually install the software
 
7:48 PM
mysql is installed @roganjosh what i could read the most relevant clue is that it has something to do with wheel
 
I've to sleep, see you all later.
 
see you ahmed
 
user10984358
I am going to sleep as well, see ya
 
rbrb guys
@HabibRehman you need to give some more info here. What, exactly, is failing?
 
wait let me share the post i just wrote
i tried to explain the issue here, let me know if i have properly explained the problem
https://stackoverflow.com/questions/61067056/error-mysql-python-1-2-5-cp27-none-win-amd64-whl-is-not-a-supported-wheel-on-th
i have an idea because the file says "win" in "MySQL_python-1.2.5-cp27-none-win32" maybe its a file for windows and not linux, not sure though
there were no other files though
 
7:55 PM
Wait, are you trying to install Windows binaries on Linux?
 
ummm o_O
 
If you're on Linux there's really no reason at all to be looking at those
 
so this is the reason it says that its not for that platform
 
Yeah, I'd say that was a pretty accurate error
 
when i search in apt its not getting any mysql-python, i even read somewhere that it doesn't exist anymore and now replaced by mysqlclient
 
7:58 PM
how about with pip?
 
when i do pip, it gives me a same error message which i wrote in the stack question
it also says
Using cached MySQL-python-1.2.5.zip (108 kB)
do i need to clear the cache?
 
I'm gonna go with "no" to that question, but actually I don't understand the error is in terms of trying to fix it
 
the first line says wheel can't setup " Building wheel for MySQL-python (setup.py) ... error"
it is same like when i write "pip install mysql-python"
this shows up pastebin.pl/view/fe4aedbc
 
8:17 PM
need more details/ debugging help stackoverflow.com/questions/60899040/…
@roganjosh Not bad. My recommendations for ambient while programming 1) Mike Oldfield's 'The Songs of Distant Earth', a 1994 album inpisred by Arthur C. Clarke's novel that critics dismissed as 'formula Oldfield' (new-Age electronica, worldbeat samples, synth, Gregorian chant). 2) if you like synthwave/darkwave it's worth subscribing to youtube.com/user/ThePrimeThanatos
 
@smci I'm giving The Songs of Distant Earth a listen now. The flying manta rays on the pictures... I've seen that before
Nm, I was thinking about Asia - Aqua and that's the well-known orca-ray
 
8:42 PM
Is there a reason why numpy chose the convention of broadcasting along the last axis rather than the first?
 
wim
can you give an example of what you mean?
 
np.arange(20).reshape(4, 5) + np.arange(5) yields np.arange(20).reshape(4, 5) + np.arange(5)[None, :]
np.arange(20).reshape(4, 5) + np.arange(4) yields ValueError: operands could not be broadcast together with shapes (4,5) (4,)
So arrays are automatically broadcasted with respect to the last axis rather than the first.
For operations like addition, etc.
Do you see what I mean?
Found a statement about it in the docs: "When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward."
Of course I've been accustomed to this convention and used it for many years now, but I'm doing an application where I've noticed it seems less convenient than starting with the leading dimesnions.
 
cabbage all
Bugreport: the "welcome back" banner does not play well nor does it conform to nightmode
 
wim
9:08 PM
@user76284 yes, I see what you mean.
 
@user76284 yes, it's probably consistent with row-major memory layout
at least MATLAB has column major and implicit trailing singleton dimensions, so the two are probably related
For instance, broadcasting shape (n, m) to shape (k, n, m) means that for each i in range(k) you take a contiguous block broadcast_arr[i, :, :]. If broadcasting went the other way you'd have differently_broadcast_arr[:, :, i] be a contiguous block, which is not usual. I'm aware this is just handwaving, it's just to show how I think the two might be related.
For the heck of it I tried looking up ancient history. I can't find numarray's documentation, but numeric can still be found. The docs says
> If the source and the target have different shapes, Numeric Python attempts to broadcast the contents of the source over the range of the target. This broadcasting occurs for all dimensions where the source has dimension 1 or 0 (i.e., is absent).
If there exists a dimension for which the two arrays have differing lengths, and the length of that dimension in the source is not 1, then the assignment fails and an exception (ValueError) is raised, notifying the user that the arrays are not aligned.
so numeric didn't broadcast implicitly, and that probably means that the history of implicit broadcasting should be in numpy proper
 
9:30 PM
Thanks for the explanation.
 
Sorry I can't be more exact. I just looked up the git log but can't find anything relevant :)
 
9:54 PM
@user76284 Because it's how Matlab does it and is consistent with row-major layout, hence good performance for common operations, like AD said. Are you just enquiring, or do you have a performance issue?
 
Not really performance, but syntax.
 
@smci you mean it's how MATLAB doesn't do it ;)
 
For my specific application I keep having to write a[..., 0], b[..., 1, :], and such.
With the ... at the beginning.
That led me to wonder why this convention was chosen.
 
@user76284 that's often a good sign that you should redesign
In a row-major array (C-order, default in numpy) arr[k, ...] is contiguous, arr[..., k] is not. And contiguous means "good for caching" which means "fast".
 
The problem is I'm dealing with arbitrary-shape batches of $n$-tensors, where the ... basically goes over the batch dimensions.
But I have to think about it.
If the batch dimensions were at the end rather than the beginning then I could just do a[0], b[1, :], etc. But then I'm forced to add in the extra dimensions when doing broadcasting operations like addition, etc.
 
9:58 PM
In the meantime I've found a mailing list thread from 2006. This message is a subjective explanation of how "the dimension padding" (i.e. the current behaviour of implicit dimensions when broadcasting) happened according to a self-admittedly unreliable source
> First, padding the dimensions.I believe that this is mostly historical baggage.
@user76284 batches are usually the first dimension. Did you misspeak?
 
@AndrasDeak Yeah, fixed.
 
OK. Because numpy/scipy's philosophy is shape (nbatch, m, n) is a "stack of matrices".
 
has anybody seen this error before in django "Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-f107x9mb/supervisor/"
 
10:20 PM
@user76284 upon rereading your example seems a bit off. The "where does numpy assume singleton padding dimensions" problem would imply that you have to add leading None dimensions to your code. That's not in your example (but that might just be a mistake in the example)
 
Not sure what you mean.
You mean where the ... is?
... being a literal ellipsis in the code, sorry if that wasn't clear.
 
That's fine, but that doesn't inject new dimensions
Leading vs trailing singletons only means omitting a None index (or more)
 
Hmm I still don't quite understand.
Which specific code are you referring to?
 
2 hours ago, by user76284
np.arange(20).reshape(4, 5) + np.arange(5) yields np.arange(20).reshape(4, 5) + np.arange(5)[None, :]
That ^ is related to implicit dimensions when broadcasting
29 mins ago, by user76284
For my specific application I keep having to write a[..., 0], b[..., 1, :], and such.
That ^ is not
Changing the behavior of padding dimensions only helps with a problem of "I have to keep injecting None padding indices for broadcasting"
 
I mean when I want to add b[..., 0] and a 1D tensor, for example.
Sorry, suppose b has 10 axes, where the last axis has dimension n.
 
10:26 PM
And what is the size of the 1d array?
 
I guess a better way to put it is, let positions be a tensor with dimensions ([some batch dimensions], 3).
I want to do stuff with x = positions[..., 0], y = positions[..., 1], z = positions[..., 2].
Do you see what I mean?
 
Not necessarily...I don't see the broadcasting.
Is your issue that when you take an item along the last slice you lose that dimension?
so that positions[..., 0] no longer broadcasts with positions?
 
One moment.
 
sure thing
I'm not being obtuse on purpose, I just want to understand your use case properly.
 
Yeah no problem, I’m trying to come up with a good, concise example of what I mean
 
10:31 PM
thanks
 
Suppose I have an array positions of shape (1000, 2000, ..., 3) denoting the 3D positions of particles or something like that.
 
Suppose I want to add a particular vector, say vector = [3.4, 5.6, 2.3] to all of these positions.
 
still OK
 
By putting the batch axes first, I can use numpy's broadcasting convention to do positions + vector.
 
10:33 PM
yup
 
So that's good. The downside is, to access x, y, and z, say, I need to do positions[..., 0], positions[..., 1], positions[..., 2] rather than the more concise positions[0], positions[1], positions[2].
So you might say, ok, let's put the batch indices last then, so you can index into the components in this way.
 
@user76284 Ah. Yeah, no, I wouldn't say that.
 
So make positions have dimensions (3, 1000, 2000, ...).
 
This issue is not broadcasting-related
 
The problem then is that I can't write positions + vector (under numpy's broadcasting convention).
The point is that if numpy had chosen the broadcast-along-first-axis convention, I could have the best of both worlds.
 
10:36 PM
@user76284 no, positions + vector would break then
 
I could let positions.shape = (3, 1000, 2000, ...) (so I can do positions[0], etc.) and I could write positions + vector.
 
ah, you mean like that
 
So I can use both positions[i] to index into the ith component and write positions + vector.
Because the "semantically important" axis comes first.
So to speak.
 
OK, I see your issue now. That's probably an inevitable consequence of the broadcasting rules and how you want to handle your data. My suggestion is to use np.moveaxis
>>> np.moveaxis(np.random.rand(2, 3, 4, 5), -1, 0).shape
(5, 2, 3, 4)
this also doesn't copy, it's basically another kind of transpose
 
Yeah I currently use the ellipsis to jump to the last axis, but it makes the code noisier and for other reasons complicates the logic in my head.
 
10:40 PM
(but if you see yourself slicing along the last dimension a lot you should still consider to refactor. Only a timing test on realish data can tell which is faster)
 
I hope named tensor axes become a thing.
 
@user76284 moveaxis would remove that
 
Does anyone know a tool that I can keep track of the command I have run? The commands include SQL, Shell, some short python code, and so on. The tool can keep track of the date and have a nice syntax highlight.
 
@user76284 I think xarray does that?
 
Probably. Hope this approach becomes more widely used in the future.
 
10:41 PM
anyway I'd be inclined to do x, y, z = np.moveaxis(...) to have access to those arrays. But note that this will create copies.
 
I’m tired of trying to mash arrays of different shape together for various operations.
 
I have a very strong feeling that moving padding dimensions would lead to problems to a lot more use cases (like pairing a stack-of-matrices 3d array with a single 2d array)
@user76284 you can always pad yourself and work with arrays of the same number of dims :P
 
That sounds horrifying with my current code :P
 
wim
@whatsnext Linux
 
@user76284 oh I remember seeing that, it seemed neat
ah, no, I saw einops github.com/arogozhnikov/einops
 
10:44 PM
@wim I use mac
 
Interesting, I’ll look into it.
 
it's mentioned in what you posted, that's why I thought I'd seen it
I don't need named arrays but I do like some fancy array slicing and dicing
 
11:12 PM
I'm trying to read a json file with: d = json.load(open('tweet-meta-by-id.json')) but am getting OSError: [Errno 22] Invalid argument
anyone ever get that before?
 
is there a way to force python threads to actually be concurrent?
like, to bypass the GIL somehow?
 
@duhaime try separating the open (you should be doing that anyway), see where the error is coming from
 
multiprocessing module @AmagicalFishy, or stackless python or pypy
 
oh, cool. yeah i'll use that instead—thank you
 
@AmagicalFishy GIL implies tasks might be concurrent, but not parallel stackoverflow.com/questions/1050222/…
 
11:14 PM
Truth is I had it as a two liner with a file handler, but both ways it's the json.load() call that throws the error
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-8-51be1016a95e> in <module>
1 with open('tweet-meta-by-id.json', 'r') as f:
----> 2 d = json.load(f)

~/anaconda/envs/3.5/lib/python3.5/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
263
264 """
--> 265 return loads(fp.read(),
266 cls=cls, object_hook=object_hook,
 
@duhaime can you reproduce with another file? Something small you write by hand as an example?
 
I cannot, other JSON files load just fine
 
Weird, I definitely can't help you there.
 
I can validate the json with cat tweet-meta-by-id.json | python -m json.tool
 
ah! noted—thank you @AndrasDeak
 
11:16 PM
This JSON file is only 3GB too
 
@duhaime Perhaps some crazy long line that messes with buffering? Check the longest line perhaps unix.stackexchange.com/questions/150800/…
 
distinctions between the words concurrent, parallel, and asynchronous. I always consider parallel and concurrent to be synonymous
 
@Todd read what I linked
 
It's a good question. The whole file is one line! That could be the problem...
 
@Todd to make it more obvious: async in python enables concurrency, but it's not parallelism
 
11:20 PM
i already understand this
 
then you should not say things like "I always consider parallel and concurrent to be synonymous", because you either don't or you misunderstand the word "synonymous"
 
Because i do consider it that way when i personally use the terms
 
So you understand but intentionally misuse the terms?
 
yes
=)
 
11:22 PM
for shaaame young master, for shaaame
 
@AmagicalFishy usual caveat: make sure your problem is CPU-bound before you try throwing parallelism at it
of course if you do it right (i.e. profile your code) the worst that can happen is that your refactor only makes it slower
 
oh yeah. it's totally cpu bound—trying to reduce 7-seconds to hopefully ~1
 
does anyone know of a fast way to bust a single line json file into separate lines? The JSON is just a 2D array (list of lists)
cat tweet-meta-by-id.json | python -m json.tool > formatted-tweet-meta-by-id.json
is slow
 
@AmagicalFishy good luck
 
correction - this data is just an object/dictionary mapping keys to lists
 
11:25 PM
ty :)
 
@duhaime how about breaking after each ],?
 
My only fear is there's lots of plaintext in the file (it contains tweets about covid) and ], could occur in natural speech
 
then you should use a json parser...no wait
Doesn't json.tool pretty-print?
 
it does
 
Does that not include breaking it out into separate lines?
 
11:28 PM
it does include breaking it into separate lines
 
I'm struggling to see the problem here :D
oh, nevermind, I missed your next message, sorry
guess I should in fact go to sleep
 
ah okay
where do you live?
 
Hungary (UTC + 2)
 
!!!
ya you should sleep!
I'm in bed at 9 lol
 
:)
I can't think of a foolproof way that doesn't involve writing half a json parser
 
11:31 PM
I saw some approaches to streaming Python earlier today that could be worth a shot
I'm just rebuilding the dang file though and have already saturated my machine's RAM so can't afford to open more browser tabs
 
I wrote a json parser... here's the code jsondata = eval(open('file.json', 'r').read())
 
ha, but that won't stream the data!
Try to open a 3GB file with that
 
@duhaime just to hide my shame with a bit of a nitpick: you should use python -m json.tool < tweet-meta-by-id.json > formatted-tweet-meta-by-id.json. I'm sure cat is not the bottleneck, but it's usually nicer to avoid its useless use
 
aha!
I didn't know one could do that!
 
What most people don't know is that you can pass in string literals with <<< which removes a useless use of echo :)
 
11:34 PM
that's neat!
 
whenever I need arithmetic in bash I do stuff like res=$(bc -l <<<"$var * 0.7")
 
i just started learning bash
 
Hmm, well Todd's suggestion made me want to just read the file in as a string and then json.loads() it, and that's throwing a JSON decode error. That must be the problem?
 
maybe
> Some JSON deserializer implementations may set limits on:

* the size of accepted JSON texts
[...]

This module does not impose any such limits beyond those of the relevant Python datatypes themselves or the Python interpreter itself.
is the size of the string read from .read() consistent with the file size?
 
yesterday I spent an hour debugging JSON deserialization in javascript because I didn't know javascript only supports a max of 52 bit integers and twitter ids are like 100bits
interesting question
once I get a little RAM I can check
what makes you wonder?
 
11:41 PM
you could just check the last few characters, and compare with the file
@duhaime well either your json is borked or .read() truncates some of your data, if we believe cpython's claim that the json module doesn't impose unexpected restrictions
unless of course there's something like "python strings can only be N long on 32-bit systems" or whatever
 
that would be wild
but I've read huge text files with no issue thousands of times
 
I don't know if .read() is allowed not to return before EOF
 
this json file is just a huge text file
 
@duhaime yeah
 
it's the json parsing that bombs
I think that file is borked somehow
 
11:43 PM
I guess it's way more likely that your json is corrupt. Not that hard to break a download of 3+ GB
 
I just wrote it earlier today with python and the process exited without issue
I wrote it on this machine
that's what puzzles me
 
"wrote the file without issue" doesn't guarantee that all the data is on disk and that the data was complete to begin with, does it?
 
I should have just stored the data as a numpy array or something
well I don't know
 
if it's a lot of strings, which it is, then you're not better off with numpy
 
it was just a python dictionary
 
11:45 PM
perhaps you could use some database next time
 
each row has 8 vals, like 4 ints, 2 strings and 2 floats
 
pandas?
 
blah I really like flat files
nein I like numpy but I don't use pandas much
 
pandas does have a json loader.. but is it just the usual json module it uses
 
I like numpy too, but I like to use the right tool for a given job.
 
11:46 PM
this file is composed of lots of little metadata files from a high performance compute cluster
ya I hear you
but how would pandas serialize a dataframe like this?
if that's the word
 
@duhaime no idea, I don't use pandas. But your data would fit perfectly in a dataframe with 8 columns
 
haha ya
 
sounds like a common issue (serializing dataframes I mean), so there has to be both a way and a lot of info about how to do it
 
but it's a nice little dictionary at the moment, where keys are tweet ids and values are some data about that tweet
can one post images in this room?
 
yes
 
11:48 PM
but.. is the json support in pandas dependent on the the regular json module
 
@duhaime sounds like you want to use the tweet id as an index, which is fine
@Todd there's three of us here and two of us have already said we don't use pandas. Go google.
 
no need to get testy
 
I'm not testy, I'm just telling you to go search answers to your own easy to answer question.
 
It's not my question man
 
3 mins ago, by Todd
pandas does have a json loader.. but is it just the usual json module it uses
1 min ago, by Todd
but.. is the json support in pandas dependent on the the regular json module
is it not
 
11:50 PM
do you iknow what a rhetorical question is?
it's when someone phrases something like a question, but what they're really doing is making a statement
 
seems like you don't know what a rhetorical question is :P
 
the problem isn't on my part
 
Anyway, I should go. Good luck with your json, duhaime
 
thank you! Here's the plot I'm working on:
 
the reason I posed that rhetorical question was as a caution that turning to pandas to parse in this json might not solve it
 
11:52 PM
ah screw the JSON
I will write all of my files as Excel spreadsheets from now on
 
yikes
 
lots of tweets
chickenbone is not pleased
 
@duhaime ehm, while pictures are OK, we're not crazy about profanity here. Can you find a tweet that isn't abusive?
 
haha, to do that I need to parse this JSON!
 
heh
 
11:56 PM
for whatever reason, that was the only tweet that was present in the old and new JSON
 
are there linebreaks in the json text?
 
eventually this will be online where you can fly through the space; it's more fun when you're a spaceship and it's not static
 
neat
 
I don't know. The file is one line, but I believe some of the tweets have \n in them
 
grep '\\n' ... can help figure that out
 
00:00 - 17:0017:00 - 00:00

« first day (3460 days earlier)      last day (1505 days later) »