« first day (3012 days earlier)      last day (2161 days later) » 
00:00 - 21:0021:00 - 00:00

00:49
I have a file with url and I want to add a string to it with the below for loop but I am getting string first then URL.
filepath = 'statelink.txt'
with open(filepath) as fp:
line = fp.readline()
with open('software.txt', 'w') as j:
for lines in line:
j.write(line+'/d/software-qa-dba-etc/search/sof')
01:45
As written, line will be a string, so the for loop will iterate through the characters of the string. I think what you want is:

filepath = 'statelink.txt'
with open(filepath) as fp, open('software.txt', 'w') as j:
for line in fp.readlines():
j.write(line+'/d/software-qa-dba-etc/search/sof')
@piRSquared What do you think, do we need a canonical explaining why df.apply is slow and should be avoided, and discuss alternatives where possible?
02:18
@ch
@ChuckIvan what I am getting is string first and then url
/d/software-qa-dba-etc/search/sofhttps://auburn.org/
@JimBeam Try using the code I wrote
nest the for in the with, and the write in the for. I don't know why that would put the url before the path
after the path*
Also make sure you add '\n' to the end of that entire line you want to write
@JimBeam
https://pastebin.com/0TJqmRrT
When you end up reading from software.txt you will want to read it and use strip() to get rid of those newline characters we added
Or read().splitlines() take your pick. Anyways try that code
02:37
I have run the code you have provided but again that is giving me the same result
It's as of writing the string first and then writing the url
also \n is putting the url and string in different lines
@JimBeam Did you overload the + operator? This is strange business
Try writing "hello" + "world" and tell me if you get "worldhello"
ok
it's showing correctly
it's happening wrongly with the loop + string
result is string + loop
but with both as string result is good
02:53
If you can go to pastebin.com paste as much relevant code as you can, create new paste and put the url here
I'll be back in a few
@JimBeam It's also possible you wrote the code wrongly the first time, didn't fix software.txt, and are looking at the top lines from that file but not the bottom (i.e. head vs tail)
File is not a big one around 500 lines with unique url in each line
03:00
If you are on a mac or linux please try > tail software.txt and tell me if that looks correct
I have checked the software.txt file from top to bottom and it's same
in terminal
Well than I am stumped Jim
it's same
But there's alot of smart people in this chat, so try later and they will find what I didn't
but again thanks for helping me.
03:01
I will say this
I don
I don't see why concatenating 2 strings would reverse their ordering, unless the + operator has been tampered with. But it may be something else entirely
Of course. Try again tomorrow morning if you can
You know what
DAMNIT
@JimBeam try j.write(line.strip()+'/d/software-qa-dba-etc/search/sof'). Man I am tired
ok take rest will update you if I get this resolved
hahaha thanks buddy that last one worked
What I think is happening is the lines you are reading from statelink.txt have newline characters (\n) on them so when you write(url + some_string) the url will be on it's own line, the some_string will go to the next line and get added to the next url. Do you see what I'm saying. I should have asked you more about the data from the start. Never trust code like that
I think what you said is nearly right but when I checked the line in the statelink is ending with \
thanks alot I have wasted alot of hours to resolve this problem.
Yeah the first file with the urls has newline characters. Just remove them before making a full url with the pathnames concatenated. I don't know why I didn't see this sooner.
03:15
Yes the line in statelink is ending with '/' because it's the end of the domain. You can either strip that as well with line.rstrip('\\n') or you can remove that first '/' from the path
rstrip('/\n')
OK
but strip() also worked
It's clean if you do:

j.write(line.rstrip()+'d/software-qa-dba-etc/search/sof')

Notice I took out a slash from the string so it flows with the url

at least to my noob standards
I'm sorry j.write(line.rstrip()+'d/software-qa-dba-etc/search/sof\n') Ok I need to shut it down
got it
That'll work
@ChuckIvan Shouldn't you us code formatting :D:
j.write(line.rstrip()+'d/software-qa-dba-etc/search/sof\n')
03:26
Nifty. Triple backticks right? I thought Andras said it doesn't work in chat. I'm pretty new here
03:56
@AndrasDeak Don't ya just see me and Chuck Ivan hacking triple and double code backticks?
:-)
@U9-Forward And I'm sure he would not approve. FYI, please "hack away" at these kinds of things in the Sandbox room
04:48
@coldspeed thanks, done it.
 
1 hour later…
06:12
but... the needs of the many outweigh the needs of the many... or one...
now they killed 511 to save one. :/
07:01
@coldspeed indeed, thanks
@U9-Forward they "work" on a single line because two backticks are a no-op. The point of triple backticks on main is multiline code.
@AnttiHaapala shouldn't have imported vampyre
Also the needs of the vertebrate... :P
07:17
the needs of the vertebrate outweigh the needs of the arachnids
 
2 hours later…
09:25
@ChuckIvan to see what does work see sopython.com/wiki/…
10:15
cbg all
10:40
pd.to_datetime("00:01:18", format='H%:M%:S%', errors='coerce', infer_datetime_format=False) why is this just returning the string as is? Just not sure what the pandas to_datetime is doing here...
%H, %M, %S
>>> pd.to_datetime("00:01:18", format='%H:%M:%S', errors='coerce', infer_datetime_format=False)
Timestamp('1900-01-01 00:01:18')
uggggggh
thanks!
no worries
never programmed in C I guess :P
But like shouldn't the error coerce to NaT? I think that's the bit that I'm finding most surprising
Yeah, it seems it should. But perhaps with no field specifiers parsing is a success (no-op)
10:49
Given I specified infer_datetime_format=False this behavior seems a bit odd to me
11:01
@shuttle87 I tried looking at the source but it's way too complicated for me to untangle. You could raise it as an issue to see if this is a bug or a feature
11:26
@AndrasDeak thanks for having a look, I opened an issue about this
no problem
 
2 hours later…
12:58
cbg, all
13:10
@AndrasDeak Thanks for link
 
1 hour later…
14:13
@coldspeed Hmm... I can see the usefulness. However, I can see many bytes wasted trying to explain "PandasObject.apply exists to make possible many tasks that need be be iteratively applied to each row or column. However, often there is a better way and learning about other methods and techniques can dramatically improve your code and performance."
Also, if you go ahead with such a Q&A, feel free to poach anything I say.
On that topic: a nice canonical about "What does vectorize mean?" would be nice. But it would require surgical dup targeting precision because no one asks that directly but it is often necessary to explain.
14:36
cbg \o
15:02
o/
16:01
Is there a "PEP 8" for SQL?
I couldn't find one the last time I checked. Probably because there are so many variations upon SQL and none of the vendors can agree on anything.
@Kevin do you really need to do that with a regex? I'd have thought a collections.Counter would have been the simplest way to verify non-repeats.
Just checking. I didn't want to get too "opinionated" with folks if there was an authoritative document that argues against me.
16:20
@holdenweb Regex is indeed highly impractical for the task.
<troll>I was digging through web archives and found some old HTML that had overlapping element nodes. Is there a good way to parse that with regular expressions?</troll>
16:35
@piRSquared of course there is - you just have to chant: Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn while doing so...
@JonClements noted... and will be begin practicing that chant now.
It helps if you're really bunged up with a cold though... makes things much easier to pronounce :)
Figure 4.8. Ph’nglui mglw’nafh Cthulhu R’lyeh wgah’nagl fhtagn! A surface whose shortest splitting cycle cuts a shortest path g times, and a closeup of the undulating shortest splitting cycle.
17:00
My jimmies will be rustled if stackoverflow.com/questions/54185908/… turns out to be "oh, actually I wrote if user_decision == hit: but I forgot to change my sample code to reflect that"
hi guys what does this command do?

list(set(data2["Word"].values))
where data2 is a dataframe
It gives you a list of the unique values in the Word column. set removes any dupes.
ok that makes sense, was wondering where all my words went, thanks
You could also data2["Word"].unique().tolist()
which might be more intuitive
17:15
I have a question about this code
class SentenceGetter(object):

    def __init__(self, data):
        self.n_sent = 1
        self.data = data
        self.empty = False
        agg_func = lambda s: [(w,  t) for w,  t in zip(s["Word"].values.tolist(),
                                                           s["Tag"].values.tolist())]

        self.grouped = self.data.groupby("Sentence #").apply(agg_func)
        self.sentences = [s for s in self.grouped]

    def get_next(self):
        try:
            s = self.grouped["Sentence: {}".format(self.n_sent)]
I think it is removing duplicate sentences, but I'm not sure...
17:32
@erotavlas It looks like that code assigns a tag to word to indicate that a group of word belongs to the same sentence. The class has a method to increment a counter and get the words in the next sentence based on said tag. Don’t see the remove duplicates part, however.
whenever I run it on my data, I get fewer sentences than what I start with
I wanted to check one particular sentence see how many times it occurs, but was having trouble with the code...basically I want to do the following
Knowing nothing of whatever data type data is, I'm suspicious of groupby
for sent in sentences:
    for word in sent:
        if word == '[COMMENT]' :
            print(sent)
Are words incorrectly being grouped together thereby reducing the total number of sentences?
sorry the data2 datatype is a dataframe (from CSV file)
they are being grouped together correctly
17:36
@erotavlas if '[COMMENT]' in sent: print(sent) -> one less loop
ok I can try that, thanks
shouldn't change behaviour though, just performance
nothing happens, when i run in jupyter no output
and reduce multiple prints of the same sentence if it contains that token multiple times
@erotavlas so what's sentences (type, shape, contents, etc)?
following along with this code depends-on-the-definition.com/…
except I don't have the third column (POS) in my own data
but basically my data is in the same format
17:40
guys I need some help with Max's syntax
def compute():
ans = max(((a, b) for a in range(-999, 1000) for b in range(2, 1000)),
key=count_consecutive_primes)
return str(ans[0] * ans[1])
---------------------

Sorry it's not editing
but I don't understand max function here
@AndrasDeak I fixed it, I had to do this

for sent in sentences:
if '[COMMENT]' in sent[0]: print(sent)
I don't two things here. I know how these two ranges are working and how key works

What I don't get is....will `count_consecute_primer(ab)`

get ab from max(((a,b ......?
and how is ab one thing if max has them separate (a, b)
@erotavlas so sent is a list of lists? Or a list of words but the match is not exact?
--------------
Lastly, can anyone tell me how to format code in this group? I'm using Mac
@AbhimanyuAryan (a, b) is one thing in python: a single tuple that happens to have two elements. Does this answer your question?
@AbhimanyuAryan please see the Illustrated Guide to Formatting Code in Chat
17:47
so once you get value from ranges which is a & b. You send them to key method right? and then return type comes to max?
And please either post code as text (after practice), or link your code from a code paste service. But don't post images of text.
First things first: do you understand the nested generator expression inside max?
"ab" is just a name. it is the object being passed (a tuple in this case) that is capable of being "unpacked" into a, b
the max function takes an iterable and returns the biggest of the things in the iterable. You may choose to pass a function that acts as key to help determine which item is in fact bigger. In this case count_consecutive_primes takes an iterable with two things in it. (a, b) qualifies as such an iterable. a, b = ab splits into two things and returns i. That i is only used to compare against the other is for every other (a, b) to let max know which is "biggest"
17:49
OK, carry on, getting a bit crowded here and you seem to have more time to type now
Another pin for my voodoo doll attuned to teachers that give assignments like 'take this obviously iterative algorithm and do recursion to it'
inb4 "well some algorithms are equally likely to be implemented recursively or iteratively". It's Fizzbuzz. Fizzbuzz should not be recursive.
haha. to be fair though, i mean, its more important to create examples that are easy to understand. and usually, its only iterative solutions that fit the bill.
Parse fizz(fizz(fizz(buzz)buzz)buzz) and count the number of times I cried.
Wanting easy examples is a noble goal, but choosing an easy iterative algorithm and throwing recursion at it is a misguided approach. It makes it harder to understand why recursion is useful, if you only use it in useless contexts
18:02
Cant argue with that. However, i will admit that i found recursion fairly daunting until i had a couple toy examples to start with. (and mostly it was just letting it sit and ignoring recursion till one day it all clicked)
cough factorials cough
To this day I find recursive algorithms hard to handle. Not because I was taught wrong, my brain is merely wired differently.
wim
wim
I tried to cast a delvote but I can't.
now jpp trying to reopen it ... wtf man
I asked them why they answered it in the first place, the response was something like "why not"
yesterday, by jpp
@AndrasDeak, Why not? The error's pretty clear. I mean, if you like you can edit the question with an arbitrary version number.
@jpp why do we need that reopened again? ^
factorial is an OK entry-level toy example, although its iterative form isn't all that complicated, so it doesn't necessarily demonstrate the full power of recursion. Towers of Hanoi is a good exercise for that, since the recursive solution is like ten times easier than the iterative one.
Hmm, does the iterative need backtracking?
probably
18:10
You could probably do something terribly clever with stacks, but not if you're a CS 101 student
well I'm not
I also can't disclaim the possibility that there is an OEIS sequence, complete with closed-form equation, that tells you when to move which disk where.
sorry, whats OEIS?
google isnt being very helpful
oh. nvm
google always knows.
how do I get the max length from the following?

len(s) for s in sentences

where a sentence looks like

[('Bronchial', 'O'), ('wash', 'O'), (':', 'O')]

so I just need to get the len of all of them but then the max value I encountered
What is the length of that sentence? 3?
18:16
yes
iterate through the list, taking the 0 index, calculate length.
If I understand the requirements correctly, try max(len(s) for s in sentences)
max(len(x[0]) for x in your_list_of_tup) #although you can do it better with a normal loop.
ok that worked, and also unexpected it found a sentence with 252 words in it
252 words? or a word of 252 length
18:18
no, 252 words
unless is counting word length
If each sentence looks like the one you showed us, my approach is not likely to be counting word length
ah. i misunderstood i think
@erotavlas How familiar are you with python?
not very, in fact I hate it so far
hears his heart breaking
18:20
The two aren't exactly similar qualities, but OK. In that case you might want to read a tutorial to brush up on the basics.
I'm kind of doing it while I work....trying to do simple tasks at a time
I think if I do it in a regular for loop I'll have better luck instead of using those shorthand methods
Frustration is a natural response when it seems like the language is obstructing you from getting things done rather than facilitating you
@erotavlas yup, list comprehensions are convenient but you should understand what happens inside first
One may shout at a piece of furniture when you stub your toe on it at 2 AM, regardless of how comfortable it is when sat upon
I currently hate that stupid colon you have to put after every for statement
18:24
that's the price for not having braces
and not after every for ;)
Currently trying to determine whether the language would remain unambiguous if colons were optional.
for i in range(10) print(i) could only be interpreted one way, still
wim
wim
yeah I think there was actually a legit discussion about this on python-ideas once
Possibly statements like for a in b: (c) could become ambiguous
the argument being indentation stays and then colon isnt needed?
wim
wim
Guido: it's too late to change markmail.org/message/ve7mwqxhci4pm6lw
im guessing frankly its nicer for the auto formatting code editors and stuff more than anything. And just as a reminder to the programmer to indent.
But i think indentation alone should make blocks pretty clear.
jpp
jpp
@AndrasDeak, Do we need it closed? I'm pretty active closing posts, often max'ing them out, but we don't need to overdo it.
18:28
i can't even do this?

print(count + " " + len(s))
wim
wim
guido actually mentions "makes it easier for editors/highlighters"
jpp
jpp
@AndrasDeak, The downvote on my (communit-wiki) answer seemed excessive.
@jpp yes. It has no code, only an error that most people won't see
@erotavlas Not unless count is a string, and len returns a string (which it doesn't). Try print(count, len(s))
and I bet the exact error message has a bunch of dupes
18:29
@erotavlas nope - you can't concatenate incompatible types... just use print(count, len(s))...
jpp
jpp
Then retitle the post? Add a sentence at the top with Python version?
wim
wim
delete your answer
jpp
jpp
Because SO is a Q&A site ?
When you found it you should have voted to close yourself rather than answering with a CW.
wim
wim
18:30
it's not a wiki style post so you should not mark it as CW
If I ask about the best Beef Wellington recipe will you answer it?
CW is probably due to the answer having been posted in comments
jpp
jpp
yes, exactly, I don't copy a one-line answer from a comment and call it my own, if it isn't.
wim
wim
@AndrasDeak 2 wrongs don't make a right
@wim not excusing; explaining
jpp
jpp
the post was +6 before it got attention here. I'm taking a wild guess and it's not because of how the question was phrased, but because that comment helped some people.
wim
wim
18:33
I actually would like Python better if the colon at the end of the for/if lines was not there, and the ability to one-liner if statements/for statements rather than indent was removed
seems cleaner and more consistent
jpp
jpp
1,719 views in 9 months is also not negligible.
How bad would it be, really, if we could concatenate ints and strings
wim
wim
@Kevin pretty bad
i appreciate python being strict about data types.
@Kevin JS bad
18:34
^ js comes to mind
wim
wim
you would have to get a consistent behaviour between '1' + 1 and 1 + '1'
"Surely we can have type coercion, and have it not become a huge mess, by learning from the mistakes of the last N languages that tried", says language N+1
wim
wim
do you get a number type or a sequence type? do you lose commutative property?
We'd be flooded with "why does my simple calculator code print(input() + 3) give strange results?" questions from n00bs
18:35
python was such a breadth of fresh air for me. i liked being told i am an idiot when i was being an idiot instead of things silently failing
Also... '1' * 5 and such...
it made me have to learn, really learn, the importance of data types.
I'd be inclined to make '1' + 1 return "11" and 1 + '1' return "11". String-and-string addition already isn't commutative, so no loss there.
but if they're in variables, and i expect a 2, i'd end up with 11 and be really mind effed about it
especially if i end up using the result for further "math" calculations with strings
i'd imagine many people would fall for something similar too
18:40
first (a,b) tuple is passed to key function. Than return types of key function comes back to max function and then whatever is the max return type (a,b) for that are passed right?
@ParitoshSingh was more of a depth of well water for me.
53 mins ago, by Andras Deak
And please either post code as text (after practice), or link your code from a code paste service. But don't post images of text.
@AbhimanyuAryan Pretty much.
ignore the part about images; concentrate on the first sentence
Andras I used I used ctrl+k and it indented my code. But does look like it did
@AndrasDeak
18:42
you can practice in the sandbox until formatting looks like it should
haha. i am not quite sure if thats a compliment or a complaint for python :P @piRSquared
wim
wim
TypeError is correct for 1 + "1" if that was changed I would be a sad badger
where's sandbox?
there's a link in the formatting guide
wim
wim
18:43
TypeError is also correct for 1 > "1" and I am a sad badger in Python 2.x
@ParitoshSingh it was more me teasing about the use of the word "breadth" instead of "breath". Me being the same kind of obnoxious I am with my family and friends (-:
Thanks Kevin
Well this just adds to my case of the Mondays... Log in and find this notification stackoverflow.com/help/user-was-removed :( -2 rep from it.
Thanks Andras
haha, i even hesitated while typing that word! pah, i blame english
18:45
Oh yes! English is the worst (-: but unfortunately the only language I speak )-:
Is a computer language analogous to spoken languages?
Well, despite its shortcomings, if you had to pick one language, its the best language to know anyways. :)
Nah, computer languages are much more strict. I'd reckon only sanskrit comes close (as is often said)
I mean a computer language is a means in which non-computers agree to communicate with computers so that the non-computers don't get confused. That is not the same task for spoken languages.
wim
wim
so new golfed way to do str(42) would be 42+'' ?
f"{42}"
and the opposite operation would be +'42'
wim
wim
18:48
that the same length as str(42)
bah, f"{True}"
lol
as for the spoken languages, i think the 2 way communication necessitates that the language is still easy to understand for any non "receiver". In its own convoluted way, i'd say the similarities end up being there.
@AndrasDeak Nope, when I'm BDFL there won't be coercion from strings, only to strings. +"foo" is a no-op, and -"foo" is "oof"
poor golfers
this is why I play tennis
also, cbg
18:53
As a concession to golfers, I will restore backticks to their former glory
Code-Tennis: Length of code has to equal 0 mod 15
With no white space fillers
Hard mode: Length should be of the form X mod Y where X, Y E {0, 15, 30, 45}
for code length to matter you have to prescribe something like "no dead code", which probably leads down a rabbit hole shaped like a halting problem
18:58
@piRSquared "What does 'vectorize' mean?" Sounds like a question people would want to know the answer to. Like the "For loops - when should I care?" post, this wouldn't be one you would use for marking dupes, but more to supplement your answer with a link to authoritative references and examples.
If making something into a string via putting it inside an f-stirng is referred to as effing then what is this f"{True}"?
My only concern is people will think the question is too broad.
boolean singleton inside an f-string
@coldspeed What we need is SO-lemmas
@AndrasDeak that's effing(True)
yes, I like the sound of that
19:00
@AndrasDeak Feel free to knive that one
it isn't a canonical, but a lemma. The apply question would also be a suitable lemma; too many OPs misunderstand its uses
yeah, agreed
We could have used SO docs to do that but... oh well
Re:"vectorize", I am not quite sure I have enough expertise to write a post. Perhaps you should take this one @piRSquared
ok, I might just get up the motivation
\o/ and Jeff can't bully you either, this isn't exactly something you put in the docs :P
19:03
(-: I like Jeff and all his cantankerousness.
19:37
Had an interesting application of binary search come up discussing a possible solution to a problem, wondering what different implementations people would come up with if presented with it. Boiled down to a basic (but maybe silly) form, the problem is: use binary search to find the intervals of identical alphabetical characters in a string.

Example input: `'00aaaaaaaaaaa000bbbbbbb0000000000000000cccc000000000'`
Example output: `[(2, 12), (16, 22), (39, 42)]`

The intervals (2, 12) are the (inclusive) intervals of the character 'a', (16, 22) the same for 'b', and (39, 42) the same for 'c'.
Wait, is "use binary search" part of the problem specification?
yea
Actual context in which it was important to use binary search here is each character in the string is an image, and this is trying to find the enter/exit frames of someone tracked in the images, so the operation to actually compute whether a person is in the frame and whether its the same person between two frames is not a cheap operation
Doesn't look like you can use binary search because I don't see how you can discard half the search space at each iteration
I'm probably wrong but that's what it seems like
id also think this would have to be O(n)
Don't you have to eventually check every single 0 block to see if something happened there? I feel like there are implicit assumptions missing.
19:41
^ this.
i feel like the additional machinery required to get a logarithmic solution actually working for this problem is not worth the payoff compared to just a simple linear scan
is the assumptiion: an object will not reenter the frame separately later in the sequence a valid one? can "a" appear again?
Ah, I guess I phrased it wrong; yeah, you'd have to iterate through every 0 block to get the starting point of an interval. You can use binary search to find the ending of each interval though.
@coldspeed depends. If it has to be fast during runtime you might spend a lot of time during dev to make that happen
or do you mean the runtime machinery?
yes. It would be less efficient than a linear algorithm for all but the largest of inputs, mostly
19:44
@coldspeed absolutely the case if I had a video. But the images are online, and I have to download each one individually
So I really don't want to download an image for every second to track things that happen on a minute-by-minute scale
cbg
I have not been able to construct a dataframe from a simple JSON. I know how it works, but this one doesn't seem to work..I'm stuck in a viscious circle of errors like ValueError: arrays must all be same length. I've tried all possible solutions, here's the JSON
{
"participants": [
{
"name": "Test 1"
},
{
"name": "Person"
}
],
"messages": [
{
"sender_name": "Person",
"timestamp_ms": 1485467319139,
"content": "Hie",
"type": "Generic"
}
],
"title": "Test 1",
"is_still_participant": "true",
"thread_type": "Regular",
"thread_path": "inbox/xyz"
}
python dict or JSON?
well, I get its in a JSON file. I get it in a dict with json.load(d)
19:53
And what shape would you expect the result to have? You have a probably-dict for which one of the values contains 1-length dicts and the other value contains 4-length dicts, and none of the keys in the latter 4-length dicts match
Oops, I just gave advice about exception handling to someone that was obviously writing a virus.
@Kevin, How did you know he/she was?
readvictim = victim.read()  # !
The pathname ending with "VirusTest" was my first clue
19:56
subtle :D
:D
huh, the Lounge is still unfrozen
^ things started to heat up
What did they do this time?
just calling each other names
20:01
@Kevin let me just quote the user whose messages were flagged...
context was confusing enough that I didn't vote on the flag
@AndrasDeak, columns = ['participants_name', 'sender_name ', 'timestamp_ms', 'content', 'type', 'title', 'is_still_participant', 'thread_type', 'thread_path']
The 'participants_name' column should be the values of the 'name' keys
├── messages
│   ├── dict0
│   │   ├── content_Hie
│   │   ├── sendername_person
│   │   ├── timestamp_1485467319139
│   │   └── type_generic
│   └── dict1
│       ├── isparticipant_true
│       ├── threadpath_inboxyz
│       ├── threadtype_rewgular
│       └── title_test1
└── participants
    ├── dict0
    │   └── name_test1
    └── dict1
        └── name_person
that seems to be the topology of your dict ^
underscored things are key_value items
Yes, how did you build this tree? Is there a software to do that?
nope, regular elbow grease
I created a directory tree and used tree. There have to be tools for this but this was the fastest in wall clock time.
20:13
tree in Unix is great. So, you just pass the dict as an argument to tree?
> I created a directory tree
$ mkdir participants/dict0
$ mkdir participants/dict1
$ touch participants/dict0/name_test1
$ touch participants/dict1/name_person
$ mkdir messages/dict0
$ mkdir messages/dict1
$ touch messages/dict0/sendername_person
$ touch messages/dict0/timestamp_1485467319139
$ touch messages/dict0/content_Hie
$ touch messages/dict0/type_generic
$ touch messages/dict1/title_test1
$ touch messages/dict1/isparticipant_true
$ touch messages/dict1/threadtype_rewgular
$ touch messages/dict1/threadpath_inboxyz
$ tree
ahh, i see
So basically...you want to flatten all of that info that I showed and make all of them a single column each, because all that information relates to a single message?
How do I get this into a meaningful dataframe? I thought if I'd ignore the messages key and just get the keys of its constituent keys as columns, that should make it a bit easier.
The problem is you have a dict of list of dicts, I'm pretty sure there's no general way to parse that into a dataframe. You have to work.
20:17
Yes, that's what I want
Do you have a lot of these dicts, amounting to the rows of the dataframe?
What I would do is write a custom function that turns that dict (of fixed form, as I understand) and return a flat dict with...uh...9 items
Yes, several JSON files which were malformed, but I've fixed that part. Now I have valid JSON files but don't know how to put them into a df for analysis.
then you can loop over your dicts (JSONs) and gather all the rows into a defaultdict(list)
might not be the most efficient approach, but it's straightforward
I thought if I can get it to work for 2 files, then I'd generalize across the whole
20:20
I'll try defaultdict()
defaultdict(list) is optional, but that's one of the easiest ways to "generalize across the whole" (I can explain what I mean later if you have part 1)
first write a function that flattens one of these babies into a simple dict corresponding to your df columns
I think I'll go for a dict flattening function, I'd worked on flattening dicts before, just need to find that script from some corner of my PC..:D
you have a fixed structure so it should be straightforward
Or I'll write one again, yes its straighforward
output = {'from_name': participants[0]['name'], 'to_name': participants[1]['name']} etc.
20:26
Yes, I'll work on it. I like JSON anyway, way better than XML..
\o/
but the tree seems off
I award myself the "you tried" star
the second list ends before the last }
oh, hold on, yours is right, mine is wrong :)
I award you the "you tried" star
20:35
:D I'll take it
Oops I probably should have used BOX DRAWINGS LIGHT HORIZONTAL instead of HYPHEN-MINUS for the horizontal bits
Replace "-" with "─" in my code for best results
20:52
@Kevin I put that disclaimer in all my documentation
Use only BOX DRAWINGS LIGHT HORIZONTAL when doing arithmetic in your code, and it will never produce incorrect output
(... Or any output, other than SyntaxError)
# -*- coding: fancy-as-yam-ascii -*-
cbg. Is there a method that I can implement for a class (maybe something like to_dict) that if I do dict(instance_of_my_class) will turn that class into a dict?
that was two underscores before and after the to_dict ...
00:00 - 21:0021:00 - 00:00

« first day (3012 days earlier)      last day (2161 days later) »