« first day (2444 days earlier)      last day (2501 days later) » 

12:20 AM
Is there a way to put a variable from a script(that is outside a function) into a funtion? How can i do this
 
I don't understand what you are asking? What kind of script?
 
already fixed it sorry for the weird question Xd
 
 
2 hours later…
2:10 AM
the future is now. Python 5 is here
 
 
2 hours later…
4:04 AM
Hi guys I have a question about performance of list vs str in python 3, basically what I am trying to do is store the positions of missing_data_points and one solution is to store in a list the positions in which there is missing data other is to make a binary string that has 1 for existing data and 0 for not existing data and then I just see what are the position of 0s to see the index of the missing data
My question is what would be better to iterate over, the big str or a list
 
 
1 hour later…
5:19 AM
"Premature optimization is the root of all evil" A list will be the most natural and versatile way to model this. Will you be filling in this data and thus updating this structure? Lists are mutable while strings are not. Iterating over a list of missing data locations is more straightforward and direct than using .find or .index to find 0's in a binary string.
 
Sounds good, thanks Paul
 
5:59 AM
A list is also more versatile. It can be a list of 0s and 1s/true and false just like your proposed strong or it can be a list of something more complex if necessary.
 
6:41 AM
@Jasguerrero you cannot change the str.
@Jasguerrero but then, for that, I'd use something else, like a bitvector
@Jasguerrero or use a bytearray.
lists waste way too much way memory, at least 8 bytes per element on 64-bit computer
 
 
3 hours later…
10:03 AM
no repro problem solved elsewhere, irrelevant comment answers stackoverflow.com/questions/44733059/…
 
 
1 hour later…
11:17 AM
Grmbl grmbl Willem
I feel like expressing myself in the style of Vinnie Jones
 
 
2 hours later…
12:55 PM
cbg
 
cbg
 
I answered a question, and it seemed fairly straight forward, but now I'm wondering if there is a better way to do it. Or if there is something I'm missing and it is actually much more difficult
 
if there's a link, I can't see it :P
 
@AndrasDeak here
I just want to know if I'm off my rocker. Admittedly, it seems a bit brute forcish..but I think that's it?
 
already the question sounds underconstrained. How to treat overlaps?
anyway, I'm pretty sure OP isn't interested in pi
> Let's take the digits of pi as an example:
 
1:02 PM
right
The main question they had though made me think that they are interested in incremental sequences
in general
 
hmm...guess we should compare it to [pistring.count(str(digs)) for digs in range(100)]? (after cutting off the stuff before the decimal point)
 
so, in one pass, single, then double, triple, etc
 
incremental sequences?
 
they were wondering in that sequence of pi, that they can figure out each single digit and how many times it shows up
but then they were asking how many times 11 can come up, and 12
 
yes, up to 99
 
1:06 PM
I decided to just go up to the maximum length
maybe I should provide a max constraint
but either way, it can be adapted for whatever
the overall solution of determining counts of a sequence of x seems to be met with my solution
seems :P
 
if I understand correctly, the list comp I mentioned ^ is essentially what they need. But for really long strings it's probably beneficial to pass once with windows of various sizes and collect the numbers...
 
hmm let me run yours
 
it's basically the other answer in a list comp :P
 
without specifying which digits belong to that count
but yeah, yours works well
 
OK, so {str(digs):pistring_cropped.count(str(digs)) for digs in range(100)} :P
 
1:11 PM
lol
and isn't that pretty much Counter?
 
perhaps using int keys is even better, depending on the application down the line
 
and can it be argued that Python's implementation would have been optimized to be better than that?
I'm stating this on pure ignorance and the assumption that out-of-the-box would be better
 
Here is a link to get the first billion digits of pi: stuff.mit.edu/afs/sipb/contrib/pi
 
umm...I guess it matters on how you're dancing around calling the Counter?
 
@PaulMcG OH!!!! let's inject that in to my code and see what happens
idjaw has disappeared in to the ether
 
1:13 PM
ah, I see, you're passing every slice and categorizing those
 
yes
 
yeah, this could be faster, I guess, because .count() has to search across the string each time
my point is that the difference is not dictcomp vs Counter, but passing bunch of times vs passing fewer times
6 mins ago, by Andras Deak
if I understand correctly, the list comp I mentioned ^ is essentially what they need. But for really long strings it's probably beneficial to pass once with windows of various sizes and collect the numbers...
I tried to say that ^
 
Need an iterator-based method to do this kind of thing in one pass
 
right!
 
my idea is "loop over starting index: loop over window length: accumulate counts"
 
1:16 PM
profit
and then....
THE WORLD
 
OK that --------^ is exactly what idjaw seems to be doing
good job, idjaw :P
no, wait, not exactly
got confused
I should probably swap my keyboard for a coffee
 
are you debating whether I'm trying to take over the world?
because I assure you, that is going to happen whether you like it or not.
 
Is there a blog post coming out of this, "Slices of Pi"?
 
oh..it was based on an answer I gave and I was wondering if I mis interpreted its simplicity
@PaulMcG if you're interested stackoverflow.com/questions/44746081/…
 
1:57 PM
Hmm, it looks like I found more trigrams than you did
('028', 1),  ('058', 1),  ('062', 1),  ('078', 1),  ('097', 1),
('105', 1),  ('141', 1),  ('159', 1),  ('164', 1),  ('169', 1),  ('197', 1),
('209', 1),  ('230', 1),  ('238', 1),  ('264', 1),  ('265', 1),  ('279', 1),
('286', 1),  ('288', 1),  ('307', 1),  ('314', 1),  ('323', 1),  ('327', 1),
('338', 1),  ('358', 1),  ('375', 1),  ('383', 1),  ('384', 1),  ('399', 1),
('406', 1),  ('415', 1),  ('419', 1),  ('433', 1),  ('445', 1),  ('459', 1),
('462', 1),  ('494', 1),  ('502', 1),  ('510', 1),  ('535', 1),  ('582', 1),
I used a deque for a moving window over the digits
73
In your loop, you step by i through the length of the sequence, so you won't see overlaps
What happens if you remove the step?
Brute force is nothing to be ashamed of, btw
For n=1,2,3, here are counts for each n-gram: (1, 10), (2, 57), (3, 73) I can eyeball the first one, there are 10 different single digits. I added a semi-validation step at the end of my script to do pi_digits.count each n-gram in my counter, but this just verifies the count of those I found, not any that I missed.
 
2:15 PM
Cbg
 
2:56 PM
OK, not only did my entire Steam wishlist go on sale, but now I'm just browsing adding more games.
This is not good.
 
<Steam as Palpatine let the gluttony flow through you gif>
good thing I don't have steam... *sneaks away to look at the sale*
 
You can buy stuff for my wishlist to get the experience.
 
hehe, thanks, I'll keep that in mind :D
 
3:44 PM
With all these eval answers sprouting like mushrooms, I had a great^[citation needed] idea for a project. Turns out someone has already done it.
there goes my plan to familiarize myself with the ast module:D
Whoaaaaaah huge TIL: Haskell Curry
 
So far: 35 in wishlist, 7 in cart.
I think I should stop.
 
I didn't see anything that piqued my interest
 
perhaps the pretty Ori one
 
@Kevin has been playing Hollow Knight for a while now.
 
3:55 PM
wait, don't you already have NieR:Automata?
or not via steam?
 
Yeah, some of them are just "it would be cool if I had it on PC because mods."
 
I'm so tempted by Slime Rancher because it looks surprisingly challenging, but I'm still debating if it will hold my attention.
 
I got the "surprisingly challenging" look from Causality
 
I have that one because I thought it was my friend's game that he's working on but it's just a similar concept.
Stop me, I think I'm going all in for everything < $5.
OK, I stopped myself. "Remove all from cart"
 
4:08 PM
awww
 
neo
anyone know good sources for downloading data leaks ?
 
OK, got it down to a good mix of puzzle, story, and 3D games. I can always buy more later.
 
@neo - "data leaks" meaning... ?
 
4:24 PM
Wow, Lyne was perfect because now I have a use for my touchscreen.
 
it's always good to have a problem to your solution
4
 
neo
4:43 PM
@PaulMcG like the latest US voters list details
 
No, I have only seen the schema released, not the actual data. For other data sets, try data.world
This is not a warez group, so you probably won't find any sources for actual sensitive data. If you want to experiment with pandas on large data sets, there are plenty of publickly available and legit data sets on numerous government web sites.
Just watched a David Beazley talk from PyCon2016, dissecting the City of Chicago Health Department published data set of Chicago Health Inspection reports - very entertaining.
 
neo
@PaulMcG thanks that was indeed something good
 
yeah, I don't think "where can I get leaked sensitive datasets" spells exactly "pandas practice"
 
neo
4:58 PM
i just want to work on some sensitive information whether i can find anything intresting
 
right.
 
I think your Centos already had Python, which you now conflicted with your other installation. Additionally, python-pip RPM package installs packages into the system python. In addition, Python 2.6 has been EOLed a long time ago already, and Python 2.7 follows it - if you're compiling your own Python, why not go for the fresh 3.6?! — Antti Haapala 9 secs ago
 
@neo please look elsewhere
 
neo
@davidism sorry ?
 
"I want to commit a crime, anyone want to help me?" :D
 
neo
5:06 PM
:D
 
umm. no.
 
oh oh...what crime are we committing?
 
ice cream in back pocket
 
just sat on the couch....forgot about the ice cream
crap...
 
neo
am i the only outsider/stranger here right now ? everyone seems like know each other
 
5:09 PM
maybe the question you should asking, whether everyone here is human
that's the real mystery.
 
some of us are almost certainly stranger
3
 
yes, indeed. Some more than others.
 
inb4 Some gets his feelings hurt
 
neo
yeah,one day i talked with a bot in php room for about 1 minute withour realizing it
 
@neo have you considered the possibility that you are a bot too?
 
5:11 PM
Take that for meta.
 
Cogito, ergo sum. Don't know about you guys
 
Andras, did you push untested code in to Antti again?
we spoke about this
 
I was going to switch on his James T. Kirk AI, must have messed up again
 
JTK AI emits all sentences in the form: Must! ... Save! ... Enterprise!
 
5:46 PM
You're using Python 2 which is the reason for this unfortunate thing. In Python 3 you'd get a TypeError. The slices are strings, and you're comparing them against ints. — Antti Haapala 20 secs ago
python 2 :F
 
6:13 PM
What is wrong with this loop? I want to print all elements.
      my_list = [1,2,3]

      print(num) for num in my_list[1:]
How the ### do I format my msg to look like code?!
 
code has to be in its own message, and then you hit ctrl-k to format it
 
Hit Ctrl-K
 
how do you want to print it exactly?
line by line?
 
And for ### you can use the word 'yam', and we all understand what you mean
 
Thanks, now it looks decent
 
6:15 PM
yam yeah
 
Yeah, just print it anyhow
but the syntax is off
 
print(my_list)
^^ what is wrong with that?
 
For one thing, the index of the first element of a list is 0, not 1
 
Ok my bad, my follow up question would be "What is :1 doing?"
So it's the index?
 
furthermore, you seem to trying to do some kind of expression to print, and the first big warning is to never do a comprehension/expression for its side effects
 
6:16 PM
@simeg it is [1:]
 
the format within accessing an index -> [1:] for example, is slicing
 
That is a shortcut notation for "starting at index 1, give me that and everything afterward"
 
Noting that index 1 actually points to the second element in the list
 
the Python 3 tutorial tells you this and answers to many other newbie questions.
 
6:19 PM
Ohh it's slicing, that explains everything. It's hard to google for special characters and I had no idea what it was doing. Thanks everyone, I'll try to RTFM next time :)
 
RTYM
 
sometimes it's hard to find out what the language calls the thing it is doing when you don't know what that language calls it 😛
I've had that in the past with other things
 
@PaulMcG Right, haha!
 
however, is the concept of slicing, called slicing in other languages?
 
@idjaw Javascript has slicing, which does the same thing I believe
 
6:21 PM
I've had many questions about '%' formatting, since it goes by the non-intuitive name of "string interpolation", and does double duty as the mod operator
 
I'm not sure MATLAB uses "slicing"
 
oh yes...when I first was learning Python and saw the '%' in strings, I had no idea how to look it up
 
We need a FAQTWBAMFITAKWTAA - Frequently Asked Questions That Would Be Asked More Frequently If The Asker Knew What To Ask About
 
I think eventually "percent in string python" worked for me 😛
 
google doesn't help either
even their "verbatim" search is...anything but verbatim
 
6:24 PM
haha @PaulMcG a dictionary that maps the python keywords to its other language equivalents
or general comp sci equivalent
"did you mean ......"
 
Didn't I just say that?
 
I was repeating what you wrote in confirmation of loving your idea
kinda like when repeating the punchline in a joke and laughing
that
internet chat is hard sometimes
 
and when idjaw repeated PaulMcG's funny idea, remember that!
 
yea....don't you EVER forget
 
Yeah, those were funny days...
 
6:25 PM
remember when we were talking about string interpolation?
 
Pepperidge Farm remembers
 
lol
 
Rbrb all, just stopped in the office for a bit, heading back home now
 
cheers @PaulMcG enjoy your sunday
 
rhubarb
 
7:25 PM
Is there a way to write "pandas.DataFrame.set_index([all_columns_except_D_and_G])"?
 
@Hatshepsut how would that work?
in case of 7 columns, how would the 5 non-D-nor-G columns become indices?
 
@AndrasDeak df.set_index([column_name for column_name in df.columns if column_name not in except_list]) ?
 
how is that an answer to either of my questions?
is it implied that you have 5 rows?
 
df = DataFrame({
'a': [1, 2, 3],
'b': [11, 22, 33],
'c': [111, 222, 333]
})

df.set_index(['a','b'])
excluded = ['c']
df.set_index([column_name for column_name in df.columns if column_name not in excluded])
@AndrasDeak i dont understand the problem
 
7:40 PM
me neither
how can you set a 2-element index for 3 rows?
 
why not?
that ^ works, does what i want
 
so do that? :D
 
@AndrasDeak are you alluding to a case where it will break?
 
No, I seriously don't understand any of your problem.
 
oh, i just wanted something that would do what i wrote above, before i realized i could do that. (though i wish there was a nicer syntax)
 
 
3 hours later…
10:56 PM
@Hatshepsut I also came up with df.set_index(df.drop(excluded,axis=1).columns.tolist()) but that doesn't look much clearer, and it incurs overhead
 
11:13 PM
@AndrasDeak df.set_index(list(df.columns.difference(excluded)))
 
better, especially since it doesn't do any unnecessary work with the entire dataframe, only its columns
I suspect that .tolist() might be more idiomatic as it's a bound method, but then again I might be wrong
list(pd_index) probably can't help but consume the iterable, whereas .tolist already knows that it belongs to an index object
there's probably no measurable difference in case of a handful of columns
 
mhmm
 
11:38 PM
Hello, one question, If I'm doing an eval to evaluate something that contains certain object, the outside import doesn't work in the eval. How I can deal with that?
 
By not using eval.
 
Can you give us a MCVE?
 
Why are you using eval? and what are you doing, and yeah, you need an MCVE as vaultah suggested.
 
I will just remove it.
I'm working on a Big Data Project, I have a spark stream that saves data to file as a pyspark Row.
And I need to read that file and evaluate it to convert it back to Row
But I will just save it as json.
 
If you are dealing with converting strings that are structured as data structures that can be interpreted in to valid Python structures, you can use literal_eval, or if it is in fact valid json, then use the json module.
stay away from eval.
 
11:48 PM
ok.
Thanks!
 
cheers
 
@idjaw saved a soul today
 
Another thing... I still need to use eval
 
Give that man a medal! And a cookie.
@OmaRPR you don't! ;)
 
trust me man....you really don't need to use eval
there's 100% guarantee you do not
 
11:51 PM
I can't use json.loads because the line in the file contains the json with quotes... This way: "JSON"
 
Example...
>>> a.first()
"{'created_at': '2017-06-25T23:40:00+00:00', 'sentence': 'SCREENNAME Literally if Trump did what she did with the DNC RNC for his case you all would demand his executi… LINK', 'tweet': '@brianstelter Literally if Trump did what she did with the DNC (RNC for his case), you all would demand his executi… https://t.co/nHXtGoKDza'}"
 
json.loads(a.first())
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
 
>>> from ast import literal_eval
>>> res = literal_eval(s)
>>> res
{'sentence': 'SCREENNAME Literally if Trump did what she did with the DNC RNC for his case you all would demand his executi… LINK', 'tweet': '@brianstelter Literally if Trump did what she did with the DNC (RNC for his case), you all would demand his executi… t.co/nHXtGoKDza', 'created_at': '2017-06-25T23:40:00+00:00'}
>>> type(res)
<class 'dict'>
>>>
where s was the string you gave me.
I mentioned literal_eval or json
I didn't know what your data looked like at all.
 
11:53 PM
Oh!
literal_eval
 
literal_eval is not eval
 
Why did a JSON export output non-JSON?
 
Awesome, I will try it. Thanks to all, and to you @id
 
@idjaw literally not
 
^^ that too
 
11:53 PM
@idjaw
 
@PaulMcG yeah...wondering that too
np
 
I don't understand why that happened
 
probably not python?
 
Single quotes are repr for Python dict, unless a value contains a single quote
 
I'm doing everything on Python
 
11:54 PM
where did that string come from
 
then wow
 
Twitter Stream (Py) -> Kafka -> Kafka Consumer to HDFS (Py) -> Sentiment (Py)
 
that's a lot of places where that string is coming from
alas, have to sleep
rhubarb
 
rbrb @AndrasDeak
 
I'm not an expert on python, I'm using it to implement a solution to analyze big data. Its one of the best technologies for it.
Thanks all for your help! literal_eval worked like a charm.
 
11:56 PM
rbrb Andras
 
@OmaRPR parsing json with json would probably be even better...
I'd figure out why it wasn't proper json if I were you
it might bite you later elsewhere
 
that is a good point
If you are talking to an API, that API response is going to be valid JSON
 
then again I'm abnormally curious
 
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
The properties are on single quotes
 
I'm fairly certain that at the very least twitter is giving you valid json
 
11:58 PM
Again...
>>> a.first()
"{'created_at': '2017-06-25T23:40:00+00:00', 'sentence': 'SCREENNAME Literally if Trump did what she did with the DNC RNC for his case you all would demand his executi… LINK', 'tweet': '@brianstelter Literally if Trump did what she did with the DNC (RNC for his case), you all would demand his executi… https://t.co/nHXtGoKDza'}"
I know @idjaw but I'm changing to a tinier json... Twitter json is TOO BIG.
52
Q: python: single vs double quotes in JSON

Bin ChenMy code: import simplejson as json s = "{'username':'dfdsfdsf'}" #1 #s = '{"username":"dfdsfdsf"}' #2 j = json.loads(s) #1 definition is wrong #2 definition is right I was heard that in python that single double quote can be interchangable, can anyone explain this for me?

parts = messages_downsecs.map(lambda x: {'tweet': x[0], 'sentence': processTweetText(x[0]), 'created_at': x[1].isoformat()})
The problem is there, I think.
I'm using python JSON
No json json
I need to convert that into JSON using the library, so the library can be used to decode it, I think.
I will try it.
 

« first day (2444 days earlier)      last day (2501 days later) »