« first day (3079 days earlier)      last day (1869 days later) » 

12:24 AM
@coldspeed But how are they the needs of the greater good? Users might get misled into concluding they need to install HAMT just to delete an item without leaking memory. I think retitling to "Delete key from dictionary without modifying the original, to return a copy" and adding a one-liner in the question body *"The standard way to delete a key/item from a dict is del d[k] / d.pop(k), per this answer...
...which is much shorter and much clearer. The root of the evil with the former question is it piggybacks a harder non-standard question ("Additionally, how can I delete an item from a dictionary to return a copy (i.e., not modifying the original)?") onto a basic and misleading question "Is there a way to delete an item from a dictionary in Python?"
That should not be allowed. It's designed to mess up both readers' minds and search engines. The simpler question "Is there a way to delete an item from a dictionary in Python?" should be deleted, since it's well-covered. Or we could just link The standard way to delete a key/item from a dict is del d[k] / d.pop(k)
 
@smci 99% of user queries will be resolved by the answer to the question in the title. For the remaining 1%, there are answers (including a note in the currently accepted one). Really, it seems pointless to muddy the question with more. But that's just me.
 
@coldspeed No, not at all. 99% of incoming queries will be people simply looking to del d[k] / d.pop(k), which merits a three-line answer, but instead they get essays about HAMTs and deep copies... very unwelcoming to new users.
If people want to ask about GC, sharing and reuse of keys between deep or shallow copies of dicts and how deletion affects that, then let the question clearly say so.
I'm busy but let me reframe my thoughts on Meta someday. This should at minimum be referenced in the first line of the former question, and probably be reopened and stand alone. Anyway, later.
 
12:42 AM
I don't think it is right to have people find their answer in the question, as you seem to suggest. Also, the first answer (nay, the first few) all answer the question (as stated) concisely and to the point. The essays are further down, for the interested few.
 
wim
1:26 AM
I'm afraid I agree with smci on this one. coldspeed's edits from 2018 onward all made a terrible mess of that question
Now it is asking two questions, and the answer section is a dogs breakfast because it's not clear whether they are answering the question that was originally asked, or the somewhat unrelated question that was edited in later
 
@wim Perhaps. But I cannot believe that it will become any better were the question to be edited again. My edits from 2018 preserve the original post, except making it clear that there are two questions being asked and answered in parallel. If you insist on changing it, I will recommend rolling it back to pre-2018. Anything else will just make it worse, IMO.
Happy to go with the majority, but really dredging this up again serves little purpose.
 
 
1 hour later…
2:56 AM
Hey, I need help...
text = 'Each and every line of text should be printed but wrapped automatically to never have a line longer than the specified length. '.split(' ')
num = 0
last = -1
line = ''


for word in text[last:]:
if not len(line) > 96:
line += text[word]
else:
print(line)
Traceback (most recent call last):
File "C:/Users/Hunter/PycharmProjects/Canon-Operator/print.py", line 10, in <module>
line += text[word]
TypeError: list indices must be integers or slices, not str
 
3:27 AM
nevermind, the logic makes no sense anyway, giving another try.
 
 
4 hours later…
7:18 AM
$ python -m timeit -s "exec('x = 1')" "x+1"
Traceback (most recent call last):
  ...
NameError: name 'x' is not defined
sad cbg
 
@Arne You need to supply a globals dict. See stackoverflow.com/a/46852244/4014959 and Antti's answer that I linked in the last comment on the question.
Or should I say, a locals dict. ;) See stackoverflow.com/a/51599308/4014959
 
@PM2Ring thx! I was already breaking my head over how to get it to work without using exec
it works =)
And here I was, thinking I was just whining about a limitation of the timeit module, instead I learn something cool about python
beautiful morning cbg
 
@HunterGuimont Word wrapping is tricky. You need to check if the desired line length will be exceeded before you append the current word. And you need to decide on a strategy for words that are longer than the desired line length.
 
7:34 AM
import textwrap ;)
 
@Arne There's no shame in not knowing how to use exec or eval, since you should rarely need to use them. ;) And you shouldn't need explicit exec or eval to do timeit stuff. Give me a minute & I'll find an example of timing expressions.
 
alright
 
@AndrasDeak True, but I get the feeling this is a learning exercise.
 
although the real timeit is this one:
$ python -m timeit -s "from itertools import permutations" -s "keys = tuple(''.join(p) for p in permutations('abcdefg'))" -s "exec('from dataclasses import dataclass\n@dataclass\nclass C:\n  __slots__ = {}\n  {}'.format(str(keys), '\n  '.join(f'{k}: int' for k in keys)), globals())" -s "c = (C(*(1 for _ in keys)))" "for key in keys:" "    getattr(c, key)"
aka are big dataclasses noticeably slower than big dictionaries?
The dictionary test was a lot tamer by comparison:
$ python -m timeit -s "from itertools import permutations" -s "keys = tuple(''.join(p) for p in permutations('abcdefg'))" -s "a = {k: 1 for k in keys}" "for key in keys:" "    a.get(key)"
 
@Arne In that case, I'd put that stuff into a proper script, rather than doing it on the command line.
 
7:47 AM
Might be cargo culting on my part, but the post which taught me about timeit claims that executing it from the command line produces more reliable results
 
@Arne Hmm. Maybe... Here's a typical example of how I use timeit on functions: stackoverflow.com/a/35451912/4014959
 
The timeit docs mention that when you do multiple timeit runs, the run with the minimum time is the run to pay attention to, the other runs just give you an indication of the variance under the current system load.
Cbg, Jon.
 
alright, that's a good one to keep on my clipboard once I have to time more complicated things again
 
If you're really keen, do your timeit tests with minimal system load. That makes a big difference on my old single core 32 bit machine, it's probably not such an issue on multicore machines. But when I run timeit tests, I turn off my music player, and if I'm really serious, I close my web browser.
 
8:01 AM
Ugh... I feel sorry for 'em but I'm not sure what to do... stackoverflow.com/questions/55295048/…
 
Here's an example that tests expressions rather than functions: stackoverflow.com/a/39192976/4014959
Another useful trick is to put a function call or expression into a lambda, like this: stackoverflow.com/a/7523810/4014959 I'm sure I've posted examples using that technique, but Google's not finding them
@JonClements Run away! That's a pretty clear case of "Lacks minimal understanding". They need some one on one tuition before they'll be capable of doing online coding challenges.
 
But I must admit that although we often see unconditional return inside a loop, it's pretty unusual to see it before the loop.
 
makes the function run faster though :)
 
Oddly reminds me of the Premature Evacuation game they have on Radio 1
 
It's (almost) the weekend and I finally find myself with some free time. Any ideas for next pandas canon, @JonClements?
 
@coldspeed umm... why don't pandas come in more exciting colours? :p
 
8:33 AM
As far as I'm aware there's red, black and white. Isn't that enough? :p
 
nope... I want mine rainbow coloured
 
a scholastic treatise on how this relates to the different kinds of curry?
 
You might want to start wishing for a different animal...
 
umm... curry... :)
 
@tripleee Unless you're talking about currying functions, I'm afraid that'd be off topic :D
 
8:36 AM
oh but you can bring in Haskell Curry as a Prophet of the Panda
 
Thought it was Kung Fu Panda?
 
I'd like to see a skit in the vein of Monty Python's Penultimate Supper skit, but as a dialogue between a programmer and their high profile corporate client.
 
Sam
8:59 AM
cbg
 
Sam
9:19 AM
I cant seem to find any docs which explain the `filehandle` parameter of this export function: https://anytree.readthedocs.io/en/latest/exporter/jsonexporter.html#anytree.exporter.jsonexporter.JsonExporter.write.
How would i go about finding this?
 
Paz
Hi, I have a question.

Let's say I have a tuple list, how do I delete a tuple from the list where the first element is equal to X?
 
Rebuild the list: new_list = [item for item in old_list if item[0] != X]
 
Paz
Is it efficient? like 'O'-wise
 
O(N), you traverse the list once and you can't avoid that
 
Paz
Got it, thank you!
 
9:30 AM
You're welcome :)
 
@Paz Removing items from lists is possible, but it's a bit slow because all the items after the removed one have to be moved down to fill the gap. That's tolerable if the list is fairly short, and you're only removing a single item, since the move down gets done at C speed. It's not so good if you're removing multiple items, since the move down happens after each removal. So it's very common in Python to just build a new list like roganjosh said.
 
I also thought I should point out that by complexity I was referring to time complexity and not space complexity. It's possible you might be limited by memory.
 
@roganjosh although it doesn't technically remove the items from the old_list - it just creates a new_list without those items :)
 
he never said it does :P
 
If you don't know the indices of the items you want to remove, then it gets even messier, since you have to search for the ones you want to remove, and if you aren't careful you may not remove the ones you expect because removing stuff from a list your looping over messes up the indices (unless you iterate backwards). That's another reason to build a new list instead of trying to modify the existing one.
 
9:45 AM
When I print the new list, they're dead to me. I don't care about the specifics of how I got there :P (well, actually I do and it is a totally valid point)
 
If you're working with a list that's so big you don't have the RAM to copy it, then you may have to o
 
just saying that sometimes, some_list[:] = (el for el in some_list if el > 3) or whatever is an option
 
If you're working with a list that's so big you don't have the RAM to copy it, then you may have to use remove. Or do stuff a different way. ;)
 
Actually, in most cases, I guess I should be advocating that since I guess it's more common to want to work with the filtered list and not keep the old one
 
9:49 AM
meh, refcounting will instantly kill the old list
 
yeah... that doesn't avoid having to keep some_list around... but it does mean you don't have multiple copies of it kind of thing
 
in most cases there's no real difference, but there's overhead from mutating the old list and looping over the genex
some_list = [...] will kill the old list anyway if there were no other references to it, so from a semantic point of view it's the same thing
 
the important bit being there if there were no other references to it
 
That last code Jon posted saves on RAM, but it's around half the speed of a listcomp that creates a new list, especially if the list is smallish.
 
but if you have other references it makes a huge deal whether you mutate or copy, which is not the trivial case
needs clarification from OP
 
Paz
9:59 AM
I'm not focused enough to actually read what you guys wrote, Basically I'm making a chat and have a list of connected sockets. When a user disconnects, I'm removing it's socket from the list.
 
There's an interesting dichotomy in saying that you're not reading the extra information we provided and then adding extra information, but anyway. It's Friday and focus is in short supply :)
 
> it takes a senior developer to write a MCVE in my experience. That barrier to entry is way too high.
That's it
I'm done
goodbye
 
It wouldn't be a bad thing to list in criteria on job adverts, though. Screening process: write a question that gets upvoted and answered :)
 
@Paz In that case, it may be better to store the sockets in a dict, with the user as the key. It's cheap to remove items from a dict.
 
Umm... I just wrote an answer with a regex that appears to work... didn't think about it much - just wrote it... but now I'm sitting and thinking of explaining it... I've no idea...
 
10:15 AM
Regex must be awesome for pups. Smash keys with your paws and whatever comes out at the end probably does something.
stackoverflow.com/questions/55297211/… unclear but answers are coming in
 
Well everyone else there is going for stuff that relies on presence of other things/ordering etc...
 
My comment was only intended as a joke about regex itself (looking at it often terrifies me; just a jumble of symbols).
 
Paz
@PM2Ring His method actually worked well, i'll keep it that way
 
@Aran-Fey heh, just downvote and close and delete and move on
 
@JonClements It'll probably help if you show how the thing works step by step. Fire up a REPL and show what the value of split is, show what the result of split[1::2] is, show what zip(split[1::2], split[2::2]) does, etc
 
10:27 AM
@Aran-Fey probably will do that later... juggling other bits at the same time... works quite nicely though...
 
I need to be more strict with new answers... If this was a 5-year old question, 90% of those answers would have downvotes by now
 
Hopefully I'm not in the 90% :p
 
Nah, yours is good
 
I routinely downvote series of crap new answers on new questions
especially (but not exclusively) if the question is also crap
 
@Aran-Fey \o/
think it's the only one that has any element of robustness...
 
10:31 AM
@AndrasDeak My learning so far has been that adding a comment explaining why the information given is incorrect, or even harmful, is not wise though. No matter how objective you try to be. My last round of serial downvoting a couple of days ago was reversed though, thankfully.
 
well, yeah
I didn't say anything about comments :D
 
Which is pretty unfortunate overall considering that highlighting bad practice is probably just as useful as upvoting good practice
 
the best way to highlight bad practice is to downvote it
you can try explaining in comments but future readers will probably ignore that, and the answerer won't usually suddenly find the path of light
 
Probably very true
The answerer learned nothing from the whole thing, though, if the answer was genuine
 
It depends on the mistake of course. If it seems like an honest mistake I'm likely to leave a comment, and I've seen a lot of positive response. If it's a crap answer with "try this: <wrong code>" there's no way I'm going to argue
 
10:35 AM
I think I'll take FGITW into more consideration in commenting on answer downvotes.
 
@AndrasDeak that or they'll find it unfriendly or unkind :p
 
well, I don't usually shed any tears for comments
(at least on main)
 
that you don't usually means you do sometimes? :p
 
That's mostly just weasel-wording :P There was one comment where I told a clueless asker in genuinely polite words that they're asking wrong, shortly after the "clueless noob indicator" was introduced. That got deleted very quickly and I was unhappy. Since then I just downvote and close vote.
 
tried editing that answer twice now and discarded the draft... meh... I can barely explain (that) regex to myself... let alone someone else
@Aran-Fey oh... did I not put the r prefix... lol... thanks
 
10:48 AM
bad pupper >:(
 
@Aran-Fey yeah... I'll go read LPTHW and improve my practices :p
 
hums in the background another one bites the dust.. :P
 
good luck with that :P
 
@Paritosh well - who wants to live forever hey?
 
Anyone who wants to break free from mortality
 
10:56 AM
we're all just waiting for the hammer to fall
@roganjosh but I (don't) want to break free from mortality? :p
 
God knows I want to break free, myself
 
i dont care about all that, i just want to ride my bicycle.
 
Then again, being under pressure all the time is tough
 
wonder how many others are going to play the game of queen song titles...
 
Doesn't matter if its one or none, because the show must go on!
 
11:01 AM
oh save me!
 
more of that jazz
 
if we keep this up !I'm going (to go) slightly mad"
 
Indeed, these are the days of our lives, we need to make the most of it
 
I'm on a roll - don't stop me now!
 
11:03 AM
why would we stop you. We are the champions after all!
 
I certainly won't; the show must go on
 
what has this room turned into all of a sudden, save me!
 
<duplicate alarm goes off>
 
I was wondering that :(
 
thats the alarm saying We will rock you!
you know what, i think im going to go listen to some queen now haha
 
11:07 AM
It's fine, Jon. You tried to play the game.
 
@roganjosh umm... you don't fool me
I think with all that - it's actually time to listen to some Queen :p
(and not the one that does her xmas speech)
 
Way ahead of you guys lol
I already had to fire up These are the days of our lives
 
spooky... that was one I was trying to look for along with "Too Much Love" :p
 
:) Wasn't that their last one?
 
How can I work Brighton Rock into this conversation...
 
11:11 AM
think I've got all the Queen vinyls in storage somewhere, got a couple of GH CD's about the place (probably in the storage cupboards)... but umm....
 
I'm pretty sure Brian May did a solo version of it after Freddie died and it was their last song together. To Google!
 
I don't think I've ever got into bohemian rhapsody or stuff like that... but things like "you're my best friend", "too much love" and "these are the days" etc... just absolutely loved... mind you - a personal fav. is definitely "don't stop me know"... I remember they use to play that on TV for snooker stuff
 
"Don't stop me now" gives me life. It never fails to lift my mood. Bohemian Rhapsody has just been so over-played that it actually annoys me.
I was incorrect about Too Much Love. Turns out it was dropped from an album in the 80's
 
Is it just me, or is this question hopelessly unclear? The cryptic meaning of the list fields is not explained, the title has nothing to do with what's going on, it could probably be done better with two one-line list comprehensions anyway, whatever is going on is not reusable or generalizable, and it just gives me a headache.
 
But May did play a version in a tribute to Freddie
 
11:18 AM
^^ I definitely want to break free of that.
 
@roganjosh NO YOU DIDN'T!
there's no such thing as too much Bohemian Rhapsody
 
@smci You're not wrong that it doesn't make sense because there's a stray 14 in the expected output (probably more, I stopped reading after that)
@AndrasDeak This is a Brexit kinda situation where I simply draw a red line that it is a good song any more.
It had its day. I kindly request that it isn't played any more unless I'm so drunk I can barely stand
 
@roganjosh The cryptic extra '14' is somehow coming from the OP's cryptic fields about some ranges of hours involving pharmaceuticals. But it's not explained and the question is a train-wreck.
 
@roganjosh I presume that's also the will of the people for which you will fight?
the people have spoken already, no need to ask them again
 
Yes, I speak for all people except yourself, apparently. Democracy states that this is irreversible and irrefutable.
 
11:22 AM
@roganjosh didn't know she played music too! Such a versatile person.
"Don't stop me now" was in a commercial here that went on for months...didn't help
 
That's unfortunate :/
 
@smci Not just you. Voted to close.
 
I mean, im listening to it right now :P i aint complaining
 
@AndrasDeak If the advert is half decent, it can help but otherwise it's a good way to kill a classic. Exhibit A for an advert done right.
 
@smci It's bad, but 2 people figured out how to answer it. But it's 5 years old, with only sixty-something views.
 
11:27 AM
haha, I remember that commercial, but not the music
brilliant indeed
 
I genuinely can't help but laugh when watching it. You may not recall the song but if it happened to play in public, I bet it would all spring back :P
 
T-mobile Hungary commercial featuring Don't stop me now
it's too meh
@roganjosh it isn't characteristic enough for me
 
I refused to watch that advert in full. It was putting a depressing tone to the song.
 
...did I stop you?
 
So yeah, if you're subjected to that all the time, then Don't Stop Me Now is ruined for you :/
It... did stop me. I was an atom bomb waiting to explode
 
11:36 AM
hm. anyone aware of an active data science related chatroom in SO?
just had a question crop up randomly in my mind. But all my searches seem to indicate there's only dead rooms around
 
My recollection is the same but there are data scientists in this room
 
@Paritosh how complicated is the "data science"?
 
@PM2Ring I mean, when I looked at the tortured code for a while, I figured med[3] is the start hour, med[2] is the count, med[6] is the hour_step.
 
@JonClements haha, not really complicated at all.
 
chat.SE might have something stats.SE-related
 
11:39 AM
I was just wondering. at what point/estimate can we say a sequence is too big for forecasting? say sequential/temporal data
 
is it related to just theory or are you implementing such things in Python?
 
@ParitoshSingh Ask away; this and the R rooms are as good as it gets. CrossValidated is for stats and modeling end of things. DataScience.SE is a wasteland.
 
there's a datascience.SE...?
 
I don't understand the question
 
@AndrasDeak yep, theres one
 
11:39 AM
@Paritosh yeah... that sounds fairly maths/theoritical
 
As in, how far into the future can you project?
 
(DataScience.SE is a *@#$&ing wasteland and beyond repair. Perhaps should be nuked from orbit.)
 
Because I don't think there's a theoretical limit on how much data can feed a model
 
@roganjosh well no, not quite. how big of a sequence can you use to predict "1" output.
 
11:40 AM
@PM2Ring :D
 
Any size you like and can handle
 
With the understanding that you can frame the problem as a sequence classification. Or as a time series forecasting
 
Testing data would tell you this
 
@ParitoshSingh What do you mean, please give an actual example. Does "too big" mean "too many samples"? "too many periods"? "physically too big for memory/DB"? something else? Please give specifics.
 
@smci if you're an active member - have you raised that on their meta etc...?
 
11:41 AM
Are we talking about financial markets by any chance?
 
nope. okay, so for some specifics i suppose
this is basically acoustic data related to some event. say, earthquakes for example.
so, i essentially have full freedom to do whatever i like, in chunks of upto 150000 acoustic signal recordings
so far, ive been working off of aggregations and treating it as a feature-engineering task for normal models like lightgbm and whatnot
but i wanted to better utilize the "sequential" aspect of the data somehow.
 
I don't have a clear enough picture of what you're trying to do
 
my first reaction to having an input layer with 150000 points to predict one output, with 4194 total such chunks however.... makes me sweat. there's no way that's working out of the box.
 
But what are the features and what is the temporal aspect?
 
But before diving into all that, i just was wondering if there's some kind of practical limit on how big a sequence gets
 
11:45 AM
(Although that would merely be a waste of a good nuke.) @Jon the problem is the tsunami of new users asking repetitive garbage questions. The real problem at this point in time is the job market rewards any Java or SQL coder who unilaterally pronounces themself a data scientist.
 
@roganjosh The idea is that perhaps this acoustic data, recorded at fixed time intervals, is leading up to predictions for the actual event
in this case, an earthquake
 
@smci that's fair... I've seen plenty of people saying they're "data engineers" or... well, just "insert something here" that sounds trendy but is backed by nothing except complete incompetence.
 
Presumably this data is muddied by the fact that the signal recordings could all be part of different events
 
complete incompetence is a strong foundation
 
no, we are guaranteed that they are continously recorded, barring equipment limitations, for the same event
 
11:47 AM
@smci I was just addressing re: the site if it's a wasteland thing, has no-one spoken up and just said we need to reform this?
 
@ParitoshSingh But what specifically are you trying to predict? say 15 seconds of earthquake signal at 1ms intervals, given the preceding N seconds? or else what?
 
watch out predicting earthquakes, you might go to jail
 
@smci given a 150000 chunk, predict the time remaining until the next earthquake happens
so, "time" remaining to event. basically, regression problem if you want to think of it that way
 
@JonClements On a serious note, is this not in itself a tsunami of people saying that say something is very wrong in what's going on the site?
@ParitoshSingh regression if each signal pertains to a single event
 
Richter 10 is a 1996 science fiction novel by Arthur C. Clarke and Mike McQuay. The protagonist is Lewis Crane, who develops a hatred of earthquakes due to a major earthquake hitting his house when he is seven years old, killing his parents. The book's title is a reference to the Richter scale, on which 10 was considered (when the scale was devised) to be the most power an earthquake was likely to ever have. The plot deals with predicting earthquakes months or years in advance, and eventually banishing them forever from earth by stopping all tectonic activity. == Plot summary == There are four...
@Andras and everyone - very good read :)
 
11:49 AM
@roganjosh you can frame the problem of a chunk into an aggregation to turn it into 1 row, 1 output.
 
You've lost me
 
@JonClements (It's been an issue since the site went live, the ratio of signal to noise is just way too high. It's not salvageable. Think of the flamethrower scene in Aliens, that's what moderating DS must be like. The only reluctant solution is the twin combination of CV(CrossValidated) is where people ask stats questions, SO where we ask programming questions. I've had some questions on CV closed for being "too programming", but that's the price we have to pay.)
 
@roganjosh lol sorry. given a group of recordings at fixed time intervals, making a set of 150000 recordings, predict 1 value, that is some continous number.
does that kinda make sense?
 
Yes, ok, that makes sense
 
Now, given that setup, theres 2 clear routes that i can think of.
 
11:52 AM
But that's a time-series problem, not machine learning
 
convert 150000 recordings into aggregated features.
that turns it into a simple, 1 input 1 output. but aggregation makes us lose info
 
Maybe something like ARIMA
 
@ParitoshSingh sure, but that's sort of an anomaly detection problem, you need to find features that aggressively reduce dimensionality (presumably sliding windows of frequency-domain features, or whatever). You shouldn't just construct a truly enormous CNN and start shoveling Terabytes/Petabytes of raw signal data into it, then complain training never converges.
 
@smci bingo. I will fully admit, this is all uncharted territory for me. This essentially is the problem i was trying to think through in the "alternative" approach
 
CNNs and TensorFlow are not the cure to all ills.
 
11:54 AM
@ParitoshSingh if the reading is of the same feature and you only have one output then, at least to my current understanding, there is literally nothing to be gained by aggregating the data
 
I was wondering if shoving 150000 points into a CNN is as bad as it sounds
 
I'm guessing you're working on Kaggle competition LANL Earthquake Prediction, right?
 
@smci haha. yep :)
 
Busted!
 
hides
@roganjosh oh you'd be surprised. the key difference vs a normal "time series" problem in how i understand them is this: i am not predicting values off of a lag. i am predicting "something else" that has a relation.
 
11:56 AM
@ParitoshSingh Yes it is. There is no indicator because the signal is the same feature and you don't have any way of representing the temporal side other than making each time stamp a feature
 
sure... I'm just checking it's been asked and addressed appropriately - "my" site is SO and apart from sci-fi which I enjoy perusing (and very rarely have a not too bad answer) - I don't actually
"really" (and shouldn't have to) care about the rest of the stack.
 
I'm not sure I'd be surprised. A time series analysis alone would not do anything, but it would provide the features needed to actually make a prediction
 
Earthquakes aren't my thing, but look at any frequency-domain features computed over time windows of various lengths (10ms, 100ms, 1s, 10s, 100s etc.). Power-spectral densities, things like that. Look at some basic literature and see what people in the field actually use. Usually on Kaggle competition that's very domain-specific, people start threads discussing introductory papers, which libraries to use etc.
 
I haven't yet looked at the kaggle competition but this is no different than algorithmic trading. Take a time series analysis, get your indicators from that, and then trigger something. Whether that's a buy/sell indicator or sounding an earthquake alarm is irrelevant
 
@roganjosh That's pretty fair, yeah.
 
12:00 PM
And look at that competition's kernels, esp. the Exploratory Data Analysis ones with notebooks and visualizations, to get some intuition e.g. NikitaGribov: Seismic signal EDA + Analysis Function. I see over 100 kernals, have you skimmed through them?
 
@smci Thank you. yeah, i am frankly amazed by how indepth people get with these things. its been really useful so far.
Yes, all the current approaches work off of the "aggregation" methods. Most of my current work is based off of the excellent kernels labelled under "beginner/introductory". I suppose the main issue i have right now is coming up with something that gives me a more intuitive understanding of alternative things to try.
 
Suggest you start writing your own EDA notebook with your thoughts (no matter how rudimentary) and visualizations, then post it for competitors' feedback. Their opinions will be 10000x more valuable than ours, because they're actually working on that task and dataset, unlike us.
 
does anyone else remember that Iceland volcano going off? (can't remember it's name) - apart from that everything else was fine - depending on the time range, that'd seriously bias stuff
 
When I'm writing a test should I store the value of an expression when I'm only testing something about the expression itself and not the end result? Compare
thing.baz[0]
assert thang
# with
_ = thing.baz[0]
assert thang
 
@ParitoshSingh That's cool, we all start somewhere, just don't be afraid to post something (in that competition's kernels) and ask for comments. Some Kaggle users are rude, but ignore rude comments and downvotes - some of the expert guys are quite helpful. That's how we learn.
 
12:03 PM
@smci hm, i suppose thats something to consider. i am a bit.. intimidated by it all though
 
@JonClements Ejyafjallajökull or something
the js and ys are probably mixed up
 
shrugs - some <insert name here>? :p
@Paritosh one step at a time :)
 
Haha, that's definitely fair. :) I think im getting to that point with programming by now, it took me a while to actually join SO too. I suppose its the same thing to do with kaggle next.
 
I remember it because it grounded my planned 21st birthday present to myself, which ended up being a good thing because the day after my birthday I was told by uni "don't expect to be having a holiday in April because of the design project" and I happened to be too incapacitated on my actual birthday to book it :P
@ParitoshSingh Because it's business. Staying on top of approaches is worth millions to industry
And the people that enact it are worth a lot of money to said businesses
 
Funnily enough, that was one reason i picked the earthquake one. Rather than go for something that helps people just make more money, perhaps we can apply ML to making a bigger impact on something perhaps more "noble?/altruistic?" (foolish?).
 
12:11 PM
There's nothing really foolish about your efforts to use data for altruistic purposes. What might happen to the approach and how it's used is out of your hands.
 
Thats very true. thankfully im definitely years away at the minimum from coming up with something groundbreaking, so i dont have to worry about that personally :P
 
@roganjosh Einstein's theory of being able to split the atom?
 
@ParitoshSingh That statement I totally disagree with
The boiling frog is a fable describing a frog being slowly boiled alive. The premise is that if a frog is put suddenly into boiling water, it will jump out, but if the frog is put in tepid water which is then brought to a boil slowly, it will not perceive the danger and will be cooked to death. The story is often used as a metaphor for the inability or unwillingness of people to react to or be aware of sinister threats that arise gradually rather than suddenly. While some 19th-century experiments suggested that the underlying premise is true if the heating is sufficiently gradual, according to...
 
Data science is just so ridiculously vast. There's so much that's already been done, and even understanding just some of it would probably take years
@roganjosh ah, in that context, yes. technology in general has been advancing at an insane pace, but it doesn't really register until we look back to 10 years ago and compare.
 
And the people embroiled in that tech race cannot be expected to perceive everything going on. There is nothing wrong with new ideas and approaches being injected into the mix vs. people going down the paths they are used to
So really, if you actually wanna make a project for yourself out of this Kaggle competition then you should have no concerns about your intent and every comment about your approach is advisory
 
12:21 PM
I remember when I first started my "career" at 16 - with no "computer science" or any other degrees under my belt... I just learnt on the job... and these days I just take it as it comes, read up on stuff, try to understand, experiment a bit, and well, while I couldn't name algorithms or concepts or all that kind of stuff, I get a job done.
still get those little "sods" outta uni. that because they've spent 4 years learning about linked lists and hashmaps and what not value it more than some decades of actually doing it... but... c'est la vle
 
Hello can I ask a question here? I have a problem with my code but cant ask on stackoverflow for 80 Minutes:
Ok im sorry
 
@JonClements You get them in Uni too. With my Design Project I mentioned, which is compulsory in Chemical Engineering to get a Masters, I was told by one person that the supervisors would "laugh in my face" at what I did. She didn't have much to say when I got the highest mark and she was contesting her 2:1. This happens everywhere
 
@Dr.Marc no sorry - the rate restrictions are in place for a reason... it's great to have you here and that you have a question, but 80 minutes isn't too hard a time to wait out is it?
 
It is i am a practicant and probably lose my job
 
@JonClements its weird, i think this is probably a culture thing too. Where i am, most of the uni folks are just scared shitless we learnt nothing useful and wont be able to work. (and frankly, its not exactly wrong, minus the "wont be able to work" part. You just have to learn on the job)
 
12:28 PM
@Dr.Marc I'm sorry to hear that, and you're clearly desperate, but we can't help you here - good luck.
 
@Dr.Marc Try to debug your own code, and narrow things down. run things line by line if you can. Reduce the problem. Assume for a sec no one can "bail" you out, what would you do? Take a second and read this . Reduce the problem, narrow it down.
 
Thank you I will try my best to run it
 
@ParitoshSingh in the world as it is now and how things work, I imagine it's f*ing scary. I don't need a degree, I don't need whatever, I can say I've been doing it for 20+ years.
I don't know what it'd be like if I was going for my first job
 
It absolutely is. The barrier to entry is really the main problem. Almost everyone is capable of learning on the job itself, but its the "first job" fear, that's pretty rough to handle.
 
If you fear it then you're probably the ideal candidate
 
12:32 PM
@JonClements what is your salary?
 
that is definitely a very personal question to ask :P
 
4 years ago I had never programmed in my life. It was clear that I had to learn on the job
 
1 head scratch per hour and a treat for every finished project
 
@Dr.Marc have you considered the possibility that you should find a job that's better suited to your skills?
 
I personally was in a pretty bad mindset at the time when i got out of uni frankly.
 
12:34 PM
@Dr.Marc What are you expecting a salary for what you want to do?
 
Fortunately, things turned out okay on the job itself.
 
@ParitoshSingh Many fields are growing faster than we can keep abreast of. Don't be intimidated. On each Kaggle competition, pick a small but achievable objective for yourself. Also, what helps enormously is find a few teammates at your level of experience, especially reliable ones who are local and can make a time commitment to get together and discuss. e.g. for this one, it would be great to find someone with any signal-processing experience, or someone who knows anomaly detection, ...
 
@Aran-Fey just one head scratch :(
 
..but really anyone suitable with curiosity and who is committed will do. Pro tip: on Kaggle never form a team with people until they've demonstrated that they can write code, and keep to their commitments (because once you join a team, you can't remove them). Give them a small task and see if they complete it.
 
I still think this room could nail a Kaggle competition as a team but I guess I've been too lazy to instigate it :(
 
12:36 PM
@Dr.Marc I'm not going to answer that, it's varied between 2001 and 2019 in many different ways for many different reasons.
 
Ah, that's definitely solid advice. Thanks
@roganjosh haha, i'd love to be a part of that and see how that goes.
 
i except 70k
$
 
Nice to know. What is the point of this?
 
Do you have what it takes to earn 70k right now? If not, what is it that you need to learn/do to get there?
Answer those questions for yourself. You do not need to tell us.
 
@Dr.Marc and what would you do for - why would you think you'd be "worthy" of a $70k?
 
12:39 PM
@ParitoshSingh What country and city are you in? Try to find teammates at any hackerspace, data-science academy, Meetup event, whatever. That's what I did. It works.
 
Not long ago they were saying that they were losing their job and, if my calculations are correct, that's $0 salary. I smell a troll
 
beeing good programmer and support my company
you have moral problems with high salary?
 
@smci Im from India, Gurgaon. I dont know if we have such events here, though i'll be frank, ive never looked around, i am a pretty shy person irl
 
currently i am not worth 70 k but ofther i studied
 
umm... okay... gotta take a while from the keyboard but yes something smells
bbiab
 
12:42 PM
@ParitoshSingh No problem. e.g. Python and Machine Learning Meetup, Noida. Meetup doesn't seem to be very big in India, I guess try Facebook, IEEE, ACM or other things to find data-science events.
 
maybe troll, maybe not, but definitely not worth too much of your time (based on their interactions here in the past couple of days)
 
@ParitoshSingh also maybe 91springboard Gurgaon "a vibrant coworking community of freelancers, startups and established small to large businesses.". My tip is go to talks, typically at the end you can stand up and announce yourself and tell them your one-line summary of what/who you're looking for.
 
@smci I didnt even know about that site. Thank you!
 
Probably not worth my time because I charge 5 head-pats an hour and I'm not sure I'd wanna put in an expenses form to receive payment
 
cabbage
 
12:44 PM
cbg
 
cbg
 
Anyway, good luck. Let us know ho you get on.
Also e.g. Which institute is best for a data science course in Gurgaon?, although Quora is overrun by spammers.
 
much appreciated, will do :)
 
There are presumably PyCon or NumPy events in Delhi?
PyCon India 2019, Oct 12-15, Chennai yes I know that's down in Chennai
 
@ParitoshSingh If you wanted to start this off then I would try contribute to a shared github/bitbucket
 
12:49 PM
But yeah, do try to post a kernel on Kaggle, even a very modest one ("my first kernel"). It's about learning, not about being #1.
@roganjosh No, that's what kernels on Kaggle are for... people can vote, comment, improve them, fork them with one-click, ensemble them...
 
But they don't reveal the winning code
There's a point in forming teams, and those teams have to collaborate somehow
 
You "can" if you wish to, and i think some competitions ask you to reveal winning code so to speak, at the end of competition.
but yeah, you can, within a team, create private kernels. With all the fork/comment/modification goodness
the only thing if anything i dislike about kernels is being unable to run them forever, but i understand why it is that way.
 
@roganjosh sometimes people do disclose killer features. But also, after competitions end, the top-10 competitors often do post writeups in that competitions forum, with working code, visualizations, explanations, what features they tried. etc. [Paritosh] So look at the previous signal-processing competitions. There are several.
 
@smci That may be, I haven't actually ever submitted anything to Kaggle. But I would be a bit shakey trying to develop something that isn't using my local tools
 
@smci oh shoot, that is a brilliant idea
 
12:55 PM
I'm not fussed whether the end result is global, I just want to be able to work on it locally as I would my own workflow
 
you can easily download the code from a kernel. Thats usually how i do it, i prefer working locally too
the only issue is, i dont have a GPU, so thats a no bueno on my local system if i ever try something stupid.
 
Ok I'm logging off now. Paritosh let us know how you get on. You can tell us your Kaggle handle/URL if you like.
 
cya, thanks again for the ideas and the convo, its been really useful.
 

« first day (3079 days earlier)      last day (1869 days later) »