« first day (3449 days earlier)      last day (1490 days later) » 
00:00 - 18:0018:00 - 00:00

AMC
12:23 AM
Does anyone know the appropriate course of action for a question such as this one?
I'm searching for a duplicate, since I could have sworn that I've seen one.
I found a few 0-1 upvote questions.
 
@AMC hammered
@wim ¿queue? There's no versionadded
 
AMC
@AndrasDeak Thanks. I'm going to keep looking for a reference/canonical answer, the more I stare at the one I found, the less I like it.
 
wim
@AndrasDeak been there as long as I can remember
 
@wim huh
@AMC I didn't use your target
 
wim
12:28 AM
(digs) new in 2.6
I also started using Python around 2.6, so "as long as I can remember" was a lucky guess.
 
AMC
@AndrasDeak Yes, I know?
I'm going to keep looking because I'm sure that this kind of question will appear again in the future.
 
OK, just wasn't sure
@wim with the frequency that heapq has come up here it's surprising that queue is less prevalent
 
 
3 hours later…
3:03 AM
:D OP insulting me in comments because i guide him :D lol
 
wim
3:45 AM
the common theme in all these is answering the crappy web-scraping questions in the first place, why bother? the questions are garbage and the users are often stupid, rude, or both..
tbf your edits and answer looks fine and all, but we have a saying "don't polish a turd" :)
the terms on that site also asks not to scrape it:
> ... use of data mining, robots, or data gathering and extraction tools.
 
I see, just were trying to help even I've already reported it as duplicated. but there's people in earth which is very rude even if you just did a good thing for them.
 
wim
@MisterMiyagi I think they are even worse than "how to regex this", because I suspect many scraping questions are either hand-holding O.P. every step of the way through some low paid job which they aren't competent enough to do without help, or even worse you're assisting them on some outright illegal scam or spammy project
 
4:00 AM
I will stop that from now on.
 
wim
@αԋɱҽԃαмєяιcαη yep, there are. but maybe you have a sampling bias here, too - those who would be willing to take on an ethically questionable web-scraping job may have a higher probability to be the desperate or rude users in the first place
 
 
1 hour later…
5:12 AM
cbg guys o/
Is there any Python lib which can extract table data from a pdf and convert it to JSON?
 
pandas.read_table
 
That reads from pdf?
 
Yes
otherwise from tabula import read_pdf
 
I found this lib called: camelot
 
5:49 AM
@wim I'd venture say you might be overly correlating rudeness with job. The internet in general is mostly rude, aggravating, and offensive. SO is an exception at keeping a professional tone. Also, I don't know if you graduated Uni or worked at large multinational companies, but frankly the latter 2 were among the rudest places I've seen...
@wim but I do appreciate your points about the web scrapping Q's...Hadn't thought about that.
 
6:23 AM
hi
 
Hello
 
I am getting some problem in deploying my flask application in heroku .
I need help.
I am following this tutorial
 
user10984358
6:37 AM
you will have better suggestions if you tell what problems you are facing
 
user10984358
not that I can help, but users here need to know what it is before they can help you
 
6:51 AM
remote: ! No default language could be detected for this app.
remote: HINT: This occurs when Heroku cannot detect the buildpack to use for this application automatically.
remote: See https://devcenter.heroku.com/articles/buildpacks
 
7:07 AM
cbg guys, could someone help me out with this: stackoverflow.com/questions/60828879/…. Thanks
 
cbg
 
Hello ! Python people ....Hope all are fine stay home and stay safe ...Just need a help on . i am working on insurance project where customer will renew the policy or not . in that i have mobile number and email id . the hypothsis is such that if mobile no and email are present . the chances of renewal are more . so in this context . i have mobile no and email which is dirty , need to clean . how can i clean or validated the mobile no and email is . please guide...
 
what do you mean by dirty
 
@αԋɱҽԃαмєяιcαη dirty means improper data there would be all 0 or all # any thing apart from mobile no , it can all alphabet ...
 
i think that your question answer itself already !
 
user10984358
7:21 AM
I think he is asking how to do that
 
@TheNamesAlc yesy ..thank you
 
ah okay, check if the mobile# include digits only
then check if it's in the same length of your desired. depending on country of your selection.
 
user10984358
pandas.pydata.org/pandas-docs/stable/reference/api/… might be of help, seems like its easy to do with regex?
 
regarding email. it's will be the same to validate an email address is in the proper format.
 
user10984358
I am assuming the input is from a csv or similar, if not you have to loop through what ever data structure you are using and then do the check
 
7:24 AM
@TheNamesAlc well, if you asked such question or even did little search, you will see a multiple answers, that's in case if i will go and say "Hey how can i check for a valid mobile number"
 
user10984358
sometimes you dont know the keyword to search, he knew he had to check them he didnt know what to use to get it done, so I was assuming it that way, but research does help
 
THANK Q ..was helpful
 
user10984358
is there any other way other than permalinking a starred chat to read the whole message? :48919866to the last question, no. There is a big divide between the magic that tech companies can do, and the reality for manufacturers. There are... to read after ...
 
user10984358
I was expecting something like a click to jump, but history and permalink are the only options
 
Click the timestamp
 
7:33 AM
then pickup the url
 
I'm not sure I know what you're asking, since to get the permalink you'd have to do that in the first place?
 
user10984358
i feel dumb
 
user10984358
i just wanted to read the message, permalink just took me to that chat
 
user10984358
not scraping or anything just wanted to read the whole message
 
7:35 AM
No worries :) it's not really intuitive
 
user10984358
It is orange in color, but I am not used to clicking url's that are timestamps
 
It's the same for when people link to comments from main btw
 
user10984358
ahh, thanks!
 
7:59 AM
Hi everyone, I would appreciate it if someone could give me directions: I'm working in an offline environment with an installation of Anaconda 4.5.11 and a local mirror of conda-forge/win64. However, conda search does not find many popular packages like statsmodels, scikit-learn, scipy.
Indeed, the online channel does not list these packages (conda.anaconda.org/conda-forge/win-64) while the conda-forge page of these packages (e.g. anaconda.org/conda-forge/scikit-learn) suggests that they should be included. Am I missing something? Why are such standard packages not included?
 
dupe stackoverflow.com/questions/60862925/… . Essentially about what is Truthy vs Falsy
 
^ closed
 
"Someone asked if ... but he didn't seem to be satisfied by the answer" I smell a thinly veiled interview question.
 
8:05 AM
that's it in the 1st place
 
Not sure what you are pointing to. I am not working with the Anaconda base environment (where these packages indeed are available), but I created a new environment for a separate project. conda install just looks for it in the registered channel (local mirror of conda-forge) where it is not available...
 
8:53 AM
unclear, OP doesn't seem to understand their homework task stackoverflow.com/questions/60863155/…
 
@MisterMiyagi Then you could mention that in a comment. Interview questions are allowed.
 
Tis was a general rambling. Nothing bad about it per se. Definitely better than the other trash my SO feed presented me with today.
Though some of our interview-preparing semi-regulars may want to take the hint that they should know the answer. ;)
 
user10984358
Is that a good question to ask in an interview?
 
hi
I have an error: local variable 'screen' referenced before assignment.
What is wrong with the following code?
 
@TheNamesAlc If it is for a Python specific position, yes. Booleans and None'ness are a very fundamental concept, and integral to code health.
I'd use it as one of several gatekeeper questions to weed people out.
 
9:01 AM
@YoelZajac Just like last time you posted, we still need a (formatted) MCVE
 
I know, don't know how to do it.
Can you explain me briefly?
 
you've already been pointed to our formatting guide, but the TL;DR is: 1) paste your code 2) press ctrl+k
 
@YoelZajac Did you google the error message? It's a pretty well-documented error.
 
import sys

import pygame
from pygame.sprite import Group

from settings import Settings
from pygame.locals import QUIT
from ship import Ship
import game_functions as gf

def run_game():

	# initialize pygame, settings, and screen object.
	pygame.init()
	ai_settings = Settings()
	self.screen = screen

	# Make a ship
	ship = Ship(ai_settings,screen)
	# Make a group to store bullets in.
	bullets = Group()

	screen = pygame.display.set_mode((ai_settings.screen_width, ai_settings.screen_height))
	pygame.display.set_caption("Ship_left")
@MisterMiyagi yes I googled.
There are many explanations, nothing regarding 'screen' variable
 
9:06 AM
screen is just a name.
 
The error is telling you that you're trying to access the screen variable before you assigned a value to that variable (i.e. the variable doesn't even exist yet)
 
The problem doesn't depend on how you name the variable, but how you use it.
 
so I need to define 'screen'... am I right?
 
Well, you're already doing that. You just need to do it earlier
 
I'll try....
 
user10984358
9:13 AM
isnt his indentation that is causing the problem?
 
user10984358
he has screen inside the function, which may not be what he wanted? or is it a copy paste error
 
Well, changing lines in this way:

screen = pygame.display.set_mode((ai_settings.screen_width, ai_settings.screen_height))
	self.screen = screen
 
but the screen is created based on ai_settings, so it must be inside the function
 
now telling me that

'self'
 
user10984358
he has self and over there, so this must be inside a class
 
9:15 AM
is not defined
 
Right. Why would it be?
variables don't magically appear out of nowhere
 
@YoelZajac Please ask yourself what you think is the value of all these things that Python tells you are undefined.
 
@MisterMiyagi closed
 
because there isn't any class, only one function in the main program. Should I create a special class only for

screen

?
 
What would that class do? What instance attributes would it have? What methods would it have?
If the answer is "none", then you shouldn't create that class
 
9:38 AM
If it is undefined and no class is needed, I guess that
self.screen = screen
is just unnecessary... am I right?
 
Yes.
Is this your own code, by the way?
 
"Python Crash Course".
 
Fitting title. :/
 
This is an exercise from the book, copying the vast majority of the text in the explanatory files, but also trying to understand.
 
9:49 AM
@YoelZajac Does that also explain fundamental concepts of programming?
 
@YoelZajac For too long code, make a gist and share link here.
 
@TheLittleNaruto looking at that link, don't understand what should I do...
 
Sorry I assumed you are aware of Github.
 
@AndrasDeak The first part explains the fundamentals, then the second part are projects and exercises based on the written projects.
@TheLittleNaruto actually would love to write my projects in github, don't know how to do it.
 
If you're already logged in, you can see three boxes, first one is optional which takes description of the code you're going to paste. Second text box takes File name with extension, for Example hello_world.py and the last one takes the code you pasted above. After all these done, you can click on "create/public/secret gist" button.
Good thing is, it also does code coloring based on the extension you would set in second box as part of file name.
 
9:57 AM
@YoelZajac it's like pastebin/dpaste etc, just on github. Code paste service.
 
ok, trying to understand..
@TheLittleNaruto hmmm... little confused.. I see many boxes in the link that you sent to me..
not only three
 
Text boxes*
 
@YoelZajac you're not logged in to github cc @TheLittleNaruto
Forget about github, you can use any code paste site
 
Uh-oh!
 
ok, I'll get acknowledged to github later.
Thank you guys for all the help.
:)
 
11:07 AM
Is Pypi working for anyone else or is it just me?
pip search is broken
and searching on pypi.org doesn't work for anything other than (presumably cached) common searches like numpy
 
> PyPI is down for maintenance or is having an outage.
it's probably not just you
 
Sure, its just that there were no other status updates anywhere that said it was having issues
 
11:24 AM
perhaps they are busy putting out the fires, or haven't noticed yet
 
user10984358
So if a website says There's also a complete Python API, but it has not yet been documented. how else can I find what parameters a method expects? need to look into source? Ref : github.com/openstack/python-cinderclient
 
user10984358
is there something like dir(object) but instead gives the parameters?
 
help(function)
 
user10984358
wow that works, i thought help was only for stdlib function
 
user10984358
thanks
 
11:37 AM
Fresh cabbages everyone, I hope you are having a good day
 
@TheNamesAlc that's what docstrings are for
hmm, OK, signature is extra
 
user10984358
so you're telling me the codes I have written using numpy docsstring format also supports help?
 
user10984358
I documented stuff but only for sphinx autodoc
 
user10984358
never tried using help on my own function, will try that
 
@TheNamesAlc yes
that's how numpy's help works
 
user10984358
11:47 AM
didnt know that, need for docstrings in my code just increased knowing one can do this, I wasnt expecting OpenStack python api to be undocumented though
 
@TheNamesAlc I'm sure pull requests are welcome ;)
 
user10984358
lol, I can say I contributed to OpenStack in my resume then. wishful thinking
 
@TheNamesAlc doc fixes and issues are really important in open source. Nothing to sneeze at.
 
user10984358
I assumed bug fixes were top priority
 
pypi is back now 👍🏼
 
12:04 PM
@TheNamesAlc top prio maybe, for core devs. But docs are important for users, and community members can tackle those issues more easily
The great thing about opensource is that if you find an issue you can help trying to fix it. That's easier to do with a docbug than with a code bug. Much less maintainer supervision needed to merge.
 
@Dodge you seem to be liking Minecraft?
cbg all
 
I have a few PRs in numpy and scipy and almost all of them are docs
 
user10984358
I might actually consider doing it then, considering I have a lot of free time now, appreciate the suggestion :)
 
user10984358
is Minecraft paid or free? I want to get in just for the fun, google tells me there is something like a classic version, will that let me join in the sopython server?
 
12:17 PM
Not sure there's ever been a free version?
 
user10984358
well classic is free but its not being updated, thats what I understood
 
@TheNamesAlc you can always ask them first if this would be welcome
And check for open PRs
 
umm... not sure... however I'd imagine a classic edition would be to play an older version locally and that it wouldn't be compatible with the latest version of the server... else no-one would ever buy it :)
 
user10984358
yeah thats what I was thinking, I guess its just to lure players in so they buy the full game
 
user10984358
@AndrasDeak a lot of devs in my org often raise bugs in OpenStack, not sure if they send fixes but they talk about this a lot
 
user10984358
12:24 PM
in case anyone is interested classic.minecraft.net , doenst ask for any signups, limited to 10 players
 
user10984358
it is more like an extended demo I guess
 
@TheNamesAlc ahhh.... yeah... that's definitely a good 5 odd years behind :)
 
@TheNamesAlc I meant if docs PRs were welcome. Issues always are.
 
user10984358
lol, let me see if it entices me, later guys
 
while loops again
i append to the queue so that queue should never be empty
but its terminating
IGNORE^^^^
 
12:45 PM
hello
 
12:59 PM
i love python
honestly best language
 
English is quite useful as well :)
@InfinityTM hi
 
print("strong disagree") :p
Cabbage all
 
1:44 PM
Does anyone know why scikit-learn is not available in conda-forge/win64? It's a standard package, no?
 
@JonClements For a blocky game, I'm surprised how quickly one's imagination takes over. Let's hope Las Vegas doesn't come up with a version of the game that involves gambling because that might be the most addictive, non-substance based activity in the history of the world ;)
 
2:00 PM
@JonClements Close: ten years. Last update November 10, 2009.
 
I was spot on for a rather large value of "odd"?
 
Being off by a mere factor of 2, that's practically a rounding error
I don't get out of bed for anything less than 10
 
Indeed!
 
Cbg
 
Clicking around the version history section in the wiki, I can't find any explicit mention of the version I remember that took place on a flat plane. I think it was an offshoot of pre-survival Classic.
minecraftforum.net/forums/archive/classic-creative-mode/… might be evidence of that era. Notice the lack of terrain and the relatively limited area of play. Unlike most versions, the world is not unlimited in size.
 
2:11 PM
@PaulMcG cbg
 
@Kevin There's a downward limitation (I know because I've hit bedrock) but do you know if there is an upward limit?
 
@Dodge you've hit bedrock?
Wow
 
Most likely yes. Every version I've ever played has had an upward limit.
 
@JonClements Daimond fever is no joke
 
In the current version, I believe the cloud layer marks the halfway point
 
2:18 PM
@Dodge I've created a monster!!!! :p
 
Bedrock isn't too hard to hit -- sea level is at about y=64, and bedrock is at about y=6
You can dig a stairwell from ground to bottom with a half-used iron pick
 
don't think I've ever hit it... but then that's probably because I never normally survive more than a few minutes at a time
 
Playing on a public server is bad because everyone/someone can see how much time I am wasting :|
 
Not quite... only reason I can see stuff is because I'm running the server and just have a peek at the logs occasionally to check it's behaving itself
 
If you're online when I sign on and online when I leave, all that tells me is that you have played at least two seconds longer than I did. So I can't judge you.
 
2:21 PM
I've been told there is a way to program in the game with levers, but the concept is still murky. Might do that at some point
 
Of course, if you erect an entire city of gleaming silver skyscrapers, that probably gives me a hint about how much time you're spending outside of my direct observation
 
redstone stuff
 
I have dreamed of a solid gold castle, I wont lie :)
 
I've never done much with redstone. It's certainly turing complete, but you need like eight blocks worth of space to make an XOR gate.
Anything capable of printing Hello World would be kilometers wide
 
umm... suppose I could login and spawn a few hundred gold blocks or something... but then part of the challenge is earning it, right? :p
 
2:24 PM
Creative Mode exists exactly so you can be an immortal god-king with unlimited resources. I think you can turn it on with a single command.
I prefer to scrabble in the dirt, myself.
 
can't beat hard graft in mine craft :)
 
There's so much dirt to scrabble in
On another server long ago, I was constructing a tremendous field of wheat, because I like constructing tremendous fields of things. A moderator that I was in the good graces of appeared one day, and with a few deft commands, cleared out the large zone of rough terrain that I had cordoned off, placed down tilled dirt, and sowed it with seeds. I was initially pleased, but soon I thought "well... I guess I'm out of projects now."
 
Lol, sorry for the off-topic, but didn't use python for a week, had to try 3 times to open an excel file ("pd.read_xlsx","pd.read_xls","pd.read_excel")
 
From the BBC: "An appeal has been launched in the Czech Republic to "save" the equivalent of 1,305,552 pints of craft beer that will go off "within weeks" if it is not drunk. [...] If you’re in the Czech Republic and think you can help, see: zachranpivo.cz/";.
It's interesting to see just how many sites are being thrown up so quickly, too
 
"savethebeer.cz", nice
 
2:38 PM
@roganjosh People take shelter for two weeks and that's how much beer is not consumed, wow! (To be fair, a million pints seems like a lot but I knew a guy in the army who could drink that in a week)
 
Relatedly, while I was acquainting myself with the specifics of New Jersey's quarantine orders, I discovered that liquor stores are explicitly considered "essential" businesses, alongside pharmacies and grocery stores. I got a kick out of that.
 
@Dodge off licenses were added to the list of essential businesses over here so they can stay open. Presumably it's to stop people sneaking into pubs
 
I guess "essential" includes both "will people die without this?" and "will people riot and storm city hall without this?"
Devil's advocate: liquor stores serve as a useful backup for medical supply stores, since you can make homemade hand sanitizer out of 70 proof or higher alcohol
 
It's definitely to stop pubs opening :) I know of 3 that, prior to the lockdown but after they were originally shut, that were planning to do it. One put on their Facebook "we hope you'll still visit when we see the back of this" meaning "regulars should use the back door". Having off licenses open vastly reduces the temptation
 
except there really aren't off licences here any more... just lots of shops that happen to serve booze
well... sell... serve isn't really the right word there I guess...
 
2:46 PM
anyone know what max_subtring doesnt go up?
 
l-r+1 will quite often be negative, yes? l can only ever be zero or one, so if r is greater than 2, then the result is less than 0
 
oh damn
sorry
 
If you move l = 0 above the loop, then it can be larger. I suspect you should also be doing r - l + 1.
Admittedly I don't fully understand what the ultimate goal here is
 
yeah this bit is wrong l += 1
 
I wonder if there's some terribly clever way to do this in O(N)... I'd probably just write a lazy nested loop and hope it doesn't time out when I submit it
 
3:16 PM
that's a dynamic programming problem
 
My brain bounced off of any DP based approaches, so I'm thinking of something sliding-windowy
Probably O(N^2) in the worst case, but approaching O(N) in the best case
Like this. (spoilers, obviously)
 
instead of looking at each character and keeping counts, it's probably faster to scan the string once and check whether len(set(chars[i:i+max_len + 1]) > max_len. Slice and sets are fast.
 
3:31 PM
Now that I write it out, I think it's just flat O(N)? I guess I assumed that the right end of the window would both grow and shrink, but it really only does one
 
@Kevin why?
 
I'm only 75% confident that I haven't missed an obvious corner case
 
this is a two pointers/sliding window problem as well
 
@Kevin the best case should be O(n), since you have to scan the string (O(n)) but have a fixed upper bound for the lookahead (the size of your alphabet).
 
@MisterMiyagi i dont get this approach
 
3:34 PM
Basically at the end of each loop you want to preserve the precondition that the window contains a valid no-repeated-characters substring. I thought that you would occasionally have to shrink the window from the right in order to preserve that precondition, but thinking about it some more, you don't. The left edge of the window only ever moves forward, and doing this can never violate the precondition, so there's no need to move the right edge backwards at that point.
 
def lengthOfLongestSubstring(self, s):
    max_len = 1
    for i in range(len(s)):
        # check if there is a *longer* substring up ahead
        while len(set(s[i:max_len+1])) > maxlen:
            max_len += 1
    return max_len
 
My O(N) assertion is assuming an idealized dict with O(1) access time, by the way
 
btw this is one of the fabled 1% of cases where range(len(sequence)) actually makes sense.
 
@MisterMiyagi lol
 
the DP approach requires a whiteboard... at least for me. I need to sketch out the table before I can code up its generation, this morning
@MisterMiyagi oh heck yeah!
 
3:38 PM
someone fetch the champagne!
 
/giphy champagne
oh damn! this isn't slack
 
No champagne for Mr. camelCase
 
I dread to think how this room would look if we had a bot with a /giphy command :)
 
if you record the last seen index you can reset the l pointer.
its wrong but it looks right
 
Hmm, I notice that my own solution's range(len(s)) could have been an enumerate, since I do access s[l] in the last line of the block.
 
3:40 PM
@Permian for some that'd be enough :p
 
It would have been more characters to do that, and introduced an additional poorly named single letter variable, but it's the principle of the thing dangit
 
@JonClements not for me
 
"string analysis algorithms that require lots of manual tracking of indices" is one category of programming challenge site questions that I consider most unlike my actual work as a programmer
 
@Permian You can check whether a string has no duplicates via len(set(string)) == len(string), since set eliminates duplicates. You can also get a substring from i of length l via string[i:i+l]. You can use this to scan through the skin, checking whether the current position is the start of a non-duplicated substring longer than the longest you have found so far.
 
If you get dizzy trying to juggle pointers and offsets and histograms, good news! Most of those problems were solved twenty years ago and you don't need any of them if you're trying to make, say, a web site.
 
3:50 PM
@MisterMiyagi genius
 
hm, checking various other languages on leetcode, I see that Rust also hast this useless impl Solution, yet GoLang directly expects func lengthOfLongestSubstring.
so, that's quite a bummer
 
Rust and GoLang (weeps)
 
in addition, I realise we've all been fooled to look at the Pythoff version of the challenge instead of Python3. I smell a conspiracy.
@Permian C also does it properly. I expect an evil plot by the OOP brigade, only they didn't quite get GoLang's approach.
 
4:09 PM
someone tipped over the java jar and it poured all over the other languages
 
4:25 PM
dupe / needs details, pick your favorite reason stackoverflow.com/questions/60848129/…
 
@MisterMiyagi note the age of the question...
although OP was last seen 19 days ago I guess
 
I firmly believe it is never too late to tell my data analysing brethren, sisters and others that they please always have a minimal ounce of understanding of the data. Just in case said data ever ends up being my data, and potentially being widly misinterpreted.
 
Yeah, sure, just wanted to note that with a question that's 7 months old you'll never know if OP will ever read it
no harm in commenting, just FYI
 
Admittedly, I was just too lazy to check.
Frankly, I find it a tad scary how many data based questions reveal the OP has not even the most basic idea about the data they work with.
 
Coders must become comfortable with the idea of shouting into the void with no hope of a response. It's an essential skill when asking obscure questions on SO, making bug reports, or asking your boss for better hardware.
 
4:35 PM
@MisterMiyagi that's just the usual business. Except with data it's more glaringly obvious when someone has no idea.
 
@Kevin Your boss is the void? Be blessed by the mighty machine spirits, may they protect you.
 
@AndrasDeak I know, the OP mistagged it as only [python-3.x[ not [python], as my comment said. Just close as 'needs details' already, unless OP refreshes it with new details. The OP's issue was whether graphivz package needed pydotplus rather than pydot. Just close, unless is reconfirmed as a real issue.
 
I know you know, I wasn't sure Miyagi knew.
 
@AndrasDeak For code, yes, but from my experience, it's the exact opposite when looking at results. If some devops scripts is bonkers, demons from hell invade your office. If some data analysis is bonkers, you get a pat on the back for figuring out this complicated problem no-one understood.
 
I had "scrapping the web" and "create this website" in mind
things lending themselves to cargo cults
 
SVG
4:46 PM
Hey guys, I am doing a project related to glucose value prediction for individual patients. But now I want to build a more generalized model combining all the patients data together, and give a prediction to the users who don't have any previous data.(eg: New patient identified today as diabetes). So the dataset contains about 50 patients with 22000 records of their blood glucose values, insulin dosage and carb intake and other related factors.
So inorder to reduce the error of the predictions I used K-means clustering to cluster similar data such that noise from unwanted patients will be reduced to a certain extent. So I clustered data in to 43 clusters and for each cluster I trained a machine learning algorithm. That means I have 43 machine learning models trained.
But it seems like messy and difficult when deploying since I need to maintain 43 pickle files and load them based on the cluster predicted by the KMEANS
 
is there a reason why you store the data as pickle?
 
That means that you've also drastically reduced the amount of data with which to train each model
 
SVG
So guys, the problem is, Is there something I am doing wrong, like is there any easy way to achieve the same functionality as above without creating 43 separate pickle files?
 
or for that matter, why you consider 43 files to be messy?
do you cluster patients or records?
 
You might be able to use the clustering output as some categorical data and combine it all into a single model
 
4:50 PM
Noise from unwanted patients? Which ML algorithm are you using? If you look into say gradient boosted trees for instance, they intrinsically take care of that by bootstrapping
 
SVG
@MisterMiyagi I cluster records
@roganjosh Yeah exactly because for different patients their blood glucose variations and carb intake is different.
 
So in other words, you throw all patients' records together, and then only select the records across all patients which are similar?
 
SVG
@roganjosh I used SVR and decision trees
 
That doesn't seem suitable for predicting the values of individual patients.
 
SVG
@MisterMiyagi Yeah true only the patients who are similar will be taken into consideration
 
4:53 PM
From the way you describe it, the very idea of patients and their progression is eliminated.
 
@SVG ok. I still don't understand the reasoning behind the clustering and separation, though
 
If you cluster all your training samples into 43 clusters. When you do get a new unseen sample, it will only be related to one of those clusters, right? (assuming K was correctly chosen) So only one of the models will know what to make out of the new sample...?
 
SVG
@MisterMiyagi Yep but for predicting new patients who oon't have any previous records. The thing is I tried the same thing without clustering, the rmse lies around 60-70 but when I do clustering and prediction my rmse reduces to 30-40
 
How would you know if there was any correlation between the parameter you clustered on and any other variable in the models? All of those potential interactions are lost
 
offtopic (too old to migrate) Question was migrated to CrossValidated back in 2017 stackoverflow.com/questions/46655406/…. Yes I know it's not Python. Just fighting the tide. Like King Canute.
 
SVG
4:56 PM
@yatu Yep!!
 
@SVG You mean the RMSE compared of predicting the clusters? That shouldn't be much of a surprise, since clustering will naturally select similar data.
 
@smci if it were migrated it wouldn't be on SO. OP just re-asked it there.
 
SVG
@MisterMiyagi Yeah exactly, So is there any way where I can create only 1 pickle file to all of those stuff like clustering and prediction
 
@SVG So the approach doesn't seem to make sense? How do you handle such scenario, in which you only have one of the 43 models' output be of interest, an you do not know which it is?
 
SVG
@yatu Sorry didn't understand. can u explain it a bit ?
 
5:00 PM
@AndrasDeak yeah yeah as I had already commented on it, 'migrated' as in the OP manually reposted on CrossValidated back in 2017. Just trying to clean things up here on SO.
 
@SVG I'm under the strong impression that your clustering approach is inherently flawed and will lead to wrong conclusions. Before polluting the channel with sarcastic remarks based on guesswork, I'd rather not give any suggestions.
 
SVG
@MisterMiyagi Yeah I can understand. Since I am a newbie to this field I also doesn't have much knowledge regarding this area.
So I might fail to answer your questions sometimes :(
 
Can the person starring all the things please stop? We prefer if only interesting things are starred, that can possibly be relevant to a broader audience.
Unfortunately stars are a common public pool.
you can collect the permalinks of the chat messages somewhere for yourself for future use
 
Thank you, Andras and Anonymous Starrer/Unstarrer
 
@SVG Like everyone else is advising, this approach is flawed and unnecessary. Most classifiers can handle clustered input. (If you had a severe imbalance between sizes of clusters, can handle that too.) Just don't do this. 47 pickle files is only a symptom, not the root problem. "in order to reduce the error of the predictions... [I threw away individual records... to reduce prediction error]" doesn't make any sense. What was your original loss function? RMSE? logloss? something else?
... Did you normalize your inputs? (I suspect you didn't). Strongly suggest you go back to the original classification problem and solve it.
@PaulMcG I starred them (at least short term, so the OP sees the consensus against their methodology; I could remove the stars tomorrow). Andras disagrees with you and me. What should we do?
 
5:11 PM
Nothing, we're already done.
but I'm not sure Paul and I disagree
 
@AndrasDeak Clearly Paul does. I can remove the stars tonight. I see lots of stuff here that other people tag that doesn't have longterm value, jokes, memes. If you think you have a consensus for what people should and shouldn't star, then edit a 'Guidelines' section into the Room Rules. I don't believe any such consensus exists. (Personally, I want to see fewer meme jokes.)
 
We don't - if we did, I would have said something besides "thank you". I don't think chat really has a mechanism for what you want to do @smci. The star board currently shows what I would characterize as "interesting comments/links/snark above the usual chat", not +1's or links relevant to a particular thread of comments.
Where did you get the idea that I disagreed with Andras?
 
@PaulMcG I can remove those stars tonight. But that still leaves lots of starred junk, memes, snark - I don't see Andras objecting to those. Suggest we table suggestions for a 'Guidelines' section in the Room Rules
 
Your suggestion is noted.
 
@PaulMcG Much love for M*A*S*H. Quarantinis, anyone?
 
5:20 PM
Evening cabbages everyone
 
No really, @smci - you completely misread my "thank you" comment (again, I'm mystified how that could be), and then put words in my mouth ("Clearly Paul does."). Please don't speak for me in the future.
 
Is there a simple way to assign a number to all possible subgroups after using pandas groupby ?
 
@PaulMcG Your post was grossly unclear: why did you say "Thank you... Anonymous Starrer/Unstarrer" unless you agreed with (temporarily) starring those posts? And if you did approve, then clearly that disagreed with Andras. I'm not writing a classifier to decode conflicting utterances in chat... meanwhile, this that I posted earlier still needs close-votes.
 
I was thanking you for unstarring them.
 
@PaulMcG I haven't unstarred them. I said I would unstar them tonight. Ok I'm unstarring them right now. It seemed the OP still was not accepting the unanimous consensus that clustering records and throwing away most of the input data to reduce classifier error a bit, was bad methodology.
 
5:25 PM
@CeliusStingher Well, there is GroupBy.ngroup, if that helps
 
...hmm, they're already gone, seems any of the ROs can unilaterally remove stars?
 
@smci yup... RO's can remove stars from posts...
 
@JonClements Hmm, that has a clear potential for abuse. When I look at starred posts, a lot of that is ephemeral and not relevant or valuable.
 
oh noes
call the star police, quick
Hmm...Starfleet might work
 
@AndrasDeak Meanwhile, this that I posted three times still needs close-votes.
 
5:30 PM
@smci - I went to that question, your link to stats.stackexchange.com (labeled "Cross-validated") is not formatted correctly.
 
@smci why do you think that it's a good idea to keep pestering me with that request while you're lawyering moderation in this room? It's not.
 
@PaulMcG Ok, I'll fix that, but that's no reason not to CV.
 
You should stop with both.
 
@AndrasDeak Don't make false statements about me again. I came here to do a morning cleanup of bad or dupe posts. I didn't "lawyer moderation". I did point out there's no consensus from the room for your subjective individual opinion about what should be starred. I had also pointed out that you had misunderstood why I temporarily starred them, and you were alrady aware I'd remove the stars. Meanwhile, there's lots other starred junk that you're ignoring. Guidelines on starring would be good.
 
I've done statistics and multivariate work in my time, but it's been a while, so I hesitate to weigh in on data modeling posts, either on SO or chat.
 
5:37 PM
@PaulMcG (already fixed the URL. Anyway that wasn't a blocker for closing)
 
I would infer from the general lack of complaints over time by most room participants on this issue (first time I've ever seen it come up, to be honest) as consensus, albeit undocumented.
 
@smci Do you have a proper dupe on this question? The question itself seems surprisingly un-unclear to me, and the dupe candidate doesn't have an answer.
 
@PaulMcG I like having useful/informative stuff in the star bar, but I also love humorous stuff, both links and chat. I don't mind if the chitchat/jokes get removed later or heavily culled down to stuff that might be funny without context of the chat, but that does seem like, well, effort.
 
That is about my take as well.
 
00:00 - 18:0018:00 - 00:00

« first day (3449 days earlier)      last day (1490 days later) »