Of isolated units, yes. Much of the stdlib is tested that way, AFIK, or was.
Basically, a mechanism to tightly couple simple, short tests, in REPL syntax, directly to the documentation (module/class/function "docstring"), which should include samples anyway, right?
you dont need to colour your datapoints, thus skipping over dealing with the "Target" class so to speak
that plot in particular however uses the colours to indicate true labels, and usually visually seeing their segregation gives us an indication of how easy or hard it would be to segregate them
OK. So my point is this: the colours in the figure you found are not the output of clustering or PCA. The colours correspond to data already in the dataset.
In your case this would correspond to you having a "male/female" label for each of your data points.
pca in and of itself, is just a "reconstruction" or representation of multiple features (say n) by new features that capture the same relevant information with less than n features.
inherently, it doesn't tell you anything about the target.
This is a series of data in the initial set, but they are divided according to the PCA philosophy. In the sense that similar dimensions are projected separately and different dimensions are distant.
sorry, same dimensions are near, otherwise are very far
if you see this iris setosa are all near , because probabily have similar dimension
I understood only half of that (my fault this time). Are you saying you're trying to come up with a different measure (and algorithm) according to which you can separate the red and the blue points?
classification usually for classification problems, we dont bother with clustering. or atleast i dont see why one would want to
"supervised" learning to be more specific.
at the risk of losing model explainability, pca is just a step of reducing the dimensions so hopefully the ml algos can do a better job separating things. but you have to do the actual model building afterwards on those variables
Ha, this reminded me of an answer I posted to a similar question. They had a huge dataset with many dimensions and wanted to cluster it. I added an incomplete answer showing them that they at least have a chance of clustering because there are gaps in the data
So I was thinking, all these really rare radioactive elements...they're rare now, but hundreds, thousands, millions of years ago...surely they'd have been much more common?
...and you know what that means? Radioactive dinosaurs! Or: Spiderman, Hulk, Daredevil, Doc Manhattan, and Fantastic Four: all legend, rather than fiction.
@toonarmycaptain More like radio-insensitive polyextremeophiles, and a much higher incidence of radiographic fungus (consuming gamma radiation as a food source)ā¦ more towards the billions of years (~4B) than millions. XP
(Though there are still fungal species with the phototrophic [via melanin] properties, notably found around nuclear meltdown scenes such as the Elephant's Foot from Chernobyl.)
Good old fungus, breaking down everything. (The pressure exerted by fungal cells as they invade neighbouring spaces is intense. Enough to punch through steel.)
True or 8x when the dinosaurs were around. Still, it was Thorium/Radium I was thinking of, on the order of thousands of years. Thorium 65million years ago...2^40625 more prevalent than today, unless my math's wrong. Or, you know, there's other factors involved *shrug*
It's a bit complicated because radioactive elements decay into one another.
I wouldn't be surrpised if the thorium we had wasn't primarily the remnant of some ancient huge thorium repository, but rather the newer product of radioactive decay chains starting from slower-decaying materials
> 232Th is a primordial nuclide, having existed in its current form for over ten billion years; it was forged in the cores of dying stars through the r-process and scattered across the galaxy by supernovae and neutron star mergers.
so maybe not
> In the universe, thorium is among the rarest of the primordial elements, because it is one of the two elements that can be produced only in the r-process (the other being uranium), and also because it has slowly been decaying away from the moment it formed.
other thorium isotopes seem different though
> Natural thorium is usually almost pure 232Th, which is the longest-lived and most stable isotope of thorium, having a half-life comparable to the age of the universe.
gotcha!
> The other natural thorium isotopes are much shorter-lived; of them, only 230Th is usually detectable, occurring in secular equilibrium with its parent 238U, and making up at most 0.04% of natural thorium.
"secular equilibrium" is the steady-state reached by a pair of a slow-decaying isotope and its fast-decaying child, if I recall correctly
I was taught some of this, but it wasn't yesterday and it was never along my main interests
@AndrasDeak This information, while genuinely fascinating, will have little impact on my intention to declare at Bible study tonight that the biblical Nephilim were actually Marvel superheros.
I'm just saying that the Earth wasn't a ball of plutonium 1Gyrs ago :P
and you might want to consider the fact that being subjected to ionizing radiation will typically kill people rather than turning them into arthropod-mammal hybrids ;)
I'm reading a pickled file, and decoding each line of it; but I am not sure what to do with the characters which then appear at the beginning of the resulted string. On Windows I'd just replace the standard newline /r/n but not sure what to do on Linux. The resulted string has some junk characters at the beginning. Any suggestions?
@AndrasDeak And you make a fair, if not quoteworthy, point. I too was taught some of that physics, though likely not to the same level you. That said, 'typically' - sounds like there's a risk-reward equation there worthy of consideration.
from pathlib import Path
class SmallDemonstration:
def __init__(self, path):
self.file_path = path
def to_json(self):
data = []
with open(str(self.file_path), 'rb') as f:
for line in f.readlines():
value = self.byte_to_string(line)
if value != '':
value = value.strip()
data.append(value)
return data
@staticmethod
def byte_to_string(input_bytes):
try:
return input_bytes.decode("utf-8", errors="ignore")
perhaps an mvce of that portion too? The key thing to realise is that if you want to avoid encoding decoding errors, you need the knowledge of both sides. the encodings must be consistent at both ends.
usually, if you're trying to fix issues at unpickling, or "reading" and seeing issues crop up with encodings, you're already one step late to where the problem actually happened.
And we probably don't need the streaming part, just the IO-relevant bits. But that's less of an issue.
@ParitoshSingh indeed. I just saw a post by deceze explaining encoding issues, I didn't read it all but it seemed informative enough kunststube.net/encoding
for me, it was ned's unipain slides that finally helped make some sense of what all these issues were all about
the funny thing is, once it all clicks, its pretty simple to follow along with. But i still remember when facing encoding errors, it was chaos to figure out anything. Mostly because it was the "wrong end" of the problem, so to speak
no it's the same file. Afterwards it'll be a very large file and I'll have some data for my analysis. I have a chunk utility and I'll chuck this file into smaller ones (like the one I sent you)
I know it's the same file. The question is whether there's 1 big pickle or many smaller pickles inside.
Anyway, this sounds like a very bad approach. At best you should keep your pickles separate and pickle.load them separately. Even better: use a database or similar data structure that lets you append to it and also retrieve part of the data reliably. Hacking along a pile of pickles with a custom parser sounds like a recipe for disaster.
It would probably even be less work to manually parse json, and that would occupy roughly the same space and be safe. Still, doing the same thing with json would be equally wrong (multiple objects stuffed inside a json aren't valid json.)
Morning folks...I'm looking to make pie charts for each column in a dataframe. The charts will show percent NULLS (50% NULL and 50% non-NULL, e.g.). Can someone point me in a direction for an efficient way to do this? Currently, I am filtering each column and then making a single pie chart, but this feels cumbersome.
Maybe there is an easier way, without filter, to generate these charts per column in one graphic?
I am woefully ignorant about visualization in Python. I assume you are using matplotlib, right? I only know that this is a thing. I don't know how to use it.
My dad asking me about using git and this being something I know more about than him, and can actually advise him in, is a pretty surreal moment (he's an electrician whose coding exp originated via using PCBs and microcontrollers in the 70/80s, and writing some C++ in Delphi in the 90s).
@AndrasDeak Well he's not really programming, he's building an online learning platform using some software...and that software wants him to set up git so that instead of completely reinstalling the software for updates he can just rebase his copy of the core code with it's config and plugins and the changes will be updated...I'm not sure how much I trust that to be seamless, but I suppose it's workable.
ā¦ I also typically let Git work out if rebasing is the best approach when pulling. Nobody wants to be this guy:
3
@toonarmycaptain The root of my "dedicated server" VMs is itself git-managed, generally absorbing some utilities from /usr/bin, most of /etc, and so forth. Post-receive hook to automatically identify all packages owning all files modified, all init.d scripts for those packages, and automatically reloads/restarts where appropriate. (Branch to checkout on first boot passed by kernel command line.)
It's rarely bit me, and has never bit me in ways Git for source projects has (where I end up duplicating the entire tree prior to an operation I deem "risky"ā¦ Just To Be Safeā¢ā¦)
@amcgregor I see it potentially biting him in terms of him having to merge the changes in over his changes or changes by plugins he's installed, or updates being incompatible etc, but maybe he'll be fine.
@toonarmycaptain A graphical diff tool (e.g. FileMerge on Mac, or even the official GitHub client) can go a long way towards helping resolve conflicts like that. Seeing sprays of <<<<<<<<<< and ========== and >>>>>>>>>>> all over the place can be intimidating. (Much easier to visually split between 'ours' and 'theirs' to pick winning chunks.)
Absolutely. Though GitHub does have some pretty impressive content-aware diffing. (E.g. it can highlight what part of an image changed.) Not sure if they gifted that functionality to their desktop app, though.
(Only happens in the "arbitrary text" token handler and one utility, shouldn't be too hard to fix.) Question, though, does next(generator) itself still raise StopIteration in the return-early case? (I suppose it would.)
@amcgregor Which is all well on good on my machine, not so good if he's only got access to a shell on a server somewhere, or if he can't install git on his hosting provider's server? We'll see.
@toonarmycaptain That's where you bring GitHub Pull Requests into play. At my work we've done the occasional PR develop→master deployment; handy to also integrate into the review cycle.
@amcgregor Oh you're saying have his repo on github, and then just reclone the repo after pulling the changes in there? I didn't think of that. Of course that might mean having 3rd party plugins also in that remote repo...
@toonarmycaptain Nope, not quite what I was suggesting, though server-side scrapping and re-cloning is totally how my app workers are built. (With a pip cache mounted between each build node.) I was suggesting issuing "updates from the upstream forked repo" as pull requests, basically, avoiding the need to manage merges locally where possible.
(Merge conflicts would need more manual intervention, of course.)
@vaultah Yikes, okay, to clarify (since I appear to have this caseā¦) generator raising StopIteration bad, what about the iterator protocol next method? I'm hoping still safe/sane thereā¦
Assuming he's got git, unless there's a merge conflict, there shouldn't be an issue. I don't understand what "issuing "updates from the upstream forked repo" as pull requests," would look like if what he's done is clone the repo to the server and modified it, without managing his changes etc in a github account. Here's the section from the manual: https://paste.ofcode.org/svPkFkvdNe9WxAXfvnZgR
NB I've never really used git aside from via desktop/pycharm/web, so I've not really executed any commands in git manually before. Working through the git tutorial is on my lengthy TODO list ;)
Visit repo on GitHub, PR tab, New, selecting the upstream repo originally forked as the origin, the fork as the target, submit. Accept PR, squash+merge (or just merge), you're updated.
Sorry, I meant remote as in the repo on the webserver, not the origin he'll clone it from on github. I still don't get how I can manage a PR on github unless I have a repo on github, rather than simply cloning the original repo on the server.
@toonarmycaptain A ha. Indeed, I'm assuming use of GitHub as a "canonical store for source code" and broker of automation around that code, e.g. your server would know to "pull" and update itself due to a push hook registered in GitHub against that repository.
ā¦ I'm clearly too spoiled by my server automation these days. ¬_¬
Oh yea, that's what I meant earlier when I was talking about having his repo on github, then recloning his webserver copy from his repo on github (I guess updating or rebasing are better terms?) with changes. Still doesn't fill me with confidence, if he's got to install plugins which change the code on his local machine, unless he has them in his repo too. Like the link I shared said, the manual for the software he's using makes it sound like you update your installation in place by running git pull origin and then running their software's upgrade applet.
@amcgregor Well, you're running a server, rather than paying for some hosting ;)
@toonarmycaptain In some cases (VMs on Rackspace, so still not bare-metal despite 3+ year uptimes). In a majority of cases, I'm utilizing heroku-like build and cloud infrastructure with native GitHub integration.
Average CMS application cost for my CMS hosted there: 0.5ā¬/day.
Yes, but my issue is that the docs mention that to do a scatter plot I must have an x and a y series. however I only want to do a 1-dimension scatter plot. pandas.pydata.org/pandas-docs/stable/reference/api/…
cbg, with python sqlite3 do you actually want to just use the execute command and input the raw SQL or is that only suppose to be used as a last resort?
That plot sounds a bit specific so I wouldn't be surprised if there weren't a built-in solution for it (it definitely isn't a thing in matplotlib). I'm pretty sure you could convert (melt? something else?) that dataframe into another one, one that has an integer column encoding the original columns, a second column encoding the colour, and a third one with the values. Then you could feed that into other_df.plot.scatter
I think that's the way to go writing raw statements can get tedious and messy fast. with psycopg2 it is easy to define the database model as python class and then build out from there. psycopg2 handles the connections, pooling, and data manipulations for you.
Don't bet on it: the coordinates that correspond to "rockA", "rockB" etc are completely arbitrary. Look at a barplot instead, that might suit your needs.
I try not to let it show just how frustrated I am at seeing the 3 thousandth poorly researched question that day, with various varying degrees of success