« first day (3121 days earlier)      last day (1810 days later) » 

12:01 AM
cbg
 
12:31 AM
rbrb
 
 
2 hours later…
wim
2:48 AM
@Arne ugh.
@Aran-Fey it could have been a good place for a packaging canonical, but it has already accepted answer after 13 hours (before I could get to it). sadly, the stuff in the accepted answer is already too dated - as Arne mentioned, setup.py should not really be recommended these days (it's replaced by pyproject.toml) and even requirements.txt should not really be recommended (it's replaced by build system's lockfile).
I didn't DV answers because that stuff does all still work currently, but in new projects we should be looking to move away from that and onto the modern tools to break the hard dependency on distutils (which is really ancient and limiting)
also, that setup.py links to Kenneth Reitz's cargo culting instead of to the setuptools guide, which makes me sad.
 
cbg
 
wim
3:19 AM
@AnttiHaapala You should not parse it from the filename. You should get it from the package metadata (which is inside the file). And don't reinvent the wheel (pardon the pun) - just use pkginfo to get it.
@AnttiHaapala yes, and not all indices canonicalize the filename correctly. so, don't use the filename.
 
wim
3:34 AM
@Code-Apprentice yeah, I see that. unhappy robinhood landing page.
@AnttiHaapala glad to see someone else say it. the query is for the session, not for the model!
@AnttiHaapala that kind of goes for flask in general. for a good design in a web framework you can basically say, ok, let's look for what flask does here - and then do the opposite.
@roganjosh sadly no. packaging world is almost entirely third-party, stdlib is only distutils which even pip does not use these days (pip uses a vendored setuptools which monkeypatches stdlib distutils). and pip is not stdlib, actually doesn't even have an api.
 
3:52 AM
rbrb
Nothing interesting again...
 
@wim yea I know :P
 
wim
I see beginner questions very frequently where the symptom is import errors, and the problem is usually bad package structure or dependency specification. The answers attracted are almost always bad (fixing the symptoms with some hack or side-effect of the import system, or misunderstanding virtualenv, or trashing the system python somehow)
@WayneWerner looks kinda like doom or wolf3d
 
 
2 hours later…
6:16 AM
@roganjosh I ended up not writing one, the answers there aren't necessarily wrong wrong. Just a bit unnerving if you have spent some time on the subject.
It's also really hard to draw the line between best practice and opinion, since packaging is in an evolving phase right now. So maybe it's better to say nothing at all instead of something that will be outdated in 6 months.
@wim since we're talking about packaging, do you know the answer to this question by chance?
 
6:52 AM
@wim I thought it was the old Windows screensaver
 
7:51 AM
import numpy as np
arr = np.array([
    ('b0263', 'a', 1, 2, 3),
    ('26ab0', 'b', 4, 5, 6),
    ('6d7fc', 'c', 7, 8, 9),
    ('48a24', 'd', 0, 1, 2),
    ('1dcca', 'e', 3, 4, 5)],
    dtype="S5, c, i4, i4, i4")
type(arr[0])
Out[27]: numpy.void
does this offer any advantages at all? why would someone want to do this?
First time encountering the void datatype. i didnt even know this was possible in numpy
does it allow vectorization? it seems to not allow arr[:, 0] notation
 
Um... don't recall seeing that one either... Its doc string does mention: "Abstract base class of all scalar types without predefined length."
 
8:05 AM
yeah no, unless i find out otherwise, ive concluded this is just not useful at all* (for direct use, i suppose internally numpy probably builds on it for all other datatypes we need to use or whatnot)
 
@ParitoshSingh inhomogeneity, yet compact storage. And can facilitate reading from binary files
That being said I've never needed structured arrays
note that arr[0].dtype is probably more informative
(Just a hunch)
 
i would assume every instance of this datatype can be substituted for a dataframe without much issues and better control
 
Could be, but that's not numpy's concern and this might predate pandas
 
oops sorry internet issues
dtype([('f0', 'S5'), ('f1', 'S1'), ('f2', '<i4'), ('f3', '<i4'), ('f4', '<i4')])
 
Better
Here's the docs if you're curious docs.scipy.org/doc/numpy/user/basics.rec.html
 
8:16 AM
oh, that rings a bell now
vaguely remembering a struct datatype in C? or C++?
 
(in case you haven't seen this before)
 
user7437554
population=[human1, human2,...humanx]
for human in population:
  write_file('human_i.txt')
 
user7437554
Hi guys, is there any way to write a filename (human_i) where the name includes information on the position in a list?
 
@santimirandarp like this?
 for i, human in enumerate(population):
    write_file(f'human_{i}.txt')
 
Structured datatypes are designed to be able to mimic ‘structs’ in the C language, and share a similar memory layout. They are meant for interfacing with C code and for low-level manipulation of structured buffers, for example for interpreting binary blobs. For these purposes they support specialized features such as subarrays, nested datatypes, and unions, and allow control over the memory layout of the structure.

Users looking to manipulate tabular data, such as stored in csv files, may find other pydata projects more suitable, such as xarray, pandas, or DataArray.
yep, that just about perfectly answers everything i had in mind. Thanks for the link!
 
user7437554
8:35 AM
@PeterVaro yes, thanks ..=)
 
user7437554
Umhmm its not working in my code
 
user7437554
Doesnt it need something like .format() ? @PeterVaro
 
@PeterVaro Haven't seen you in a bit mate - how you doing?
 
user7437554
8:54 AM
This is the original, just in case some of you want to help me to rewrite it
 
user7437554
 
user7437554
(the code works)
 
@santimirandarp maybe you're on a python version that doesn't support f-strings yet
the equivalent code for f'human_{i}.txt' that works on python < 3.6 would be 'human_{}.txt'.format(i)
The snippet you posted uses string interpolation instead, which is yet another way to format strings that should be avoided in favor of .format() or fstrings
 
user7437554
@Arne thanks for the explanation. I'm switching from python 2.7 to 3.6
 
yes, please
 
user7437554
9:01 AM
this code is executed in 3.6 though
 
then fstrings should work. what's the error message?
 
user7437554
I supposed the f was a typo, I guess that's why (have never seen that way of writing)
 
user7437554
But I'm not sure how to re format the whole snippet
 
user7437554
  for gene in population:
    mol=prepare_molecule(gene)
    Chem.SDWriter('mol_%i.sdf'%(population.index(gene))).write(mol)

  for i,gene in enumerate(population):
    mol=prepare_molecule(gene)
    Chem.SDWriter(f'mol_{i}.sdf')).write(mol)
 
user7437554
Something like that?
 
9:05 AM
f'mol_{population.index(gene)}.sdf'
you can write any code in the squiggly braces of an fstring
 
user7437554
amazing
 
user7437554
thanks man
 
by default it will try to call its __str__ use put that instead.
no problem, happy to help
fstrings are pretty neat, in my opinion the best new feature in recent time
 
9:23 AM
@santimirandarp The new f-strings basically use the same Format Specification Mini-Language that the format function and .format method use. You don't need to memorize all that stuff, but it's a good idea to get familiar with the main features.
 
9:33 AM
@Arne they're nice... some people do seem to try and shove too much into 'em though
 
True. I draw the line at what can go in the squigglies at any of
- more than 60 characters total
- contains a comprehension
- nested fstring (also I think this one doesn't even work)
correction: nested fstrings do work
f'{f"{list}"}'
"<class 'list'>"
but.. should they?
 
9:55 AM
@Arne Maybe they shouldn't. It could get very messy.
OTOH, nested replacement fields are useful, so you can dynamically set field width and precision. But they only permit a single level of nesting.I can't think of a reason why you'd want deeper nesting for that, though.
 
puzzle: Is it possible to get more than two levels of nested fstrings?
 
triple-quotes and probably escaped quotes
ah, no backslashes
>>> f'''{f"""{f'{f"{list}"}'}"""}'''
"<class 'list'>"
 
4 seems to be the hard ceiling for now
currently trying shenanigans with eval...
>>> inner = 'f"{list}"'
>>> f'''{f"""{f"{f'{eval(inner)}'}"}"""}'''
"<class 'list'>"
 
mine eyes... they bleedeth! :)
 
what was the old wisdom again.. shoot yourself with small bullets every day to build up an immunity against bigger ones?
 
10:09 AM
cbg
 
cbg
 
So, the good news for today is that I finally found out what causes us all to get logged out of my site randomly even when active. The bad news is that I seem to be some passive victim that can only watch on while Flask-Session goes nuts :/
 
> I supposed the f was a typo,
:|
 
It just keeps pumping out tmpztxw24dk.__wz_cache files every 5 seconds or so that I can't even delete myself, until the directory reaches the session limit and everything gets wiped, including the valid session files
 
This resonates with me because last night an asker told me "no, you are overcomplicating this, this is really easy to do with for loops". They were asking for a vectorized solution and they wouldn't even show their loop.
 
10:13 AM
@Arne sounds like an even stranger variation of building up tolerance to arsenic or something :p
 
after a while your code nose gets accustomed to the stench and you no longer feel the code smell
 
user7437554
@PM2Ring Thanks :)
 
10:28 AM
how do i upcak in python 2.7 a,b,*c=[,1,2,4,5,6]; such that a=1,b=2,c=4,5,6
 
Is there anything special in the .__wz_cache file extension that anyone is aware of? Some kind of name mangling?
I can't delete the files, either in Python or manually, because Linux insists that these files don't exist, even though I'm clicking on them and something really does clear them away when the directory hits a certain threshold
 
"Linux insists that these files don't exist" sounds highly suspicious
Symlinks? Permissions?
 
Github search doesn't find it, even though I know where it is
I have a gif of the behaviour
Or perhaps I'm not using github search properly
 
github search is poop, it can't search for code...
 
10:36 AM
I tried hosting the gif on giphy but the quality is awful. Will SO allow gif uploads of 1/2MG?
 
use proper git with find and grep
@roganjosh MG?
the limit is either 2 or 5 megabytes
 
MB, oops
Am I ok to upload here? I'm not sure how big it will look, but you won't be able to see anything with my giphy link
 
you can edit the message afterward if the one-box is too bad
 
You said it's linux, right?
 
10:39 AM
Yeah, Debian I think
 
so why not forget about the silly windows and see what's really going on in a terminal? :P
Also, what issue is this causing again, what are you trying to solve? That other things also get deleted along with the cache?
because if so then the X in the XY problem is probably that the flask session directory is being used as the cache directory
 
So I have a file-based Flask session. Something keeps pumping those nonsense files out (there are legitimate session files). When the directory hits a threshold of 500 of those files, it wipes the directory, including the genuine session files, and all of us get logged out
The genuine cache files are at the top of the directory. They have names like "919b410b1fce956a13cda9246ce0fc34"
 
what are "genuine cache files"?
 
Files that hold actual session data for logged-in users
 
yeah, OK, so that's exactly what I said
 
10:44 AM
Umm, I'm not following then
 
...have you read all my recent messages?
I mean, isn't the FileSystemCache you linked a different story from the session cache you want to keep?
 
Here is a problem. A sequence of 1s and 0s is given. When I have hit a 0 and then a 1, I need to print 1 0 in that positions in a different place and copy that list/string to the first one and again repeat this process a given number of times. I have written this code, but it's showing TLE
t = int(input())
for _ in range(t):
    ar = list(map(int,input().split()))
    n,z = ar[0],ar[1]
    a = list(map(int,input().split()))
    aa = ' '.join(str(x) for x in a)
    #print(aa)
    while(z):
        aa = aa.replace('0 1','1 0')
        #print(aa)
        z-=1
    print(aa)
 
Yeah, but I'm looking through my config and there is nothing that should make Flask cache anything. I don't import a cache, nor do I use it anywhere, and it would be bizarre that it suddenly decides to cache in a directory that has an arbitrary name that I chose
 
Can you try actually assigning a separate directory for that cache?
@AshisGhosh TLE?
 
Time Limit Exceeded
 
10:49 AM
ah, another coding challenge site
 
Err, it doesn't look like Flask has any parameters for cache and I don't even have Flask-Cache installed
 
Is that thing part of flask-cache?
can you grep flask_session in whatever config stuff you may have?
@AshisGhosh when you put back the print in the loop, does it print what you expect it to print in each iteration?
 
@AndrasDeak ok so I need to store it? will not that result in Memory Limit exceed?
 
It'd probably help to see the actual text of that challenge... you seem to be doing a lot of splitting and onverting to ints only to join it back to a string again... and that while(z) is a very inefficient way of doing replacements...
 
@AshisGhosh I'm talking about debugging your own code locally.....
 
10:53 AM
Length of every sequence can be 10^6 order
 
I'm getting somewhere. So it is Flask-Session [github.com/fengsp/flask-session/blob/master/flask_session/…
 
@AshisGhosh just like an other problem, you need an MCVE
@roganjosh I would expect the session part not to self-destruct using the cache...
 
As would I :P
 
How many cached files come from the session? Perhaps the bug is that all those _wz_cache files are being created.
if there should only be a dozen session cache files at most then the 500 limit should be more than enough
 
There are ~5 genuine files, as in, representing actual people. Seemingly nobody needs to even be doing anything for all the __wz_cache files to spring up
 
10:57 AM
:param threshold: the maximum number of items the session stores before it
                  starts deleting some.
this suggests that it should be possible that they don't get obliterated all at once, by the way
 
Yup, agreed
 
How low-level is this part of flask-session? Can you track down how this class and the corresponding cache gets instantiated?
 
I'm trying to look through Werkzeug there, but I won't necessarily know what is the main caller here, because presumably it's something to do with gunicorn that's causing this rhythmic creation of files. It's not user activity, and there are no calls coming from the front-end that are automated
 
doesn't sound too encouraging
 
Anyway, got to go for a little. I'll have to have a longer think. In the meantime, I'm more curious about how I can't even delete the files myself, yet something can
 
11:01 AM
for that you should look at a terminal
see what ls -la and stuff gives you
 
Here is another code, it's showing NZEC. Though it's running for given test cases. Can anyone tell me where the possible error occurred? I'm new to the online judge
https://ide.geeksforgeeks.org/69Mc9TDeg9
 
when reading in a pd.read_csv is there an option to ignore any columns with an empty header?
 
@ThelurkerLurker you might be able to get away with using usecols= with a callable... something like: df = pd.read_csv('somefile.csv', usecols=lambda col: bool(col.strip()))...
 
thanks @JonClements will give it a try
 
11:11 AM
Hi! I'm new to python and stuck trying to extract some information from a list of dicts. Normally I'd use numpy and pandas but I'm trying to do this with native python. I'm trying to get the most frequent product as well as sum the prices for each. I've tried to use itertools and groupby but I get stuck because it tries to sum a string. Here is a small example of the list:
[{"product":"banana", "price":1.00},
{"product":"apple", "price":2.00},
{"product":"orange", "price":1.35}]
 
if memory is not an issue I'd use a defaultdict of lists, with items like {'banana': [1.00]}
then as you encounter the same fruit again you append the corresponding price to that value, and in the very end you can find the longest list and compute the sum
 
I've been trying to do something like that but I'm not familiar with defaultdict and haven't been able to get it to work. (normally I'd just read this as df and be done) so I'm completely out of what I normally do.
 
that's when you get to learn ;)
collections.defaultdict takes a callable (typically a type) as its argument, and the result is a defaultdict which uses the corresponding type to initialize values when a key is being referred to without having been defined beforehand
>>> from collections import defaultdict
>>> dct = {}  # usual dict
>>> dct['k']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'k'
>>> ddct = defaultdict(list)  # defaultdict, with a list default value
>>> ddct['k']
[]
>>> ddct
defaultdict(<class 'list'>, {'k': []})
For all intents and purposes it's like a dict that makes sure you can't access invalid keys. You could do the same thing with a regular dict and some boilerplate for handling missing keys.
 
I don't see the point, if you know what the keys are (and thus none are invalid)? (beside the issue, but appropriate to the topic)
 
for me to know what the keys are I'd have to iterate your list of dicts twice
 
11:21 AM
I don't follow, why twice?
 
Anyway, if this is all new to you I suggest implementing it first with a regular dict. Initialize a dict, loop over your list of dicts, for each product check if the given fruit is already in the dict, and create a new list for the given fruit if it's new (otherwise append the new price to the existing ones).
@JasonWilcox nevermind, this is a bit too handwavy and vague and probably not worth the effort to clarify now
 
11:34 AM
im having some memory trouble and im trying to use chunksize to help with this - but when i include it im getting an error - im not sure what its doing but it sounds like its turning my dataframe in to a TextReader and I am unable to include a new column
for f in all_files:
    frame = pd.read_csv(f, skiprows=[0,1],low_memory=False,chunksize=20000)
    frame.insert(0,'filename',os.path.basename(f))
    datalist.append(frame)

data = pd.concat(datalist, ignore_index=True)
im getting an error on insert
 
If all you're doing is concating all the chunks together at the end - you're not going to avoid memory trouble...
In fact - you're compounding the issue - as you'll have all the data in chunks in a list and then try to create a dataframe object on top... so you'd have more than doubled memory requirements...
 
o no!
ok i actually located the source of my memory troubles
data = data[data.columns.drop(list(data.filter(regex='Unnamed')))]
 
errr... interesting use of list and filter there :)
@ThelurkerLurker did the usecols= I suggested not work?
 
no unfortunately not
 
@JasonWilcox please see the pinned message on the right (if you're on a desktop browser) for formatting multiline code in chat
 
11:46 AM
grouped = {}

for product in fruits:
    name = product['product']
    price = product['price']

    if name not in grouped:
        grouped[name] = {name: 1, 'price':price}
    else:
        grouped[name] = {name: 1, 'price': price}
like that?
 
You can either do what I suggested and keep a list of every price (which naturally leads you to the count and sum after the loop), or you can keep two variables (the count and the subtotal) separately in any way you like. Either way in the else: block you have to use the existing value of the dict for grouped[name]...
@JasonWilcox if you mean formatting, then yeah
 
my issue isn't lack of comprehension of the method to get what I need, its implementing the code
 
I'm not sure how the two can be separate
You can only implement a clear-cut algorithm you already have in your head. If there's any vagueness you won't know what to implement.
 
Do you know how to operate and run a forklift? (probably not, since many people have never done it), but if you wanted to move something I'm sure you'd understand to move the forklift to the palette, lower the arms, move forward, and then raise the arms to lift the palette.
Same thing with the code. I understand I need to aggregate by product name and sum the prices, as well as count the entries of each product (both of which could easily be done in a df or sql) but I'm not familiar enough with the specific code using basic python to do that, and my attempts in the last 6 hours have been rather fruitless.
 
What kind of answer are you looking for then?
 
11:57 AM
Can anyone give any idea about finding factorial of a number in O(n*log n) ?
 
I'm trying to explain how to raise the arms to lift the pallet but you're insisting you already know how to drive
 
even dropping 1 column give me memory issues with
data = data.drop('Unnamed: 101',axis=1)
 
@ThelurkerLurker how much of your memory does the dataframe occupy?
Around or more than half?
 
how do I check that?
 
Umm, there are probably programmatic ways out there on the internet, or you can look at some sytem monitor for a rough guess (assuming your df is the only large object that eats memory)
 
11:59 AM
hello
 
the reason why I asked because before data.drop(...) gets reassigned to the new name you have two dataframes: the original one and the dataframe returned from data.drop. So that's necessarily double the memory. Plus if there are other references to data then the original dataframe won't get deleted but it will stick around in the background, so to speak.
@Andy hello
 
im not sure im using jupyter notebook - its sitting at
36% total memory
 
hmm, then there's probably overhead from data.drop
 
let me try restart my notebook and try again
 
is it possible to construct the "rest of the columns" and constructively select those from data?
(when I say "possible" I mean "feasible")
 
12:04 PM
that also gives memory error
 
selecting the other columns?
 
newdf = df[['a','b'...]]
 
@ThelurkerLurker right... well... how did you use it - what column names are you actually getting... You said "blank", but it appears pandas/openpyxl might read it as "Unnamed..." going by what you've shown in your attempted drop statement
 
Ugh, this file issue is nuts. I can open the file and view its contents - nothing, but it tells me it doesn't exist when I try to delete it and the only code I can find that actually cares about the extension could only be using os.remove() and bang, the whole contents get obliterated... but I can't use os.remove(). The situation seems genuinely impossible :(
 
@ThelurkerLurker weird, I would expect that to give you 2x memory at worst, and if you have 36% now it shouldn't be more than seventy-few...
 
12:05 PM
So maybe not pd.read_csv('blah.csv', usecols=lambda col: 'Unnamed' not in col)...
 
perhaps there's some ulimit-like memory limit that applies to the notebook
@roganjosh when you try to delete it from a terminal?
 
@ThelurkerLurker what does df.columns give you as it is now?
 
Let me try
 
didn't you even check ls -l as I suggested?
 
this is my full list:
array(['Age @ Bord', 'Age @ Loan', 'CapitalAmount',
       'ClientDepartmentAndUnit', 'ClientEmail', 'ClientEmployeeNumber',
       'ClientFirstNames', 'ClientID', 'ClientPrevEmployeeNumber',
       'ClientRank', 'ClientSurname', 'CurrentStatusReason',
       'Date of Birth', 'Difference Expected vs Collected',
       'Difference as % of Loan Amount',
       'Difference between Loan Amount & Insured Amount', 'EmployerID',
       'EmployerIndustryTrade', 'EmployerName', 'EmployerType',
       'ExcRecDataReceiptAmount', 'ExcRecDataStatus', 'Gender',
 
12:07 PM
rm tmp3a3h6tfo.__wz_cache doesn't work, ls - l lists only the legitimate session files
 
ls -la?
it should also not list them then
 
Same, just the legit files
 
hmmmmm
what the yam
Are you inside the directory? Asking due to possible directory permissions
 
yep
 
And then you have rx permissions for the dir, right?
 
12:09 PM
I should have all permissions. I could chmod 777 and try?
 
@ThelurkerLurker wowsers... lots of blanks... if you don't need 'em, then use: pd.read_csv('blah.csv', usecols=lambda col: 'Unnamed:' not in col) on reading and see if that cuts out the cruft at the offset...
 
@roganjosh nonono
 
hahahah... the old... "this is getting really annoying now... let's just chmod -R 777 the_entire_bloomin_directory and I'll worry about it later" approach :p
 
yup :P
 
I'm literally at that point :/
 
12:10 PM
you could try a sudo ls to be sure
that's non-destructive
 
nope, doesn't see them
 
OK, that's what I expected. I've got nothing.
 
At this point my options are either to whack up the threshold, which is ugly at best, or completely change my session setup
 
or...figure out where the phantom files are coming from and where they are? :P
 
They are, apparently, empty and 0 bytes, but then I have no idea if anything is true at this point
 
12:13 PM
you could try on chat.SE for something linux-relevant, see if they have hints what and where such files might be
 
I might try that. In the short term I have to leave, so I'm just going to up the threshold because it seems to be getting faster. IT is going to monitor the network traffic to see if anything might be causing my server issues (but I can't see anything)
 
in before filesystem problems :D
you can use up the number of available inodes in a directory with too many files
you'll start seeing a rise in the size of the (nigh-)empty directory
U&L chat seems active
 
Well the other part to this is that I don't see how any of the python code could be doing the obliteration (it's written to not do that and even if it wanted to, it doesn't seem it would have the tools to do it). I haven't actually counted whether the obliteration coincides with the 500 threshold or whether it's just happening when there are "a lot"
And I won't have chance now until next week. Oh well, cheery note to leave on :)
 
@roganjosh Why not? The cache implementation you linked had a deletion method, and it only spared "management elements" or whatevers
 
Yeah, os.remove(), which doesn't work for me
 
12:18 PM
because the files don't seem to exist
until you figure out where and how those files are you can't make educated guesses about the code that deletes them
it's a very sound guess that the thing deleting it is the same thing that creates it, the cache thingie
 
Maybe the file is somehow locked to the process that creates it, so the code really is seeing the 500 limit, but somehow nobody else really can
 
When you drop a row from a Pandas Dataframe is it still held in memory somewhere hidden?
 
@roganjosh I didn't know that was possible, which doesn't mean it's not
 
Anyway, thanks for your suggestions, Andras
 
no worries, too bad I can't be of real help
you can take a look at the mktmpwhatever call that creates these files in the cache thingie...
@Andy df.drop will return a new dataframe by default, so the old object is still there as long as there are references to it. If you pass inplace=True the original dataframe gets modified (mutated). If you don't pass inplace=True but rebind the old name to the new dataframe then the old dataframe might be destroyed if there were no other references pointing to it.
 
12:23 PM
@Andras ok thanks, so if I am using inplace=True then the row is dropped and destroyed
 
I would think so, though I'm not a pandas user so I can't be sure.
 
I always worry with Pandas dataframes that i am copying or duplicating data unwittingly
 
that can easily happen, because most methods return copies, but then again most methods support the inplace kwarg for exactly this reason
Python's reference counting means that you often don't have to care about this. If your dataframe is only 100 MB large then df = df.drop(...) vs df.drop(..., inplace=True) makes no difference.
 
ok great,
 
(assuming there are no other references to df of course)
 
12:27 PM
yes this is where it gets confusing :S
 
references aren't that sneaky usually
 
@JonClements @john Thank you this worked! :)
 
df = pd.read_csv(...)
df_orig = df  # new reference to the same df
df.drop(..., inplace=True)  # the original dataframe is mutated, df_orig and df are still the same object
df = df.drop(..., inplace=False)  # df now refers to a new dataframe, df_orig refers to the old one, they are independent hereafter
 
ah ok clear
but also confusing in practice
thanks
 
no problem
 
12:30 PM
seems with Pandas its very easy to screw this up
 
also references can be created by, say, putting your dataframe in a list or a dict, but that rarely happens with dataframes
@Andy it's more like that you should have a good understanding of native python first, then learn the quirks of pandas. Conflating the two will be confusing.
this is true for most third-party frameworks
 
ye i just posted a question where I was using a dataframe where one column has an arbitrary object and the other columns data to be passed in that obj as argyments
and i had some helpful comments from @JonClements which are making me rethink it
 
@Andras ok cool
I am still relatively new to Pandas and Python
so good to know it can just be a bit confusing and will come with time
 
You'll get to grips with it soon ;) Here's the go-to reference about references and mutation: nedbatchelder.com/text/names.html
that's pretty important to understand this section of native python, so I strongly suggest giving it a good read
 
12:35 PM
@AndrasDeak super helpful
exactly sort of article i need
 
glad to hear that
rhubarb for a while
 
@AndrasDeak what does df default if not specified?
inplace
 
\o cbg
 
@ThelurkerLurker To make something inplace - you have to explicitly set it to True if supported... the default is to copy otherwise
 
m8_
12:51 PM
@roganjosh, I went with another implementation since my list was small. Not sure if you gave up on trying or not, but I'm good to go
 
1:05 PM
@AndrasDeak that article was fantastic
really helpful
 
@m8_ oh, the last I remembered was that you said you were changing the data, was waiting till you replied sorry and then had to go. Glad it's sorted though
 
1:19 PM
@JasonWilcox umm... You might want to pad that question out with the things you've been given and tried here... I thought Andras had already given you an answer or a really good starting block?
('cos right now - I can see that getting downvoted as it's really not clear what you don't get how to apply... plus we've already chatted about it here?)
 
m8_
ah gotcha, no worries. Thanks anyways!
 
@JonClements apparently I'm no forklift driving instructor
 
1:35 PM
Today I am annoyed by questions whose input and output don't match. "I have a list which is something like [1,2,3,4], and I want to find the sum, which is something like 12"
 
those are the best because you can close them as both unclear and no MCVE
 
@Kevin it's something like 12 give or take a bit :p (If they work for the government - that's pretty accurate enough :p)
 
If you point out that 1+2+3+4 isn't 12, they might get snippy and say "I know that! I was just giving a general idea of the structure of the problem. You should be able to figure out the real result from context". But if you go ahead and answer the question with "try sum()", they'll say "That won't work, because I don't want the arithmetical sum, I want the quantum aleph rosenstein sum. You should have been able to figure that out because obviously sum([1,2,3,4]) doesn't return 12."
Today's perpretrator is Using a list of dicts to find most frequent and min/max of a sum, whose expected output's structure has no resemblance to the requirements stated in the question, and whose numbers literally cannot be derived from the input no matter what you guess the actual requirements to be
 
@Kevin who was in here talking about it - got some advice and then just asked the same thing they did here on main anyway :p
 
only after telling me that I'm answering their question wrong, because they know what to do, they merely don't know what to do
I asked what kind of answer they are looking for then; still waiting for an answer
from the question on main I suspect it's "a complete one I don't have to think about in any way"
 
1:42 PM
@JonClements things have never been better! how's yourself?
 
this is the second asker in two days telling me in completely unhelpful ways that I'm answering wrong
@PeterVaro haha, is your mental state getting worse? ;)
 
Peter's a trendsetter and self-replies will become the hot new thing of 2019
 
@AndrasDeak this happens when you (a human) try to do what a machine should..
@Kevin LOL
 
@Kevin capital observation old bean, insightful as always
 
gosh, isn't it annoying that the chat of SO hasn't been evolved a bit in the last 5 years?
 
1:44 PM
there was like, one feature that got implemented
 
(so frustrating to use it actually..)
 
You may as well get angry at the ocean for being salty
 
@Kevin how do you know I'm not?
 
@PeterVaro just the usual mate... :)
 
Hmm, true
Anyway, that's not a value judgement. Each of us is free to scream at the sea as much as we like.
Just don't expect Poseidon to return your call, saying "I thought about it, and you're right"
 
1:47 PM
no, because old Pos' is a jerk
 
that's exactly the main problem with these so called gods.. they are very unreliable, aren't they?
 
They're all about the "working in mysterious ways" aesthetic. I'd appreciate a ticket resolution system, myself.
 
@JonClements it is amazing, that we, old farts, are still around here, isn't it? reminds me of, I haven't see @Ffisegydd for ages -- what's happened to him?
 
@Kevin I'd hate to see Zeus' backlog
 
My last communication with Fizzy indicates that he's alive and in one piece
 
1:52 PM
that's good to know
 
@Peter he and Brian are off gallivanting around in the time machine... ;p
bbias
 
@AndrasDeak Quite a lot of child support claims I imagine
 
anyway.... "we, old farts"? Speak for yourself mate... I'm still a pup :p
 
pup farts are still lethal
 
1:53 PM
@JonClements I knew you are going to offended by it :P
 

« first day (3121 days earlier)      last day (1810 days later) »