8:43 AM
2
A: How count occurrences *across all columns* in a polars dataframe?

HericksTLDR. You can use value_counts after unpivoting the dataframe into a long format. df.melt().get_column("value").value_counts() Explanation Let us consider the following example dataframe. import polars as pl df = pl.DataFrame({ "col_1": [1, 2, 3], "col_2": [2, 3, 7], "col_3": [1, 1,...

 
@jqurious Indeed - I've edited the answer to rely on the default. Thanks!
 
Huge Thanks this is exactly what I was looking for! However I ran into an issue- with.value_counts it is counting the 1 in 1 but also the 1 in 10 and 2 1's in 11 etc. so my value count for 1 is artificially high- how could I account for that?
 
@user24556897 Do you want to count all 1‘s (even those in 10 and 21) in the same bucket? Could you provide us with a sample input along with the expected output to clarify on the issue?
 
@Hericks As an example: the data set is 10 columns of data, the values in range of 1 to 100, the melt worked exactly as I wanted to bring the 10 columns down to the 2 columns, being value and frequency. I want it to count the frequency of each number 1-100 in the dataframe- the issue has arisen for the values 1-9, it counts the frequency but currently it includes 10 as 1 and 11 would also count as 2 * 1. So my values in the range 1-9 are showing as extremely high frequency. Is it .unique() that I would need to fix this?
 
@user24556897 Sorry, maybe I am misunderstanding, but pl.Expr.value_counts won't count the digit 1 in the numbers 10 and 11 also to the digit 1. You can simply check as follows. pl.Series("numbers", [1, 10, 11]).value_counts(). The corresponding count is 1 for all numbers (1, 10, and 11). Especially, the count for the number 1 is not 4 and the counts in your dataframe should not be artificially inflated.
 
8:43 AM
@Hericks I'm confused- when I run the check- the count works but when I implement the code: counts = df.melt().get_column("value").value_counts() I am getting artificially high numbers- am I missing something? For instance I have 318 rows in my CSV but I am getting 378 frequency for the value 6, each row doesn't have repeat values so the highest possible frequency should be 318.
 
Hi @user24556897, could it just be the case that the value 6 appears in more than one columns? It could appear in more than one column.
You can also do a quick sanity check. df.with_columns((pl.all() == 6).sum()).sum() If this returns 378, there are 378 6s in your df.
 
No it definitely only appears in one column. I manually inputted the data- so unless something funky is happening in the dataframe but it shouldn't be-
I ran a sanity check but it spat out weird numbers: ┌──────┬──────┬──────┬─────┬─────┬─────┬─────┐
│ 1 ┆ 2 ┆ 3 ┆ 4 ┆ 5 ┆ 6 ┆ 7 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 │
╞══════╪══════╪══════╪═════╪═════╪═════╪═════╡
│ 7222 ┆ 7850 ┆ 4396 ┆ 628 ┆ 0 ┆ 0 ┆ 0 │
└──────┴──────┴──────┴─────┴─────┴─────┴─────┘
 
Sorry, the approach above double-counts. Can you try df.select((pl.all() == 6).sum()).sum()
Notice the .select instead of .with_columns.
df.select((pl.all() == 6).sum()).sum_horizontal()
 
8:59 AM
That produced the answer 83...
 
Can you share an example dataframe with a mismatch?
 
Do you mean the dataframe showing the high frequency?
shape: (35, 2)
┌───────┬───────┐
│ value ┆ count │
│ --- ┆ --- │
│ str ┆ u32 │
╞═══════╪═══════╡
│ 6 ┆ 378 │
│ 20 ┆ 67 │
│ 13 ┆ 57 │
│ 23 ┆ 66 │
│ 24 ┆ 63 │
│ … ┆ … │
│ 12 ┆ 60 │
│ 18 ┆ 62 │
│ 17 ┆ 77 │
│ 22 ┆ 63 │
│ 15 ┆ 54 │
└───────┴───────┘
 
No, ideally the original dataframe (or the head of it already product a mismatch).
So, I can verify.
 
Here is a chunk of it- I have subdivided off a couple columns
shape: (314, 7)
┌─────┬─────┬─────┬─────┬─────┬─────┬─────┐
│ 1 ┆ 2 ┆ 3 ┆ 4 ┆ 5 ┆ 6 ┆ 7 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╪═════╪═════╪═════╡
│ 9 ┆ 18 ┆ 22 ┆ 23 ┆ 31 ┆ 34 ┆ 35 │
│ 4 ┆ 6 ┆ 20 ┆ 21 ┆ 24 ┆ 25 ┆ 30 │
│ 1 ┆ 7 ┆ 10 ┆ 19 ┆ 20 ┆ 25 ┆ 31 │
│ 6 ┆ 9 ┆ 15 ┆ 18 ┆ 19 ┆ 29 ┆ 34 │
│ 3 ┆ 7 ┆ 10 ┆ 12 ┆ 13 ┆ 24 ┆ 25 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
Is there a way for me to paste on here and retain the formatting?
 
The output of df.write_json() would be helpful.
Then, I can read it back in.
 
9:20 AM
I am not getting an output- where would the .json file go?
 
print(df.write_json())
If the no file is specified a string is returned
 
I got hit with a message too long- hold on
 
df.melt().get_column("value").value_counts().sort("value") gives me 64 for the value 6
Which is the same as df.select((pl.all() == 6).sum()).sum_horizontal()
 
9:40 AM
I ran it again- I am getting 378 still
This is my code: pastebin.com/3LG0a7AA
 
Ah, I see. There is a minor bug leading to the wrong result: You call .melt() twice.
 
Ugh that is so annoying!
Thankyou so much for being so patient with me! I am a Junior dev so still learning!
 
10:09 AM
No worries! Glad I could help and cool that you are using polars - its quite a neat library :)
 
10:22 AM
Yes it is a severely underrated library! Thought it best to get across it sooner rather than later! Appreciate the help!