« first day (4091 days earlier)      last day (872 days later) » 

12:02 AM
And this is why we shouldn't work on holidays :P
 
Hey, I got my concurrency working in Java at 6am this morning. What is circadian rhythm?
 
what is a holiday? :p
 
I think I read about them once. Something about sun and no work. Utter nonsense, if you ask me :P
 
What is the sun!? Why are you all trying to confuse me just after midnight!? :p
 
<only just realised it was after midnight>
 
12:11 AM
My rabbit is running laps around the living room...
 
What's the rabbit called?
 
it probably just received news about the kind of things that roganjosh keeps as pets :p
 
Not any more. I only have Monty, my cat, these days. I got him 2 weeks after I started programming so the name seemed fitting
 
His "official" name is Stetson (like the cowboy hat), but we call him all sorts of things - Rabbit, Bunny, Bunzo, Booplesnoot, Fuzzbutt, stuff like that
 
He seems to be doing ok, given that I can't even extend my legs without protest :/
 
12:17 AM
 
Stetson's cute!
 
He is. He's a great pet - minds his own business most of the time, but is always willing to get pats, handles my long days at work, doesn't make any noise, he's litter trained, and he's just fun to be around.
 
Can he come train Monty?
 
@MattDMo uh oh... careful... else you might summon poke :p
 
Unfortunately, one thing he does not do well is travel. He hates going in his crate.
 
 
3 hours later…
Avv
3:18 AM
@roganjosh. Nice cat
 
 
2 hours later…
4:51 AM
Nice h̶a̶t̶ cat @roganjosh
 
 
1 hour later…
5:55 AM
We now have 3 buns - Auggie, Eeyore, and Sweetheart (all "fixed")
 
6:12 AM
@Kevin Can you add a notification to pop up on all the windows?
 
6:58 AM
is there any way to combine these two append files.

sys.path.append(os.path.join(code_dir , 'files', 'tools'))
sys.path.append(os.path.join(code_dir, "files", "report"))
 
7:21 AM
Not sure if you can sys.path.extend, but
sys.path.extend([dir_one, dir_two]) seems like it would work
 
8:03 AM
@WayneWerner Thanks
how to make any_string text a hyperlink , these is been used to send mail in outlook

<td><a>{}""".format(any_string)+"""</a></td>
 
'<a href="{0}">{0}</a>'.format(html.escape(any_string))
 
8:26 AM
Thanks I modified little

<td><a href="{0}">{0}</a>""".format(any_string)+"""</td>
 
So all this time, all you've needed was to put a string inside another string? I'm sorry, but that's ridiculous. You need to work on 1) your python skills and 2) your question asking skills
 
9:00 AM
cabbage
 
@Aran-Fey Thanks for the help this what I needed It worked
 
 
4 hours later…
12:51 PM
cbg
If I had a grid like this and these buttons where put into a 2d list based on their coordinate(starting from 0) then in what way can I index out everything in the first column except the first item only? I tried something like lst[1:][0] but it gives the wrong stuff out
TL;DR how could I index out 'Li', 'Na', 'K', 'Rb', 'Cs', 'Fr' from the 2d list set upon their coordinates
 
Either lst[0][1:] or [row[0] for row in lst[1:]] depending on how your 2D list is organized
 
[row[0] for row in lst[1:]] worked but why didn't lst[1:][0] work?
 
Because that's the same thing as lst[1]
It's like saying "take all rows except the first one and then take the first one of those"
 
Oh, so my understanding of 2d slicing is actually wrong
 
For the record, python doesn't have 2d slicing
 
1:05 PM
Oh, maybe that's why. I've been expecting it to be slicing since I used :
 
Python doesn't even really have 2d containers. You're simulating a 2d container by putting lists inside another list, but that's all. As far as python is concerned, it's a 1d structure (that contains other 1d structures)
 
So if I had this thing inside a numpy array then would it work my way?
 
No. You'd have to do arr[0,1:] or arr[1:,0]
 
Ah yea, something like that
I studied slicing like this with numpy array and I thought it'd be possible with python lists too
But it makes sense, thanks mate :)
 
If Python lists could do everything that numpy arrays can do, then we wouldn't need numpy arrays :-p
 
1:12 PM
Numpy arrays cant hold other objects right? Only numbers?
 
1:24 PM
The supported data types are here
 
Great
 
@CoolCloud they can but they should never
 
It might be a fun project to write your own array-like class that can hold any kind of object
 
How would the semantics of the result differ from a list?
 
I will defer this question until my caffeine levels are higher
 
1:45 PM
The indexing of numpy would occasionally be useful for lists.
 
it'd be nice... not quite sure how you'd achieve it with numpy style slicing though as it'd require the list to have some sort of "shape" to apply the slice to...
I'd be quite surprised if someone hasn't already written something that introduces a class that introduces numpy like slicing without it being numpy by now...
 
It looks like someone may have gone a little further
 
umm... looks quite interesting
 
2:18 PM
I have installed a python library which also contains a cli tool , under linux everything works fine , under windows the cli tool name gets not recognized, do I need to something to use the cli tool under windows ?
 
I don't follow - "under windows everything works fine , under windows the cli tool name gets not recognized" is a contradiction?
How did you install the python library?
The chances are that you just need to add something to your PATH
 
What library is it? For future reference, please include this kind of information in your first message
Many libraries install their cli tools by putting a .py or .exe file in your python install's ./Scripts directory. Most likely that directory is already in your PATH, since you probably wouldn't be able to run pip otherwise
 
On my PC, pip installs commands in %APPDATA%/Python/Python310/Scripts
 
If you can't find the tool in your Scripts, then maybe it didn't install right, or maybe the library put it somewhere else on purposse
 
3:13 PM
Do we have a (good) canonical for NoneType errors when using Beautifulsoup? i.e. two-level .find() where the first one didn't find and returns None so the second one raises a TypeError
 
3:50 PM
Firefox uses .sqlite files to save persistent user data such as cookies. Can Python's sqlite3 module read those files? sqlite3.connect(the_filename) executes without complaint, but cur.execute("SELECT sql FROM sqlite_schema ORDER BY tbl_name, type DESC, name") gives me no such table
I got that sql from google so I'm not confident that it's the "right" way to view the tables of a sqlite db
Oh good, the db exists and Python can read it, it's just that specific query that's bunk. cur.execute("SELECT * from moz_cookies") gives me meaningful data, for instance
 
Maybe "SELECT sql, type DESC, name FROM moz_cookies ORDER BY tbl_name"?
 
I'll try it... Same message, sqlite3.OperationalError: no such table: sqlite_schema.
For the time being I downloaded a sqlite explorer GUI, which does a decent job of letting me poke around. In the long run I'd like a sqlite3-only approach, but it's not blocking my progress or anything
Ah, just noticed your edit. If I run that query on moz_cookies, I get no such column: sql
Here's an almost-MCVE. You need to fill in the blanks in PROFILE_DIR. pastebin.com/93yQZmqX
 
4:29 PM
Oh no, the localStorage db encodes the URL that each key/value belongs to... How am I ever going to crack a code as clever as moc.ebutuoy.www.:https:443
 
4:47 PM
I think it's www.cutebuoy.om?
 
That URL encoding - oof, tough break there. Time to call the NSA (Not-Very-Good-At Secrets Association).
 
5:01 PM
cbg for a moment, friends
@Kevin does .tables work in sqlite3?
 
I'm afraid not: sqlite3.OperationalError: near ".": syntax error
 
Hrm... I thought that's what it was
Oh - I meant the command line app, not as a query
i.e. sqlite3 /path/to/moz.db followed by .tables
(It's one of those weird commands that only exists within the command line app, not actually the language)
 
the docs for sqlite itself confirm that .tables will show you tables. As you say, I think it only works on the command line.
I haven't verified this locally because my shell doesn't recognize sqlite3. Probably some kind of PATH weirdness.
 
Well, this suggests that nothing is super weird
❯ sqlite3 /tmp/foo.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
sqlite> select sql from sqlite_schema;
Error: no such table: sqlite_schema
sqlite> CREATE table foo (int boo, string foo);
sqlite> select sql from sqlite_schema;
Error: no such table: sqlite_schema
sqlite> .tables
foo
sqlite> .schema foo
CREATE TABLE foo (int boo, string foo);
 
Curious
 
5:08 PM
Is this more like what you're after?
sqlite> select * from sqlite_master;
table|foo|foo|2|CREATE TABLE foo (int boo, string foo)
 
That output looks useful to me, yes
 
Looks like this is another thing
sqlite> pragma table_info([foo])
   ...> ;
0|int|boo|0||0
1|string|foo|0||0
 
Cool, cur.execute("select * from sqlite_master") gives meaningful data. Nice find.
 
I'm guessing that the two of those combined should be useful!
FWIW, my brave search was "sqlite query equivalent of .tables"
 
Isn't it just:
 
5:10 PM
I feel it... Real Ultimate Power is mine
 
import sqlite3

conn = sqlite3.connect("testing.db")
c = conn.cursor()

c.execute("CREATE TABLE IF NOT EXISTS my_test(col_a String)")
c.execute("CREATE TABLE IF NOT EXISTS my_test_2(col_a String)")
conn.commit()
c.execute("SELECT * FROM sqlite_master WHERE type='table';")
print(c.fetchall())
c.close()
conn.close()
 
I didn't think to google anything like "equivalent of .tables" because sqlite.org/cli.html says:
The ".tables" command is similar to setting list mode then executing the following query:
SELECT name FROM sqlite_schema
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
ORDER BY 1
 
Additionally, you can use print([item[0] for item in c.description]) to give you the column headers, so you don't have to SELECT *
 
... And I already know that sqlite_schema doesn't exist in my db
So I assumed that googling for an equivalent would just give me this same broken query
 
sqlite_schema should just be an intrinsic property of every sqlite3 db?
 
5:14 PM
Incidentally, I wonder what "list mode" is. Can everyone tell that I'm not in "read the whole documentation top to bottom" mode today?
 
The schema table can always be referenced using the name "sqlite_schema" says this but I determine that not to be true because I also can't seem to select it in my example. Hmmm...
 
Well, apparently the docs are a lie :)
 
Wayne's experiment at chat.stackoverflow.com/transcript/message/53731189#53731189 may indicate that sqlite_schema will get quietly created under mysterious circumstances
 
so... it looks like sqlite_schema nee sqlite_master would be most accurate
good times
 
"The name has been changed to "sqlite_schema" as of version 3.33.0." I had hoped this would explain the issue, but sqlite3's docs say it "requires SQLite 3.7.15 or newer", so I'm back at square one
Oops, I misread the version numbers.
Ok, 33 > 7, so there is indeed some wiggle room for sqlite3 to not support sqlite_schema.
That'll teach me to do lexicographic comparisons
 
5:52 PM
This might be an artist you'd be interested in @PM2Ring here
I've listened to a few of her songs. Very stripped-back but an excellent voice
 
Avv
6:32 PM
Hello guys, I am trying to match columns in same data frame from `col 1, col 2` such that their label is different.

import pandas as pd

df1 = pd.DataFrame(
{
'Column1': [1,2,3,4],
'Column2':[10,11,12,13],
'Label':['apple', 'orange', 'apple', 'orange']
}
)

target = pd.DataFrame(
{
'Column1': [1,2,3,4],
'Column2':[11,10,13,12],
'Label':['apple', 'orange', 'apple', 'orange']
}
)

I am just looking for one match between columns `col 1` and `col 2` such that their labels are different, which is enough. So we cab only have either `(1, 13)` or `(1, 11)` but not bath in the target data frame. I
 
Interesting. Are the labels always "apple" and "orange", or can there be more?
 
Avv
Usually there are more! But I am just trying to solve this sample first. I used cumcount above to prevent repetition probably by having either (1, 13) or (1, 11) but not sure how to match columns though!
target frame is one possible solution to that.
Labels there are just to make it easier to solve the issue. So they can be discarded once we match columns
I tried to use shuffle, but it's random and some rows might still be the same or matched with columns of same label
 
Right
Potential algorithm: group the elements of Column2 by label. Find the largest and second-largest groups. Remove one element from each of them. Insert both of them into the result array, but with their positions swapped. Repeat all previous steps until the result array is full.
 
Avv
Let me think about it. Thanks. I will try to solve it and post back to you with code.
 
6:49 PM
has anyone heard of the cs50 course?
 
I have not
 
Avv
CS50: Introduction to Computer Science | Harvard University
This is taught at Harvard
Never took it though
 
neither did i
a newbie asked me if they should take it
so i was looking for someone who has taken it, to discuss some aspects
but nvm, thanks anyways
 
@Letsintegreat It is by Harvard(and free), but I'd say its boring. I did take a look into it, but did not find it interesting
 
7:06 PM
That reminds me, I wanted to check if there were any machine learning courses available from one of the bigwigs like MIT. There's Machine Learning with Python: from Linear Models to Deep Learning, but I don't think it starts until May of next year.
 
Avv
@Kevin. Are you looking to take this pls/
 
I might audit it, but I'm not interested in submitting work and receiving a grade
 
I assume you're already aware, but the course by Andrew Ng is basically the biggest hitter in the field
I don't know if that's exactly the right link; I think he's across multiple platforms
 
My eyes have passed over many tutorial links but I can't tell good from bad
 
Although I've just seen your comment about not wanting deadlines, so maybe all of his courses are out
 
7:11 PM
@roganjosh Seems nice, but it is not taught using python I guess
 
I don't think that really matters. I also seem to think that he might dip into Tensorflow with the examples, but I could be well off on that
Ah, nm, it's Octave/MATLAB
 
Oo then it must be advanced to an extent
 
Separately, I was surprised to see how Octave fared in Julia's benchmarks. That's quite a discrepancy
All benchmarks should be taken with a pinch of salt, obviously, but it's still interesting to see
 
8:02 PM
Python is a "nice to have" but my goal is language-agnostic wisdom, so anything I can run will suffice
 
8:20 PM
@roganjosh octave is free.
Not fast, not nice. Free :P
 
Ha, this is true, but so are the other languages it's being benchmarked against (other than MATLAB)
 
@Kevin today I bumped into lfortran that is built on LLVM and can make your fortran code interactive like python, or so they say :P
 
Actually, I'd need to check Mathematica on that statement
 
@roganjosh I mean if you ask me for pros, I'll tell you it's free
@roganjosh mathematica costs an arm and a leg
octave is "being able to run your matlab code even when your academic license runs out"
 
I guess that's fair. So it's really a fall-back vs "I've identified the best language for my new project and it's Octave"?
 
8:24 PM
that's my impression, yes
 
In which case, I will wind my disbelief down a couple of notches :P It's not an ecosystem I'm familiar with
 
octave is not very fast, and it lacks features compared to MATLAB, but it's trying hard to keep up to an extent, and to be compatible in the MATLAB -> octave direction
There were selling points such as implicit broadcasting in octave, but MATLAB caved after a few decades and introduced that too. Now the main selling point of octave (to me, a non-user) is being able to do foo(bar)(baz) (since function calls and indexing both use parentheses in MATLAB, and chaining them like that is only allowed in octave)
then again idiomatic octave would spell that as foo( bar )(baz), gross
ah no, that's not it
let me look it up
 
I suppose to be fair to Octave, Matlab isn't actually stellar in those benchmarks in the first place given the £1000s it can cost
 
foo (bar)(baz). My bad.
 
Then again, the GUI it gives for things like machine learning (just the facet I've actually used) are somewhat useful, and I definitely wasn't shooting for speed at the time I was using it
 
8:30 PM
@roganjosh it probably depends on the MATLAB version and the exact code they used for the benchmark. There are some really weird optimizations in MATLAB that can make a huge difference.
@roganjosh octave has a GUI too, although I always avoid it when I rarely use it.
ah, you mean something more specific, like a toolbox
 
If I were programming in Matlab now, I would also avoid the GUI just because I "know" how to program. Back when I was just an engineer, the whole thing would have been impenetrable for me in general
 
I used the curve fitting toolbox with its fancy GUI quite a lot, but then I ended up preferring programmable approaches anyway
 
If you'd asked me in my early 20s whether I thought I'd ever understand the code that the machine learning toolbox (you assumed correctly) spat out - I'd have been confident in saying "no" and, probably, "and I don't really care"
Which is a shame, because the months of Excel that could have been removed had our course just taught us something more practical than some matlab program to see whether ships would collide on a linear trajectory (totally unrelated to anything we were studying)....
 
and you poor engineers didn't realise that you only needed a paper and a pencil... :P
 
I think modern Engineering might not be quite as Stone Age as you might think :P
rogan-give-me-an-abacus-and-I'm-set-josh. It has something of a ring to it, I guess
I have a feeling that I've already pointed you to the formatting guide for code in chat, @Avv. Please read and apply it. There's a sandpit at the bottom of the link where you can test things out before posting here
 
Avv
8:45 PM
I just labelled every match I got with NaN so that I don't match it again. I separated the original table into 2 dataframes one with col `` and nother with col 2`.
 
Avv
@roganjosh. I formatted it with ctrl + K
 
It wasn't formatted. That's indisputable. Therefore, you haven't taken the guide on board
 
Avv
It's working on sandbox

Sandbox

Where you can play with regular chat features (except flagging...
 
@Avv It wasn't
 
Avv
8:48 PM
for i in range(len(df1_col1)):
    for j in range(len(df1_col2)) :
        if((df1_col1["Label"].iloc[i] != df1_col2["Label"].iloc[j]) and (df1_col2["Label"].iloc[j] != "NaN")):
            match = {'Column1': df1_col1["Column1"].iloc[i], 'Column2': df1_col2["Column2"].iloc[j]}
            print(match)
            df1_col2["Label"].iloc[j] = "NaN"
            result.append(match, ignore_index=True)
            break
 
@Avv Now it works
 
Much better, thank you :) I guess that it didn't translate over in a copy/paste
 
Avv
@Kevin. That's how I did it. I used drop but got some issues, so probably this work now, what do you think please?
 
How many labels do you have and what's the ballpark duplication rate?
 
Avv
I have more than 445 labels. Some duplicates over 300 while other just 1. So it varies. The above approach works in O(n^2), but that what I got now!
 
8:53 PM
If you have 10,000 unique labels and each is only duplicated once, we could probably just brute-force it and still be faster than that approach. If you have 10 unique labels and each one is duplicated 50 times, then we have a different kind of issue
@Avv oof. Frustrating middle-ground :/ I'll have to have a think
 
Avv
The above data needs systematic way to solve it, it has no well-defined structure (different labels counts) :(
 
It feels like it can be reduced to O(N) but it's not guaranteed to give a feasible solution. Then again, I'm not actually convinced that your approach can also guarantee feasible solutions in the first place. If I have 5 instances of label "a" and 1 instance of "b" (and no other labels), what are you going to do?
 
Avv
@roganjosh. Right. So if I have 3 oranges and 2 apples on data frame 1 and same thing on data frame 2, then after the above code, I will get only 2 oranges matched with 2 apples and 2 apples matched with 2 oranges. So one will be lost.
It seems okay in my case as I want to match rows with different labels. So, once I finish I will have pairs originally from different labeles
 
Ok, so now it's looking an awful lot more like a combinatorial problem
 
Avv
:(
You are right, some data will be lost
 
9:02 PM
Can you give me a bit of context on what you're doing and what the consequence of a dropped row is please?
 
Avv
That's the problem I have, I have no control over it
For example, pair of cloths
I would like to match different cloths together. Originally, the dataset has all pairs with same label (t-shirt with t-shirt) and so on.
However, I would like to have the opposite, match different cloths together (t-shirt with shirt). I hope it's clear now?
I posted 3 posts about this before:
https://stackoverflow.com/questions/70500734/matching-different-items-from-two-columns?noredirect=1#comment124630072_70500734
https://stackoverflow.com/questions/69518646/merge-two-python-dataframes-and-avoid-adding-same-match-twice-before-moving-to-t
 
In theory, we shouldn't have really been discussing this here because our room rules suggest that you need to wait 48 hours before bringing questions from main here. Given that it's Xmas for me, I'm gonna let that slide, but please keep it in mind for future
I assume "cloths" is actually "clothes"? I'm not being overly pedantic here - cutting fabric into patterns to maximise the number of clothing units you can make from a single sheet is its own optimisation problem, so we need to have that foundation in place, at least
It looks like you just want to match outfits?
 
Avv
Sure I will keep 48 in mind. Thanks.
Yes matching different outfits together.
The original dataset has all outfits that are the same (shirt with shirt, t-shirt with t-shirt, trouser with trouser, etc.)
Matching opposite pairs will definitely lose values as you see I have different counts of labels.
If I have just one count for each label, I won't lose anything.
 
What is the end goal? You want indiscriminate pairings, so most of them would "look" ridiculous in fashion
 
Avv
The end goal is to train a neural network to recognize different pairs. I don't want indiscriminate pairings, I want all pairs to be different pls. It's not allowed to have pair of same type together.
 
9:14 PM
I'll get a green top with orange sweatpants, and that could be the only combo I even consider for the green top
@Avv It's indiscriminate in the sense that your only criteria is that the pairings be two different items of clothing
 
Avv
Yes sure.
 
Beyond that, you don't generate anything like the combinations of things that people might buy - your pairing is arbitrary and doesn't look at anything close to the solution surface that's possible
And if your combination happens to be the green top with orange sweatpants, your best bet in your training set would be 1 drunk maniac at best
 
Avv
haha
It's not about recommedning crazy outfits together, for a classifier to detect crazy outfits
It's similar when a robot want to detect broken pieces
 
Sure, but I have reservations about how you're generating the training data. I don't think it's going to give you the search space you want. Most of them are probably crazy combos
 
Avv
That's fine for me now. That is what I have to do. No control :(
 
9:22 PM
Ok. Well, with that in mind, I'll have a think
 
10:04 PM
@Avv Before I start, this is pretty interesting. I have no idea how that works
It might just be a greedy match and there might be no guarantee on ordering (though I can't think of things in pandas that don't have fixed ordering. SQL is a different beast)
Also, stuff like this from cs95 worries me a bit that doing this in pandas is going to explode memory
 
Avv
10:47 PM
@roganjosh. Thanks! I will give it a shot and respond back to you if anything odds happen. Meanwhile, code is still running (30 minutes) on my code sample above :/
Have ~17k rows.
 
Then it's already gone kaboom
 
Avv
around 390 million matches :/
Do you think SQL would be better in these kinds of problems !
I almost forgot SQL, but probably I can recover the concepts within a week
Till now, pandas get most of my work done
 
Well; SQL isn't going to fix your problem here. There's things to keep in mind - where possible, you shouldn't have data in your dataframe in the first place if it could be filtered out in the first place with SQL because that doesn't have to pile everything into memory all the time so you should filter down as much as possible first
But your problem is still combinatorial. I don't know whether I'll have time to mock up a solver tonight. I guess I can try, but I'm making no promises. The != join is interesting, but I don't know how it works and I have concerns about the model you want to build on the back of it, even if it does work
 
Avv
I will try thanks. I see SQL does a lot of work while data in HD w/o moving it to RAM, which is great!
 
Going back to the green top/orange pants combo - you're just generating that randomly and then presumably hoping that there's some consumer purchase for the combo? I've taken a lot of leaps, but you're recommending stock for a retailer to hold?
 
Avv
11:00 PM
I think it's time to have some SQL back on deck
 
If there was a boom in green top sales, I'd never see it from your model because it paired it with something ridiculous in your model. So, you'd tell me not to sell green tops?
 
Avv
hahaha
 
I'm using a ridiculous example, but I'm being serious
 
Avv
The whole goal is to teach neural network to recognize different
It can with very high accuracy recognize similar, so now it's time for oppsite
 
I don't know what that means
 
Avv
11:03 PM
You can use that in product recommendation websites like Amazon for example
You can use it to recommend crazy outfit
This is one byproduct of this work
But my goal is to train neural network if this seems a good match or bad match
This is my end goal
 
I don't want to recommend crazy outfits, though. If the green top in isolation was a big seller, I'd just want to know that. Not only how it pairs with other items of clothing?
 
Avv
Yep! You should add a margin to tell how crazy you would like to go
For example, we can not allow very crazy outfits
 
Retail is not my area of expertise, I work more in industrial settings, but this is the kind of stuff the company I work for works on, so I have a passing knowledge
I'm trying to get my head around exactly what actionable stuff your model is going to give to a retailer
 
Avv
Even in industrail domain, you might have similar issues like recognizing if products on a production line are similar
Many applications can be worked on
This can also be used to tell if something crazy is going in in the CCTV
 
Not really. I can disaggregate the production schedule to individual units that can do work. That is definitive - everything has a Bill of Materials, so my disaggregation is definite. Yours is not - you've said yourself that there's millions of combinations but you're not feeding that into your model
 
Avv
11:09 PM
For me, I should have only dissimilar pairs, that is why I added label
 
We risk going in circles here
 
Avv
in the original data, all rows are matches, so the label describe what kind of match as I explained earlier. Yes. Circles you are right
 
You have 17k lines, and you've determined there are 390 million combinations (I suspect there could be more). Your result will be 17k rows in a dataframe (at best, assuming we don't drop anything). That's a pretty small subset of possibilities with which to feed a model, and the vast number of combinations probably have no sales data anyway
Maybe it would help to take a step back - how exactly is the model going to work? I'm not trying to be difficult, I'm just trying to make sure you don't waste time before coding this up. At the moment, it looks like you want to just randomise purchase pairs to get a minuscule subset of the combinations people could buy, and train on that. I just don't think that will work
Even something like Pearson Correlation could be used to make things a bit more targeted than that?
 
Avv
@roganjosh. It's abstract from my end now in the sense that I am asked to train neural network to recognize different pairs. So feeding network with similar pairs would produce erroneous model.
I will follow up with you after that if you like. But I am also asked to add a margin of how dissimilar they are so that we don't have endless dissimilar pairs. This is another issue
 
Is this a school/college/uni project?
 
Avv
11:21 PM
no it's not academic, it's a project
the code I posted will solve it for now, so I will go with it temporarily as it's not the main problem
 
There are lots of problems :/ But I will suspend my interrogation :P
 
Avv
I just want to get this step done as it's nothing compared to the problem :/
Now, spent alomst half day on it :(
@roganjosh. Thanks for your help.
 
No worries. I'm going to play with this random assignment because I'm just curious but something feels off with the premise to me
 

« first day (4091 days earlier)      last day (872 days later) »