« first day (42 days earlier)      last day (2876 days later) » 

2:00 PM
Np, thanks for showing up, I have posted this meta now meta.stackoverflow.com/questions/326494/…
@Andy as you see we are running duplication comments in this chat and also SD regex to test. I saw you had done some wonderful work.
 
If you talk to hichris in the charcoal room, he may have such a dump. At last check, he's been grabbing all comments posted to SO for a while. I think he still is.
 
Ok thanks, in your dev did you make a library something we could steal?
 
I built this to interact with the API, but it's python based. Looks like you're using Java, right?
I'm interested in this duplicate detection. Is it just reading the "possible duplicate..." auto comment right now? Or is it actually suggestion a duplicate exists based on post content?
 
@Andy for now only possibile duplicate comments, but @Sam is working on an algorithm to find them also, when he is ready we will implement it.
 
That sounds awesome. I've tried a couple different times but can't get anything accurate enough that I'd test on the site with.
 
Yeah its all java based and all the api stuff is already running, I will see if I can get some rude/abusive comments and when I have it, I bet I will have some questions on the machine learning. You used this lib mahout.apache.org/users/algorithms/spark-naive-bayes.html correct?
@Andy I will notify when we have something up and running for testing.
 
Yes, that looks correct.
 
Oh hey Andy o/
 
2:12 PM
@PetterFriberg I appreciate that. I'd love to see that working. I'll also show a bit of professional jealousy if you get it working. :)
@Tunaki Hey.
 
@Andy If We make it work you need to send the greetings to @Sam, I'm will just call his api.
I think he has it on a github
Do you know anything about this lib "Weka"?
 
The link is purple in Google, so I've looked at it, but I am not a Java developer so I can guarantee I haven't played with it.
 
@Andy ok, thanks for showing up, I will gather some stuff if I can.., start playing around and then get back to you when I step into troubles... If you have ideas (more then the great post on meta) let me know.
 
Sam
Hiya
 
2:18 PM
I appreciate the ping. I'm always interested in seeing what others are doing to automatically find "the bad stuff" (tm pending)
Hi @Sam. Long time no speak.
 
Sam
@Andy Yeah, it's been a while :)
 
Thanks Andy was interested in your dev.
 
@Sam Hey
 
Sam
Well the algo works, it's just inefficient atm.
@Tunaki o/
 
Really? Cool! How accurate is it?
 
Sam
2:21 PM
It's hard to say precisely, as I can't currently load anything more than 50k posts.
(The algo grows in accuracy if you have a larger post set.)
 
shakes fist at Bill Gates 640kb!
 
Sam
If anyone's able to optimise these queries, I'll be much obliged.
 
starring will take a look
 
Sam
Thanks :)
 
2:29 PM
SELECT EXISTS (SELECT 1 from posts WHERE postid = {postID})
hmm
do you need that SELECT EXISTS?
 
Sam
@Tunaki I guess I don't
 
INSERT INTO globalterms (value)
SELECT value FROM gterms
WHERE NOT EXISTS
 (
  SELECT value FROM globalterms
  WHERE globalterms.value = gterms.value
);
 
Sam
Currently the largest bottleneck is this one.
 
That is not good
 
Sam
2:34 PM
Tuna's reaction to seeing my code: Ugliness, ugliness everywhere
 
You want to have a LEFT JOIN
and check whether the joined value is NULL
 
Sam
Oh
 
SELECT value FROM gterms LEFT JOIN globalterms ON globalterms.value = gterms.value WHERE gterms.value IS NULL
nvm
 
:D
@Sam Maybe the WHERE localterms.value = nv.value? This is text.
 
Sam
2:37 PM
@Andy Yep :D
@Tunaki It's an int
 
Oh
 
Sam
The only thing in the db which is text is the tags column in the posts table.
 
Scratch that then
do you really need all those temp tables?
 
Sam
Nope
I just don't know what else to use.
 
Let's take the first one gterms.
So, that table is created with a single row, having value <queryTermHashes>
 
2:42 PM
@Sam can't you do something like this in postgree (as in mysql) I do not know if its faster tough
UPDATE localterms lt
inner join globalterms gt on
gt.value = lt.value and
gt.postid = lt.postid
SET lt.vector = gt.tf * gt.idf
Then I do not really know what the fields are its just an example
 
Yes you don't need a with there also.
 
@PetterFriberg Yep that looks good
 
Then set index on all the value and postid
 
oh wait
> // Special thanks to Tunaki, stackoverflow.com/users/1743880.
Woot
 
2:45 PM
He will need to change that now ; )
 
Sam
@PetterFriberg gt doesn't have a postid column.
 
Do you get the idea anyway?
 
Sam
Yeah, I think so.
 
maybe just remove that join, I taught since this SELECT postid, localterms.value, tf * idf henche where not localterms I taught global : )
 
Sam
@PetterFriberg ;)
@Tunaki single column*
<queryTermHashes> is replaced with a set of values later on.
 
2:47 PM
@Sam and single row right?
oh
 
So, first off, you can have
 
type faster
 
wait
do you have an unique constraint of globalterms.value?
 
Sam
It's the pk, so yes
 
2:52 PM
if so, you can use an upsert statement
great
 
Sam
wonders what an upsert statement is
 
update and insert : )
 
INSERT INTO globalterms (value) <queryTermHashes> ON CONFLICT ON CONSTRAINT <name of constraint> DO NOTHING;
 
Sam
Oh
 
That Postgresql way of upsert = update or insert :)
 
2:53 PM
Yeah in mysql I always use that its the fastest
 
Sam
Great :)
 
you don't need any temp table ;)
 
Sam
\o/
 
2:54 PM
Yeah mysql has the same also, a bit different syntax
UPSERT is not standard SQL but lots of DB supports it
Will look at the rest tonight :) need to work now
 
Sam
Sure thing, thanks :)
I'll update what I've got now...
 
well make sure it still works after that also :D
 
Sam
xD
 
Query isn't official Tuna approved. It wasn't me if it blows up.
 
2:59 PM
no dupes for nathan... come on @queen
 
@PetterFriberg I don't judge people. I'm very enthusiastic. I'm just kidding around.
 
Yeah I noticed them.
 
3:33 PM
Hiya
@queen k
@petter You made it to the CB!
 
Sam
Hiya
 
> Yep, I guess you can delete this one to avoid bringing shame to your family name for the next centuries. This Q/A probably won't help anyone, so no need to keep it.
Reply for a typo
 
@Petter, @Kyll incomming possibile rude/abusive comment for testing
@MarkRansom Visual Studio has tons of bugs. And if you disregard the fact that it doesn't even claim to be c++11 compliant, then you'd think it has a ton more. Visual studio is a fucking joke of a c++ compiler that can't even bother to implement 6 year old language lfeatures. — xaxxon 45 secs ago
 
C++, Rude comment, ignore.
 
3:48 PM
@BhargavRao CB?
 
Community Bulletin, Hot Meta Posts
 
aaah ok, yeah lots of people like to read them with some popcorn.. I'm hopping undo gives a push : )
 
Yep.
 
4:01 PM
@que [java]
 
@Ferrybig in java scanned 1097 questions between Jun 02 12:55 and Jun 20 17:59 filtered and ordered: 20 in batch 1188
 
@PetterFriberg ^ last Q of that batch is a deleted question
 
Too Broad queen, Too broad
 
Sam
@PetterFriberg So something like this?
UPDATE localterms
SET vector = localterms.tf * globalterms.idf
FROM globalterms
INNER JOIN globalterms USING (value);
 
4:05 PM
@Ferrybig Yeah happens sometimes (it was deleted after api called was done), it will also give an error in cv count
@Sam in mysql its like this
wait
 
UPDATE localterms lt
INNER JOIN globalterms gt ON (lt.key = gt.key);
SET lt.vector = lt.tf * gt.idf
but I do not know postgree @Sam
 
Sam
Ok, np
I'll just fiddle around some more :)
 
it does not work in postgree?
 
Sam
Yeah ik
The syntax is different.
 
4:09 PM
@Tuna wake up... you know the syntax?
 
Sam
Lol
He said he'll check it later.
No rush
 
you can't update with inner join
unfortunately
 
Sam
:(
Darn
 
@Tunaki really?
 
Yeahg
But isn't it just
UPDATE localterms SET vector = g.tf * l.idf
FROM globalterms g, localterms l
WHERE g.value = l.value
 
4:12 PM
yeah that should work to
 
Sam
Yeah
 
no wait
UPDATE localterms SET vector = tf * idf
FROM globalterms
WHERE globalterms.value = localterms.value
in English
 
@Tuna thanks go to sleep mode again : )
 
for all local terms, we get the global terms having the same value and multiply two fields
and update the localterm back
 
Sam
Yep
Another gold star for Tuna.
 
4:14 PM
Did you test it? :D
 
Sam
Yeah
It works fine.
:)
 
Woot.
I'm sure we can get rid of all your temp tables :)
 
Sam
Much better than:
 
@que done
 
Sam
WITH new_vector(postid, value, vector) AS
(
  SELECT postid, localterms.value, tf * idf
  FROM localterms
  INNER JOIN globalterms ON globalterms.value = localterms.value
)
UPDATE localterms SET vector = nv.vector
FROM new_vector nv
WHERE localterms.value = nv.value
AND localterms.postid = nv.postid;
xD
 
4:15 PM
you can't tell a with isn't needed here
because the pre-populated table isn't doing any group by operations
 
@Ferry You see it's missing the deleted post cv count
 
Sorry @sam for posting while you wanted to post an annotated code
 
Sam
lol np
 
meaning, it isn't snapping multiple rows and grouping them to a value
 
Sam
Right
 
4:17 PM
You'll soon be a PostgreSQL expert @Sam :)
 
Sam
Thanks to you guys. :)
 
Is it faster though?
Probably so but you never know...
 
Sam
I'll have to check.
It should be faster.
 
Sam
afk for a bit, I'll benchmark the queries when I get back.
 
@queen k
 
@queen k
 
4:58 PM
@Petter, @Kyll incomming possibile rude/abusive comment for testing
when someone hits minus and leave not comment that is same if you piss on me. fucking idiots. — pregmatch 1 min ago
 
5:37 PM
Clickbaiting works! Got my first Guru badge stackoverflow.com/help/badges/18/guru?userid=4099593 :D
Hiya @gunr2171
 
@BhargavRao hello
 
@queen [java] dupes
 
@Tunaki scanned 1999 questions between Jun 17 14:39 and Jun 20 19:41 filtered and ordered: 6 in batch 1189
 
@que done
 
5:47 PM
@Tunaki Thank you for your effort, you reviewed 6 questions, I counted 6 (100%) close votes and 3 questions closed
 
@Petter
in SO Close Vote Reviewers, 12 mins ago, by Tunaki
@Petter I'm thinking we need a userscript that goes through all possible-duplicate notification that got feedbacked and bury them somewhere. What do you think?
This means I need an API to check if a post had feedback.
Relying on the chat message only doesn't cover all cases (like feedback after 2 minutes)
 
6:42 PM
@Tunaki Tuna all confirmed and non confirmed are on the databaseù
Most of all maybe its better to only give link to rude/abusive comment
 
Yep, so just a simple Rest API that returns k or f or empty feedbacl
 
this is the table dup_response
REST API... hmm we need a webserver for that Sam?....
we will kill Sam this way
 
:)
Well killing Sam has always been my goal since the beginning of this project.
3
 
Either for now we just leave them, or we just archive all of them, if we need stats I have all the data.
with a k or f..
 
From a JS userscript, I can't directly select into the table :)
Even if I could, it would be soooooooooooo horrible I couldn't write it
By the way, I hadn't thought about anonymization for the rude comment dump...
 
6:48 PM
Yeah I think I need to stop put all the info in chat, after Brad comments it seems like the comments are not public, SO likes to hide and forget about them
The solution is to only output link to comment...
 
Apparently, yes.
 
hence if delete the link will not work.
I honestly taught that all was public and rude/abusive was related to the coontext
hence outputting them in chat they loose context and are not rude/abusive in chat
I will try to do this modification tonight.
 
Sam
@Tunaki So that was your plan!
Ok, lemme backread....
 
:)
 
6:56 PM
Killing you : )
 
Sam
@PetterFriberg Just tell me what you need it to do, and I'll see what I can come up with. :)
 
@Tuna is this really a priority for now, I would let Sam work on the dups?
 
Clearly
 
@Sam I would wait with this FR until a rainy day comes by.... its about moving the possibile dupe messages... if it gets a priority for SOCVR yes... but until then meeh
 
Sam
7:06 PM
Oh
Might be a candidate for our cv-pls message achiever.
Right
 
Sam
Time for benchmarks
 
Yeah, but go ahead with you fun
 
@Tuna and @Sam I will start to use Sam's waffle method some (to get an initial dataset, that I can classify by hand after). Meta is not giving much hope but my objective to tell that we are up to this stuff is achieved.
 
7:15 PM
Yeah we shoudl't get our hopes up on Meta
Nothing will probably come out of it
but at least it's there
 
Well the true objective was to tell that we are looking in to this stuff.... and if they would like to help us they could : )
 
Sam
@Tunaki Yep, about 20-30% faster.
 
\o/
 
Sam
Only tested it against a table with 65k records though.
 
7:20 PM
If you hit perf issue because of the sizes of the tables, the time will come to create indexes
but not needed for now
 
Sam
Yeah
 
let me take a look at those other queries
So CREATE TEMP TABLE gterms was solved previously.
On to CREATE TEMP TABLE queryterms
 
Sam
Let me push my changes.
 
ooooo
 
Sam
@Tunaki Forgot to update this.
 
so you're calling GetQueryVectors that creates another temp table
hmm
that will kill you
 
Sam
Ok, done.
 
this GetQueryVectors will need more thoughts.
Moving to the temp table idfs
 
Sam
Yeah, it's not a simple one. :/
 
7:36 PM
well this one is straight forward
WITH idfs AS (
  SELECT value, log(2, <postCount>.0 / count(*)) AS idf
  FROM localterms
  GROUP BY value
)
UPDATE globalterms
SET idf = idfs.idf
FROM idfs
WHERE globalterms.value = idfs.value;
 
Sam
With all the things
 
it is pretty much the same as creating a temp table... I'd expect it to be a bit faster though
and take less memory
 
Sam
Oh, ok.
 
I've built document classifiers with over 30 categories using tools like Apache NLP and this would be relatively simple to accomplish provided that you have a mostly pure source of good and bad comments. — Technik Empire 24 mins ago
I have invited if he is interested to dev some : )
 
7:42 PM
Your post has 2 cvs :D
 
Sam
> This question does not appear to seek input and discussion from the community. If you have encountered a problem on one of our sites, please describe it in detail. See also: What is "meta"? How does it work? 2
lol
 
Hehe Exactly.
 
: ), yeah I saw that don't worry it will be reopened : )
 
Sam
afk
 
@queen [python] dupes
 
@BhargavRao scanned 1996 questions between Jun 16 17:34 and Jun 20 21:49 filtered and ordered: 17 in batch 1190
 
@Sam I don't think you even need a SQL query for GetQueryVectors
the goal of the method: takes a IDictionary<value, count> and returns IDictionary<value, float> and another float queryLength.
If I'm reading right, both dictionaries will have the same keys
what changes is the value, which is idf * (count * 1.0 / max) for the returned dictionary
idf is the one from globalterms. There's only one here, it's the PK.
count is the value of the incoming dictionary so we already have it
max is the maximum values of the incoming dictionary
so you can just load globalterms in a dictionary and look it up to get the idf
and the rest can be LINQed from the input dictionary
right?
---- wall of text end ---
 

« first day (42 days earlier)      last day (2876 days later) »