« first day (38 days earlier)      last day (2872 days later) » 

@queen k
 
Kill'em all
@Sam we have Yvette's blessing so when you have some doc or instruction let me know and I will try to call the api
 
Sam
Sure thing
Just optimising the heck outta everything atm.
 
: ), we need to fill Tuna with dupes so e refuses to enter the room
 
Sam
lol
 
@queen k
 
Sam
10.5k posts/sec \o/
Let's see if we can go faster.
 
go Tuna go... and now you like to instant close off-topic and mcve to meta.stackoverflow.com/questions/326250/…, you will need some more votes...
I can get them to in here I guess :D
 
@Sam Seems enough....
 
Sam
@PetterFriberg Not when you need to process 30 million.
 
You like to also check old ones?
 
Sam
Some dupe targets are pretty old.
I need to make sure I also include them in the search.
Atm it'll take ~45 mins for one query.
 
seems interesting... I need to check out your code later... to understand what you are up to..
 
Sam
4:48 PM
This is the main query:
CREATE TEMP TABLE tempposts AS
SELECT posts.postid, posts.length, localterms.value, localterms.vector FROM posts
INNER JOIN localterms ON posts.postid = localterms.postid
WHERE localterms.value IN (..., ..., ...); # The incoming posts words (hashed).

CREATE TEMP TABLE queryterms
(
	value  int4   PRIMARY KEY,
	vector float4
);
INSERT INTO queryterms VALUES (..., ...), (..., ...), (..., ...); # The incoming posts words (hashed) with each word's vector.

SELECT postid, SUM(vector * (SELECT vector FROM queryterms WHERE tempposts.value = queryterms.value)) / (length * <The incoming post's lengt
It's probably ugly.
Where localterms is created like so:
CREATE TABLE IF NOT EXISTS localterms
(
	id     bigserial PRIMARY KEY,
	postid int4      REFERENCES Posts(PostID),
	value  int4      REFERENCES GlobalTerms(Value),
	tf     int2,
	vector float4
);
(postgresql btw)
Unfortunately I think I've screwed up the similarity calculations.
(Previously I was calculating everything in C#.)
 
@Sam be sure to put index on all search keys as postid, value, lenght, sim that is an easy way to speed it up.
 
Sam
Sure thing.
 
Sam
@PetterFriberg Is there some way to make this more elegant?
SUM(vector * (SELECT vector FROM queryterms WHERE tempposts.value = queryterms.value)) / (length * <The incoming post's length>)
 
4:56 PM
Yeah thats the worse part
 
Sam
Yeah, I thought so. xD
 
thinking, I need to imagine the structure
The first vector is from tempposts?
 
Sam
Correct.
 
so its always the same for the record.
surely no need to multiple on each sum...
 
Sam
No, each record has a different vector value.
Here's a small dump of tempposts:
 postid | length  | value  |  vector
--------+---------+--------+----------
      9 | 19.5836 |    864 | 0.187707
     39 | 74.8015 |    864 | 0.750829
     39 | 74.8015 |  28357 |  1.13924
    109 | 95.3455 |    864 |  3.56644
    109 | 95.3455 |  28357 |  2.27847
    260 | 63.5686 |    864 |  1.68936
   1037 | 36.2826 |    864 | 0.375414
   1010 | 45.9522 |  28357 |  1.13924
   1040 | 34.8206 |  28357 |  2.27847
   1528 | 58.2363 |    864 |  1.68936
   1528 | 58.2363 |  28357 |  3.41771
   1760 | 41.7471 |  28357 |  2.27847
 
5:01 PM
aah ok
 
Sam
value and vector were joined in from the localterms table.
 
Sam
So what I'm trying to do is sum all of the vectors for a post which are also in the query table, and then divide the result by the post's length (after multiplying the length by a hardcoded value).
 
Maah, your solutions are like that or to try to inner join the the table instead or to try to do view with pre-calculated result, the key I think is to have index in both tables on column value
I would probably have tried an inner join.
 
Sam
An inner join with queryterms?
Oh
Yeah, I see what you're saying.
 
5:11 PM
but I do not know which is faster... you need to test these stuff.
 
Sam
Benchmark all the things!
 
Yeah, check what query takes time, and work your way down, worst first...
but at a certain point there is only that much you can do...
 
Sam
Yeah
 
need to run.. have fun :D
You can always use memory load stuff in memory :D
 
Sam
Cya soon
 
Sam
@Petter You'll be pleased to know I've found something less hideous:
CREATE TEMP TABLE queryterms
(
	value  int4   PRIMARY KEY,
	vector float4
);
INSERT INTO queryterms VALUES (..., ...);

CREATE TEMP TABLE tempposts AS
SELECT posts.postid, posts.length, localterms.vector AS lterm, queryterms.vector AS qterm FROM posts
INNER JOIN localterms ON posts.postid = localterms.postid
INNER JOIN queryterms ON localterms.value = queryterms.value
WHERE localterms.value IN (...);

SELECT postid, SUM(lterm * qterm) / (length * ...) AS sim
FROM tempposts
GROUP BY postid, length
Also it's faster.
 
7:15 PM
@Sam Yeah seems much cleaner
 
Sam
~100,000 posts/sec. :)
 
: ), that need to enough...
next step is to load all in memory : )
 
Sam
lol
 
do you already have that api up'
send me some instructions so I can have some fun this weekend
 
Hiya
@queen [python] dupes
 
7:26 PM
@BhargavRao scanned 1995 questions between Jun 13 21:17 and Jun 16 21:23 filtered and ordered: 20 in batch 1089
 
Sam
Hiya
@PetterFriberg Not yet
 
Sam
@PetterFriberg It might not be ready by then.
 
Hiya BR!
 
Sam
o/
 
7:27 PM
Howz the day? Quiet/Noisy?
 
Sam
Lotsa dupe notifs.
 
I have been at a client... trying to figure out what they need... so I have no clue
 
Sam
And I've made progress with UniStack.
 
UniStack is the dupe finding algo, rite?
 
Sam
Yeah
 
7:31 PM
Progress implies some good news is near.
 
Sam
15 mins ago, by Sam
~100,000 posts/sec. :)
So far ^
 
SEDE?
 
Sam just need to work all night so I can have some fun this week-end
 
Sam
@BhargavRao Local SO post dump.
 
I did Sheet Loads of work in the last 2 weeks of May and I am almost a month ahead of schedule.
 
Sam
7:33 PM
Nice :D
 
What about Yvette's Ada? Any progress there?
 
@PetterFriberg haha yeah. Lots of possible duplicates
 
Hi @ryan \o
 
7:36 PM
My son is asking why I laughing... so lol
 
greetings!
 
Sam
7 hours ago, by Yvette
so the whole bot project was neglected
Read a few lines above/below
 
That's a bit sad. :/
 
Sam
Yeah :/
 
7:39 PM
Yeah I had a chat with Yvette, I guess she is a bit sad but I hope also relieved and when she have time we can see how to stitch everything together.
 
@Sam You also have the Jammy Account, So you can run code on that.
 
Sam
Yeah
 
naa its a open api, I'm stealing it it all!!!!!!!!!!
 
Sam
It's all open source :p
 
Oh WoW! Queen's in the testing facility!
 
Sam
7:41 PM
Ofc, you'd need to populate the DB and run it somewhere.
 
You can prep her up for a trial run in SOCVR for a day to check.
 
Actually I'm we will test this week-end I hope
 
Just ping me when you run her there. :)
 
We will only to dupe notifications in tags that RO allows... chat.stackoverflow.com/rooms/114671/…
so you need to sneak in python....
 
Hehe, Queen will be here or in Campaigns for that (as I am the only regular Py hammer there, if you ignore JonCle).
 
7:46 PM
she is already running in SOCVR Testing facilty with that setup.... BR we need to get the dupe notifications into python to..... work on it ; )
@que list tags
 
@PetterFriberg In this room these tag are available
 
There are so many dupes no room can handle them all... specially if we get Sam's algo up and running to... we need to convince each room to work some.
 
Sam
With my algo you'll get a % match.
So you could filter out anything < x.
 
Yeah poor Tunaki has been pulled for all the java stuff
 
There are many java hammers in SOCVR
But note that Tuna is like the hammer for Java. Currently 3rd on the most hammerers
So we will need to make poor tuna even more poor @ryan :P
 
7:49 PM
@Sam Sure, but do you see how many dupes we get only in these tags.... SOCVR are worried about to many dupes, hence I have implemented the tag block... it would be nice for SO if each room had thier dupes, but I guess this is just a dream
 
Did Pham ever make it to the SOCVR?
I remember Sam saying that there was a poor hit ratio.
 
Sam
I haven't worked on Pham for a few weeks now.
The account's currently being used for a different project.
UniStack is one of the core functions of Pham. I've just ripped it outta the project so others can make use of it.
 
and others are about to use it : )
 
Sam
:)
 
What's the history behind the sobriquet of that project?
 
Sam
7:56 PM
Ha, it's pretty long.
Well
Longest out of all my projects.
So, the project started in Sep 2014. I was (more) active in the MSE Tavern, flagging spam, etc. as Smokey was primarily running there. However, I felt I could improve on SD's design (and accuracy). Fast forward a few weeks and I made the first bot which checked answers (and was faster than SD).
Fast forward a few more weeks and it soon became apparent that he was too inaccurate (for my liking). So I decided to explore the world of NLP (we had been previously using regexes with a custom weighing system, built by yours truly ;) ). A couple more weeks later and I finally found a promising classification method, sadly the libraries were too inefficient, so I had to abandon that idea, which is why the project remains untouched.
Pham was also the first bot I ever made.
So that's the story of Pham, @Bhar.
 
@Sam do you still have some nice stuff we can test on comments?
 
Sam
Regexes?
 
@Sam Regexen!
 
I liked the idea of Pham, Does it cost a lot of space to keep the bot up and running?
Perhaps when you are free you can refactor Pham.
 
Yeah I'm still wondering if we could do better on comment, we only get the boring regex cases... either I need better regex or something more advanced.
 
Sam
8:06 PM
@BhargavRao The old (v1) Pham was a piece of cake to run (resource wise).
@PetterFriberg Let me have a look.
@BhargavRao I plan to, I haven't abandoned the project. :)
 
Do you have a transcript where the V1 was running?
 
Thanks. I recognize Zephyr there ;)
 
Sam
@PetterFriberg 1 and 2. Pham was more aimed at finding spam than offensive stuff though.
@BhargavRao Yeah, me and Andy started on our projects together.
 
@Sam ok thanks anyway, I will have a look and see if I can integrate anything... but my guess is that to effective it needs to understand offensive phrase even if not a swear word is used
 
Sam
8:15 PM
Exactly.
The main thing I was trying to improve upon was understanding context.
Hence me switching to NLP.
afk
 
Sam
Pizza :D
 
I've yet to have dinner ;_;
 
@que 10 [java]
 
8:20 PM
@PetterFriberg in java scanned 1098 questions between Jun 02 16:36 and Jun 16 22:18 filtered and ordered: 10 in batch 1090
 
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 20 questions, I counted 20 (100%) close votes and 9 questions closed
 
@que done
 
@PetterFriberg Thank you for your effort, you reviewed 10 questions, I counted 10 (100%) close votes and 10 questions closed
 
@que 10 [java]
 
8:25 PM
@PetterFriberg in java scanned 1099 questions between Jun 02 16:36 and Jun 16 22:24 filtered and ordered: 10 in batch 1091
 
@queen [python] =4cv
 
@BhargavRao in python scanned 1071 questions between May 25 15:39 and Jun 16 22:24 filtered and ordered: 20 in batch 1092
 
@que done
 
@PetterFriberg Thank you for your effort, you reviewed 10 questions, I counted 8 (80%) close votes and 2 questions closed
 
@BhargavRao python slacking? 20 at cv4 : )
@que index [java]
 
8:30 PM
Starting to index tag: java
 
@queen [python] =4cv
 
@MorganThrapp in python scanned 1069 questions between May 25 15:39 and Jun 16 22:24 filtered and ordered: 17 in batch 1093 There are other questions locked in open batches
@PetterFriberg Index of tag java completed
Scan statistics tag [java]
  Date    # questions  # closed       CV1       CV2       CV3       CV4        PD
-------------------------------------------------------------------------------------
26 May             49         7     0 (0)     0 (0)     0 (0)     0 (0)      1(1)
27 May            692        87     0 (0)     0 (0)     0 (0)     0 (0)      3(3)
28 May            459        44     1 (1)     0 (0)     0 (0)     0 (0)      1(1)
29 May            446        61     3 (1)     2 (2)     1 (0)     0 (0)      0(0)
 
@que index [python]
 
Starting to index tag: python
@PetterFriberg Index of tag python completed
Scan statistics tag [python]
  Date    # questions  # closed       CV1       CV2       CV3       CV4        PD
-------------------------------------------------------------------------------------
26 May             35         4     0 (0)     0 (0)     0 (0)     0 (0)      0(0)
27 May            491        50     0 (0)     0 (0)     0 (0)     0 (0)      1(1)
28 May            303        32     0 (0)     0 (0)     0 (0)     0 (0)      0(0)
29 May            313        39     3 (1)     0 (0)     0 (0)     0 (0)      4(4)
 
@que 5 [java]
 
8:34 PM
@PetterFriberg in java scanned 1096 questions between Jun 02 16:36 and Jun 16 22:32 filtered and ordered: 5 in batch 1094
 
@queen done
 
@MorganThrapp Thank you for your effort, you reviewed 17 questions, I counted 16 (94%) close votes and 16 questions closed
 
@queen [python] =4cv
 
@MorganThrapp in python scanned 1094 questions between Jun 05 19:16 and Jun 16 22:33 filtered and ordered: 20 in batch 1095 There are other questions locked in open batches
 
@que done
 
8:38 PM
@PetterFriberg Thank you for your effort, you reviewed 5 questions, I counted 3 (60%) close votes and 2 questions closed
 
@Ferry I left 10 for you : )
@que 5 [c#]
 
@PetterFriberg in c# scanned 1099 questions between May 28 07:31 and Jun 16 22:33 filtered and ordered: 5 in batch 1096
 
@PetterFriberg You don't call the army ;)
 
:D, arrrg I close as question I did not like to close....
 
@queen ignore
 
8:41 PM
@BhargavRao The batch has been deleted, questions will be displayed in next batch request.
 
@queen [python] =4cv
 
@BhargavRao in python scanned 1094 questions between Jun 05 19:16 and Jun 16 22:39 filtered and ordered: 20 in batch 1097 There are other questions locked in open batches
 
@queen done
 
@MorganThrapp Thank you for your effort, you reviewed 20 questions, I counted 19 (95%) close votes and 19 questions closed
 
@queen [python] =4cv
 
8:42 PM
@MorganThrapp in python scanned 1094 questions between Jun 05 19:16 and Jun 16 22:39 filtered and ordered: 20 in batch 1098 There are other questions locked in open batches
 
@queen done
 
@MorganThrapp Thank you for your effort, you reviewed 20 questions, I counted 2 (10%) close votes and 2 questions closed
 
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 20 questions, I counted 5 (25%) close votes and 5 questions closed
 
Hory Sheet @morgan 15 of the last 20 are struck on cv4. We need another guy.
@queen [python] =3cv
 
8:46 PM
@BhargavRao in python scanned 1097 questions between Jun 05 19:16 and Jun 16 22:45 filtered and ordered: 20 in batch 1099
 
Where's wajdi when you need him? :P
 
Sam
Back in my day, sheets were used to write on. Now we just use 'em to curse.
 
Lol :D
 
Sam
I'm not that old though. xD
 
20+ is too old :P
 
Sam
8:48 PM
Well I'm 18. :p
 
Morgan crossed 22 this week
 
I closed this wrongly I'm happy if you take a look at it: stackoverflow.com/questions/37604065/…
 
I'm out of votes for today.
@BhargavRao Sure did!
 
Now I can drink slightly more legally!
 
8:49 PM
@Sam Ah, The sweet age when you get license to drive :D
 
Sam
@MorganThrapp Happy belated birthday!
 
Thanks!
Twas back on Tuesday.
 
Sam
@BhargavRao Meh :/
 
HAPPY BIRTHDAY!!!
 
Sam
8:50 PM
Nice :D
We should have a SOCVR version of that.
 
@Sam happy birthday.
 
@Sam True True, Who wants to drive when you can stay at hom in front of the comp :D
 
Sam
Yep xD
@ryanyuyu Not for another 4 months.
 
remind me in 4 months
 
@Bhargav Hey, your birthday is a day after mine!
 
8:51 PM
@Kyll 21 June 96 97 95 it is
 
@BhargavRao Meh, a year sooner =p
Wait
Got my dates confused
A year further in deh past
 
meaning that his is a year after yours?
 
@BhargavRao Yes. @ryan Yesh too.
 
@Kyll I'm from Jun 94.
So is Morg
 
8:54 PM
We'll have the same age for a day \o/
 
Man. It's not often I'm the oldest in the group.
 
@que 15 [java]
 
@Ferrybig in java scanned 1099 questions between Jun 02 16:36 and Jun 16 22:52 filtered and ordered: 15 in batch 1100
 
in Python, Jun 13 at 19:04, by Bhargav Rao
@MorganThrapp June 94, The Month when geniuses were born. ;)
 
@ryanyuyu It's not often I'm not the youngest =p
 
8:54 PM
well other than probably petter who has a family
 
@ryanyuyu Yvette's there :P
 
@BhargavRao snorts
 
@que done
 
@PetterFriberg Thank you for your effort, you reviewed 5 questions, I counted 3 (60%) close votes and 3 questions closed
 
"back in my day, we used floppy disks that were actually floppy"
 
8:55 PM
@ryanyuyu Come to the R Room where everyone are 40+ :D
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 20 questions, I counted 9 (45%) close votes and 0 questions closed
 
@queen [python] dupes
 
@BhargavRao scanned 1995 questions between Jun 13 23:07 and Jun 16 22:52 filtered and ordered: 19 in batch 1101
 
Sam
Like dem?
:p
 
yeah
 
8:57 PM
@ryanyuyu and @kyll, in socvr I can not but here I can plz check out this stackoverflow.com/questions/37604065/… I made a mistake closing it and I'm trying to fix it : )
 
But really mostly the less floppy 3.5 inch ones
 
Still wondering if hardcore backward-compat or just Photoshop
 
@PetterFriberg actually already voted
 
Sam
@Kyll ikr
 
@PetterFriberg You can cast a reopen pls in SOCVR
 
8:58 PM
he did
 
I'm not proficient whatsoever in that tech, can't really vote
Off to me bed, night!
 
Sam
Cya
 
bye kyll
 
Sam
@PetterFriberg It's open now
You should ping me if you've got anymore C# stuff.
I didn't notice your req, until just now.
 
Thanks, will do.... sorry for bugging but closing correct question is one of my main goals... so I need to fix that misstake
 
Sam
@PetterFriberg np
 
@queen k
Sigh, Need some one to dupe hammer this. I was the guy who edited the py tag stackoverflow.com/q/37824439/4099593 :(
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 19 questions, I counted 19 (100%) close votes and 6 questions closed
 
@que tags today
 
@PetterFriberg You can give me honest feedback.
 
9:07 PM
@que stats tags today
 
Tag statistics today
   nr [tag]             Reviews  CV virt.  CV count    Closed
-----------------------------------------------------------------
   1. python                136       116        90        57
   2. android                71        58        58        10
   3. java                   25        21        21        14
   4. bash                    4        13         4         3
   5. c#                      5         3         3         3
-----------------------------------------------------------------
 
@queen [python] =2cv na
 
@BhargavRao in python scanned 1096 questions between Jun 05 19:16 and Jun 16 23:03 filtered and ordered: 9 in batch 1102
 
@BhargavRao yeah thats how we like pytonn... :D
@que stats tags
 
Tag statistics all time
   nr [tag]             Reviews  CV virt.  CV count    Closed
-----------------------------------------------------------------
   1. python               2074      2009      1630       722
   2. java                 1413      1386      1124       599
   3. c#                   1347       827       827       343
   4. ios                   665       677       522       204
   5. android               595       476       474       249
   6. php                   323       267       267       169
 
9:09 PM
@PetterFriberg Yup, Soon more guys will join :P
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 9 questions, I counted 9 (100%) close votes and 0 questions closed
 
@queen 3 [python]
 
@BhargavRao in python scanned 1085 questions between Jun 05 19:16 and Jun 16 23:08 filtered and ordered: 3 in batch 1103
 
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 3 questions, I counted 1 (33%) close votes and 1 questions closed
 
9:11 PM
@queen 2 [python] =1cv na
 
@BhargavRao in python scanned 1086 questions between Jun 05 19:16 and Jun 16 23:08 filtered and ordered: 2 in batch 1104
 
@que 5 [c#] =4cv
 
@PetterFriberg in c# scanned 1099 questions between May 28 07:31 and Jun 16 22:55 filtered and ordered: 5 in batch 1105
 
@queen done
 
@BhargavRao Thank you for your effort, you reviewed 2 questions, I counted 2 (100%) close votes and 0 questions closed
 
9:12 PM
@queen stats me
@queen remove
 
400 closed : )
 
Out of 792 :D
Anyway time to sleep. Bye
@queen bye
 
cya
 
10 more of BR and 10 Tuna I guess there would not be a review que anymore...
Cya BR
 
Heh, That's not too far :)
 
9:13 PM
@que ignore
 
@PetterFriberg The batch has been deleted, questions will be displayed in next batch request.
 
@que 5 [c#]
 
@PetterFriberg in c# scanned 1099 questions between May 28 07:31 and Jun 16 23:12 filtered and ordered: 5 in batch 1106
 
@que done
 
@PetterFriberg Thank you for your effort, you reviewed 5 questions, I counted 4 (80%) close votes and 4 questions closed
 
9:22 PM
@que done
 
@Ferrybig Thank you for your effort, you reviewed 15 questions, I counted 15 (100%) close votes and 15 questions closed
 
Thanks Ferry, java is under check mate
@Ferry did you notice that I had voted on all (and not today or yesterday).. which just goes to prove the inefficiency of the CVR que... those would never have been close (vote would just have aged away)
 
@PetterFriberg Did not notice that, but I'm a little tired at the moment
 
@Ferrybig I reopen the batch, just to check... trust me, I noticed the same in C# they all have @RichardSlater on them.
@que index [c#]
 
9:32 PM
Starting to index tag: c#
@PetterFriberg Index of tag c# completed
Scan statistics tag [c#]
  Date    # questions  # closed       CV1       CV2       CV3       CV4        PD
-------------------------------------------------------------------------------------
26 May              8         0     0 (0)     0 (0)     0 (0)     0 (0)      0(0)
27 May            596        46     0 (0)     1 (1)     1 (0)     0 (0)    13(12)
28 May            266        20     1 (1)     0 (0)     0 (0)     0 (0)      7(7)
29 May            290        20     0 (0)     1 (1)     1 (0)     0 (0)      6(5)
 
@queen k community
 

« first day (38 days earlier)      last day (2872 days later) »