« first day (399 days earlier)      last day (2996 days later) » 

user3956566
5:30 PM
hello
 
Sam
Hey
I got my matching algo working
 
user3956566
omg!
 
Sam
But...
 
user3956566
I was too excited to stay asleep :D
 
user3956566
But???
 
Sam
5:31 PM
:o
What time is it for you?!
 
user3956566
4.31
 
Sam
... it's terribly slow.
:o
 
user3956566
erm not saying am/pm
 
Sam
I can probably guess
 
user3956566
Yes I'm not surprisd it's slow
 
Sam
5:32 PM
Takes 8 mins 15 secs to check one post
 
user3956566
how doesn smokey manage to be so quick
 
Sam
So yeah, I'm optimising it atm
@MsYvette He just compares posts to regexes.
 
user3956566
ttytt that is why I wanted phrases, as smokey is so quick and uses regex
 
Sam
I'm comparing a post to more posts
 
user3956566
yep
 
user3956566
5:33 PM
I know
 
user3956566
that was why I was looking at phrases
 
Sam
But, I ran those benchmarks on unoptimised code
So we'll see
 
user3956566
I think the speed is going to be our biggest problem
 
Sam
Yus
 
user3956566
I've even been wondering if we filter by tags
 
user3956566
5:34 PM
so we don't check every post
 
Sam
!
I didn't consider that
 
user3956566
well to begin with, the prototype is to just focus on one post
 
user3956566
I know that is not useful in gathering speed requirements
 
user3956566
but I have been wanting to add this when people are discussing dupe targets
 
Sam
Yeah
 
user3956566
5:36 PM
The idea behind this bot is to focus on wildly large dupe targets
 
user3956566
it's to catch those posts that have been asked 100s of times before
 
user3956566
I would anticipate, that we would need a troop of bots to really cover a good range of bots
 
user3956566
remember when I said we would need troops
 
Sam
Hmm
 
user3956566
But
 
user3956566
5:38 PM
This is not an issue yet
 
Sam
Not sure that'll be accepted with smiles
 
user3956566
It's irrelevant at this stage, as we have no proof of concept
 
Sam
true
 
user3956566
also it's possible that if we really did cover a wider range of dupes, any room would be flooded and there wouldn't be enough close votes.
 
user3956566
Not without a larger range of participants from SO
 
user3956566
5:39 PM
BUT
 
user3956566
this is where my clever idea comes fomr
 
user3956566
if this bot is successful, you could have bots posting purely into chat rooms for specific tags
 
Sam
in Tavern on the Meta on Meta Stack Exchange Chat, yesterday, by Shog9
@MsYvette see what you can come up with for matching algorithms
 
user3956566
if the community of that room wanted them
 
Sam
in Tavern on the Meta on Meta Stack Exchange Chat, yesterday, by Shog9
algorithm first. Bot comes second. An inaccurate bot is just an annoyance.
But that ^
 
user3956566
5:41 PM
yes I know
 
user3956566
but we need to test the algorithm on a bot
 
user3956566
which we are
 
user3956566
for speed and accuracy
 
user3956566
I honestly think the best way is a regex checker
 
user3956566
I would like to see what SO uses, to get the pop ups when people are writing a question title
 
user3956566
5:42 PM
something like that, it's fast
 
Sam
@MsYvette Yeah ik, just reminding us what the goal was. :)
 
user3956566
I'm not as stupid as I sound sometimes
 
user3956566
I know exactly what the goal is :D I created the goal
 
user3956566
in terms that it's my mission
 
user3956566
34
Q: Remove the incentive for FGITW to answer well known dupes

Ms YvetteTLDR If a dupe target has n number of duplicates closed against it, that if future questions closed to that dupe target are answered, all the rep is nullified, this way there is no reward for lack of research or potential rep whoring. A query given to me by Shog9 of answers posted to duplicat...

 
user3956566
5:44 PM
Trying to overcome this
 
Sam
yeah
 
user3956566
so we are fighting against speed FGITW
 
user3956566
but we do need accuracy or a success rate similar to exisitng bots smokes
 
user3956566
smokey I mean
 
user3956566
there are false reports, but people are ok with that, due to the number of succeses, so it would be good to get some stats
 
user3956566
5:46 PM
I might ask them in Charcoal
 
Sam
ok
 
user3956566
Now I know that's a lot of regex, but it would be a piecemeal operation and I think if the bot proves viable, how we determine the dupe targets would be on meta by community consensus
 
user3956566
we don't need to trawl for dupe targets
 
user3956566
You know.. ok we have bot.. please post you dupe target... top x posts are done first in experiment
 
user3956566
You see this was given to me by a high rep user interested in the bot idea
 
Sam
@MsYvette I don't follow.
 
user3956566
meaning the number of dupes coming from just that one phrase. But obviously that one phrase alone would need to be coupled with x?
 
user3956566
so I would go back to that user and ask for input pardon the pun
 
Sam
Oh right
 
user3956566
this is why I've been pushing for phrases all along
 
user3956566
5:51 PM
the nullreference exception, could have two key phrases of regex
 
user3956566
@TheLostMind are you still awake?
 
user3956566
I've gone purely on the success of smokey and how smokey works
 
user3956566
keeping it simple
 
Sam
I've been using regex for years. And if I've learnt anything, it's they have no concept of context.
 
user3956566
that is why we test it on these two post with one dupe target
 
Sam
5:52 PM
Sure, you can add to the regex to catch some cases of inverse meaning, etc.
 
user3956566
to see what we need to get a percentage success rate
 
user3956566
inverse meaning?
 
Sam
How would you do that with a regex?
 
user3956566
nullreferenceexception and all the variants that can make up that phrase
 
user3956566
and object not instantiated bla bla
 
Sam
5:54 PM
@MsYvette "I have this code <blah blah blah> which doesn't throw a nullreference exception, but instead it throws BlahBlahException, why?"
 
user3956566
actually seaching on the errors
 
user3956566
@Sam you are throwing up the hypotheticals I wanted to reveal with a test case
 
user3956566
to see what the results ore
 
user3956566
this is where regex can help
 
Sam
;)
I've had past experience with something similar.
Just saying to be mindful of this.
It does happen.
 
user3956566
5:55 PM
this was the whole point of doing a test run with one dupe target and two post to use for key phrases
 
user3956566
it's a matter of examining the top dupe target and then other highly relevant dupes to find the key phrases most likely used when asking that duplicated question
 
user3956566
the rest is irrelvant, the rest of the post rather
 
user3956566
I had this idea with nullpointerexceptions, except it was too extreme
 
Sam
Extracting the key phrase(s) won't be easy.
 
user3956566
That's not a problem at this stage at all
 
Sam
5:57 PM
Well
 
user3956566
the problem is proof of conept
 
Sam
Anything language related is difficult.
 
user3956566
sorry typos, very tired
 
Sam
lol
np
 
user3956566
I know love
 
user3956566
5:58 PM
that's why we need someone keen and good at regex
 
user3956566
and this is not an overnight project
 
user3956566
if we can test and tweak the speed and accuracy for one dupe target
 
user3956566
and take it from there
 
user3956566
we need to make the same test with regex for speed, as is
 
user3956566
to see how much of this is the post to post causing a lag
 
user3956566
5:59 PM
or whether our bot is too slow generally
 
user3956566
that way we do not exclude the possibility of post to post comparison until we really knoe it's not viable
 
Sam
Benchmarking regexes will be... interesting.
 
user3956566
see if we can produce speed results similar to smoeky on some basic regex phrases, they don't even hve to be intended for accuracy
 
user3956566
but to see if the speed is there to begin with
 
user3956566
@Sam agree
 
user3956566
6:00 PM
what do you think of my overall vision
 
user3956566
it's from one post, to potentially the whole site, but cutting it back to how we begin?
 
user3956566
the first few steps
 
Sam
@MsYvette I'm not sure. I can see potential pit falls and advantages.
But hey
That's why we're testing
If one algo fails, we can make another
If that fails, we can make more, explore other potential ideas, etc.
It's too early to say anything more.
 
user3956566
oh there will be plenty of pitfalls and we cannot foresee them all
 
user3956566
and you never know there may even be advantages we haven't foreseen
 
user3956566
6:03 PM
you see
 
user3956566
a room like C++ and that tag, may not welcome a dupe hunter bot
 
Sam
Indeed
 
user3956566
but to begin with, we can target the most ridiculously high targets with 1000+
 
user3956566
and we can manage those in our room, without being specialists
 
user3956566
you and I alone could work out whether a nullreferenceexception Q is a dupe
 
user3956566
6:05 PM
you don't need a gold badge is what I mean
 
user3956566
where as some of the more unique posts, you do, like Cimmanom and sass, but then there's not so many dupes there by comparison, so we don't need to worry
 
user3956566
it's not about stomping all dupes, just grabbing the excessive ones
 
user3956566
and many of them are throwing errors - so it's easier to find them
 
user3956566
and which is why I don't understand the OPs don't search
 
Sam
Right
 
user3956566
6:07 PM
I think you will find we grab two types of question with this type of algo that don't belong
 
user3956566
one in excess will be code dumps that are off topic anyway
 
user3956566
and some unique and good questions that happen to be getting the error and it's not an obvious cause
 
user3956566
Hi!!!
 
user3956566
we're still testing regex!!!
 
user3956566
bwa hahahahaha
 
6:08 PM
I am now
 
user3956566
I had a feeling we would need you @TheLostMind
 
Yes?.. @MsYvette - go ahead I am listening
 
user3956566
anyway Sam osrry to talk your ear off, I appreciate you listening
 
user3956566
I've been thinking about this for some time
 
Sam
Lol, np
 
user3956566
6:09 PM
months
 
user3956566
@TheLostMind the post to post atm is taking 8 mins per post
 
user3956566
so I want to mock up some regex to test
 
user3956566
I will give you the two phrases
 
user3956566
hang on I will open VS
 
user3956566
6:10 PM
what I need the regex for is:
 
user3956566
the exception 'nullreferenceexception'
 
user3956566
can be worded in a cariety of ways
 
user3956566
as one word
 
user3956566
separate words and with typos
 
So basically null reference exception or nullreferenceexception or nullreference exception
 
user3956566
6:11 PM
so there are two phrases we want to do a test search on
 
user3956566
wait I will get them
 
user3956566
> "Object reference not set to an instance of an object"
 
user3956566
adn
 
user3956566
> "NullReferenceException"
 
user3956566
6:12 PM
so what we want to do right now is a speed check against the post to post algo
 
user3956566
so accuracy is not important at this point
 
user3956566
it's to catch phrases, like smokey and also all under the tag c#
 
user3956566
@Sam so we need to limit the search to that tag for now?
 
Sam
For the regex-based algo, sure.
 
user3956566
there will be plenty of quetions incorrectly tagged, they're unlikely to get passsed the unclear vote
 
user3956566
6:14 PM
so we're not trying to catch 'crappy ' questions, just dupe of this target
 
Searching for NullReferenceException in a string is fairly simple
 
user3956566
@TheLostMind in keyphrases.cs that is the class where I wanted to store our regex for comparisons
 
Sam
I'm going to test the nlp-based one for all tags, since I need to get the performance to something decent.
 
user3956566
thanks Sam :D
 
user3956566
xoxox
 
6:15 PM
@Sam - Go Sam! :P
 
Sam
:p
 
user3956566
I just talked to Sam at length about my idea
 
Sam
Also
I'm going to put up a new model file.
The old one is inaccurate.
 
user3956566
ok
 
user3956566
@TheLostMind are you cool with those directions?
 
6:17 PM
@Sam - "
Benchmarking regexes will be... interesting." .. I used 8 regexes aainst a 1million useragents today.. took 5 seconds on 1 core.. not bad for performance :P
 
user3956566
if you need to read from here
 
Sam
@TheLostMind Nice
 
@MsYvette - on my way.. gimme 5 mins
 
user3956566
@TheLostMind well smoke detector covers the whole site with it;s helper bot,
 
user3956566
@TheLostMind oh no! I don't mean to rush you I was meaning do you have Qs
 
6:18 PM
@MsYvette - I do.. will ask you once I understand your comments completely
 
user3956566
@TheLostMind ah k
 
@MsYvette - 2 min more... :(
 
user3956566
@TheLostMind I'm around, making coffee
 
I might need some as well..
So yes.. I read through your comments ..
 
user3956566
and?
 
6:25 PM
OK.. we find all questions with NullReferenceException and say its dup of some old highly voted NRE question..
 
user3956566
yes
 
Ada started.
I found a message: 'Code not executing when condition is accomplished' on stackoverflow.com
 
user3956566
but what the bot does is come into chat like smoke detecotr, possible dupe of x and the post link
 
Ada started.
 
I am just thinking how to efficiently search for ""Object reference not set to an instance of an object""
 
user3956566
6:26 PM
accuracy is something we will tweak depending on results. I think it won't be hard on this particulat error
 
user3956566
> "not set to an instance of an object" "Object reference Exception"
/// "Object reference not set to an instance of an object"
 
Ada started.
 
But yes.. it can be done.. So, I wil have to go through the entire post and look for phrases like the 2 we are talking about
 
I found a message: 'Qualtrics: How to easily insert many different images into a repeating set of questions?' on stackoverflow.com
 
user3956566
No I have goen through the post for you
 
user3956566
6:28 PM
you don't need to go through the posts, unless you want to, at this stage I will be doing that
 
Then what are we trying to match against?
the title of the posts?
or some test text?
 
user3956566
the two phrases here
 
user3956566
against both the title and body of the posts
 
user3956566
just liek smoke detector
 
user3956566
we will grab posts under and use your regex to compare them
 
user3956566
6:30 PM
if there is a match that post will be reported into this chat room as potential dupe
 
OK.. hold on I will open a c-sharp regex editor online
 
user3956566
:D
 
user3956566
@Sam so the idea of my KeyPhrases class was to have a list of regex and tags and then the bot searches on this combination
 
Sam
Ok
 
user3956566
doh java tag
 
Sam
6:33 PM
But shouldn't that be moved to the matchers dir?
 
user3956566
yes
 
Sam
And implement IMatcher
 
user3956566
I'll do that now
 
Sam
ok
 
user3956566
ok, I'm sccared to run and do anything without the ok :D
 
Sam
6:34 PM
lol
 
user3956566
we need a meme for merge conflcits and my fear of :D
 
The phrase "NullReferenceException" .. does it usually come at the beginning or the end (in the posts you've seen / stacktracce)?
 
user3956566
push
 
user3956566
hm varies
 
user3956566
they will often post a quote with that phrase
 
user3956566
6:36 PM
in a stacktrace
 
can't bank on that.. could lead to catastrophic backtracking :P
oh.. i see.. Ok.. will use a reluctant quantifier as of now.. will see if we need to change it later
 
user3956566
with VS the program will break, and there will be a pop up with the exception, that the user can copy from
 
user3956566
so stack traces as such may not be posted, like android
 
yes.. I know that :)
ok
Can I have more phrases @MsYvette to test with?..
 
user3956566
@TheLostMind for these posts?
 
6:47 PM
Well right now .*?NullReferenceException.* and .*?Object\\s+reference\\s+not\\s+set\\s+to\\s+an\\s+instance.* are the only ones I can think of.. A compiled regex should work ok
with case insensitive flag (?i) in java
 
Ada started.
I found a message: 'Wordpress - post offset jumbling posts' on stackoverflow.com
 
RegexOptions.IgnoreCase flag in C#
 
user3956566
@TheLostMind yep case insensitive
 
Sam
@TheLostMind (?i) also works in C#
 
user3956566
yes c# LOL
 
user3956566
6:48 PM
you are asking?
 
user3956566
oh that is Sam.. I though you were The Lost Mind as well
 
user3956566
I am tired
 
Cool.. Yes. use (?i) with the regexes.
 
Sam
Lol
 
lol
 
user3956566
6:49 PM
I saw @TheLostMind and assumed it was him talking 0>o
 
Sam
5.87 seconds!
From 8 mins 15 secs.
 
user3956566
really!!!!
 
user3956566
EXCELLENT
 
Sam
Yus
 
Awesome :)
 
user3956566
6:50 PM
is that with tags
 
Sam
Nope xD
 
user3956566
@TheLostMind do not leave
 
Sam
With tag filtering it'll be quicker.
 
lol
 
user3956566
what did you do and why isn't the code pushed mister
 
Sam
6:50 PM
afk for a sec
 
user3956566
he makes a million % improvement and goes off k
 
@MsYvette - Youve a regex expert in Sam here :P
 
user3956566
@TheLostMind he doesn't think so
 
user3956566
and he's a brilliant mind
 
That doesnt matter miss :) -- > I meant the fact that he doesnt think so
 
user3956566
6:51 PM
he is regex resistant.. I need someone regex happy
 
user3956566
trust me
 
user3956566
sob
 
lol. I dont blame him..
You need to get some rest.. Seriously.
 
Sam
And back
 
user3956566
I am just exhausted but want to do things
 
Sam
6:52 PM
@MsYvette Nature calls...
 
user3956566
ok you make a zillion % improvement and then leave and odn't tell us what you did
 
Sam
Lol
 
user3956566
@Sam too much information!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! LOLOL
 
Sam
I needed to make sure it worked
Before pushing
 
user3956566
are you pushing this????????????????????????????????????????????????????????????????????????????‌​?????????????????????????????????????????????????????????????????????????????????‌​????????????????????????????????????????
 
Sam
6:53 PM
I will be
 
user3956566
That;s my silly overtired typing
 
user3956566
do you get so tired, it's like yr drunk
 
Sam
Mem usage has risen, which is the trade off between that and speed, so.
 
user3956566
wth difference did you make
 
user3956566
so we can only ever check one post lololol
 
Sam
6:55 PM
Ok, so the main performance hit was reconstructing posts from the format in which I was originally storing them.
 
user3956566
bloody bots
 
user3956566
ah
 
user3956566
where'd he go
 
Sam
I now store an "extra" easier-to-process collection along side the original collection.
 
user3956566
k
 
Sam
6:56 PM
It's still not as fast as I hoped.
I'll push what I've got now.
 
user3956566
it's better than 8 minutes
 
Sam
Then get back to optimising.
 
user3956566
k
 
Sam
Yes :D
 
user3956566
and we will still do phrases
 
Sam
6:57 PM
Yeah
 
user3956566
and we will have a race
 
user3956566
who will win at the finish line
 
Sam
Also, goodnews
 
user3956566
ttytt it would be good to have options
 
Sam
The model file is smaller
 
user3956566
6:57 PM
yes I love good news!!!!
 
user3956566
ah yes
 
Sam
by 7 MiB
 
user3956566
nice and so we add that to bin, as a raw file file not within it's folder?
 
user3956566
why do we throw all the stuff we need in the bin
 
Sam
Yeah, just place it in the same dir as settings.txt
lol
 
user3956566
6:59 PM
yep
 
user3956566
programming, the only place where you keep the things you want in the bin
 

« first day (399 days earlier)      last day (2996 days later) »