last day (2900 days later) » 
14:00 - 21:0021:00 - 00:00

9:00 PM
@Drew Then just launch the bot, I'm only having some fun
I will continue to have some fun since I want to find vote count on possibile duplicate target to display in file
 
6 hours ago, by Petter Friberg
Drew I'm not really worried about the database and to tell the truth I hope that it the future it can be avoided (hence SO gives us possibility to filter on cv and flags).
^ no DB, and a change in the API. Is that the suggestion?
 
that's a possible convenient future
but, right now, we don't need any change
A db can store gazillions of questions
 
A user makes a request, and we have no DB, and the API changes "feature requests" are going to handle that?
Is that what was said 6 hours ago?
 
Nope, I'm saying the DB is no worry, since I can't see the problem (lets see if I hit a wall) and I like the bot to question the DB only for 100 question_id, then go through api, if Shog9 gives us a hand we can avoid the DB and the bot can go directly to api
 
yep ^
each step at its time
 
9:04 PM
so if Shog gives us a hand and changes the API?
is that what you are saying Petter?
 
we can't rely on the answer of that question today
 
@Drew I doubt that.
 
I am trying to get some clarity from Petter
On one hand he says all the info is available in the API.
The above says, hey maybe it isn't
 
nope
those 2 things are not contradictory
Look at it this way:
We're today. The plan is to have a nice bot that 1. dumps to chat possible dupes live 2. queries the API and stores info in a DB about close vote count + tag
Now we extrapolate
This works fine for a couple of month
 
no I'm saying we will use db for now (and its easy do fill it, tomorrow I can fill one and you can see), but the bot should not rely on this info, its should go by api... So when so get tired of us calling the api to fill a db they implement a filter.
 
9:07 PM
Then a CM comes in and says "Hey that's really great, let's change the API"
 
you guys are smoking some serious wacky weed :p
 
So they do and we change our bot to drop the DB and query live
 
then we can stop filling DB with questions!
 
To me, this is the plan. Both short term and long term.
 
lol
 
9:09 PM
But anyway, if that doesn't happen
 
@Drew What is wacky weed?, problem with filling a db, SO will not consider to implement a filter in version 3 of api...
 
Sam
And I'm back.
Can I have a TL;DR?
 
we'll still have a nice bot that 1. dumps to chat possible dupes live 2. queries the API and stores info in a DB about close vote count + tag
@Sam Yeah read my last messages :) from here chat.stackoverflow.com/transcript/message/30412324#30412324
 
Where is the discussion between Gothdo and Shog9?
 
Sam
Thanks :)
 
The truth is that they do not really need to change api, they only need to implement a filter...
 
and even if they don't, we have a working thing. Bullet-proof and kosher.
 
@PetterFriberg Which effectively means changing the API.
 
you will never get API changes. Forget that
 
well I do not really care about this now, if they like us to query the api 2000 times a day since we can not filter we will do that
 
9:12 PM
^ that exactly
and it is explicitely allowed to do that since you just need to make sure you don't go over your #requests quota
 
a few users blew Tiny's quota daily btw
 
So let's take the numbers
 
Sam
We have 10k reqs
 
what's the quota?
 
well the api quota for the db filler is no problem I have already done the math
 
Sam
9:14 PM
10k
 
per day?
 
Sam
Yes
 
1 tag one day is 10 request
 
really
maybe turtle
 
You just need to implement a good strategy I can do that and again what is important is that since bot goes through api, the db does not need to be perfectly updated
 
Sam
9:15 PM
@PetterFriberg That's 1k Qs?
 
Yeah java have about 1K question a day
 
Sam
Right
 
Petter, I have been doing this for quite a while. Not 2 days
 
So 10k per day is more than enough
 
Sam
This is peanuts, we have plenty of requests
 
9:16 PM
Yeah and the bot needs to do 2 api request for every user that wants a batch
 
Sam
Even at 20 reqs per tag.
 
put the bot will have another api key
 
so this is not an issue
 
so what is the issue @Drew?
 
9:18 PM
the issue is that you guys need to catch up on the development side of it to see it in practice
 
Tomorrow I can do some testing, we just need to find the correct strategy (at what time is the possibile duplicate arriving)
Ok tomorrow I will catch up and put some data on-line
 
fair enough
 
Track 20 tags with highest volume. And offer timely updates to that data
Get that into a DB
 
Sure, my target is 40
 
ok do the top 40. DB them all. And when a request comes in, have that data not more than 1 hour stale delivered to them in 10 seconds
 
9:20 PM
DB can store billions and billions of question. Even the whole 10m is nothing for it.
 
We do not store every question!
only cv and possibile dupe question
 
track 20k questions in a DB
 
Can't see the problem
 
I don't see why this is difficult
 
and know that you will not be tracking all duplicate candidates
show us how that progresses
 
9:22 PM
?, well since I'm having fun I will try it and download the code so you can see
 
Sam
Pessimism will get us nowhere.
 
who is being pessimistic?
 
Sam
I'm just saying.
 
@Drew what was the specific performance issue you had?
 
I am just confused by Petter's comment about "1 tag one day is 10 request"
And wondering what he is talking about
 
9:25 PM
1 tag as java has 1000 question, 100 questions per request gives 10 request
 
so you are ok with 1 day stale data
 
The strategy to update db, is querying often on latest 1000 questions (to get new data quickly), querying less often on older data and then updating questions already in database (querying directly on those question, if they have not been cleared by bot)
 
Java has a pool of 14000 questions that then slide out of visibilty and age away
 
With 10K api request I can not see the problem, the system really only needs to be tuned
 
so the data is stale
 
9:27 PM
I only care about the last 20 days (there are no cv left on older questions)
 
Sam
Early optimisation is the root of all evil.
 
Why stale?
 
Sam
We don't need to worry about maximising data freshness atm.
 
1 tag one day is 10 requests
that is stale
 
With 10k requests, you can query all last <5 days questions every 5 minutes probably
 
9:27 PM
I do not care that db is not perfectly updated since bot wilò query on 100 question, get live data and serve the best 10
@Tunaki There is no really need to do this
 
Like I said, play with it and see how it progesses
Some call that pessimism I call it keep us informed
 
Do we really care about a new cv vote (it did not have cv vote) on a question 5 days ago, every 5 min?... no we just need to catch in maybe 2-3 hours
 
we can calculate that
1 request to the API returns 100 questions
 
Exactly we need to tune the strategy for quering the api, I wan't to catch possibile duplicate fast.
@Tunaki correct
 
so let's say we focus on the last X days
 
Sam
9:30 PM
3 mins ago, by Sam
Early optimisation is the root of all evil.
 
@Tunaki and you can also query on 100 question_ids in on api call
 
Sam
Get the bot "working" first, then we can worry about performance.
 
yup
but I think we can get down to some serious real-time here
 
I can setup the service that query for data, so we get some and understands when close vote and possibile duplicate happens, if I have time I will do it tomorrow
 
of course, the more "live" the data is, the shorter the window of tracked questions is
 
9:32 PM
What I want is that the one who develops the bot implement the api call
The DB is just because of the filter problem
 
no that is not true
you need a DB regardless
 
In the project I have download you have some brutal code for api call, tonight I will download also api call on questions_id
 
@Drew Is stale data for 2-3 hours a concern?
 
not for me. For dupe hunters sure. 5 answers sneak thru
then again I don't care that they sneak thru. A guy was helped
 
bah, they can answer after the question is closed anyway. It doesn't matter if it's closed or not for those guys.
 
9:36 PM
This is why you need correct strategy for dup hunter (query latest 100 question in tag more often), I would bet the problem is with these
 
you need to scan them frequently not just once. The dupe call takes a while
the dupe call meaning the guy calling it out
 
Immagine that I query 40 tags every 2 min (last 100 question) how many calls are this?
 
there are 720 2minute segments per day
 
A bit too much Petter, that's 29k calls per day
 
make it 5 minutes and I'm home free
 
9:41 PM
pretty much
 
except you can't update the actual questions then
 
what do you mean?
 
all you are doing is getting last 100.
 
Sam
@PetterFriberg 5 mins is perfect
 
What does that leave of the quota for updates
 
9:42 PM
actually you an also have different service handling different tags
 
Java has 14000 questions you will track. If all you ever do is blow your quota on the last 100, you never update squat
 
I can do some counting if you like but I'm pretty sure I can find a strategy to serve a bot with 100 farily good question,
 
I thought it was just decided that if you scan the last 100 per tag every 5 minutes you just blew your quota
You have nothing left to actually update anything
 
Remember maybe only 1 or 2 times a day you query the last 5-10 days, to find new dupes and cv, the rest of the queries are targeted on questions in db and to update latest cv count. Then we can do only 25 heavy traffic tags so bit it (since we want updated info every 5 min, or we update every 10 min)...
 
Sam
...
 
9:50 PM
so a person can't use this system for show me cv 3 or 4 and have any decent results
 
Sam
Can we please not get bogged down with formulating exact numbers here?
 
Its all about making the best of 10K queries to have farily good info
@Drew Remeber that DB returns 100 questions to bot, bot query api to get live result and present the 10-20 best one!
 
what is the API call for the last 100 questions
 
@Sam Tomorrow I can put down some number, then with experience we can tune the api cals
 
or rather most recent
 
Sam
9:53 PM
@PetterFriberg Ok
 
that's by activity but you can change sort
 
/questions/unanswered/my-tags
that one
As I mentioned to Sam, I don't use the API call for that one. It blows the quota
Thus my meta question about pagination. Which exposed a bug
Which brings up screen scraping. Which some think is never needed
 
well tomorrow I will put down a document about query strategy to have best result
 
And the onion gets unraveled and unraveled
I would code away for a while to get best result. Just sayin
Nothing beats experience but doing it.
Seeing time difference and aging
Frequency
 
I'm pretty confident that with 10K api request (1.000.000 question scans) the db can be fairly updated. I maybe wrong, so I will just test it
 
10:00 PM
cool beans. Good luck.
 
thanks
 
the heck
I'm gone 5 minutes and it's goes hayway
Make it 10 minutes check, it's good enough
that leaves plently of calls for the rest
I don't see the fundamental issue here.
 
screen scrape page 1 for each tag. Trust me.
save your quota for what matters
 
Make it 30 tags, make it the only 10 tags have 1000 questions a day, make it different service querying different tags...
 
well, if we hit a wall with the API, we'll improvise
but until then, let's stick to what the system gives us
 
Sam
10:04 PM
(Did some more work on the readme. Still gotta finish off the bottom of the second section.)
afk
 
btw
Smokey does it pretty well and it uses the API purely AFAIK
it doesn't scrape anything
and there are a lot of answers and edits that bump questions each day
so if all else fails, we can ask how ProgramFOX did it
 
note also that we can query api with out tag and get only last 100 question (in one query) make it that we like 200 question, then we can do it every 1 min
I do not know when the possibile duplicate occurs in what time frame, I guess 90% within the first 30 min but need's some data to check
Its just about the correct strategy..
 
but those are tags you dont care about
 
How cares, just skip them, if your target is the 40 high traffic tags, I would guess that 90% of questions are in one of those tags.
 
point taken
which API call returns comments (strings)
 
10:08 PM
Then one I sent to you before...
just set the filter, if you have download my project you can se the api filter
 
never had that in my filter
cv count is all i care about
 
yeah, but as you know I want the possibile dupes also and that's my only way
 
that is why I asked as a guy can say "this is a dupe of url"
 
we will miss it and who cares, we will get the other ones...
normally people flag or close vote
 
well if you are going after comments, why not pick up on a few variations
 
10:13 PM
sure but that's fine tuning.., lets get it up an running first : )
just for info from what I have seen the possibile dup happens fairly quickly with 1 hour, but I will collect some data on this.
 
the damage is done by then
 
The comment was intended as scan strategy (hence only last 1h of question or maybe last 30 min) need to be scanned to find them quickly
 
Sam
@Drew So what? Users can still post answers to a Q up till 2 hours after closure.
 
^ yeah
 
I think we can alert a dupe hammer within 5 minutes if the comment is passed on a question in his tag... but I guess I need to prove it
 
Sam
10:31 PM
Wait
Has this whole discussion been about warning users as soon as possible about dupes?
(@Petter)
 
I do not know, I guess its about whether you can do this by using api or not.
if 10K api request is enough....
 
Sam
I know just the thing. Websockets.
Use a websocket to listen in for when a new Q is posted. Then get the Q's data via the api.
 
it would be nice if you could listen to new comments :p
 
Sam
You can
On a per-question basis.
 
: ) I think we better just do are best querying 1.000.000 questions every day...
 
10:37 PM
yeah but I'm guessing there's a limit on the number of questions that you can bind to
 
Sam
Yam's doing fine at 400+ Qs.
 
bah that's perfect then
 
I guess you will need 10.000 Qs (listening to comments) and we need to query api anyway to scan for cv count
 
it's a mix of both
 
sounds fun
 
Sam
10:41 PM
Using websockets is just an idea.
 
and it's fun to code :p
 
sure, I will test tomorrow, put up some temporary api strategy that we can discuss.
 
^ that's an example of creating a websocket in Java @Petter
 
Sam
Man, that's so ugly.
 
:(
 
10:44 PM
Nice, but can we really attach one to each incoming question for 1h
 
it will be roughly 2000 questions?
 
Man, that's so ugly.
 
Sam
We'll have to experiment :)
 
Look at those braces. Hugh. Blargh. Ewwwwwwww
 
10:47 PM
lol, I love it all : )
 
Sam
@Tunaki Allman style for life.
FOR LYFE
 
It's ugly :p
 
Sam
let the battle begin
 
Hello @Laurel o/
 
Hey @Tunaki
 
Sam
10:51 PM
Hiya
 
I found two bad answers by the same person on the same question posted within 5 minutes of each other. I flagged a mod, but I figure I should also gather a mob. Here's one: stackoverflow.com/a/37105581/6083675
 
Sam
Wat
 
I think you hit the wrong room. You probably want chat.stackoverflow.com/rooms/41570/so-close-vote-reviewers but we're not a mob :)
 
Yes thanks.
 
11:11 PM
Bed time for me, if we actually hit an api quota problem (I think with correct strategy of searching we will not) another solution is to use data.stackexchange.com for question older then a week (hence not scanning these by api.)
 
there are a lot of solutions
let's make a simple thing first and build on that :)
 
Sam
Yep ^
 
right, lets sleep on it : )
 
14:00 - 21:0021:00 - 00:00

  last day (2900 days later) »