last day (22 days later) » 

5:04 PM
20
A: Triage: How to vote if more information is needed

Shog9The goal for Should Be Improved: a third path for lackluster questions This observation hits on the intention for Should Be Improved: Without this information it is not impossible, but probably rather difficult for others to come up with a good answer. There are a lot of questions that can...

 
The new text is far better than the previous one.
 
So if a question needs additional information from the author, and the missing information is information that cannot be edited in by anyone else, how does choosing NI work to accomplish that? What are the reviewers in the NI queue expected to be doing? How is the author going to know what is expected of them? How does the post leave that queue after the author does fix the post?
 
You can leave a comment if you want, @Servy. Eventually, we'll be feeding these into another queue where reviewers will be asked to provide either edits or guidance; I'll elaborate on that in a future post. Most importantly though, there's no shortage of information already available on how to ask... If the author wishes more people to see his question.
 
@Shog9 I'm simply curious on how the actual NI queue, that posts marked as such get fed into, should handle these posts. Assuming users in that queue are given options to mark as post as OK, edit, comment, and flag, what should they do when a post doesn't have enough information? They can comment, but then what? If they then mark it as OK it'd go to the front page without enough information. If they vote to close as unclear then there's the issues you've mentioned here. They'd almost need a button specifically for "wait for author to edit" to press after commenting.
 
Why on earth would we give folks in such a queue the option to mark a question as OK, @Servy? At that juncture, either the post has problems that need to be corrected, or is confusing enough to convince three previous reviewers that it had such problems - either way, someone needs to edit it. I'm actually leaning toward "edit" as the only action that does anything...
 
5:04 PM
Why on earth would we give folks in such a queue the option to mark a question as OK? If there was an out of queue edit, that or if you put the post in front of people after the first edit to see if that one edit was enough, this means that OK is really, "we've already done enough" to distinguish it from, "we still need to do more". So that brings us right back to the original question. If the only option is "edit", what should that person do if the post needs an edit from the author? There isn't any edit that they can meaningfully make.
 
Nothing, @Servy. That would have to apply to any question you didn't think you had a good shot at salvaging.
 
@Shog9 So the in Triage queue items that can only be edited by the author shouldn't be marked as NI, because there's nothing that the users handling NI posts can do about it?
 
user3717023
"must therefore be removed from the site" will make some folks hesitate when they recognize a decent, but duplicate question. How about "must therefore be closed or deleted"?
 
I would consider adding a comfortable option like "Unlikely To be Improved", with sole purpose to walk away from three-point scale, as was argued in this awesome analysis (internally that option could mean the same as Unsalvageable)
 
If you're 100% positive that a given question is unanswerable without input from the author, you should vote to close it @Servy. Realistically though, I don't expect that much care from folks in Triage - in my experience, good questions missing one key piece of information are rare, while awful questions with too much information (most of it irrelevant to the problem) are common. If a handful of unanswerable questions end up in NI (and then get deleted because no one can answer them) along with thousands of technically-answerable but painful-to-read questions, no harm done.
 
5:04 PM
@Shog9 It still makes sense to have the button descriptions match that though. NI really is for questions that the community can fix without the author. Unsalvageable is for posts that either can't be fixed at all, or where only the post author can fix them. As it is, the directions are incorrectly stating that posts that only the author can fix should be NI.
 
Not overly concerned about folks waffling between Unsalvageable and NI, @gnat. Bigger concern is too much stuff landing on the Looks OK side. Another button might help there, but I'm skeptical: that's a good answer, but this isn't a survey; past experience has been that folks' initial responses vary quite a bit over time - we can't play tricks to compensate for new reviewers without skewing the results from old ones... Or making this really hard to understand/predict for folks like yourself.
Nope, @Servy. Expecting reviewers to determine (in roughly 20 seconds) what "the community" can/can't fix is ludicrous. That's a decision to be made, but not here; there are other tools and systems for that.
 
@Shog9 Then the NI queue needs an option to close questions that it can't fix; it can't just have an option to edit as the only thing to be done, if you're expecting people to actively push questions into that queue that cannot be fixed by the community.
 
well your reasoning sounds compelling, especially in the light of updated explanations for triage categories. Only thing that keeps me somewhat skeptical about current approach is a past failure to involve more contributors in a similar close votes review (which was also totally logically structured and presented)
 
@Servy The flow chart in the question that introduced the triage queue showed a "VLQ flag" path out of the improvements queue to another triage review. Remains to be seen what will be implemented, but that's a good sign that you're not alone in this concern.
 
New text looks good. I am not sure I like the "should not" part but you said you aren't worried about too many of those anyway. As for the "Looks Okay" sentence, if that is of concern, perhaps it could include "without any editing" on the end of it. Looks OK for questions that can be found, understood and answered as-is without any editing
 
5:04 PM
Why, exactly? We don't need to build every tool imaginable into every specialized review queue - you can't vote to close from the suggested edits review or Late Answers queues either, even though it's possible to find posts in both that need it. We're not doing anything to limit the ability to vote/flag for closure outside of review, @Servy - problems can still be handled in the usual way.
 
@Shog9 So then what's the expected workflow for a question that cannot be answered without additional information from the author? You're proposing that such questions go to NI, not Unsalvageable. From the NI queue the only action available will be to edit the question. So now we have a question that only the author can edit, and the only possible action to edit the post. What happens then? Everyone just skips the post until it ages out of the queue?
One of two things needs to happen. Posts that only the author can salvage need to be marked as unsalvageable instead of NI, or the NI queue needs to have some tools to deal with posts that only the author can salvage. Either is fine, but actively pushing posts to a queue and then not giving them the tools to solve the problem that those posts have is problematic.
 
I need to just take a bit of time and post my proposal for NI, @Servy... Along with some numbers. You're worried about posts that no one can help, but there will be thousands of questions in NI at any given time, the vast majority of which will require a fair bit of work to fix. This can't be an efficient assembly-line operation; it's Mother Teresa's leper colony. You're hand-wringing about the possibility that lepers might die. Think about that.
 
@Shog9 Isn't the whole point of Triage to make sure that by the time posts get to NI we can be pretty certain that we can help them? The input to Triage is the leper colony, the NI queue is the people that get into the hospital because they're both sick and salvageable. But you're right, the system won't be perfect; even if people shouldn't push unsalvageable content into NI wouldn't the queue need some means of saying "sorry, triage was wrong, we can't fix this, it's too far gone"?
I'm bringing this up now, before your proposal for NI, because the resolution to this problem may (or may not) be to change triage, not NI. The longer we wait to make changes in what types of posts get put where in Triage, the harder it will be to change it later.
hello
 
@Servy so here's the problem - there are (on Stack Overflow, at least) a lot of questions that in theory could be good, answerable, useful questions.
They just need a good edit. From the author, from someone else, from both...
But...
There are more such questions than there are folks who can/will edit them.
 
So, in all honesty, I'm actually rather okay with pushing questions that require clarification from the author into NI, I just want to make sure NI will have the tools to deal with them, because it sounds like it probably won't. I'm much rather see NI get the tools to handle them than any other resolution though
 
5:09 PM
To be clear, there are also more of these questions than there are folks who can/will close them.
So right now, they just sorta... clog the tubes.
 
agreed
 
On a well-run site that gets... Let's say <=40 questions/day, this isn't a problem. It only takes a handful of people to process these questions.
Once a site grows much beyond that, the overhead of just processing that many questions starts to eat into how much time anyone can spend fixing / closing them.
 
I mentioned this in an earlier comment, but on way for someone to "handle" an item in the NI queue, when the problem is that it needs clarification from the author, is to have some action along the lines of, "post a clarifying comment, and then wait for the OP (or someone else?) to edit the post; the post will be paused from the queue, not feeding anyone, until an edit reactivates it, allowing it to be either edited again, closed, or fed to the homepage
an "unclear" close flag could be applied after some period of time if it's not edited
if there is some tool, whether it's that or something else, that allows people in the NI queue to handle items pushed in requiring clarification, then that could work out really well
 
Don't even need that. Question languishes for x days in NI without any edits? Just delete it.
 
That means that the question is fed to lots of reviewers, all of which have to skip it
and since there are a lot of questions needing clarification, as a NI reviewer I could end up having to skip tons and tons of questions to find one I can edit
 
5:14 PM
So, in the <6 days since we rolled out Triage, 4297 questions have been marked as NI.
 
if most of the actually salvageable posts in the queue get handled quickly, it could become a real problem
 
Current stats:
Name               Upvoted Downvoted Answered Deleted Closed  ClosedOrDeleted
------------------ ------- --------- -------- ------- ------- ---------------
Looks Good         33.10 % 10.62 %   35.27 %  4.61 %  5.28 %  9.69 %
Should Be Improved 6.89 %  39.75 %   23.37 %  6.45 %  18.32 % 23.09 %
Unsalvageable      1.20 %  80.36 %   7.43 %   19.76 % 65.51 % 73.65 %
 
interesting stats
almost all of it looks quite promising as far as the triage reviewers are concerned
 
Name               Upvoted Downvoted Answered Deleted Closed ClosedOrDeleted
------------------ ------- --------- -------- ------- ------ ---------------
Looks Good         1035    333       1103     144     165    303
Should Be Improved 296     1708      1004     277     787    992
Unsalvageable      10      671       62       165     547    615
^^^ Raw numbers
 
it's a bit to sad to see so many problematic posts getting answers regardless
but I realize most people don't share my view that bad questions shouldn't be answered.
 
5:16 PM
@Servy keep in mind, we're not yet doing anything to hide these
 
yep
 
Taking them off of the front page should put a noticeable dent in that
 
agreed
 
And hopefully give a bit of a boost to the Looks OK group
But let's say the numbers don't change.
 
like I said, it gives me a lot of faith in the reviewers, that complaint was on SO users in general
 
5:18 PM
Let's look only at questions triaged as NI, that haven't been upvoted or answered or deleted or closed. 4297-296-1004-992=2005 (yes, would be somewhat higher since those groups overlap a bit, ignore for now)
2005/6 == 334 questions per day
Let's say we only keep stuff in NI / helper queue for 2 days.
If no one reviews, that's over 600 questions in the queue at any point in time.
 
so what if we just do something with skips other than have the user ignore the post?
if X/Y people skip a post, then we have experimental evidence that the post is not salvegeable
or at least, that the NI queue isn't capable of salvaging it
 
Sure. We could kick it out of the queue faster, or deemphasize it.
 
so rather than having a question getting continually bounced around
 
Or as you suggested, we could have a "comment" button that... didn't give you credit for the review unless the asker edited; that's messy, but might be feasible.
 
So the idea of having you comment is really a way of saying, "defer reviewing this post until later" where later is either when the OP replies to the comment or edits
 
5:23 PM
Or we could do nothing. With a queue that large, and editing the only available action, stuff is gonna drop out faster via aging than via reviews.
Because... There's another problem.
Questions where someone other than the asker could fix it.
If they know the topic. And they have the desire to devote the time to it.
 
It'd just be nice for people in the queue to have some way of indicating the difference between, "this posts looks fixable, I just don't think I can do it" verses, "this post doesn't look fixable by the community"
if the difference between those options could simply adjust the items priority in the queue, perhaps?
The less effective means we have for conveying this, as mentioned earlier, is skips. A few means the former, a lot means the latter
 
@Servy the problem is, that's still a really hard decision to ask someone to make. If I know the topic well, I can usually make that call - when I review close votes, I filter by tag and it's reasonably easy. Until I hit a question about a library I've never used. Then I gotta skip - I just don't have enough information to decide if the question is really lacking or if someone with sufficient knowledge of the topic could understand it easily.
That's why I excluded answered questions above, btw: in that case, it probably makes more sense to just pop up a message in front of the person answering & ask them to fix the question.
They're almost certainly better equipped to do so than someone trudging through review
If the options are: "fix it" and "skip", then I have one very simple decision to make: can I fix it?
 
I'm just trying to think a bit on the macro scale
The whole goal of the queue here, after all, is to do a better job of helping a reviewer find posts that they can fix than they could without the queue
 
If you add another option, suddenly I have a decision tree:
can I fix it? -> yes -> fix it
can someone else fix it -> yes -> skip
can the author fix it -> yes -> explain how
close
Which assumes that folks don't just pick the low-effort option... Which a lot of them do/will
 
If this queue gets to the point where I'm skipping more questions than I'd skip over when looking through, say, /questions for a tag I'm active in, then I'm better off not using the queue
 
5:32 PM
Closing isn't low effort. Editing is extremely high effort. "Someone else fix this" is very low effort.
 
agreed
it could not count as a review
I'd be fine with that
 
That works in our favor for Triage. Which is why we built it that way.
But... It hasn't worked out well anywhere else.
@Servy exactly.
A lot of people aren't willing to edit. A lot more people probably shouldn't edit.
We need this queue to self-select for people who both want to edit and are good at it.
 
Off-topic, but since you're here... What's up with this?
 
...
you been keeping up with some of the other back and forth's George and I have had?
on other posts?
Short version, there were a few actions that I've seen George take that look like just innocent mistakes
things that I can really see a mod doing before looking closely at a post
but where it's clear, upon a deeper explanation, that the action isn't quite right
 
@Servy yes
Starting to think you two are a married couple IRL
 
5:40 PM
So I sent two emails to the community team
last I checked, I haven't gotten a reply from my last email
not sure if it's you responding
there was no username in the signature
 
I'll check; we have a massive backlog in team@ right now
 
figured
The original ticket went in in the 26th, I got a response 7 days ago, and replied 5 days ago
I was expecting a reply sometime later this week, ish
It was a long reply...
 
500 tickets in the queue right now; one sec...
 
want me to post it here?
not sure if it's too long...
The message was sent Dec 4th 9:42 CST, if that helps you
 
@Servy found it
 
5:49 PM
k
 
There's also another one from a user complaining about you.
But I shouldn't talk about that.
 
I'm not surprised
or bothered, for that matter
 
 
2 hours later…
7:24 PM
@Servy I've written a rather long response, and I've realized that I need to send it to George as well. Do you mind if I just copy him on the response?
 
that's okay
 
If you'd rather keep your previous messages private, I'll just send it separately.
 
they don't need to stay private
 
8:21 PM
@Servy sent
 
so a few points to that email
one is that the community did delete the post. The mod declined the flag, but the community deleted it
next, while you're quite right that we both feel that the question is problematic, the issue is how to go about dealing with a problematic question
I'm asserting that when a question is lacking sufficient information to be answerable, one should comment indicating that it is lacking information, and possibly VTC. George is saying that it's fine to post an answer saying that the question doesn't have enough information.
one of the core principles of SO is that when you go to SO you find an answer, not a post telling you that the question doesn't contain enough information; the actual answers are completely separated out from everything else used to get to an actual answer.
It's the whole reason NAA exists.
You're basically saying that NAA flags as a whole are pointless and not worth enforcing
 
@Servy so... There's actually a lot of history there. The question went through LQ review twice, and got in total 3 NAA flags. Along with another flag.
Between the first and last review, the question was closed, a lot of downvotes showed up, and there were some comments left.
 
8:36 PM
true, and that's partly why I could have understood the declination at first as simply being a mistake; a post with a bunch of code and an explanation looks a lot like an answer at first glance. its only when you put it in context that you can see that it's not actually answering the question.
Once some comments were added to explain this context, and reviewers in the queue had that additional information, they understood the flags and deleted the post
normally this would be a case where I'd say a custom flag would be nice, as it'd add context beyond a normal flag
 
@Servy No, I'm saying NAA on a Not-question (sometimes I really miss "Not a Real Question" as a close reason...) is par for the course.
Folks post not-answers on questions all the time, they get deleted, everyone goes on with their lives.
 
agreed
 
When you get something that looks like an answer, but isn't because the question itself is bogus then all bets are off. Undefined input -> undefined output.
You'll occasionally see some drunk/troll posting nonsense and some other overly-enthusiastic soul posting a mock "answer" to it. And then someone flags the answer.
Yeah, technically that flag is correct - it isn't an answer.
Buuuut... There is no answer to drunk ramblings.
You flag and it all gets deleted.
Only mods and 20K users have explicit permission to delete answers, but the vast bulk of answers are deleted by not-mod and not-20K users.
Because deleting a question also deletes the answers.
 
note that this isn't drunk ramblings, it's a question that has some real potential, there's just some information that's either missing, or was incorrectly provided in the question. I wouldn't expect the whole thing to be deleted, just improved
 
It's deleted.
It was closed, and deleted by the system
 
8:44 PM
Hadn't seen the deletion, but the point remains that it could easily have been improved to demonstrate the problem
 
In theory.
But it was never edited by the author, who hasn't been seen since the day after it was posted. Presumably he saw not only the answer that demonstrated he'd posted the wrong code, but also the comments, the close reason...
This is the most common outcome by far.
FWIW, that question would've entered triage if it were posted today. No telling where it'd end up, but...
 
It's reasonably likely that in trying to figure out what it took to replicate the problem, he found his own solution, and didn't bother to go to the site to say so
 
Very likely, I'd say
 
note that I'd probably OK it in triage
I wouldn't take the time to run the code and see that the code doesn't reproduce the problem
 
No, I don't think Triage would've discovered the underlying problem
However, the question looked pretty crappy in the first revision
And frankly, not a lot better in the last.
Although I do wonder how time-of-day will affect the outcome of Triage
 
8:49 PM
so to hop back on topic, your primary argument is that the question is likely to be (and eventually was in this case) deleted, so who cares if there are bad answers, they're not worth the time to remove, even if they're violating the rules?
 
@Servy not even. I'm saying it's usually a waste of time to try to figure out whether an answer is bad if the question sucks.
If the answer sucking actually informs you of a problem with a question, even better
Some folks like to post answers that consist of "See: <link to other SO post>"
They're terrible answers. But if the asker accepts them, that's a pretty good sign that the question was a duplicate anyway.
So, bonus...
 
The answer didn't inform the reader of anything that the 5! other comments saying the same thing didn't
so, yes, those link only answers can tell you that there's a duplicate, but if there's already a VTC as a duplicate, then it's not adding value
and those earlier comments were posted 20 minutes before the answer
There were 5 people using the correct tool, and one using the incorrect tool.
 
Not quite.
The first comment was from the same guy who first flagged the answer (and raised the meta discussion).
He didn't vote to close until two days (and two flags) later
In fact, all but one of the people who left the comment could have voted to close.
None of them did, at least not until much later
 
presumably they assumed that the person posting the question would just edit it to provide a reproducible example. I haven't found a lot of people beyond the two of us that are as trigger happy with the close button.
some, not a lot
 
And that's fair
It's their vote, their call
But the fact remains, you have a potential problem in front of you: a question geared to attract crap answers
Then you see it turn into an actual problem: a crap answer has now been attracted
You have a tool to prevent this.
You can use it, or refrain
 
8:58 PM
Well, 5 out of 6 respondents used comments to indicate that the question was missing information; only one answered
and it was a new user that presumably isn't familiar with SO's guidelines for answering
 
I'll wager that's no longer the case
 
:D
 
Again, it's the voters' prerogative to vote or refrain
 
but I'd also like to draw your attention back to George's answer to the meta question.
He's actively encouraging people to post these answers
 
So is T.J., for that matter
 
9:00 PM
see the last few lines of his answer
One major point here though is to imagine this issue from the perspective of the flagger
getting a flag declined with a reason of, "this is a bad answer, but it's just not worth deleting because the question is crap" is something I'd expect people to live with. Getting a flag declined saying that the crap answer is an acceptable answer and isn't worth deleting is very different. Personally I think that this answer is worth taking the time to delete, but you saying, "it's crap, but not crappy enough to delete" is radically different than a mod saying, "this is great stuff*
and of course, this one post is already resolved; I'm concerned about all of the future posts that get correctly flagged for being crap, that get declined, or all of the people that start posting crap answers because they saw a meta post telling them to
so the actual declined flag bothers me a lot less than a meta post saying the flags are wrong, the posts shouldn't be deleted, and people should absolutely post answers like these
 
@Servy So... This may just be a particularly odd situation, but the flagger perspective here is more complicated than I first realized.
I'm not sure how much of this I should post, given flags aren't public
But it wasn't a NAA flag that got declined
It was a custom flag, asserting something that... Was not strictly true.
 
the meta post says he flaged as NAA, and then a custom flag later
 
Ah, cool - he did quote the flag there
 
Is his quote of the flag text in the meta post not a real quote?
 
The quote is accurate, the response was omitted
So the meat of the flag is simply,
 
9:12 PM
do you have a problem sharing the real response?
 
> this post isn't an answer, it's simply an "I tried your code and it works fine for me" comment.
The response was,
> Declined: a moderator reviewed your flag, but found no evidence to support it
Which is one of three pre-defined "decline" responses available to moderators
 
the post isn't saying anything (meaningful) beyond that the code doesn't work for him
 
No, he also demonstrates that the code works
 
the code is not meaningfully different from the code in the question
 
Which is a subtle but important distinction, given folks actually do post "this works fine" as answers.
 
9:14 PM
the code in the question is already a complete code snippet that can be executed as is
and doesn't demonstrate the problem
the only changes to the code he made was wrapping it in a function and alerting it, neither of which are necessary to run the code
for all intents an purposes, he just copy-pasted the code in the question.
his changes would be no different than renaming a few variables, or changing the whitespace
 
Just to be clear, I don't disagree that it was a crap answer
 
yet you don't have a problem with a mod not only preventing people from getting rid of the crap answer, but actively encouraging people to post similarly crappy answers?
 
@Servy lemme put it this way: If I answered that question, I would do everything possible to reproduce the problem, MythBusters-style. That wouldn't necessarily mean it was reproduced, or reproduced in a way that matched the asker's problem, but I'd at least have some fun with it and perhaps leave something of value to others.
The answer there didn't do that. Or really, do anything other than demonstrating beyond a shadow of a doubt that the code presented in the question did not produce the effects described in the question.
It was a crappy answer.
But that's because the question allowed for crappy answers.
You can't post a test-case that doesn't produce the results you say it does and expect folks to conjure up a test-case that does.
Nice if that happened, but... It'd be nice if my garbage disposal turned onion skins into gold leaf too.
 
I'm with you so far; I certainly wouldn't have expected that question, as is, to get a good answer
so now that we know that the bad question has in fact attracted bad answers, as per our expectations, what do we do?
we can close the question, and post a comment asking for additional information
 
9:40 PM
@Servy close it.
Optionally come back in a couple of days and vote to delete it.
 
And the bad answer(s)?
 
Downvote
They'll get deleted along with the question
 
and if the question is edited to provide a reproducable example (after reopening the question, if needed)
 
Ah, now you can flag the answer
Or just leave a comment noting that it no longer applies, if you want to be polite about it
Or both
 
the fact that an answer doesn't apply isn't grounds for deletion
the whole idea here is that a comment is being posted as an answer, and, being a transient comment, unlike an answer, easily becomes obsolete
the whole idea of the separation of comments and answers is so that that messy clarification comment noise is kept separate from answers
 
 
1 hour later…
10:53 PM
@Servy we're not talking about a clarification though. Let's be clear: this hypothetical scenario is essentially describing a chameleon question
Realistically, the guy almost certainly just miscounted the spaces in his code, and inadvertently corrected the problem copy-pasting it into the question. There's no version of it that will merit a better answer.
 

  last day (22 days later) »