« first day (111 days earlier)      last day (1313 days later) » 

3:56 AM
@Scratte how did you resolve it?
 
 
3 hours later…
6:44 AM
Saying that I worked it out was not entirely true. Kevin B pointed out if was probably a timing issue here :)
 
7:21 AM
if you don't want jQuery, then you can use this:
const originalOpen = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function () {
  this.addEventListener('load', function() {
    if (this.responseURL.match('/review/next-task')) doStuff();
  });
  originalOpen.apply(this, arguments);
}
SE makes a /review/next-task call in order to fetch the HTML which should be added (the buttons, the description, etc.)
> The load event is fired when an XMLHttpRequest transaction completes successfully.
 
 
1 hour later…
8:46 AM
Nice :) I worked on a modified version of one of Samuel's scripts though. So the UI tells me when the review is an audit.
 
yes, you have to check for the isAudit property in the JSON this call returns
 
Yup.. and then I change the text of the Skip button :) Which I finally managed to do. So now I have a tiny user script that on average took 30 minutes per line.. and I only actually added one line myself :D
 
When you say "this call", you mean the one in your code snippet here?
 
the one SE makes to /review/next-task/review_id
I'd also like your help with a few Java warnings I get, but probably some time later today
 
8:59 AM
(function() {
    'use strict';

    // from github.com/samliew/SO-mod-userscripts/blob/master/…
    function listenToPageUpdates() {

        // On any page update
        $(document).ajaxComplete(function(event, xhr, settings) {

            var button = document.getElementsByClassName("js-review-actions")[0].lastElementChild; // The Skip button

            // Get additional info about review from JSON response
            let responseJson = {};
            try {
^ that's the code I went with..
I'm not entirely sure how I can modify it to use your suggestion though.
Since I need the response text and I get very confused about what I see to be optional parameters on methods and functions.
I mean does originalOpen contain the response?
Hmm.. no, can't be. Must be this.something-I-don't-know-about-yet :(
 
so you want all your reviews to be audits?
you can use do stuff(this.responseText) to pass the text as a parameter to your function
then you can JSON.parse() it
and if you're going to use that way, then you'll need to remove $(document).ajaxComplete(function(event, xhr, settings) {})
 
9:15 AM
It's working as is.. I didn't even bother to put the (button.innerHTML == 'Skip') into the if-statement :D
But I may fiddle with it later to try it your way.
@double-beep Ahh.. that's the key. Thanks :)
I like fixing java warnings :) I'm going to guess it's about using raw types instead of generics or deprecated methods..
 
 
2 hours later…
11:26 AM
@Scratte it's more complex than you think :)
it's not actually a Java warning, but a warning a tool throws
the project is here
 
@double-beep Oh. I don't use tools except Notepad++ and Windows Command Prompt
Which tool?
 
utils.DatabaseUtils.aMethodThatUsesResultSet() may fail to clean up java.sql.ResultSet [...] Obligation to clean up resource created at DatabaseUtils.java:[line xxx] is not dischargedPath continues at [... lines of file ...]
I'm getting this ^ for each method that uses a ResultSet
(in utils.DatabaseUtils)
(e.g. checkVandalisedPostExists)
 
I remember this from earlier.. it's using try-with-resources, so I'm not sure why you're getting those warnings.
 
but not with ResultSet, right?
 
Hmm.. I would expect it to though. They seem to return booleans. maps, ints, not the resultsets.
What is the tool that is complaining?
 
oof, and I thought I had searched enough :)
thanks
 
I know more. I think of better keywords. No worries :)
 
Nullcheck of conn at line 157 of value previously dereferenced in utils.DatabaseUtils.createBlacklistedWordCaughtTable() [utils.DatabaseUtils, utils.DatabaseUtils] At DatabaseUtils.java:[line 157]Redundant null check at DatabaseUtils.java:[line 159] RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
 
There's also an open issue here: github.com/spotbugs/spotbugs/issues/493 :)
 
11:47 AM
I get couple of those ^^
 
Same plugin?
 
yup
 
Ehh.. it's complaining about } catch (SQLException exception) { ?!?
I don't see a null check in that method
I think it's the same issue. It's getting confused about the try-with-resources statements.
like 157 is part of the statement Statement stmt = conn.createStatement()) {
 
hmm... ok
don't worry though :) there are more
 
Fantastic :)
 
11:52 AM
Write to static field bugs.stackoverflow.belisarius.services.ApiService.backOffUntil from instance method bugs.stackoverflow.belisarius.services.ApiService.setBackOffUntil(JsonObject) [bugs.stackoverflow.belisarius.services.ApiService] At ApiService.java:[line 51] ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
that doesn't seem to be an FP
(at least to me)
 
FP? False Positive
 
false positive
 
Not my impression that this was a multi-threaded application. But I didn't check.. is it?
 
check services.MonitorService
 
I see.. it's starting a Executors.newSingleThreadScheduledExecutor() which means only one thread apart from the main thread
If course to be sure, I'd have to read all the code :D
 
12:03 PM
would appreciate it if you tell me any possible simplifications/bugs/... you may encounter
 
Then I'd have to manually read all of it, which.. is a bigger project.
I'm not sure how the ApiService is getting initialized yet.
 
I don't see any services being auto-attached.
 
and startMonitor() from the main class: github.com/SOBotics/Belisarius/blob/…
@Scratte hmm?
 
You can say that an interface is being implemented by a class and register that, so that when one wants to use an implementation, one can just ask java for any registered implementation.
they used to do that in a file (can't remember the name of the file now though). Now one can do it via a module definition.
 
12:10 PM
that was... even more confusing
 
Your project is not a module :D
and I see no servies file to autoregister classes with interfaces :D
I can't remember the exact details, but I doubt you need to know.
I think you can fix your warning by using an atomicLong instead of a long :)
But I'm not sure how the tool works..
That will make it not write to a static variable and will make the value threadsafe :)
This one: java.util.concurrent.atomic.AtomicLong
private static AtomicLong backOffUntil = new AtomicLong(0);
Then use backOffUntil.set​(jsonObject.get("backoff").getAsLong()); in the setBackOffUntil method.
and return backOffUntil.get(); in your getBackOffUntil() method
I see no reason to use compareAndSet​ as I do not expect a race condition to be a problem here. Even if one thread sets it to 0 and another thread sets a value right after, but it suppose to be in reverse, the worst that can happen is that your problem will just back off a little, no?
There's an AtomicInt for your quota too :)
 
and that was the next warning
 
Heh... obviously :)
@Scratte program.. not problem :)
 
12:26 PM
is it AtomicInt or AtomicInteger?
 
AtomicInteger, sorry :) There is no AtomicInt :)
 
(quotaRemaining is AtomicInteger in utils.StatusUtils)
 
I do not even think you need to set the initial value of 0 in the constructor
Documentation says "Creates a new AtomicLong with initial value 0.". But I do it anyway, to kind of say to myself and others that I did think about it and I picked it deliberately.
 
I created
public void setQuota(int quota) {
    quota.set(quota);
}
and used setQuota(json.get("quota_remaining").getAsInt()) accordingly
but I get int cannot be dereferenced
 
How is quota defined?
 
12:36 PM
private static AtomicInteger quota;
 
Ohh.. this is a new method you've added then?
 
yes
but quota is also updated in JsonUtils
 
This is called shadowing :) You method cannot see the instance variable. Try this.quota.set(quota);
 
so getQuota() is redundant
and setQuota() should update StatusUtils.remainingQuota
 
No.. it's not :)
..hmm.. now I'm confused. getQuota is public, so anyone can check it.
 
12:39 PM
but it's not used anywhere in the code
 
Then it's useless :D
 
instead, it directly accesses StatusUtils.remainingQuota
 
Is there another backoffuntill too then?
 
is StatusUtils.remainingQuota.set(quota); good then?
 
That's a design problem. Seems there's not a lot going on in StatusUtils
I'm confused as to why it's initially set to -1
 
12:43 PM
the owner "stole" some code from other projects
@Scratte because the SE API quota can be actually 0
 
What does 0 mean?
..and is this variable ever set to -1 in any part of the code?
 
Either keep them both where they are, so keep them both in the other. no? Unless you think the backoff is way different. I'm not sure.
 
backoff can be handled in ApiService instead of JsonUtils.get
 
How does Natty handle backoff?
 
12:48 PM
because:
bugs.stackoverflow.belisarius.utils.JsonUtils.get(String, String[]) calls Thread.sleep() with a lock held [bugs.stackoverflow.belisarius.utils.JsonUtils, bugs.stackoverflow.belisarius.utils.JsonUtils] At JsonUtils.java:[line 42]Another occurrence at JsonUtils.java:[line 34] SWL_SLEEP_WITH_LOCK_HELD
@Scratte JsonUtils.handleBackoff()
where root is the JSON the API returns
can probably do the same, but in ApiService
 
@double-beep Not unless your application is singlethreaded.
If the API sends a backoff, it wan't all your threads to stop, no? :)
So keeping it as a variable that all your threads can see is probably the best thing.
 
right
 
I didn't check to see it it's an Instant. I mean if the API tells to backoff for 120 seconds, it doesn't really make sense to add 120 into the long, right? How will any threads know when those 120 seconds started?
Where is getBackOffUntil() being used?
Personally, I'd probably remove the small class just to hold the quota, and keep it in the ApiService. It seems to belong to the service.
Unless you plan to have several different implementations of services that share that same variable.
 
That was unexpected..
So it gets some miliseconds. It sleeps those + one full second.. I suppose that will ensure that it's always keeping the contract.
 
the backoff is in seconds
so it has to be multiplied by 1000
 
I'm getting old.. I could have sworn that was a little + :D
 
I see the StatusUtils.remainingQuota being set too here. Which.. could just be the ApiService, as that's already imported.
 
1:26 PM
yes, I've already removed that
 
You're fast :)
Next warning... :)
 
still haven't found a way to handle backoff
ApiService.handleBackoff(json) or leave it as is
 
What's wrong with the way it's done? Doesn't all the thread use this same variable?
 
ok
last warning:
bugs.stackoverflow.belisarius.utils.JsonUtils.get(String, String[]) calls Thread.sleep() with a lock held [bugs.stackoverflow.belisarius.utils.JsonUtils, bugs.stackoverflow.belisarius.utils.JsonUtils] At JsonUtils.java:[line 42]Another occurrence at JsonUtils.java:[line 34] SWL_SLEEP_WITH_LOCK_HELD
 
Yes.. this is when I just want to dig a big hole and hide it it :D
So.. now it's all about intent. Do we want the code to hold a lock while it's waiting for something? Or do we want it released and reacquired?
This is the same code that we looked at with the "little +"
The method is synchronized, so whatever thread is in it is holding a lock on the object itself.
Does it make sense to release the lock and let another thread go into it?
 
1:35 PM
meh, it's probably fine as is
 
Yes :) It doesn't.. since it's only sleeping because of either the backoff or doing a routine 100 milisecond sleep.
You can get rid of the warning by implementing your own lock and wait/notify routine around a while loop.
 
does the newest commit look good?
 
I see you added SITE_STACK_EXCHANGE :)
 
that's an older one, right?
 
Perhaps.. I think I'm getting confused about the latest version
Thought it does say "double-beep committed 2 hours ago"
 
Github confuses me. I can never find the commit history on it
 
click commits
 
I see no issues with this, but I'm not a perfect human compiler (I don't know the difference between + and *, you see :)
 
:)
CI passed, which means there weren't any compilation errors
 
Ahh.. there's a link there in the little clock and "148 commits". I did not notice that.
CI? What is that?
 
1:48 PM
Continous Integration
(the tick before the commit title)
 
Ohh.. that looks like some job setting all that stuff up
I haven't spent a lot of time on GitHub.. I've only used a local version and a GUI :)
 
you don't have a GH account?!?
 
No :) Amazingly I'm still alive. :)
 
why?
 
I don't need one. I don't write libraries for others to use.
I'm very impressed that you got this project going though. I was stuck on the swagger thing and you just figured it out.
 
1:57 PM
no I didn't
this figured it out
 
..worked it out :) Found a way :) Made it work :)
@double-beep I remember you posted it before and then gave the code here too. I don't have access to that :)
 
can you have a look at Belisarius.java? I think that some improvements can be done, especially in getPostsWithLatestRevision
(e.g. I'm catching Exceptions!)
 
.. you mean LOGGER.info("Error while trying to get latest revisions for some posts", exception); is not enough? :)
 
I think it's bad practice to catch Exception
and the logic in getPostsWithLatestRevision is messy
I'd also like your opinion in utils.DatabaseUtils, too, since it's a quite big file
 
Depends.. if you look at the logs and want the program to keep going, it's fine, no?
If you want it to die.. then obviously don't catch them. Ohh.. it keeps going on runtime exceptions..
What do you want to happen?
 
2:09 PM
do not catch any unnecessary exceptions
but also don't crash for a trivial exception (e.g. can't parse JSON)
 
The easy way is to remove the exception handling one by one and see what the compiler says. It will tell you what exact exception is being raised by methods down the line.
When you can catch those and either do a default as the end or not at all.
I need to go for a while.. I'll be back later :)
 
2:58 PM
I've removed some files and simplified the code
 
 
2 hours later…
4:55 PM
Did you commit?
..and did you make a decision as what you want to happen with the exceptions?
 
5:08 PM
I'll leave them for now
@Scratte yes
although, I've not seen any thrown in the logs
 
5:33 PM
What's your decision?
 
leave them
 
Why am I seeing "Latest commit 6a193ba 6 hours ago"? :)
Or is it the same as the one I looked at?
 
oh, that means that the last commit which changed the file was 6 hours ago
the newest commit didn't touch this file
 
5:36 PM
Oh..
How can do such big cleanups in such short time? :)
 
...but make stupid mistakes? :P
 
None that I can see.
You created some static methods and removed some files.
 
pretty sure the logic in getPostsWithLatestRevision is flawed
@Scratte yeah, that commit was easy
 
@double-beep what file is that in?
 
Belisarius.java
 
5:45 PM
Are you going to tell me what you think is flawed or do I need to close flag as "Needs details or clarity"? :)
 
I'll need to dig into the API doc for that
 
It's a while loop in a while loop :D
 
yes, that doesn't sound right
the Iterator should probably be replaced with .entrySet()
the ids should be an array of arrays (each containing 100 ids)
then foreach of those use apiService.getLatestRevisions
> {ids} can contain up to 100 semicolon delimited ids
 
The iterator is the only safe way to remove entries
 
yes, but that's probably not needed
 
5:51 PM
else you get concurrentModificationException..
There's not a lot of comments in this code, huh? :D
Hmm.. there's another loop in the inner while loop.
 
that's required
 
I'm not so sure. It's taking each item in the map, then it's looping the result to find out if it matches the item in the map.
..but it's a map. So why not loop each item in the result and find that in the map?
 
que?
 
Maybe I'm not understanding it right. You have a map of id's and titles, right? Then you string up the id's and ask the api for revision on it?
 
here's what the bot should do:
- Fetch [/posts](https://api.stackexchange.com/docs/posts) every some time to find newly created posts.
- Get the ids and fetch [/posts/{ids}/revisions](https://api.stackexchange.com/docs/revisions-by-ids)
 
6:04 PM
Heh.. I was just looking at that one method :D
 
sorry for the delay, continuing my monologue
 
I'm not entirely sure what's going on in that method. But I can see that the api doesn't return posts and their revision. It just returns every revision on all the posts, not in order.
..or maybe I need to look at that again
 
from the latter method, it needs to fetch the edit summary (comment), the two bodies (lastBody and body) and the titles (if they exist, title and lastTitle)
those are run through different filters
however, since title is not returned via this method, if it hasn't been edited AND in order to register a vandalised post, you need a title, I need to somehow find it
I noticed it is returned in the /posts method, so I created a hash which contains the ids and their titles
I think it was previously a list or an array
 
Well.. that's just silly, since titles change all that time
 
also, since the HTML isn't readable, I needed to find the markdown of the post
 
6:17 PM
Can you tell me what posts that method is suppose to return?
 
then, I realised I could get it by parsing the HTML of "https://" + sitename + ".com/revisions/" + revision_guid + "/view-source"
which means I had to also fetch the revision guid from the previous revision
 
what's a guid? :) Global uni... id?
 
@Scratte it's the revision's id
which is a string
e.g. DB207FCC-5E19-4BCA-872C-ADD2541753E3
 
OK :) I'm not if you want to optimize the method or if you think it's not working properly
 
@Scratte api.stackexchange.com/docs/… see this example
it's a JSON array of revisions
 
6:23 PM
OMG!!.. post it 4 and post id 7! :D
 
just bored to search for new ones :)
 
I just used two random ones :)
I don't understand the result.. or rather I need to look at the structure.
OMG!! Postid nr.4 is yours :) <-- unless you're the editor or something
 
I'm the editor
> This type represents that state of a Post at some point in its history.
each revision is an item of the array
e.g. post 4 has 36 revisions, so the response should have at least 36 items
 
Ok.. so every time it says ' "user": {... ' it's a new entry
 
yes
if you find that confusing, then you can edit the filter by clicking [edit]
 
6:29 PM
I got confused with the tags edit
Look at those two: 63829575;63835355
I see a items then a [ then { a { and a } and a } <-- balance. So I assume the new { is a new item.
 
each revision item contains information about the editor
 
which sometimes start with "tags"
 
which is a JSON
if the tags have been edited, then you should expect that to appear
but in this case, it's the initial revision
 
Yes, that much I did understand. I also see there is no order to the result. revision 4 and 7 are intermingled.
I take it ""revision_number": 1," means first revision :)
Ahh.. I see the order is the creation_date
newest first..oldest last
oops.. :)
Are you returning a list of post with the second to latest revision?
 
6:49 PM
@Scratte yes, the creation_date of the revision
@Scratte hmm?
 
Ah.. no just the prevRevisionGuid
You didn't tell me what this list is
 
which list?
 
private List<Post> getPostsWithLatestRevision(Map<Long, String> idsAndTitles) that return list :)
 
oh that list
a list of Post objects
 
I think you're making the method word harder than it has to. It has to look at the same json entry again and again and again... 100 times, unless it's lucky that the first element is the postid of the map.
Just explain in works :)
what happens when the current revision of a post is the last element and there's a previous revision in the next response?
..also I do not understand how that's handled in the api.. getting the next 100 elements of a request of ids
 
7:04 PM
the API returns a has_more parameter in the response
 
I see that. But the loop just runs the same method: apiService.getLatestRevisions(String.join(";", postIds));
 
yeah, that's a bug
 
Ahh.. ok :)
 
@Scratte that's also an edge case, which I didn't handle properly
but I was aware of that
 
but.. if it did return the next 100 element, then the loop is a little flawed as well
 
7:06 PM
yep
I'll work on that
 
How many id's do you ask for?
 
why can't I understand you today?
:P
 
I'm dark and mysterious? :D
How many postIds are in the request to the API? :)
 
@Scratte which request? To /posts or to /posts/{ids}/revisions?
 
private List<Post> getPostsWithLatestRevision(Map<Long, String> idsAndTitles) <-- this has ids. How many does it usually have?
@double-beep Yes :)
 
7:10 PM
I don't know/haven't investigated
will also have to check if getPostIdsAndTitles() isn't flawed
 
OK. I think the iterating over the map of idsAndTitles and flawed. Iterating over that will have to check every json for every id. That's id*json iterations in total
 
the API provides a fromdate parameter which we can pass to make getPostIdsAndTitles() less complex
 
Even so.. I'd probably try to create a map from the response with the post id and the two latest revisions in it.
Let the map be defined outside the loop. Then it doesn't matter if the two revisions your interested in are in two separate responses.
Like a map of <postid, List<jsons>>. Then for every json, you can find the postid in the map. If you get null, create it and insert the json as the first element in the list. If you get a return, check the length of the List. If it's 1, insert your json as the next element in the list. If the length is two, you're done with that post, since you do not care for the 3rd revision down the line.
When the response is exhausted, you can just stream the map and create your List<Post> to return :)
 
7:33 PM
YES!.. I got an audit :)
 

« first day (111 days earlier)      last day (1313 days later) »