« first day (113 days earlier)      last day (1311 days later) » 

9:52 AM
Testing what [joomla.se] Stack Exchange: Joomla Stack Exchange becomes
 

Sandbox

Where you can play with regular chat features (except flagging...
do not forget to post <\s*a[^>]*>(.*?)<\s*/\s*a>!
 
10:09 AM
@double-beep I think I did that one in there..
 
10:45 AM
ah, plz send teh coffee
I spent hours debugging the code I wrote to fetch posts from the API using fromdate
but (from|to)date define creation_date range
and I had to use min
 
10:59 AM
Huh? "Needs details or clarity" :)
 
check that
and now I'm going to fix getPostsWithLatestRevision
 
11:19 AM
Want help with the map?
 
not yet
 
@double-beep If you want my review of your changes, you need to allow for a few hours. I'm not sure I have that today.
 
sure, no problem
 
How does the API work for splitting up the results?
does it return a value that needs to be sent with the next request?
Have you considered putting in a few comments here and there? :) So others know what you are trying to do?
what is the intent of && !postIdsAndTitles.containsKey(postId) && !postIdsAndTitles.containsValue(title)?
I mean what if you have two posts with identical titles?
 
@Scratte not possible
 
11:33 AM
Is that a site limitation?
I've seen it btw.. it's absolutely possible to double post.
 
cross-posting with the same title it absolutely possible, but the bot fetches the posts from a specific site
@Scratte with pages
there's a has_more field returned, which means that there are items in the next page, too
@Scratte yeah, the original code didn't have any comments at all
 
@double-beep Ahh.. yes, but there are exceptions :)
 
how can I put the list to the map? Do I have to create one, add the JsonObject, and then use .put(postId, list)?
or is there a simpler way?
and what if the list isn't empty? List<JsonObject> oldList = oldlist; oldList.add(newJson); hashmap.put(postId, oldList)
 
Ahh.. the map<Integer, List<JsonObject>>?
 
@Scratte wrong room?
 
11:45 AM
map.get(..) gets you the entry, right? Then use the result to get the value and just add(JsonObject)
@double-beep No.. double post, identical titles.
But don't just add it.. check to see if the value length is less than 2 :)
You can use map.put(value, new ArrayList<>(List.of(jsonObject))); for new entries
Don't just use map.put(value, List.of(jsonObject)); because List.of(..) returns an immutable list.
@Scratte OOOPS.. get(key) gets you the value directly :) So no need to find it :) just use it, check the size() and add() directly on the result of the get() :)
Very very roughly.. :
 Map<Long, List<JsonObject>> map = new HashMap<>();
 ...
 for (jsonObject in all your jsonresult) {
   List<JsonObject> list = map.get(postid-of-json);
   if (list == null) {
     map.put(postid-of-json, new ArrayList<>(List.of(jsonObject)));
   } else if (list.size() < 2) {
     list.add(jsonObject)
   }
 }
 
12:17 PM
that also reminds me: post id needs to be an integer, not long
@Scratte what after list.add()? I probably need map.put(postId, list), right?
 
@double-beep No :)
Your key is already in the map.. so you just need to add the second jsonObject to the list
The logic is this: If the key,value pair is not in the map, add it using the postid as a key and the json as the first item in the list. If it's already there, you just want to add the json to the list, unless there are already two items in the list, since you don't care for older revisions.
 
12:33 PM
yeah, I didn't expect that adding an element to a list would also update the map
 
@double-beep The reason you do not want to map.put after adding to the map value is that you know the key is there in the map and that the value is the list associated with that key
@double-beep It doesn't update the map :) It updates the list that is in the map :)
 
and what if you wanted to fetch a list from map, then add some stuff in it without touching the map?
 
@double-beep What do you mean without touching the map?
 
without updating the list in the map
 
The map is basically just key - value pairs. Both the key and the value in the map are references to objects.
So when you get the references to any Object inside the map, you can make changes you want to those Objects. The references will still point to the Object no matter if you changed them or not.
If you want to modify the map, you need to either remove items, update them (you can't update the key! You can tell the map you want another value be associated with the key) or add new items.
But you do not want a new value be associated with your key. You want to keep the reference to the list.
Am I making any sense?
 
12:41 PM
yes
but I guess that map.put(postid, list) would override the key
 
@double-beep It would replace the value for that key :)
Which doesn't make any sense, since the value you have, the list, is already the value for the key.
 
the old value would be a list with the newest revision json
the new one would be the old one + the second newest revision
map.put is just a useless step
(if I understand correctly)
because the list in the map is also updated if list is updated
 
Yes.. :) get(key) just gets you the reference to the list where the first element is the first jsonObject.
So when you add to that list, there will be two elements in that same list :)
Of course you need to use map.put(...) there is no returned value for map.get(key) because that means there is not already any entries for your key. That is my first check. I see if list is null.
 
yes, I've understood what the code does
\o/ BUILD SUCCESS
 
Wow :)
 
12:52 PM
o_O that avoided all the backoffs
 
Huh? :)
 
before this change, I got a backoff at every request I made
 
Hmm.. not sure that's a good sign though. It could be the code is just horribly slow :D
 
I'm not sure if it was the change in backOffUntil in ApiService or in the methods in Belisarius.java
 
Ohh.. it could be that the old code didn't actually look at the backOff..
 
12:56 PM
here's how my logs looked like:
INFO  JsonUtils - BACKOFF received. Timeout for 10 seconds.
INFO  JsonUtils - Received an API response.
INFO  JsonUtils - BACKOFF received. Timeout for 10 seconds.
INFO  JsonUtils - Received an API response.
INFO  JsonUtils - BACKOFF received. Timeout for 10 seconds.
INFO  JsonUtils - Received an API response.
 
Ahh.. yes. The method that tells me how old I'm getting with the +/* operator :D
 
INFO  Belisarius - Error while trying to get latest revisions for some posts
java.lang.NullPointerException
 
That's very unspecific. Any information of where that comes from?
 
PostUtils L57
 
Is that newPost.setRevisionNumber(post.get("revision_number").getAsInt());?
 
1:01 PM
yup
 
There is no revision number.
 
it may be absent
but when is it absent, though?
ah, on Q closure
 
Else.. you just put some code around where you create your post object and log your jsonobect to see it.
Ohh :) I see :)
 
1:20 PM
Spot? :)
Is that a duplicate bot for testing purposes?
 
yes
changes pushed
 
That's how Stack gets all those nice edits.. bots of high reputation users :D
 
lol
 
Did you fix the paging of calls to the API? So you get the first 100 posts and then the second 100 posts? :)
 
yes
 
1:52 PM
Awesome. Next is finding out why you are no longer getting backoffs.
Is it because the code is better.. or because it's just very slow :)
 
> While not strictly a throttle, the API employs heavy caching and as such no application should make semantically identical requests more than once a minute. This is generally a waste of bandwidth as, more often than not, the exact same result will be returned.
^ probably related to this
hmm... isn't List.of supported in Java 8?
looks like it was added in Java 9
 
2:24 PM
@double-beep Yes. I do not think it was part of Java 8
 
equivalent in Java 8?
 
Since: 9
@double-beep Hmm.. the long way :)
Declare.. then add :)
List<T> list = new ArrayList<T>(); list.add(element);
 
2:53 PM
Arrays.asList?
 
3:03 PM
e.g. postIdsAndJsons.put(postId, new ArrayList<>(Arrays.asList(revisionJson)));?
 
3:52 PM
Or you can have a map of Longs and JsonObject[2] <-- array :)
 
doesn't new ArrayList<>(Arrays.asList(revisionJson)) work?
 
is revisionJson an array of one element?
 
it's a json object
contains only one item
 
Arrays.asList(this-must-be-an-array) :)
Its going to get very.. nasty?.. postIdsAndJsons.put(postId, new ArrayList<>(Arrays.asList(new JsonObject[1] = {revisionJson})));
postIdsAndJsons.put(postId, new JsonObject[2] = {revisionJson,null}); seems simpler, no? And then work with the array instead of a list.
just that arrays don't have size(), they have length and it's an attribute, not a method.
and you'll need to check for the value of the second element instead of the length.
since that is always the size of the array, if I remember correctly.
 
how can I define revisionList?
 
 
1 hour later…
5:30 PM
@double-beep Where?
 
hmm... I realised that backoff was never set to 0
so if one backoff was received, then for every request, there would a be a timeout of 10 secs
forever
 
That will get you 6 requests a minute :)
 
5:45 PM
@Scratte to replace List<JsonObject> revisionList = postIdsAndJsons.get(postId);
 
5:58 PM
@Scratte does that look good?
if (revisionList == null) {
    List<JsonObject> firstRevision = new ArrayList<>();
    firstRevision.add(revisionJson);
    postIdsAndJsons.put(postId, firstRevision);
} else if (revisionList.size() < 2) {
    revisionList.add(revisionJson);
}
 
6:28 PM
Sorry.. I was away.
Yes. That looks fine :)
But I think you can just use revisionList = new ArrayList<>(); revisionList.add(revisionJson); postIdsAndJsons.put(postId, revisionList); because you set it for every loop :)
But that's just optimizing without needing to :)
 
I've pushed my changes
you can see them on GH now
the bot seems to work fine
thank you VERY much for your help
would appreciate any additional/general comments
 
7:00 PM
I'll look at it.. hold on :)
Please also give me time to figure out how to get to your changes :D
Latest version is e5e7be6? <-- that looks like a version change in the pom.xml :D
 
@Scratte right
 
I'm not going to check that the RegisterPostRequest.contentId is a long and not a Long :)
 
@Scratte ?
 
You changed postRequest.setContentId(Long.valueOf(post.getPostId())); to postRequest.setContentId((long) post.getPostId());
 
7:15 PM
@Scratte some bytes less :P
 
:D It all matters, I suppose :)
I find this change a little funny: executorService.scheduleAtFixedRate(() -> execute(), 0, 60, TimeUnit.SECONDS); to executorService.scheduleAtFixedRate(this::execute, 0, 60, TimeUnit.SECONDS);
because when I see those :: I always change them to ` -> ` :D
 
lol
 
I just find them prettier :)
 
#EveryByteCounts
 
That's readability for me :) and the characters count is the same :P
 
7:22 PM
I find method references more readable
 
I've had a lot of math. I understand functions :)
I also like the it's obvious that nothing goes to execute.. in () -> execute
method references are a little more tricky.. they have different meanings depending on the type of method.
I answered a post about it once. "Double::compareTo" means "(d1,d2) -> d1.compareTo(d2)"
I find the first confusing.. the latter simple to understand.
But.. it only works if d1 is a Double.
Actually that entire post was a clash between the experienced and the learner. I saw it, and I saw the Answer and I just knew there was noway the Question asker was going to understand the Answer. So I added another Answer and got 2 downvotes in less than 1 second after posting :D Then I got an accept, so I knew that at least my explanation had worked for users that was trying to learn it..
 

« first day (113 days earlier)      last day (1311 days later) »