last day (47 days later) » 

12:45 PM
Right.
What's the craic?
 
Jon's twisted autobahn websocket client still works, which is nice
I've got it sitting here spewing out all python tag activity at me
 
No longer dependent on twisted - uses asyncio instead :)
 
I would ignore a lot of what Kesh does, at least in terms of databases.
 
Oops, you're right.
 
It was built with Mongo in mind, when now I think there are better solutions.
 
12:47 PM
I don't think we should be duplicating stack overflow data locally, just what we've done to the data
 
@Ffisegydd yup - we were originally lazy to avoid having to normalise any data
 
@IntrepidBrit For now, yes definitely. But in time we'll need at least some batch data for ML classification.
But as I said before lunch, let's forget the ML for now.
 
I was thinking the other day - we could probably use elastic to store some data to find potential dupes
 
I've got a lot of experience with Neo4j, a graph database, which might suit the structure.
 
Ultimately though - some local storage will be required to detect deleted questions
 
12:49 PM
Aye - so if we just store the question/answer/comment ids in the database, we can also store what actions we've performed on that data
 
That'll be a module in the DAG, but for now I'd leave it.
 
In the case of link rot, we can store that we've looked at question 38099764 today and reported any link rot for a user to deal with
 
@IntrepidBrit yup - won't hurt to store the more recent data in full and cronjob clean ups - then if we need anything - we can retrieve later using the API
 
And then say 3 months down the line, nidaba rechecks for link rot
 
Have it be stateless, one module listens and passes it to anything listening, another module can do some basic stateless analysis, and then maybe pass it to a 3rd for writing to chat or whatever.
Ah if you're doing linkrot then we'd need a history.
 
12:51 PM
Yeah, it just needs to be as simple as "date last checked" & the id
@JonClements Hm, you're right that dupe hunting would need some storage
currently googles neo4j
 
Does it make sense to make the link rot library a new project in GH outside of Nidaba? If Nidaba is to be aimed at ML, might make sense to make it a new project.
 
Just to state the obvious - For dupe finding, the biggest problem will be doing so efficiently. Would it be a good idea to use a "term-frequency inverse document frequency" (henceforth called tfidf) to identify similar documents?
Yeah, I guess that makes sense Fizzy. But maybe expand it to cover all tag maintenance, like finding canonicals and duplicates
 
Yeah TF-IDF is a common NLP tool that would be used alongside other tools too.
Like tag analysis, etc.
@IntrepidBrit Those seem like very different things and would require quite different tactics though.
 
I'm pondering if it's worthwhile involving some of the bot developers from the SOCVR room - they're also doing similar work and already have functional stuff...
 
Aye, from what I can remember it was pretty snappy especially if we use something like rainbow. But that was from a few years ago almost a bloody decade ago
@Ffisegydd In terms of program flow, it would follow a similar pattern of checking old posts periodically?
Which would make sense for a "janitor" bot
@JonClements Aye, sounds like it would be good. What're your thoughts Fizzling?
 
1:00 PM
Yeah the linkrot program would work slightly differently to things like Nidaba. Nidaba takes new questions and does some analysis on them, whilst this is going through the history and doing some retrospective analysis on them to see if they haven't aged that well.
 
I know ArtOfCode (from my interpretation) sounded quite keen on the project when I linked it
 
@Ffisegydd Aye. So then we'd have the hard project of naming a third module!
 
I'd rather not get others involved just yet, not sure how much time I'll have to dedicate (can't obviously speak for Intrepid) and getting other people in can set expectations.
Plus I heard they smell.
 
Fair. Especially that idjaw geezer
 
Yeah. Sitting there, the lego swine.
 
1:02 PM
Grrr shakes fist
As for a name, either Janus or Lu-Tze might work :P
 
@Intrepid well you could have two DAGs with two sources, one takes new questions and one goes over old questions.
 
Would janitorBot plug into Kesh then? Or do it's own thang? waits
 
Both sources push their content to a message queue and then other modules listen to whatever they need to.
So Nidaba would listen to new questions, and Janus listens to old.
 
@IntrepidBrit Umm... not exactly sticking with the current "naming theme" :p
 
Yeah:
Nidaba (DNÍDABA 𒀭𒉀, DNIDABA 𒀭𒊺𒉀), also Nanibgal (DNANIBGAL 𒀭𒀭𒉀, DNÁNIBGAL 𒀭𒀭𒊺𒉀) or Nisaba, was the Sumerian goddess of writing, learning, and the harvest. Her sanctuaries were E-zagin at Eresh and at Umma. == Mythology == === Place in the Pantheon === As with many Sumerian deities, Nisaba's exact place in the pantheon and her heritage appears somewhat ambiguous. She is the daughter of An and Urash. From Sumerian texts, the language used to describe Urash is very similar to the language used to describe Ninhursag. Therefore, the two goddess may be one and the same. Nisaba is...
 
1:05 PM
Ah!
So Kesh is the template
Make sense now. I tried googling it
So what about Dullu then?
Mesopotamian religion refers to the religious beliefs and practices followed by the Sumerian and East Semitic Akkadian, Assyrian, Babylonian and later migrant Arameans and Chaldeans, living in Mesopotamia (a region encompassing modern Iraq, southeast Turkey and northeast Syria) that dominated the region for a period of 4200 years from the fourth millennium BCE throughout Mesopotamia to approximately the 10th century CE in Assyria. Mesopotamian polytheism was the only religion in ancient Mesopotamia for thousands of years before entering a period of gradual decline beginning between the 1st and...
> "Generally, the god's well-being was maintained through service, or work (dullu)."
 
I'd go with that.
 
> So Kesh is the template

Fecking auto-correct. I meant temple
 
Nice.
 
Okay. Would it make sense to go over new posts to make sure all urls are correct?
 
1:09 PM
How do you define "correct" in this sense? Working?
 
Essentially yeah - we get an acceptable HTTP status code
 
Yeah probably would.
You could have a third message queue that is shared, both new and old, maybe?
Or be able to register a module against multiple message queues.
@Intrepid do you have write ability to the sopython git repo to create repos?
 
I think this is one I would let someone else make a decision on - I'm going to put my hands up and admit I haven't much experience in using message queues
Only one way to find out
 
@Ffisegydd he does now :)
 
I can!
oh
Cheers Jon
 
1:15 PM
I think we can trust you by now :p
 
Dangerous
Maybe I'm a PHP plant
 
DANGER ZONE!
4
 
We're using a BSD license for the other projects yeah?
 
No idea?
I prefer Apache myself but have no strong objections.
 
Yeah, looks like kesh uses a BSD 3-Clause
I'll stick with that just now, can always change it later
But I don't think you can go from Apache to BSD later
Was it you Fizzy who found neat 'default project structure'?
 
1:23 PM
cookiecutter
We need an overall name for the projects. Nidaba is ML, Dullu is the tag stuff, kesh is new question source, I think the old question source will be eridu, etc.
 
Thanks, downloading now.
Mesopotamia?
 
I was thinking Project Cradle.
Mesopotamia was the cradle of civilisation.
Also, and I realise this is getting slightly ahead of ourselves and quite hipstery, but we should docker the sh*t out of this.
 
I like that
And sure, with the Docker. I'll add that to the pile of "high level" wizardry I need to learn ;)
 
:P
Makes it much easier to manage microservices like this.
 
1:40 PM
That works
 
@Intrepid what were your thoughts on accessing old content? Just go through the history over and over again?
So it starts at post #1, then #2, then #3, then... then #N, then #1, then #2, then...
Slowly accessing old content so to not flood the API.
 
Ultimately, yes.
 
As a basic idea, in reality you'd probably rather get 100 questions at a time and then add all 100 to the queue to be processed, then wait N minutes and then getting 100 more, etc.
Rather than making an individual API call for each one.
 
Yeah, agreed.
But if by some miracle we're processing through all the python tagged questions "quickly", I wouldn't want to revisit them too often.
Maybe put like a 3 month or something cooldown on previously checked information.
 
Yeah, we could do a back-of-the-envelope calc to make sure it doesn't get out of hand.
It might be possible to lookup recently edited questions more frequently.
 
1:52 PM
So, what I'm thinking (in the short term) storing answer/comment/question ids in a sqlite database, along with the date last checked
And grabbing updated information from kesh
But having dullu being in charge of historical API trawling
Once we've found a link with a bad http status code, we check it a few times in a row before alerting anyone of the dead link
Was thinking of posting it to a website, so that a human can look into the link, test it themselves and then edit the question if need be
 
I made a repo called eridu that I thought would do the historical API trawling.
And then dullu would do the actual analytics.
 
Makes sense
 
And they'd talk via a message queue.
brb meeting.
 
k
I'll hold off doing anything until you get back
 
2:10 PM
> One of my biggest stackoverflow pet peeves is flask questions that are actually javascript questions -- Corvid 29/06/2016 15:08 (BST)
Something we could actually look into with Project Cradle ;)
 
was wondering what was going on with those team deletions :)
good to see there's a plan and fizzy hasn't gotten on the gin early :)
 
We've decided that Firkin Gin is an acceptable compromise. Whisky infused gin
I also didn't realise that Fizzy went on a rampage
 
2:24 PM
inspired presumably by the hierarchy of diagram he linked on wiki :)
 
Which wiki?
 
2 hours ago, by Ffisegydd
Nidaba (DNÍDABA 𒀭𒉀, DNIDABA 𒀭𒊺𒉀), also Nanibgal (DNANIBGAL 𒀭𒀭𒉀, DNÁNIBGAL 𒀭𒀭𒊺𒉀) or Nisaba, was the Sumerian goddess of writing, learning, and the harvest. Her sanctuaries were E-zagin at Eresh and at Umma. == Mythology == === Place in the Pantheon === As with many Sumerian deities, Nisaba's exact place in the pantheon and her heritage appears somewhat ambiguous. She is the daughter of An and Urash. From Sumerian texts, the language used to describe Urash is very similar to the language used to describe Ninhursag. Therefore, the two goddess may be one and the same. Nisaba is...
 
2:40 PM
Of course - and here was me looking around Trello and the GitHub wikis looking for a hierarchy diagram involving our systems interfacing
 
 
1 hour later…
3:41 PM
So, all important question: What are we going to use for messaging?
 
3:53 PM
ICQ/MSN ? :p
 
Jon jon jon. MSN messenger no longer exists.
 
Seriously? What year is this again? :p
 
2030.
 
Wow - when I take a nap - I really take a nap don't I
 
Afraid so.
I'm surprised you could resurrect this room
 
4:08 PM
On the plus side - I'm still a puppy - how are your grand kids doing? :p
 
Evil lil' turds
 
Probably what you get for givin' em' milk instead of whisky in their bottles when younger
 
Didn't you hear? Back in 2022 when Scotland became independent, we pissed off the English so much that they stopped selling us malt. As a result, there's no more Scotch Whisky. What's left is worth around £4000 a bottle
I ended up having to give them gin, which probably explains their evilness
 
Oh gawd - I don't even want to know what happened to Fizzy's kids then
 
Didn't you hear? Fizzy was killed when nidaba went sentient
 
4:18 PM
Oh beep - you mean we invented skynet?
skynet does like puppies though, right?
 
Well, we don't need to worry, yet. Both the Martijn AI and nidaba are fighting over the whole internet. When one is victorious, we need to worry.
 
goodreads.com/book/show/889071.Mother_of_God isn't a bad read - you just reminded me of it
 
Ha!
That sounds awesome
Might need to get myself a copy
 
Probably on ebay cheap - been looking around to get a few missing in my FF collection
 
I should really use my goodreads account
adds to my read list
 

  last day (47 days later) »