« first day (3390 days earlier)      last day (1557 days later) » 

9:02 PM
I figured it was worldwide, but still
@JBis hahahahaha nice
 
9:19 PM
Does MySQL not scale?
 
you can only scale MySQL horizontally with read replicas
writing only scales with increased CPU/RAM/I/O
that's why pretty much every large content generation platform like Facebook uses some NoSQL db
 
shit
 
Google and Facebook ended up building custom scalable SQL Dbs
 
well this is an issue, I already started with MySQL
 
what is it for?
 
9:28 PM
My SE clone
 
ah well
a passion project is not likely to be able to afford a scaling solution out of the box
mongodb is open source
 
@DaveS guess what ad came up on my FB timeline
 
PHP?
oh Out of Africa
 
Depends underwear for the elderly? errr I mean, oh Out of Africa
 
@ballBreaker One time use?
 
9:31 PM
@JBis An SE clone will not need Facebook/Google levels of scale if you do it right
So in short, if you're in charge, best get ready for needing Facebook/Google levels of scale
3
 
@RaghavSood 🖕🏻
 
hahaha
 
@RaghavSood hahahahaha
that was brutal guys XD
 
Does your puni circuits support unicode characters?
 
Jan 21 at 14:36, by Raghav Sood
I've also recently solved the issue of people here using emoticons
Jan 21 at 14:36, by Raghav Sood
I have uninstalled them from my system
 
9:33 PM
Do you ever smile, Raghav?
 
@RaghavSood Sure. But should I switch to postgres?
 
Tim
yes
 
you should just give up
 
:'(
 
@JBis It won't really make a difference at this stage, but postgres tends to be better in general
 
9:36 PM
@JBis see what I told you a few days ago?
2 days ago, by Mehdi
better learn the most quickly from him before he turns into a cynical adult like many people in our industry XD
 
For a SE style site, you'll likely be fine with any battle tested RDBMs - postgres, mariadb, sqlite (a bit harder for this one)
 
postgressql also doesn't scale write horizontally without a custom solution to do it for you
 
You don't need scale for this
SE is extremely read heavy
 
true
 
The only writes you care about are new posts, upvotes, comments, flags
For new questions, there can be a short delay
 
9:38 PM
and chat
 
Upvotes should be instant to the logged in user who performed the update, and not necessarily realtime to others
The normal way to approach this is to have a redis or memcached layer as your "fast" database, and treat it all as a two layer storage - if it is not in the cache, then hit the db
Most of your reads for a given time frame will go to active questions, so most of them will hit the cache
 
@RaghavSood but if it theoretically blows up I don't want to have to remake it
 
Similarly, most writes will be on recent posts, which can also hit the cache immediately, and persist to the db asynchronously
 
@JBis don't worry it wont
 
If you use postgres with a good caching mechanism and well a well written app layer, you will likely not have any issues until far beyond SE's current size
 
9:41 PM
this is too hard :(
 
write it to learn and have fun
 
People are used to looking at Google and Facebook scale stuff and thinking you need to be aiming for that
 
don't expect to compete with stack exchange
 
most of the time young people have a hard time gauging what's possible in a realistic amount of time.
 
That's usually around four orders of magnitude away from even the largest normal sites
 
9:41 PM
you can solve those problems later
 
You can serve a million hits a day from a $5 DO instance for most mobile apps
 
you don't have enough experience writing massive systems so you think that it's possible for a single engineer to do so, but it isn't.
 
An SE clone will need no more than 3 servers to operate at several million hits a day
You could run the whole thing off a $30 VPS for a year or two probably
 
@RaghavSood especially with the amount of traffic it won't get.
2
 
Ok so I'll switch to postgress and deal with other issues when the come up
 
9:44 PM
@CarlAnderson Ok, boomer :D
 
I'm not that old
 
@CarlAnderson whats that supposed to mean
 
seriously though, I think your aim should be to create a well functioning MVP, 1 brick at a time
 
@JBis when I was in high school I wanted to make an entire Dragon Warrior style RPG. I thought that (since I had written some simple programs) it would be easy to do, and started planning out all the maps and monsters and such. Eventually when it came time to actually write it I realized I was in over my head.
I never wrote it because it was too much for me to accomplish by myself
 
2 days ago, by JBis
thanks, i do understand that this will most lilkey be a failure but at least i'll get some experience
2 days ago, by JBis
I'm really not the best person to create this. But I really want to work on something and I figure if I beat Codidact and TopAnswers with an initial working prototype people will join.
 
9:47 PM
but keep going
 
it would have been more fun to let him go down the google levels of scale path guys
 
Letting kids run around with too much tech is how you get snapchat
6
Massive waste of computing resources with 0 benefit to literally anyone
 
imho, I think his priority should be the app code itself at a localhost scale
 
I don't get why they are so successful. Their app sucks. UI is awful. UX is worse.
 
they solved a "problem", people adopted it
 
9:49 PM
Mehdi is right
 
cause people thought it was safe to send dick pics with it
everyone rushed to send dick pics
turned out it wasn't safe but now people use it
 
And wtf. Everyone is so concerned about privacy meanwhile the default is to share your location with your entire friend list.
 
Don't even think about scale until you can post a question, answer it, accept the answer, and upvote/downvote on localhost
Anything else is meaningless
 
so just lie about how people can safely send dick pics on your new stack exchange and you'll be a hit
 
@Mehdi so don't switch to postgres yet?
 
9:51 PM
I wouldn't, you wouldn't even feel the difference if you did
 
Switch to postgres, then forget about all this and come back once the MVP works
I would
 
have a nice day guys
see you
 
lol, see you
@RaghavSood why?
 
MySQL is not ACID compliant in most operating modes, among other things such as sketchy unicode support
 
for part1, does it matters for a 2 queries / minutes? second point is a fair point
 
9:57 PM
loading the home page is going to be a bunch of queries
 
pagination, never load everything
 
It does not matter at JBis scale, but when you get to the point of even a small beta, you will run into pipelining issues
 
to be completely dynamic store the entire rendered html page in your DB
that way you don't need to make code changes to change it
 
@JBis The homepage should be like 3 queries at most; logged in user's rep, Select title, tags from questions order by posted desc limit 25; (optional) sidebar list of Qs
@DaveS That's what I do for some of my projects, render a page and stuff it in redis or s3
You can eat up huge traffic spikes with negligible overhead
Wouldn't do that for an SE style forum tho
 
it was a joke but yeah static hosting on s3 is great
 
10:00 PM
site query, user query, questions query,
bunch of joins on those also
 
@RaghavSood thanks for the info, I'll look that up!
 
alright thanks for info and (lack of) inspiration!
cya o/
 
@Mehdi Tbh, the biggest user facing difference a programmer would notice is that Postgres supports stuff like SELECT DISTINCT ON, FULL OUTER JOIN, mysql doesn't
 
full outer is helpful
 
All of the scaling, sharding etc. issues can be categorized as "It's the devops/sres problem, not mine" :D
Though until Postgres 11, MySQL did sharding better
 
10:04 PM
|| afk postgresifying a mysqlfied application
 
@JBis bye JBis
 
@RaghavSood how would you feel the difference usually?
 
I'm talking about the programmer being the user there, not the actual end user
As a programmer, the extended SQL support is nicer for common queries
MySQL has some SQL stuff that postgres doesn't too
In reality, if you are building a truly huge system (not SE), you should site down and compare them
As a default option, Postgres is usually the better choice because more parts of a normal workflow will just "work"
As an ideal option, SQLite beats most things if you're not serving 500 req/s
Or even if you are, but can shard it to less than that (1 sqlite db/user, for example)
 
I meant the difference between a dbms that does sharding well vs not
 
Ah
Will answer in a bit, need to clear some stuff at uni first
 
10:08 PM
no rush! good luck!
 
10:23 PM
Alright, I'm back
Postgres had some limitations around sharding by range, making it difficult to rebalance data if one particular range grew too large, or if you wanted to go from append only to random insert modes etc.
But that's mostly been solved in 11 and 12
And honestly, by the time you hit that kind of problem, you have enough resources to deal with it
 
sharding and the depths of a db system are a black box to me, I've been reading some literature about how databases are built internally, it's madness :D
 
The very basic sharding you'll see is essentially symlinks
You'd have a, for example, transactions table
Except that table doesn't really exist
You'd actually have multiple smaller tables, such as transactions_2010_2011, holding a year's worth
And when you query transactions, based on the date range, you'd hit the right shard
Since the underlying tables are separate, they can be put on different servers, different drives, etc.
 
so basically like partitioning per unit of time?
 
If you do a query that spans multiple shards, you'd hit more underlying tables
That's one way to do it
Another way is by predicate
So you might put one column in one shard in its entirety, second column in another, and so on
That's more commonly done by graph databases
Where your queries tend to require fast access to all values in a "column" (edge, facet, node-key, etc.)
If you do that in an relational system, you will usually end up duplicating the primary key on each shard
So if your table is id,name,email,dob,dislikes_dave(default=true), then your shards would be id,name,email, and id,dob,dislikes_dave(default=true) since you need to be able to link rows across shards
 
makes sense, is this something you configure so it's done automatically or actually manually do?
 
10:31 PM
Yet another way is hash based -> you have a master hashtable of pkey,hash%distribution_function, with hash%distribution_function being the shard id
And then multiple shards of id,data1,data2,etc.
This is similar to the range based sharding, but avoids pitfalls with clustered data
@Mehdi You would usually configure it manually depending on your data distribution
And query pattern
You generally want hot queries to only hit one shard where possible
So based on that, you'd choose between vertical (column, predicate) sharding, horizontal sharding (rows, range based), or vertical+hash, horizontal+hash
This is the easy mode of sharding tho
 
that's pretty interesting, guess I found my reading of the week :D
 
If you're looking at Facebook style stuff, where all your shards interact with each other because that's how real life social networks work, then I can't help you
I don't understand that complex sharding :D
 
@RaghavSood Whats your thoughts on all the complaints about NoSQL?
like writing data doesn't actually write data
 
I think that anyone complaining about any tech at a general level is dumb, unless that tech is PHP or npm
Writing data also doesn't actually write data on basically every filesystem in use, you don't see those same people shifting to tape drives
 
go on..?
"Writing data also doesn't actually write data on basically every filesystem"
 
10:37 PM
NoSQL DBs are great, SQL DBs are great
 
NoSQL has its uses - there are good NoSQL databases out there (SQLite, for instance, makes a great relational, NoSQL, document store, filesystem all in a single embedded library). NoSQL isn't ideal for a lot of usecases, but works just fine for others
 
you need to understand your access patterns before you pick one
 
@JBis All filesystems use a write cache - when your program writes to a file, you are actually just updating this cache in memory - it is usually only written to disk when the cache is full, or your program releases the file handle. But releasing the file will not always write immediately, it may get queued, and instant power loss can lead to data loss
That's why when you want an atomic file update, you usually write to a temp file, fsync that, then mv the temp file to the actual file, then fsync again
 
i've seen that before. That explains a lot.
 
10:39 PM
That way, you won't end up with a partially written file
The main issues people claim with NoSQL are inconsistent writes - consecutive read-write-read-write queries might access stale data, or overwrite already updated data with bad values computed based on a stale read
 
so how do you decide between NoSQL and SQL db?
 
Those are real issues, but you usually pick a db to avoid them if your query pattern will run into those
Mostly based on the type of your data - if your data is relational, you will be better off with a relational db
 
when is data not relational?
 
Stuff like postgres has decades worth of performance and integrity work built into the db - if you try to replicate the same queries on mongo or a kv store or whatever, you'll be several orders of magnitude slower
 
stack exchange wouldn't be hard to write in a non relational manner IMO
er
 
10:42 PM
> The reason that SQL has this reputation for being slow, is that processing a complex query on a large dataset, inevitably takes time whereas NoSQL simply doesn’t provide the ability to run slow complex queries in the first place.
 
Relational data is referential - you will usually see it when a lot of your data references other parts of your data (posts belong to a user, votes belong to a user, deletions belong to many users, flags belong to a user, flag results belong to a mod, etc)
NoSQL is more commonly used in document use cases - instagram, for instance, might store comments, likes, shares in a sql db, but images and their metadata in a NoSQL db
A single image, along with its metadata, is self contained
 
@RaghavSood I don't understand this
 
You could easily store votes in a document though using an array of user ids
as long as you don't need to query the vote history of a user it's fine
 
You can, but as your query surface grows, you will end up writing your own query engine, and still be slower than postgres or mysql
But realistically, for a lot of applications, it really won't matter
 
that's why the primary consideration is your access patterns
I.E queries
you can't use a NoSQL DB effectively unless you completely understand all the queries you're trying to perform
 
10:45 PM
I'm still not understanding when data would not be relational
 
but the advantages is massive scaling capabilities and data flexibility
You don't join tables in a NoSQL
you look up documents
 
@JBis When you like a post on instagram, they essentially need to store the facts that JBis liked post 123, post 123 is owned by Dave, post 123 has 20 likes. These all relate to each other -> You're involving a users table, a posts table, a likes table, and a likes_count table - If JBis likes a post, you want all of them to update, or none of them to update. You don't want partial updates or stale data
Your action of liking has relations
 
Yes, got that.
 
For the image itself, once it has been created, it is a self contained object - the image and metadata belong only to that image, so you put it in a single unit called a document
If you then resize that image to create 5 different scaled versions for different devices, you can put those in the same document
Updating that image document doesn't require you to update any other data set
 
I see.
 
10:48 PM
A question and its content could be stored in a document db
 
Any change you make, such as size, filters, croping etc. apply only to the data in that document
 
Seems like a very narrow usecase
 
but the likes and votes would be better in a relational db as raghav says
 
Sure, but usually you'd pick one db and stick to it as far as reasonably possible
 
It's broader than you think as you can restructure information traditionally thought of as relational and not need to do it that way
 
10:49 PM
For simple use cases, you can trivially use both
But with an SE style use case, where it is heavily relational, using a relational db for everything simplifies life
Even tho individual sub components might fit a NoSQL or kv store better
 
How (could you?) would you design instagram likes/posts/comments in NoSQL
 
Your overall system works better as relational
 
I guess eventually you would use both NoSQL and SQL for SE
 
You store the information about likes and comments on the post document
 
The post document would be huge
 
10:50 PM
it's not necessarily smart but it can be done
that's not a problem with NoSQL
they support massive documents, scaled horizontally, that's the point
 
How is a large document not an issue?
 
because the document is just a key look up of a partition of memory
 
Could you store a post with 1,000,000 likes in memory?
@DaveS Oh so its doing relational behind the scenes?'
 
no you're not doing relational
lets take likes
you'd do it as an array of user IDs and check that array for your ID
if it's visible it's liked and if it's not in there you didn't like it
and when you like it you write to the document back your ID
but you can't really query every post you liked when you do it that way
 
i'm scared, there is a staff user in the js room
 
10:53 PM
it's only stored on the post
but you could also have a user history document
but it's getting stupid, this example
Relational is better for something like likes
wikipedia though, is a great example
edit history, only indirect references through https, lots of independent documents
 
The user id array is basically a foreign key
 
no keys don't work that way in nosql
DynamoDB for instance can be though of best as a hash table
your Partition Key (primary key) goes to a spot in memory
then you have a sort key which lets you get a subset of rows with the same partition key
to perform different queries with a different partition key you have to build a new index (hash table)
but because you're not relating anything you can split your data on as many instances as you want
so it's very reliable and performant at massive scales, with almost no work
 
But going back to your like example, you are basically recreating foreign keys
 
likes would just be an array of unique values to prevent the application layer from letting you like something you already did like and return the count of likes
but it wouldn't be good for doing algorithmic sorting on number of likes or anything people use likes for
except a foreign key lets you join data together
you'd have to perform a new query yourself to get the user document from the like ID to do anything with it
it's not a foreign key really in any sense of the SQL context, it's just a key
you can't think about NoSQL in terms of SQL or you're missing the point entirely
 
@DaveS yeah exactly
 
11:00 PM
Here's a good example of a NoSQL DB
We have a middle man API layer for one of our clients, they want to collect the data being returned by our 3rd party api service cheaply
so when the end user hits the API, the responses are stored in a DynamoDB by their ID so we have a relatively accurate copy of all data in the 3rd party API without a costly integration
and we don't need to build any schemas or anything
we just say here is this object, here is the ID, dump all the data at this address
we can then export/work with it later anyway we want
 
ah i see how that could be helpful for logs
 
@DaveS perfect thanjs
 
11:23 PM
I have returned
 
why?
 
good question
 
Because the halting problem is only unsolvable to you guys
I always return
 
the title is awesome - especially the second part in parentheses XD
 
I feel like the crisis was being a millionaire yet still choosing to work for Facebook.
(I didn't watch it)
 
11:32 PM
this guy is a sarcastic genius XD
his videos are about humour camouflaged in CS related topics
 
ha it's funny that I assumed it was serious.
 
he's a millionnaire and use to work for fb though XD
 
yo mehdi this guys life is depressing :(
 
Although, I suppose that means he likely lives in San Francisco and is already broke.
 
japanese ex-wife kidnaps children is a thing and it's sad
 
11:38 PM
wait... so all that stuff is serious?
 
looks like it
 
@DaveS yes :(
I saw a video of him seeing his kid finally
 
is it a joke?
 
it's a deadpan joke using his life as material
 
Quickly going back to what you guys said about the SE clone being too complicated...While competing with something as large as SE would be extremely difficult, simply building a clone I don't think is. The basic idea isn't complicated at all.
I think creating a working clone is very realistic even with a single developer.
As Rag pointed out, cost wouldn't be too bad either.
 
11:51 PM
Computing is cheap
 
Although I think it would be more than $30 a year
 
LIES
It'll be more if you grow to several thousand active users
So, LIES
 
Increasing ram on a basic GC instance is like $100
 
That's why you use Digital Ocean, Vultr, etc.
 
DO is $60 a year for the most basic
And with GC i get the first year free
 
11:59 PM
Building a basic functional SE prototype is probably do-able, handling close votes, flags, moderation tools doing it well and doing it at scale is probably beyond the means of an individual developer
eventually maintenance eats into programming
 
I'd agree with that.
 

« first day (3390 days earlier)      last day (1557 days later) »