« first day (3262 days earlier)      last day (1672 days later) » 

12:04 AM
am I really this dumb? How do I access the archives on this site? c-span.org/organization/?419/Archives
I see no button to search through the archives
 
Isn't it just the search bar on the top?
I imagine their archives are just their regular library
And you're looking at a page describing it
 
I can't search by dates with that
 
Stay in school, kids
 
alright, I'm an idiot
:47363576 yes, yes you are
 
12:39 AM
sup gents
 
Great start to a Saturday, prod issues while the CTO is afk and the rest of the team is asleep
I'm now being punished for not learning k8s properly
 
ow shit, whats the issue with k8?
 
If I knew, it wouldn't be a punishment, now would it? :D
 
true, true
have you finished your test with the prolog?
 
Yup, didn't even fail
I actually aced the prolog part
But turns out there was non-prolog stuff on it too
I should go to class more often
 
12:52 AM
lol no
fucking awesome
dont waste your time, except math and crypto
best part of my uni was cisco ccna, microsoft mta, cryptology and math
rest of the course where kinda mneh
 
1:25 AM
Well, sure, but I still need to pass them to graduate
 
yeah, that sucks
 
Well, at least prod is mostly working now
 
well you might learn some new stuff along the way
 
Plus, managed to wake up other people who know what they're doing with k8s
 
lol
you really have no soul :D
 
1:30 AM
If prod is down, the humans better be up
 
true, need to achieve those 999 sla's
im off to get some sleep its 3:30am overhere
good luck and have fun :)
 
Is your prod running fine?
 
I have no prod atm, so i can sleep like a baby
 
:(
Night, then
 
 
1 hour later…
2:41 AM
 
 
2 hours later…
5:03 AM
/o
 
5:39 AM
Greetings
 
6:12 AM
LIES
 
haha
 
Network Manager was being weird, so I replaced it with wicd, then connman
Now, I'm not sure how much of my network actually works
Not ideal
 
@RaghavSood i got a architecture problem., i have doubts can i ask you?
 
Sure
I might disappear tho if my wifi dies again
 
camera can capture photos upto 3 in a second. its stored in a folder say camera folder. you have fileobserver to observer changes in the folder. you need to upload the files to server.
How do you architect the same
say in 1 minute you get 180 photos and also you need to handle network connectivity issues.
 
6:23 AM
How fast do you need them on the server?
 
How does google photos sync photos?
how fast does not matter. it can happen any time later the syncing part
 
Google just uploads them one by one
 
and some the file upload fails. you need to upload it at some point later you need to know which one fails
 
But Google doesn't need to upload them at a rate of 3/sec
 
yes. so they batch wait for network and then upload?
 
6:25 AM
A very simple solution would be to run n parallel uploads depending on available bandwidth, have the server acknowledge each file it receives
As long as your average upload is > 3/s, you're fine
 
hmmm.
 
If the photos are likely to benefit from compression, you could do a batch upload every 10-60s to save data, which in turn would improve your average speed
The main issue here isn't flaky connections, but the amount of data
It's easy to hash a file, and ask the server if it has a file with the same hash to detect broken/missing uploads
But images tend to be large, and on a fair number of connections you might have trouble uploading them as fast as you can take them
Resulting in an ever increasing backlog
 
interesting.
 
But yeah, when dealing with large amounts of data files, hash tables are an easy way to detect duplicates/make sure each file is processed at least once
On the client side, hash each photo as you take it and save that hash and filename
On the server, as each photo is received, hash it and save the hash to a db
Then you can have an API to query if a hash exists on the server or not
If it doesn't exist, upload the file, otherwise move on
 
right. i was asked this in an interview. except for the hash part i was right.
 
6:31 AM
There's no real "right" answer here
Lots of different ways you could do it
Hashing is just a fairly simple one to implement
 
yes. there are lot of solutions.
 
A lot of it would depend on the other factors too - you might need different uploads for different scenarios
 
so how does google photos work - the uploading part only
 
For example, if you have a 1 GBps network, but an old HDD for the filesystem, you want to put your upload process somewhere that it can access the in-memory images from the camera from. OIherwise, your sync time will be dominated by disk access instead actual network issues
But, you also need to plan for the network going down and have a backup uploader that can cleanly sync batches from the disk
 
yes. reading from disk takes time. yes that can done with workmanager or jobscheduler
 
6:34 AM
@Raghunandan I'm reasonably sure they just do 1 file at a time as long as the network is available, splitting it into chunks so that they can resume paused/failed uploads without starting from the beginning of the file
That is largely more of a concern with videos
Since a video is more likely to fail mid upload simply due to filesizes and time
 
so if a file uploads is at 50% and you network connection is lost the upload resumes from the 50% chink when network comes upload. so the file size and the uploaded size is communicated using headers
so both the client and server knows what part of the file is received and sent
 
Yes, you could do that - HTTP supports it directly
Content-Range is used to specify what data you want, although in your case it's flipped since you are initiating the upload vs. the server initiating a download
 
hmmm. i answered that part as well.
 
That is byte-level tho, in most cases you would split a file into X MB chunks and sync those, then reassemble the file server side
That can help you speed up the upload of large files vs sending each byte sequentially if you have multiple parallel uploaders
 
hmmm. makes sense
 
6:38 AM
It's also broadly how torrents work
 
if you are sending bytes in parallel the server also needs to know about the order to reassemble
 
Yup, chunks are numbered
Usually a chunk is defined as number, length, data
 
shit i need to brush up my network knowledge the tcp and network layer stuff. i remember this reading in my college days
 
Generally, as for an upload system using chunks, I'd build something like:

1. N workers processing incoming data and dividing it into chunks, computing the hashes etc - they output a stream of `[fileid, chunkno. chunklength, chunkdata]`
2. M upload workers that read in chunks from the stream from 1. and upload them
3. A server side process that receives chunks, organizes them by fileid, and receives metadata like `[fileid:filehash, fileid: numchunks]`
4. Once all chunks for a file have been receives, it will stitch them together and verify the filehash matches
That allows you to upload an arbitrary number of files from an arbitrary number of sources
So you can write a single uploader
And then write two inputs, one for new images as they are taken, one for old images that weren't synced and are being pulled from the hdd
They both simply need to preprocess the data into the chunk format, and the uploader will sync them regardless of where they are from
 
hmmm.
 
6:44 AM
That's largely how most backup tools work
You generate a list of files on the disks you are backing up, convert them into blobs, and maintain a list of files in each blob, then upload everything
Then when you need to update a file for an incremental backup, you look up which blob it is in, and update only that blob+manifest section
You can even keep the previous blob to have versioned backups
It's also somewhat similar to how git objects work
Except that process is heavily optimized for text based files
 
hmmmm. blobs never thought of that
 
Blob is just another name for chunk, kinda
The difference being you usually have multiple chunks for a file
And multiple files in a blob
File backup systems often have to deal with thousands of very small files
It makes more sense to combine them into a single blob for tracking and syncing purposes
Whereas chunks are smaller pieces of larger files like images and videos
 
That's just a regular upload
But looks like they only intend for it to be used for small amounts of data, so that's fine
 
hmmm.
they do batch uploads
 
6:59 AM
They have resumable uploads too
 
yes. query the failed upload and resume from that chunk
 
Hello
 
Greetings, friendly-three-letter-agency agent
 
aw how did you know it was me. did you see the red light on your camera on?
wait you don't even live in the states....
 
I have a cover on my webcam, I wasn't born yesterday
 
7:12 AM
who said FBI was the three-letter-agency?
 
I said friendly, so it's obviously not the FBI
And no matter which agency you work for, physics saying that light can't pass through opaque materials takes precedence over whatever backdoor you've come up with now
 
this is NSA I got your camera, your micro, that photo of you from high school that you didn't want anyone to see
yeah you know the one
 
Good job, you've downloaded my profile picture
 
lol
 
hello
 
7:18 AM
Greetings, human-who-must-buy-additional-hardware-because-apple-insists-on-a-closed-ecosystem
 
:( true story
What if you could code ios apps on linux or windows? would that be bad for them?..
 
I hear IOS supports java now
 
 
4 hours later…
eLi
11:16 AM
i dont understand what is happening, github logged me out of my account, and i cant logging
i cant commit
 
Zoe
@eLi You've been suspended.
Or had your account deleted
 
eLi
it's not deleted i can see it
 
Zoe
Check your email.
@eLi Open in an incognito tab
Sounds an awful lot like a block. Got stuck there once for suspicious commits or something. Contact customer support
derp. You have a different link on your profile
Nvm ^^"
 
eLi
thats in incognito
 
11:34 AM
did you work with OCR? Like building one? :D
 
 
2 hours later…
1:32 PM
Today I went out to buy a bike
I was sitting in the dealership and I saw a chair with a nazi flag on it o.0
 
what bike? something specific?
 
Well, it's a lightweight bike. 125CC
Yamaha ybr
 
I consider buying one, maybe next year
 
It cost me about $1750 in total
It's nice for city commute, really good steering
 
ok, I thought you mean 'bike', not 'motorbike' ;]
anyway, good for you
 
1:39 PM
thanks
 
1:50 PM
I really want to buy a parrot but I am afraid I will not be able to entertain it ;_;
 
 
3 hours later…
eLi
4:52 PM
Why not buy an ipad for it :D
 
5:33 PM
ipad has no fur
 
eLi
6:30 PM
what best icon to present politics menu :D
dustbin ? :D
 
7:25 PM
LIES
 
8:18 PM
o/
 
LIES
 
eLi
23:23 PM here let me go back home
 
Why bother
Just stay the night
 
eLi
am so tired today
weekend i got tired than week days
Good night you all
See you after few hours
 
9:03 PM
LIES
 
 
2 hours later…
10:34 PM
Greetings, Timothy
 

« first day (3262 days earlier)      last day (1672 days later) »