« first day (3291 days earlier)      last day (1882 days later) » 
01:00 - 19:0019:00 - 21:00

01:10
Raghav needs his fiber optics cables to see the light
01:43
@RaghavSood Theres a whole suite of those types. iCal is another that follows similar format.
02:33
Yeah, I know, but they're mostly terrible
 
2 hours later…
 
1 hour later…
05:11
Grettings!
Does facebook app comment screen use physic based animation. Is that just a bottomsheet dialog? Any ideas?
05:59
Looks like a custom popup window with custom animations. My guess its not a bottomsheet dialog.
06:31
Hello people!
06:51
Hi.....
07:22
@Mauker lol
 
5 hours later…
13:00
only 121 difference remain in upvote
the new coc post will make history soon i guess
 
2 hours later…
15:53
timmy
16:26
posted on October 20, 2019

Android Weekly #384 🤖 #outlook a{ padding:0; } body{ width:100% !important; margin:0; padding:0; } body *{ -ms-text-size-adjust:none!important; -webkit-text-size-adjust:none!important; } body,.wrapper{ background:#ffffff; color:#505050; font-family:Arial, sans-serif; font-size:12px; } table,table td{ border-collaps

Yeah, it can be somewhat annoying to work around protobuf restrictions
But for the most part, it works
I actually never used it , but this doesn't seem very reassuring
It is overly alarmist - you shouldn't have any issues with using protobuf for normal CRUD or a bit more complex stuff
I use it a fair bit for some of my stuff, works fine mostly
Occasionally need to architect something in a slightly weird way, but you'd be hardpressed to not have trade offs in any system
It does break down when you get into super custom stuff tho, so I likely won't use it for some upcoming projects
If you look through his other posts he does seem to be angry about a lot of stuff :P
And while not strictly wrong from a purely theoretical point of view, building stuff in that way might put it out of reach of most people's understanding
Somewhat hacky but completely working code > theoretically sound, but incomprehensible code
True :D
For my use cases tbh I never really needed the performance provided by protobuf, so Serializable was good enough
Protobuf is less about performance, more about interoperability
It allows you to define types and data structures once, and then use them in multiple languages
Most languages do their own thing for serialization, protobufs sort of create an interlanguage method to pass stuff around
It's basically like JSON, but more efficient to transmit, and slightly more efficient to decode/encode
17:12
But you still can't send large binary objects right? (Like images for ex)
I mean... you can. Nothing prevents you from having a giant optional byte field to send stuff
But you generally wouldn't want to do that, since if you send a 100 MB byte field, the message deserialization will block until all 100 MB is received, which can have a fair bit of performance issues (esp. if you have a flow something like "check if message.type == DISPLAY; display image if yes"
Ideally, you'd push the large binary data into some object store (s3, redis, memcached, whatever), and share a URI within the protobuf message, so that the client can decide if it wants to pull the full data or not
Understood
What is your use case for using protobuf? (If you are at liberty to say)
We use it to define the API spec and types for the gRPC server
Although the protobuf stuff ends there, since everyone else interacts with it over a JSON Gateway to the gRPC server
But it makes updating the types and api definition super efficient
I've posted about it here earlier
Why not just define a schema somewhere? A meta store or something?
The way it works is that I can define something like: gist.github.com/RaghavSood/96391334e55e23778335eba1d8b5c25c. After that, I can run a script to autogenerate the golang interface, the gRPC server stub, gRPC client stub, a swagger/openapi definition for the API (which is used to generate the ruby, C# etc. clients for the JSON api automatically), the JSON gateway which converts incoming JSON to the gRPC call along with payload validation, etc.
So I only need to write the protobuf service definition to get server code, client code, validations, etc. Super efficient, very little (manual) boilerplate
@MehdiB. The protobuf definition is the schema, with the added benefits of tons of automatic code gen
17:24
Hmm, I was reading about Avro schemas and it seems like they're used to do the same
Validating a payload a system sends and receives (like used in Kafka)
I believe Avro is more oriented to data storage and processing, it doesn't cover services as well as protobuf (which only covers it via certain extensions/plugins). Seems like it was designed more for moving stuff around data warehouses/map reduce clusters than for exchanging information between networked microservices as protobuf was designed for
But i'm not really familiar with Avro
There's probably a far bit of overlap in valid use cases
That's what I dislike about Data related technologies
They all overlap on 75% of the capabilities
:D
That's why most places end up with their own standards over time
It's hard to quickly understand what's different between X and Y
That's what interns are for
Give two interns competing specs, gladiator-battle for a full time job
17:31
Hahaha
I lately learned about Apache Pulsar, and it seems like Kafka but with a separate storage layers, that's it. I asked the guy what's different concretely and he gave a 10min answer about storage and implementation detail and that the code base is completely different
I need to start learning about all this stuff properly
Otherwise indexing my 250 TB of data is going to be difficult
Unless...
Hahaha
The main issue I end up facing is that while stuff like protobuf/avro is great for transporting things safely/efficiently, it really breaks down once you start indexing the data into some database
It's easy to go from encoded->indexed, but not always easy to go the other way around losslessly
Since usually while indexing you might end up creating derived fields, groupings, etc. which are created for a reason
So my current approach is a massive kv store of actual raw data, and a pointer in each database to the raw incoming data. That way, if my database needs additional info, I can always go back to the source data/look up alternative indexes for it in other database types
Gives me considerably more comprehensive and faster querying at a cost of roughly 5x the storage
we always have 3 buckets / data sources per use case:
raw data (events, untouched except by the producer) -> processed data (data processed for general use cases) -> per use case specific processed data if needed (that would take data from either the raw or the processed bucket)
My flow is mostly similar:

1. Raw data coming from blockchains
2. Processed into common format
3..N: Postgres, dgraph etc. indexing it
17:44
yeah so that's pretty much it
But so expensive :(
48 TB servers would cost me like $600 a month
And that's the cheap part
Bandwidth :(
I assume 1 and 2 are quite cheap right? (are you using an object storage like s3?)
Object stores are too slow
Has to be local hdds at a minimum
The kv stores storing 1 and 2 are hit fairly often
And latency is a big factor here
Processing a transaction can involve a recursive search of all transactions it references, which could be several thousand additional GETs
If each get is 10ms...
What I really need is a massive server with like 1 PB of memory on two power channels
Unfortunately, I don't have access to the GDP of a small country
Shifting my job queue to redis because downstream people using my API can't fix their shti
I made the grave mistake of trusting users
I must live in exile
17:50
rookie mistake
I had hoped that since my users are mostly competent devs, they would create efficient, scalable systems
@RaghavSood wouldn't that be a good use case for a graph db?
@MehdiB. Yes, that's what dgraph is - but in some cases I need to go back to the source data
That's why I'm storing stuff in 3 different dbs - each is optimized for certain query patterns, and most of my usecases involve all three
So an enduser query will hit all three datastores, so all three need to be fast
@RaghavSood sqs with spot insances <3
Eh, I already have redis running, and it's all on GCP
17:54
GCP has something similar?
They have a pubsub system that would work
But I'm trying to not rely on vendor specific stuff since it is very likely we will have to move this to a bare metal machine in the coming months
Likely going to go from about 1m/events a day to 100m in the next 3 months or so
@RaghavSood would be fun to work with you on this stuff :D
thats the thing that sucks on cloud, you are tied to vendor specific services
17:56
Which is the point at which I need to redo it all
@MehdiB. This is a side project, so you're welcome to pitch in once I get the basics up :D
nice! :D
what is it about?
Which should be in the next month or so, once I'm done with exams and stuff
I intend to index most/all blockchains, and various associated metadata and sell it for profit to people with broken moral compasses
So mostly governments and financial institutions
To counteract the evil I'm doing, data will also be provided to universities and not-evil startups at a lower price
what kind of data? :D
The problem I'm trying to solve is that apart from Bitcoin/Ethereum/other mainstream blockchains, it is not cost effective for people to run APIs and hosted services for smaller coins - it will cost you more than you will get out of it, and most exchanges run wallets for smaller coins at a significant loss
yeah im interested as well, if i get some free time
18:00
It also leads to half hearted implementations, which leads to security bugs since they don't want to waste a lot of time on a small roi
I'm trying to build out a system which is generalized enough to ammortize the per-chain cost over all of them
Since most blockchains are fairly similar
il let you do the coding part, since i have no idea what your talking about :D
You then index stuff like blocks, transactions, addresses to create APIs which can be used by exchanges, blockchain wallets like @MarkO'Sullivan's, banks etc. to process user deposits and withdrawals
Such stuff exists for Bitcoin etc., but not for most coins outside the top 20 (out of 2000+), so if you provide it at a reasonable cost you're looking at a fair user base
right, i guess il have to delve a little bit more into blockchain stuff
noice
Plus, you can index other stuff - network data (nodes, peers, etc). Price data (order books, trades, etc), all of which have value to trading and analysis firms
so as i said
scraping the internet
18:03
Plus, with all the raw data, you can provide filtered subsets to academics who want to study network effects, economics, inefficiencies etc. in blockchains, which they are currently unable to do as data is hard to get, and fairly expensive
Yeah, sort of like a mix of Google and Bloomberg Terminal for blockchains
The pricing works out fairly well too - most exchanges spend ~6-10k a month on their blockchain infra. If you can provide a hosted service, giving them access to more chains for 5k a month, and get 4 exchanges to pay you, you're already pulling in far more than your own costs
You can also additionally run user facing services with this data, such as a financial site and explorers, which can bring in ad/subscription revenue
Plus, university tie ups for academic data, and trading houses can bring in even more
All of that without significantly increasing your own infra costs, since the underlying data is the same, just the query patterns change
as you said bandwith will be the most expensive part (after that initial cost for the boxes)
So even with 3 customers of each type, you could pull in ~50k a month easily
Yeah, I have ways around that
For customers with heavy, large queries, I plan to pitch them a $100k unlimited access, locally hosted set up on their network
Mostly for research places
Which can also be shared among universities (so all universities in a geographical area could collectively get one)
For other use cases like exchanges etc., the bandwidth is not that high since they will read a very, very small fraction of the data
with their own on prem installations
On the order of a few hundred MB a day
The main cost is ingress - blockchains can collectively produce several hundred GB a day
And that's just the raw data, the network overhead for the decentralized protocols is very high
wow! so it seems one cannot work on such project without blockchain knowledge.. :D
18:08
Almost definitely can, if you want to work on the data analytics/processing parts
i think not necessarily
Once you have the data, processing and querying it is much the same as most other data sets
there is a lot of other stuff that needs to be done in a sistem with that scale
Yeah, that's why this is a summer project
Can't do it in a weekend
Over a couple of months, possibly
It's not like Google took 22 years to get to their current scale :D
right :D
as i said this would be super awesome to work with
18:12
I should probably get some funding for this tbh
Gonna be hard to scale without it
Uni only gave me 5k :(
I could spend that much on coffee
yeah did you think about the hardware?
Rapsberry Pis ought to do it
Actually looking at dedicated machines :P
You can get a ~48 tb raw dedicated server for about 500 a month
A few of those in a ZFS pool ought to give you speeds of 1GB/s+, which should be enough
but is that a box or vm?
18:13
Baremetal
noice noice
so you would need like 5 of those
and i guess you will probably have to ping them about networking
Probably closer to 7 - five for my data houses, 1-2 for for the actual blockchain nodes. Plus more regular servers for the APIs
you can bulid apis in the cloud
Latency :D
it would be way faster if they were on prem
yeah
18:15
And cost. If they're in the same datacenter, bandwidth is free
Plus, the APIs should be okay to run on dedicated hardware - this is not a usecases where you'd expect wild variations in request rates. You can provision around a mean request rate with reasonably good assumptions that you won't need to scale up or down randomly
im not worried about apis
The thing is that if I can get a bunch of funding, it would definitely make more sense to just buy the servers and rent a rack - even with dedicated machines, hosting will run ~5k a month just for hardware, plus bandwidth
that should be relatively easy part, indexing and queuing and searching will be pain in the ass
You would spend less on a two year timescale if you just bought it
I'm interested Raghav, finally something I can extensively learn on at scale :D
18:26
I don't think I can afford you guys tho :(
meanwhile I'm debugging a garbage aws log
@RaghavSood stock options my man :D
The strike price is 1 BTC per share
@RaghavSood yeah im willing to offer my services for free
my best friend is also balls deep in crypto
18:31
TIL we're best friends
and he is nerd as well
yeah why not :D
@RaghavSood I'm good with your offer, I trust your fairness (reverse psychology)
Excellent
Good news Ivan, Mehdi will be funding the infra for the project by exercising his shares
1 share ought to give us a month of runway
so how many shares do I get?
At 1 BTC a share, as many as you want :D
18:34
there is a government budget here is serbia offering 32m euros
for it projects
Nice, that ought to be enough for a few months
but they require already finished products
That seems counter intuitive
"we'll give you money only if you don't need it"
yeah, i dont have exact details
but it doesnt make much sense
Working on assignments for job interviews are never fruitful
18:38
@Raghunandan how come?
i did like 5 in previous month
the time lines seem unrealistic.
hm it took me like 2-3 days to complete all of them
and after submitting the assignment you never get feedback.
Might be hard in India, but I'd generally avoid places that give out assignments or have annoying whiteboard interviews
asked me to do the above screens in 60 -90 mins
also store the data locally in db for offline usage.
18:40
well yeah thats a little bit unrealistic
i could do it in maybe 3-4 hours
i could to the functionalities in 2 hrs. But i am still working on the animation
nerd level 1000 talk
what is this protbuf shit i see?
@Raghunandan try motion layout, should be a lot easier
@MehdiB. hmm doesn't seem like the performance benefit are worth the hassle
every performance benefit is worth the hassle :)
18:42
LIES
^
one once i agree with raghav LIES
@IvanMilisavljevic and the bouncy spring animation also cases when handling cases of not swipe dismiss when you have the rv scrolling
10/10 should hire raghav for server management system
bottomsheetbehavior does all that. but need to modify the same
18:44
maybe try going with bottom sheet with motion layout
Hello people, I have a question if anyone would like to help, Is it a good practice to wrap a few lines of codes in a function that is not going to be used anywhere else? A function name that speaks of it's job using it's name
It could be
I generally move any code that doesn't fit into the "intent" of a function elsewhere
For example, I'm currently writing something that queue jobs into redis
yeah making function is a good practice
18:45
The overall function is a drain for an in memory event queue
also function might be useful in future
So this function's purpose is to connect to redis, and push the job
so yeah future proof code is a good idea
But converting the in memory job to a redis job string is in a separate function
Even tho it is only called inside the loop of this one function
But since it is an encoding step, vs a redis queue operation, it gets split
What if the code is not going to be used anywhere else? Does it still matter? What are my other options if any?
18:46
elasticsearch have what they call the FAB theorem (some version of the CAP theorem)
@TaseerAhmad It may still be worth it, especially if the parent function is getting very long due to lots of smaller steps
In most such cases, you won't lose any performance - the compiler will just inline small functions
But you will get more readable and composable code
@MehdiB. That looks nice, I'll check it out once I'm done with this queue stuff
Ok, thanks.
i think if the function is only used once there is a very high chance compiler will inline obviously depending on some restriction
@RaghavSood he went deeply into the math that lead them to some design decisions and architectures and tradeoffs, pretty cool stuff
too big article
tho worth a read
18:50
im following elastic from the beginning, they really have insane engineering team
@IvanMilisavljevic that's why I wished I worked on something with this much scale, you can't learn that stuff theoretically just by reading
right, thats how i learned the most of the tings i know
what high scale you worked on?
spending endless hours in cold noisy data centers
18:54
wow
i did entire networking for a company with over 200 network devices
maintained 2 dc for a supermarket chain with over 200 stores around serbia
ofc i didnt do it by myself i was a junior back then
but worked with some really awesome people that taught me a lot
can't edit postgrasesql.config
permission error
don;t klnow why
but have 758 permission
strange. now installing GUI on server -_-
sudo su; nano postgrasesql.config
idk if nano is already installed or not
when I opening it via vim
it show only tilda~
shows**
sudo su; vim postgrasesql.config
18:59
also having to deal with HP proliants forces you to learn a lot about server hardware
@IvanMilisavljevic cool
01:00 - 19:0019:00 - 21:00

« first day (3291 days earlier)      last day (1882 days later) »