« first day (3687 days earlier)      last day (1262 days later) » 
01:00 - 16:0016:00 - 23:00

1:20 AM
Ok, I think I have a better test (as well as some much-improved code in the RCU class). I would still enjoy hearing any feedback.
 
 
4 hours later…
5:18 AM
hey guys
how can i make a HTML + CSS rendering engine?
 
cbg guys o/
 
5:46 AM
cbg
Ah, @TheLittleNaruto - I think Martijn and I had some suggestions for your nested dict question.
 
@PaulMcG Yes please tell me.
 
Scroll back in the chat, it's not far, it was a quiet day.
 
Alright
 
@Praveen Not fully understanding what you are asking for, since it sounds like you are thinking of embarking on a major major project.
Like you want to write your own browser or something.
 
AAB
5:55 AM
@roganjosh @JonClements When I google a bit I see a lot of people of saying how NoSQL databases like MongoDB are better than RDBMS. Some posts claim that MongoDB should be the DB of choice for python and node.js, Is it because it's easier to write code in it in python as in user does not need to know SQL and all you have to do is create a json object?
I am just looking into MongoDB and somethings look nice, for eg in case of RDBMS say I want to retrieve product information like product_name, price, URL, vendor_name I would need to have a table for vendor_name, vendor_id, one for product_name and product_id, finally a table that has foreign key relation with product name, vendor name table, price, URL. I need to do a join between the 3 tables to return this data. whereas in MongoDB all I have to do is create a JSON document with all 4 fields.
 
Welllll, there is a little more to it than that. So you are going to replicate the vendor_name and vendor_id across all the product documents for that vendor? And are you sure you'll get the right vendor_name to go with the vendor_id every time? What if the vendor changes his name? You'll have to modify all the product documents.
 
When the data gets bigger, queries will not be easy to make in MongoDB
For example look at this scenario: dba.stackexchange.com/q/262602/203899
 
What about things like the vendor billing address, and invoice history? It sounds like you'll eventually have a vendor document, which you link to by vendor_id from the product documents - a lot like a join between tables once you are done.
Not to say that MongoDB is totally impractical, lots of people use it. It is best to think about it though using regular data concepts, such as normalization, and object relationships. Just because you don't define a schema up front doesn't mean you don't have to do some planning.
 
True ^
 
AAB
@PaulMcG thanks for replying
 
6:08 AM
I think there is a MongoDB tutorial about writing a blog application, with attached comments. It would be very helpful to go through that, and then see how you would do the same using a simple relational db like SQLite
There is a very nice SQLiteDB visualizer tool that makes experimenting and dabbling with it very easy.
I also looked at MongoDB a few years ago, and went through their intro/associate training course - but I haven't had a chance to really try it in anything of real substance, and I'd have to do a major brush up to get back into using it. But I did like the document model for some things, over the traditional SQL relational db.
Note that nowadays, relational db's have ways to store searchable JSON data columns, so the lines are blurring between the two.
 
AAB
@PaulMcG the other claims of MongoDB is the ability to handle high traffic easily and run nested searches like {d1:{d2:[to_find]}} are these claims always true or this could depend on a few things as well?
 
I'm thinking specifically of Postgresql and Oracle. MySQL maybe has this? And I'm pretty sure SQLite does not (since its name ends in "lite").
It always depends on things. Relational dbs handle high traffic also, but they will be slower at joins compared to a Mongo query that can get a document and all the associated data in one gulp. But you really need to think hard about that document design.
 
AAB
I was trying to do a Full text search Postgre had to_tsvector and to_tsquery to achieve that I needed to create a column with the token info of the same and run queries on that column for search, the only diff in MongoDB was I just had to select the field make it Text index and then a method to search away. It seemed easier in MongoDB but I dont know about any performance difference, so far it seems like it make the most of it I need to be just as careful with it to make the most of it.
 
So probably a better example for Mongo than your product <-> vendor mapping might be something like vendor -> invoice history.
You might have separate vendor and invoice documents, but within the vendor document keep some minimal invoice history, like the invoice summaries from the past 'n' months. In the invoice document you could keep all the line items - in contrast with a relational db where you would have an invoice table and a line-items table. So here is a clear win for MongoDB, no joins needed to reconstitute an invoice and its line items.
In the vendor doc, the invoice history might just have the id, date, and billed total, to make it easy to summarize information for a vendor without having to do any join or second lookup. Then for more detail, then you would chase down the invoice id's to the separate invoice docs.
So that would be much like a join between vendor and invoice tables.
My general design process (which has not really seen that much with MongoDB yet, so... whatever) would be to do a regular table design with foreign keys, then start thinking in documents instead of rows, and what typical joins could be incorporated into the document by either replicating some data, or just moving the whole related data into the document (like I did with the line items above).
Even in relational design, it is common for performance to denormalize the design in places where joins for summary info can be expensive - so what you do is copy some data from the joined record to the original record, at the cost of duplicate data storage and taking on the burden of ensuring data consistency across copies.
The advantage in something like MongoDB is that you can denormalize/copy/subsume multiple joined records into a parent document, which is (or at least used to be) frowned upon in relational table db's.
In short, the answer is "it depends..."
 
AAB
6:29 AM
@PaulMcG the reason I read on the same was storage is cheap and CPU power expensive
hence the new approach to store data etc.
@PaulMcG thanks for the suggestion I will keep your approach/suggestion in mind appreciate you taking your time and responding :)
 
Replication isn't just about consuming storage - you also take on the responsibility of keeping things consistent, like the example where a replicated vendor name has to be updated in every copy if the name changes. You might make the judgment that "well, that may be true, but it is also very infrequent, so I'll just deal with it when it happens." But that is the thought process/trade-offs you think about when doing this design.
Good luck!
 
AAB
@TheLittleNaruto thanks for the link as well :)
 
@AAB sure :)
 
AAB
@PaulMcG True would I be very wrong in assuming that doing an update like that will be super fast? I understand I could have the vendor_name in many documents across collections but would updating still be very slow and expensive?
 
Doing an update like that would be slow and expensive. You have to find all the documents containing the vendor name, and then read/rewrite the entire document (I think MongoDB does some optimization here, but that is speculation on my part). But you do that because you save much time in avoiding the need for joins in thousands of routine queries that would have had to get the name over a join if the name were not replicated.
 
6:49 AM
I am trying to decode a string which is retuned from mongodb query which is stored in 'items' variable: items = b'\x00\x01\x00\x00\x00\xff\xff\xff\xff\x01\x00\x00\x00\x00\x00\x00\x00\x06\x01\x00\x00\x00\xfc/pKMAAB+LCAAAAAAABAD==\x0b'
Tried the below code block to decode it:
v = base64.b64decode(items)
print(v)
decompressed_data = gzip.decompress(v)
print(decompressed_data)
But it results in the below err:

gzip.BadGzipFile: Not a gzipped file (b'\xfe\x92')
How do I uncompress it?
 
You have to chase down the code that wrote that data and then reverse the encoding/compression steps that it used. The trailing '==' does look like base64 encoding, but all those leading nulls and binary bytes don't.
 
@PaulMcG Exactly what I wanted to do. Thank you :)
Now I need to assign certain weight to each condition.
So instead of returning True/False it would return that weight.
This weight value vary for each condition
Say cond1 will have weight 1, cond2 will have weight 2 and so on..
Like this: (lambda: record.foo1 != "bla1", 1) ?
 
Yes
 
Okay thanks
 
I deleted my comment because my suggested post didn't have tuples - but you intuited my meaning anyway.
 
7:02 AM
^^
 
Yes, make the list a list of condition/weight/attrname tuples, whatever you need. Then you can just walk down the list until you get a matching condition.
 
Guess this tuple solved one more thing which I wanted to add to condition
 
Since all the conditions are just comparing a particular attr to a literal value, you could rewrite them as an (attrname, test_value, weight) tuple, and instead of using a lambda, just do if getattr(record, attrname) == test_value: return weight
But you have grasped the basic pattern, and this gets rid of your cascading elifs and makes future extensions more manageable.
 
Yes true exactly what I wanted.
@PaulMcG Ok will try that
def validate_record(record):
    conditions = [
        ("cond1", lambda: record.foo1 != "bla1", 1),
        ("cond2", lambda: record.foo2 != "bla2", 2),
        ("cond3", lambda: record.foo3 != "bla3", 3),
    ]

    for cond in conditions:
        if getattr(record, cond[0]):
            return cond[2] # returns weight
    return -1
Is this correct?
 
Except in your original test, the attrs (which you have as the 0'th element in each condition) would be "foo", "foo2" and "foo3" (I renamed "foo" to "foo1" to make the output line up pretty).
If you do the attrnames, then you don't even need those lambdas now.
And your for statement can unpack each condition into separate vars, which is more readable than the indexes.
# assuming you drop the lambdas and your conditions look like:
#  ("foo1", "bla1", 1)
for attrname, test_value, weight in conditions:
    if getattr(record, attrname) == test_value: return weight
 
7:14 AM
attrnames as in named-parameter ?
Ok got it
 
No, as in the name of an attribute in your record
 
Yes right
 
Hi guys, does anyone know how to make gaps between bars of bar graph in matplotlib?
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({'Delhi':[55000],'Mumbai':[20000],'Banglore':[19000],'Hyderabad':[37000]})

df.plot(kind='bar',align='edge',width=10)

plt.show()
 
Your google would be just as good as my google on this one. Maybe someone will pop in later with more matplotlib-fu.
 
Yes ive tried google, thats why I tried out width=10
 
7:19 AM
Is there something like a spacing or space or sep argument maybe?
 
No there isnt.
 
7:30 AM
What if you did width=0.8? The links I found in google seem to treat this as a proportion of the allocated space for a bar where each bar gets 1.0 of space.
I've got to check out for this evening - perhaps someone might come in later with other suggestions.
rbrb
 
8:18 AM
such question is getting answers away from the main question being asked by the OP.
 
cbg
 
9:04 AM
@PaulMcG Yes so how can i start?
 
9:59 AM
It's crazy that events like these are non news nowaydays: troyhunt.com/inside-the-cit0day-breach-collection
Also did somebody ever face the problem of finding out what website was breached? I would kinda like to know if it was something I care about, but it only says email breached, not on what service/website
 
cabbage
 
Hi all. I want to reduce json-string size by representing integers as pairs of bytes but it is only worse due to its b'\xff' notation. What is the right way?
 
@Powercoder quite likely - the right way is not to bother doing that?
(I'd imagine unless you've got very large numbers in general, most of the time you're going to be increasing the size)
 
10:35 AM
Is is strange? I dont want my hundreds of integers to take up space by the number of their digits, if they can be 2 bytes only.
 
what's the range of your integers?
 
Up to 60k
 
@AndrasDeak I mean I saw that, but there a millions of website, how can I figure out where MY account has been breached?
 
Floats are even worse with their precision
 
@Powercoder Why pairs of bytes? Hex is denser and practically the same encoding.
 
10:41 AM
@Hakaishin do you think that would be a good thing to publicly disclose?
"Here's where this password was taken from"
 
@MisterMiyagi hex takes 2 symbols to represent 1 byte
 
You're over optimising and adding too much complexity... json parsers between systems can naturally handle numbers and floats... if you end up trying to encode them (you're not going to get an advantage doing that anyway) - you're going away from standard json and putting overhead on the system creating it and the system receiving it to know it should do something to interpret the data...
 
@Powercoder compression?
 
@Powercoder Bytes are represented as pairs of Hex, so...
 
I mean, it's in the data breach anyways... So the baddies have it, why not give it in a nice api format to the users. Or am I now expected to change the password on around 150 websites?
Or what am I supposed to do?
 
10:42 AM
@JonClements please, "you don't need it" is not an answer
 
hey guys
 
@Hakaishin because 12.5M out of 13M don't know about the breach and have that same password still...?
 
I already use a pw manager and have uniqueu pws, but I still don't want somebody have access to my cooking recipe site or something. And the only way I see this go I either know where to change the pw or change them all, which just seems unfeasable
 
I need to reduce message size as much as possible
 
@Hakaishin you can search for your password in breaches. Where you used that: change it.
 
10:44 AM
Packing and unpacking methods does not matter
 
I looking for a good software design pattern. can anyone help me? thanks in advance.
 
@Powercoder then this isn't the way - maybe look at protobuf or something... and also don't forget if you're sending json to systems or getting it from systems, you can always look at gzip compression which'll get handled automatically by most systems without complicating the format of the data itself
 
@AndrasDeak but I have random generated ones, which would again leave me to search 150 pws, I guess the trouble is a bit less of searching so many instead of changing so many, but only by a tiny margin
 
@Powercoder Then just compress the entire message?
Compression is better the more data you compress. Splitting the data and compressing each individually is much worse.
You might also want to use a different format, e.g. BSON
 
I'll take a look, thx
 
10:46 AM
@Hakaishin download the hash dump and do your search locally.
this is something only you can do as per my earlier argument
 
As I know, send and read original but optimised data is faster than compress, send, decompress and read. Time is everything here.
 
but wouldn't I then need to know the hashing algorithm the websites used?
Just to clarify that I understand you right, you suggest download the hash dump, hash all my 150pws then search that hash in the dump and then I know on what website my pws have been breached?
 
@Hakaishin no. The dumps are plaintext. Troy hands out partial hashes with a well-defined hash, probably md5
@Hakaishin yes
 
@Powercoder You just said message size is everything. Which is it?
 
Both :)
 
10:49 AM
haha, ah tradeoffs :P And what function to use for the trade off, a problem we humans are very bad at
 
@Powercoder do you know the thing about free lunches?
 
Either way, using standard compression on the entire message is much, much, much faster than anything you can cook up yourself.
 
I need to transfer large message but parted on ~300 bytes
 
It's also going to be much, much, much faster than what your network can deliver.
 
Does it also have to be JSON? Not a very efficient encoding for numbers.
 
10:51 AM
It doesn't
 
@Powercoder This doesn't really change the content of your messages – socket's don't care what is contained in their payloads.
 
The rest is ok, so I'm looking a way to deal with numbers
 
You might want to clarify why you need to chunk the messages like that. ~300 bytes seems a very odd chunk size.
 
For speed ;)
 
Ah, my bad ;)
 
10:54 AM
@AndrasDeak Sorry I'm confused by this, are the dumps now plaintext or md5 hashed?
Ah I guess you are referring to haveibeenpwned.com/API/v3#Searching by range. Man I wish I could just pay somebody to check this
 
300 bytes is system restriction on message length. I need to pass large messages in real time. Transfering is the longest part here, right? So I want to make message as shorter as possible to minimize transfering operations. Also, I wouldn't want to waste the gained time on compression.
 
@Hakaishin clearly. I'll try to explain from laptop.
 
300 bytes might take a moment with one of those... but... :)
 
Sending more data at a time makes the compression more efficient. Sending smaller packets at a time makes the compression less efficient. So it's either or. To increase compression efficiency you need to send larger chunks of data
 
@Powercoder isn't your real bottleneck the overhead?
 
11:02 AM
also isn't this premature optimization?
 
@Hakaishin probably not, but it might be futile
 
@Hakaishin ofc I'd compress the whole message and then shatter it
 
@Powercoder you have to wait for the last batch anyway, right?
 
Yes
 
So I don't see the issue. Compress the json, send in tiny batches, reassemble, decompress the whole thing once.
I'll only believe this doesn't help if you try and time it.
 
11:07 AM
I'll try, but still
 
You said transmission time is longest. So send less data.
Fewer batches.
 
@Powercoder So you are working with some embedded hardware or similar? Usual payload sizes are 4096B and up.
 
5-digit int takes 5 bytes if it is string and only 2 bytes if it is int16
 
@owgitt Don't you think you're in the wrong room?
 
I need to translate the above code to python. How?
 
11:10 AM
@Powercoder no... a 5 digit int takes 5 bytes as an integer, but 6 bytes as a string because of the "'s...
 
Again please
Oh you mean eol character?
 
@owgitt that's a lot of unformatted code in a different language. Please post that elsewhere.
@owgitt we're also not a code translation service
 
But why is it 5 bytes as integer?
 
Int16 is 2 bytes and can store values in range 0..65535?
 
11:13 AM
Erm, that means you are going to add the encoding to each number?
 
@Powercoder JSON wise... the number 65535 can be literally 65535 (5 chars) and automatically recognised as a number... if you encode it, you need to put it in a string... which means it becomes "ffff" (6 chars)
 
This is exactly what I'm trying to deal with
 
@Hakaishin so the breaches are plain text, that's why they're a problem. Troy Hunt is nice enough to shoulder the responsibility of giving fuzzy access to the passwords inside. You can submit queries on the website for convenience, but if that doesn't work for you (for instance because you're wary submitting valid passwords anywhere) you can download a hashed dump of passwords from haveibeenpwned.com/Passwords under "Downloading the Pwned Passwords list".
 
so in this case you're complicating the json and making it larger - that's what I'm pointing out... @Powercoder
 
In other programming languages are no problems to paste bare bytes in a string
 
11:17 AM
download via torrent if possible
@Powercoder you should probably use other languages then, if that works :)
 
Sad to know :)
 
This has nothing to do with programming languages, this is about JSON...
 
@Powercoder Python is no problem pasting bare bytes into a string either...
 
12GB is so low for so many PWs, It's always amazing comparing images to text data. Images are sooooo huge
hmm 5 peers, sad
 
@MisterMiyagi it is happening by strange way, wouldn't you say?
I mean it adds b'' around it
 
11:20 AM
I think the point you're missing @Powercoder is that a string in JSON has to have " around it... so you're always adding 2 bytes in the payload straight off the bat... so the number 0 in JSON is literally 0... why if you encode it as hex and put it in a string it becomes... "00"... which is much larger
 
@Powercoder Are you looking for def encode_i16(num): return chr(num // 256) + chr(num % 256) perhaps?
 
And represents single byte as 4 symbols if it is not printable
 
@MisterMiyagi or binascii.hexlify ?
 
@JonClements Oh, right
 
Course you, useful builtin helpers destroying the need for carefully crafted helper functions!
 
11:21 AM
Can someone who knows network stuff say how much overhead is involved in sending 50000 300-byte messages?
 
@AndrasDeak Does "shudder" count as an answer?
 
@MisterMiyagi bane of my life
 
My hunch is that micro-optimising the message pieces won't lead to any kind of significant improvement.
whereas reducing that to 500 compressed messages would make a dent
 
Again, 4096B is a pretty standard message size when being pessimistic. At 300B We're talking about sending at least 10 times as many messages.
 
@AndrasDeak, probably so, it is my personal interest
 
11:23 AM
Realistically, 50 times.
 
@Powercoder what is your personal interest? The micro-optimisation instead of compression?
 
Not instead
 
I don't know a thing about networking so I can't be sure, but to me it seems the kind folks here are bending over backwards to entertain a willful XY problem.
 
@AndrasDeak it's a quiet day at the moment? :p
 
@AndrasDeak I guess they are unsalted? Since no salt is mention and it wouldn't make sense? So now I just for loop over my pws hash them, and loop over the downloaded hashes and for each collision I know I should change it? Isn't' the collision likelihood quite big when only using small partial hashes?
 
11:26 AM
@JonClements Far be it from me to ruin everyone's entertainment :P I just like us to all be on the same page.
@Hakaishin actually I may have misremembered the "partial" hash part. Just read what's written at the link... it should be self-explanatory. But yeah, you'd eventually take the dump of hashes, and grep your own hashes inside. It's a solved problem to grep a bunch of things in a file.
"bunch of things" being your own passwords, "a file" being the dump
 
Talking of pages... the TCP Segment Structure on en.wikipedia.org/wiki/Transmission_Control_Protocol looks nice :p
 
ugh, grep, my arch nemesis. I'm gonna try the slow python solution if that doesn't work I will revert to grep
 
@Hakaishin see you next month
 
alright, I will go with grep if it's that futile
 
then just loop over your own hashes and grep one by one, in bash
 
11:29 AM
@AndrasDeak didn't mean to waste anyone's time. The question was if I can make non-printable symbol to space 1 symbol instead of 4 like \xff. Thanks for answers though, I'll use gzip or something like.
 
@Hakaishin I just imagined a 100M-iteration loop over a file :P
@Powercoder as I said I'm not sure it will help, and your requirements are quite strict. That's why I think the only way is to try and see.
 
@Hakaishin Python isn't too shoddy if you mmap the file and then you can use re.findall/re.finditer on the mmap'd object...
 
in any case it seems like a lot quicker thing to implement and try, rather than adding a custom encoding/decoding layer to your messages
 
sorry what is nmap? I find a bunch of port scanners with that name?
 
mmap not nmap...
 
11:32 AM
ah
 
11:49 AM
Looking at so many pws in clear text makes me uncomfortable
 
You mean your own?
you could probably hash them all without having to collect the plaintexts in a file :P
 
yes :D Also I just realized I still have quite a few duplicate and non random pws in there, time to change them. Also while changing one I noticed our uni allows you to localize the websites calendar to different calendars like hebrew and hijiri
@AndrasDeak I mean the firefox lockwise export is plaintext, so don't know how else to get them out of lockwise
 
Ah...
I'm using gnu pass so I think I could fetch each password and hash them on the fly, writing to a file
 
does it come with autofill for browsers?
 
12:05 PM
@Hakaishin it lives in the command line :)
 
:D
 
is trio.open_nursery() is equal to anyio.create_task_group() ?
 
12:18 PM
@JonClements Brings back memories: I taught TCP/IP for Learning Tree for about ten years from 1993.
@JonClements You're full of good ideas- I'd quite forgotten that trick.
 
@Hakaishin yeah, imagine trying to find out which passwords are compromised by comparing them against leaked hashes, only to accidentally have your own system compromised and your plaintext passwords leaked. :D
 
lol, it is literally one large file.... 12gb file haha
 
@holdenweb aww shucks... blushes... what can I say - it happens occasionally :p
 
12:34 PM
Having a bit of trouble with Heroku. Locally, I am able to create migration scripts for 13 tables and the application runs well. However, when I push my changes to heroku, only 11 tables are shown. I have tried running heroku run flask db upgrade but this shows ERROR [root] Error: Can't locate revision identified by 'b824fd5038af'.
There are couples of suggested solutions I have looked. Bottom line being to delete the alembic_versions. This file does not exist, and the fact that it is working locally, I do not understand why the changes cannot be registered with Heroku
 
12:46 PM
I am facing trouble with nginx and flask. Locally my ML application is running but after deploying it on flask it throwing a 504 timeout. I looked up online and I was unable to find the solution.
 
@MisterMiyagi the OP on meta isn't clear as the original post is that
 
1:01 PM
morning cabbages, folks
 
1:23 PM
Oh god testing 1 pw takes longer than a few minutes even with mmap.rfind. Is there a more efficient way on Windows?
 
how many passwords are you checking - and how big is the file you're scanning?
 
ok so 1PW is around 5min. This makes 1000min, which I guess I could let it run over night... maybe
209
the file is 12gb
i'm using mm.rfind(search_string), not sure if there is a faster way
 
12GB of data is aa lot to string-search through. Any way you could use an index?
 
i know db indices, but that is only usefull if you do the operation more than once. I need to do it only once
I'm checking my pws against a leaked db of pws
 
you might be faster searching for all 209 PWs at once.
 
1:29 PM
ok 1pw already has that functionality (in Watchtower)
 
@Hakaishin it takes that long to pass the file. That ^
grep -e pw1 -e pw2 ...
 
@inspectorG4dget i dont have 1pw
@AndrasDeak what do you mean with to pass the file?
 
@Hakaishin N iterations. Searching m passwords takes M*N
 
sorry, I seem to have misunderstood "testing 1pw"
 
Check each password in one pass
Grep might also support patterns in file
 
1:33 PM
How do I check each pw in one passs? Right now I have this and I thought this already loads the leaked data only once?
    with open(leaked_data_path, "r+b") as leaked_data:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(leaked_data.fileno(), 0)
    with open(pw_path) as pw_file:
        pw_data = get_pw_website_pairs(pw_file)
    compromised_pws = []
    for row in pw_data:
        print("Testing pw: " + row[1] + " from website: " + row[0])
        hashed_pw = hashlib.sha1(row[1].encode()).hexdigest()
        found = mm.rfind(hashed_pw.upper().encode())
        print(found)
        if found != -1:
 
build a pattern of all PWs
 
how big is leaked_data?
 
12gb as said before?
 
... and is it just a text file with URL and pw hash?
 
per line pw hash:number_of_occurences
 
1:37 PM
how many rows in pw_file?
 
209
 
ok part of the problem definitely is that you're doing a str.rfind on a memory mapped file (that's what mm.rfind boils down to)
how much RAM in your machine? Alternatively, how much RAM do you want to use?
 
@Hakaishin hash all your passwords at once... so you get hashes = ['hash1', 'hash2']... then scan the file once using found = re.findall('|'.join(hashes), mm)
 
@JonClements Any way to get progress on this? How do I know it wont run for 24h?
@inspectorG4dget 64gb, as much as possible :D
 
it'll take mostly the same time as it does now....
 
1:40 PM
so what's the benefit?
 
I think I have a faster solution
 
and because you're sequentially scanning forward in a file it'll make things much more suitable or mmap and disks generally aren't optmised for keep reading backwards from the end of something
@Hakaishin the benefit is... image the system loads 256mb of file into memory each time... right now, you're scanning until you find something, and if that appears very late in the file, it might be paged out by the next time you scan the file...
... using this approach, what happens is, you look for the 209 hashes all at once in each chunk you load until the end of the file... you're not jumping around and making the most of effectively sequentially reading a file
 
honestly, "the set of all your own passwords" should be a lot smaller than the totality of leaked passwords
so why not build a set of your own passwords, and then read the other file only one line at a time, and just do a set lookup?
This way, you completely bypass the issue of having to lookup through a large file, instead simply iterating through it once
I suppose one issue could be if i misunderstood the case entirely, and this is not feasible for "x" reason.
would love to understand the x in that case :P
 
@JonClements Because I guess any solution will take a while, I would like a solution which I can interrupt in case I gotta turn of the pc or something. With my current one I can just discard the pws I checked, with the other approaches I'd have to let it run in one sitting
I'm a bit confused why my one long standing password, which I used a lot doesn't show up as breach, especially because somebody in Russia logged into my Spotify account with it, which prompted me to change all old pws. And stupid Spotify still doesn't have 2FA
 
leaked = {}
with open(leaked_data_path, "r+b") as leaked_data:
    for pw, occ in csv.reader(leaked_data, delimiter=":"):
        if pw not in leaked: leaked[pw] = 0
        leaked[pw] += int(occ)

with open(pw_path) as pw_file:
    pw_data = get_pw_website_pairs(pw_file)

for row in in pw_data:
    url, pw = row
    hashed_pw = hashlib.sha1(row[1].encode()).hexdigest().upper().encode()
    row[1] = hased_pw

for prog, (url, hashed_pw) in enumerate(pw_data,1):
    print("Prog:", format(prog, ',')
 
1:47 PM
@Hakaishin the benefit is that you have overhead from iterating a 12GB file
most of the time is not spent checking strings
grep -f pw_file dump_file, there
 
yeah, but with a machine with 64GB RAM, that file should just be sucked into main memory
 
one pw hash on each line in pw_file
@inspectorG4dget ah, I missed that
 
otherwise I would have recommended reading the file in chunks and iterating over all unfound passwords
 
but I presume 62 GB is taken up by windows
 
xD
 
1:48 PM
bwhahahaha
 
@inspectorG4dget an OS unless tuned is unlikely to want to go that mad on its file system cache
 
honestly I'm kinda mad that windows takes 50GB from my 500GB primary disk
that's waaaay too much
@inspectorG4dget I'm running this right now, let's see how it goes
I guess each python process will run in it's own core and have it's own memory?
@inspectorG4dget lol, memoryerror
 
umm... that version isn't using mmap....
 
2:06 PM
@inspectorG4dget hashlib.sha1(row[1].encode()).hexdigest().upper().encode() is suspect
two encodes in a row
I don't mean the memory error; rather a type error
if sha1 expects bytes then its hexdigest is probably also bytes
Huh, apparently bytes have an .upper method. That's weird.
 
hexdigest() doesn't return bytes
 
@ThiefMaster :O
Also weird. Carry on then :P
 
Hello can you please brief me about current discussion
 
I'm trying this right now: How long would you give it?
with open(leaked_data_path, "r") as leaked_data:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(leaked_data.fileno(), 0, access=mmap.ACCESS_READ)
    found = re.findall('|'.join(hashes).encode(), mm)
    print(found)
 
@YeshwinVermaTheProgrammer that sounds like you are too lazy to read the history?
 
2:10 PM
There's a name error on the next line so I didn't just assume that it works
@YeshwinVermaTheProgrammer no
 
@AndrasDeak yeah, couldn't copy paste it, but just fix the name error and then it runs
 
@Hakaishin still memory error?
 
the findall approach no, the inspector approach I didn't rewrite to use mmap
I'm kinda unsure which will be the fastest
 
@Hakaishin grep
 
@ThiefMaster oh hey - how you doing?
 
2:13 PM
Which windows doesn't have...
 
generally doesn't hurt to install git bash when working on windows...
 
I'd recommend ditching mmap. Read the leak file in chunks and check each chunk against the unfound passwords
 
this makes me wonder how bitwarden can check if my pw was leaked in 1s
because I read that they also use the pwnd db
 
@Hakaishin trie?
Or just a db more efficient than a file
 
I see
 
2:20 PM
@Hakaishin I just found this site, it can give you some very detailed information about where your account and password have been revealed.
 
Now I'm already invested in this program :D
@PaulMcG ok, nvm this website is amazing :)
They had the breached pw, thanks :) Now I can crosscheck if it's also in the pwnd dataset
Luckily it's a very old one and I don't use it anymore anywhere, atleast it's not in the pw manager, I wonder what old service got hacked
Yes, it was that one probably thanks :)
 
3:27 PM
does anyone know if python-dbus comes installed by default (it did on wsl at least) or should I list it as an explicit dependency (or if I should do that anyway)
I have a very old commit where I stopped listing it explicitly but for the life of me I can't remember why
 
I wonder if any password breach checkers use bloom filters. They're space efficient sets that save on memory by only replying "no" or "maybe" to membership tests
@aadibajpai dbus does not come installed by default on my machine. You should probably list it as an explicit dependency.
 
sounds like the perfect application fit it
 
@Kevin which OS, btw?
 
Windows 10
 
oh I don't think it would on windows, it's a linux thing
sorry forgot to mention that
btw side note is poetry really that painless to use than setup.py based stuff?
 
3:37 PM
I suspect a fresh CPython download for Linux wouldn't have dbus included, because it's not listed in docs.python.org/3/library/unix.html. But perhaps there are other distributions that do bundle it, for example wsl
 
@aadibajpai there's probably no "installed by default on linux", ever
 
i use pyscaffold and it's fine, dont know about poetry
 
python-dbus/oldoldstable 1.2.0-2+b3 amd64
  simple interprocess messaging system (Python interface)
I do have python3-dbus, which might have to do with debian transitioning to python 3 by default
 
@aadibajpai nothing in packaging is ever truly painless. But I'm using poetry since ~2 years now and am experiencing a lot less pain than before.
 
Even if every linux box in the world included dbus in its system python, I think you ought to list it as a dependency, even if it's just for the principle of the thing
 
3:40 PM
^^ concur this
 
Being able to be run against non-system pythons is also a big plus
 
There's also containers, which generally crash all your assumptions into a wall, pour napalm on the wreck, and blow it up with a nuke.
 
(Or does linux's system python share its site packages with all user-installed Pythons? Not sure how that works)
 
@Kevin It works very well. :thumbsup:
 
@Arne very well, time to switch finally
@AndrasDeak makes sense
@Kevin fair I guess I'm going to do that
honestly this is confusing because it looks like debian and ubuntu have their native packages for dbus python as well but there's also pypi.org/project/dbus-python which I currently use even though they straight up say this may not be the best binding for you
 
3:46 PM
@aadibajpai as usual
I've migrated to ignoring system packages and using pip for everything. Almost always an option.
And many linuxes have different kind of package managers, or no package managers at all. Hence shaky assumptions about "all linux".
 
about the set_password in
class User:
password_hash = "None"
def set_password(self, password):
    self.password_hash = password
why does it set the attribute if called without having a return statement ?
 
Setters usually don't have a return value, so that's normal
 
Can we first clear up why you think a return statement would be required?
 
I thought a method/function without a return statment returns none
 
True, it does return None.
 
3:51 PM
It does, but the return value is completely unrelated to what else the function does.
 
If you're thinking something like "but doesn't returning a falsey value typically signal that a problem occurred?", nah. Lots of functions return None on a success.
 
what I have trouble to understand is that the value is set inside the function, and gets to the outer scope of the function /I thought you needed a return statement for this
 
@real_hagrid what do you think should happen without a return?
 
Perhaps you're thinking of the nonlocal statement. Without nonlocal, ordinary assignment statements can't change the binding of a variable created in a higher scope.
Fortunately you don't need that here because this isn't an ordinary assignment, it's an attribute assignment. We can assign to the attributes of the self object no matter where those attributes were initially created.
 
Be aware that the value does not get to the outer scope. It is set on self, which you could consider to be a scope itself.
 
01:00 - 16:0016:00 - 23:00

« first day (3687 days earlier)      last day (1262 days later) »