« first day (2047 days earlier)      last day (2902 days later) » 

10:00 PM
@rightfold oh boy
 
what the heck - these 2 files have similar hexdump but one is in chinese:
pastebin.com/idaQ5cK4 <-- the hell's going on here
 
nwp
it is utf8 and the small difference is in the selected codepage?
 
@doug65536 Some people with XX or XY lack fingers, have extra toes, what not. It's really that simple: nature is free to fail/be creative. Don't assume
 
@OneRaynyDay why is the last part different?
 
@OneRaynyDay what tool X is interpreting them that is giving you one in chinese? ... how does tool X determine text encodings?
 
10:11 PM
@VermillionAzure The last part's different because that's what I'm trying to emulate - there's 15 bytes of random uint8's
And then 3 bytes on their side/4 bytes on my side of tiny encoded data of longs
@ellisbben it's simply notepad - they're opened with the same instance too
Refer to maybe an hour before when I chatted "Also, hair pulling moment when my stream impl built for a library doesn't work because of a single byte of difference FML"
 
gotcha
what OS / FS are you working with? I don't know whether anybody holds text encodings in extended attributes / resource forks / metadata, but it springs to mind
 
@ellisbben I'm working with windows. This stuff is done in a company visual studio environment :)
 
@OneRaynyDay have you tried attrib <filename-here> in a cmd?
 
@ellisbben Ah, I don't know this command. It came out with A ...<dir to my file>
I usually use linux at home
 
@OneRaynyDay I'm not super-familiar with it either, so :)
try it on the other file, see if you get different output
 
10:22 PM
@ellisbben I did - both says the A ...<dir to my file>
 
well, that's no help then :(
sorry
 
@OneRaynyDay DOS attribs: RASH: Read only, Archive (been modified), System, Hidden
 
@ellisbben it's no problem! Thank you though :)
and yup @doug65536 I searched it up afterwards. Thanks for the info tho
 
I assumed you probably knew, just mentioned because it's ancient DOS stuff
 
Ah, gotcha. Thanks
 
nwp
10:31 PM
is htop telling me that my process uses 20 TB of virtual memory?
 
no
 
20TB of address space
reserved range of addresses with no pages mapped
debug tools can be crazy with VM allocations
 
nwp
ah, ok, that makes more sense
 
they allocate crazy sparse arrays
 
nwp
10:34 PM
I'm using boost memory mapped files, I imagine they do stuff like that too
 
yeah, sounds right
 
@JerryCoffin holy thank you
So it was ANSI all along
I decoded the chinese character file and acquired the nuclear launch codes from the US ty bush
 
Actually, in this case it looks like Wikipedia does a decent job:
Bush hid the facts is a common name for a bug present in some Microsoft Windows applications, which causes a file of text encoded in ASCII or its superset (such as in a Windows code page) to be interpreted as if it were UTF-16LE, resulting in mojibake. When "Bush hid the facts" (without newline or quotes) is put in a new (pre-Vista) Notepad document and saved, closed, and reopened, the nonsensical Chinese characters "畂桳栠摩琠敨映捡獴" appear instead. While "Bush hid the facts" is the sentence most commonly presented on the Internet to induce the error, the bug can be triggered by many sentences wi...
3
 
@nwp if someone complains about quota violations (exceeding limits set by admin) it is probably that. unlikely though, it isn't done often
boost might degrade gracefully anyway (use a less massive size), no idea
@JerryCoffin notepad is one of the worst programs that build with windows. It is essentially an edit control, and a menu
when wordwrap is on, it appends one character at a time, and recalculates the width of every character added so far, from the beginning O(n^2), to see if it should wrap yet
 
10:43 PM
@doug65536 Yup. A notepad clone takes about a day to write (if you use the raw API--quite a bit less if you use a framework, even MFC).
 
try opening a large file with notepad, if you dont ming pegging a core in getting glyph metrics redundantly n^2 times
 
Could it be that Notepad is meant for notes not large files
 
@CatPlusPlus Just barely possible... In fact, at one time (16-bit Windows) it simply didn't support files over 32K.
 
@CatPlusPlus Could it be that Notepad is absolute and utter shit
 
It works just fine as a scratch buffer
 
10:47 PM
You can scratch my buffer, mister
 
notepad being bad is one thing, but microsoft has infinite resources, they can do better than notepad
 
What for
 
because people actually use it
 
@doug65536 it's not a worthwhile business decision
 
Clearly that means it works fine for themn
 
10:48 PM
@doug65536 For varying definitions of "better", they have--many times over (e.g., WordPad, Word, VS, Multipad, etc.)
 
You know, notepad has been awarded in 1990's as one of the greatest tools ever made, but whoever made it didn't claim the award
I'm sure it's because if he did he would live a life of misery and ridicule
 
The fact that Notepad wasn't even ported to Metro should tell you how much Microsoft cares about further developing it
 
#feelsbadman
 
nwp
I think programs like notepad, paint and calc were meant to show off how to use the winapi and were originally intended as tutorials, people were not supposed to actually use them
 
Also nobody has infinite resources
 
10:49 PM
still doesnt justify why microsoft still keeps it
 
software developers have some obligation to pay attention to what gets used, and make sure that stuff is good, if they care
 
@doug65536 They don't
 
No they don't, also you're not paying for Notepad, it's just included as a courtesy
 
And the software developers are probably not in charge of the actual software they create, much like engineers are there to do jobs, not choose them
 
caring implies you make used stuff good
 
10:51 PM
Almost nobody cares about Notepad, why would anyone spend time on it
 
we're talking about guaranteed sales. how bad would they have to make windows to push the industry away from installing it on every machine
 
And it works fine for what it was built for anyway
 
@doug65536 So why spend more time on it?
Microsoft probably wants to edge out anyways into other areas, like enterprise and toolings and AI, etc.
 
what about cmd.exe
that is really bad
the language is actually usable for complex things, but it is very clunky
 
Superseded by PowerShell
And there's a limit what they can do with it due to backwards compat
 
nwp
10:58 PM
@doug65536 same goes for cmake, make and just about every programming language
 
@doug65536 There's quite a bit more to it than that. For example, if they included an editor that could be considered a decent editor in its own right, including it with Windows would probably qualify as bundling. As it happens, "bundling" is one of those things that's considered really bad if it's done by anybody in a monopoly position. In other words, it's (quite literally) probably illegal for Microsoft to improve Notepad too much.
 
@JerryCoffin But if you put it as open source for free and then publicize a bunch of information on it... well... hm....
 
@VermillionAzure open source? Microsoft? Did I hear that in the same sentence
 
sure I would be against adding features. it should be minimal. but it has a completely naive wordwrap implementation that recomputes the width of every character from the beginning for every character. that is illegal to fix? they left it like that for decades
 
@OneRaynyDay Haven't you been following?
@doug65536 There is no reason to fix it
The people that want features will move onto a free alternative
 
11:03 PM
my issue is their way of just leaving it like that for a decade
 
@doug65536 There's no point to improve it from a business perspective
If you just look at opportunity cost, the cost of "fixing" it outweighs the work needed to do it
They can funnel that towards working with big data, cloud, tooling, etc.
 
They'll fix it and literally nobody will even notice
 
Exactly
 
@VermillionAzure I have, but I still think they're not open source enough compared to all the other companies out there
 
@OneRaynyDay To be honest, it's amazing that they're even moving at all
I respect Microsoft nowadays for being able to change it's culture to the point of being able to do that
 
11:08 PM
Fair enough - but I'll only acknowledge them when all of their employees start growing neckbeards and looking like stallman
 
They have .NET, Chakra, parts of Visual Studio on GitHub. They purchased Revolution Analytics and integrated R into many different products.
 
@doug65536 Is it illegal to "fix"? If they just did an improvement to the underlying edit control (so it worked with anybody's instantiation of an edit control) almost certainly not. If they did it as work on Notepad proper...that would be tougher. Chances are pretty decent that it wouldn't be illegal--but depending on the court, it might be. "Hey boss: we want to you to spend money on a task that can't possibly make us any money--but we really think it probably is legal."
 
(jokes, but pls I'm hoping they'll put a lot more stuff into open source)
 
@OneRaynyDay They most likely will. The fact that they have .NET on GitHub and Chakra is HUGE
 
they could estimate how much time their own devs spend staring at hanging notepad windows, loading a trace, and multiply that by their salary rate. that alone would pay for it
 
11:10 PM
@doug65536 How would 0 times whatever number you choose justify anything?
 
I worked there. people use notepad to load logs/traces. lol.
they know they can change the association. like everyone else they can't be bothered
 
This is not a technological problem
 
@doug65536 If I were Microsoft, I'd spend time on analytics, creating a strong OS with extremely good performance and compatibility while being stable and conducive to development, and focus on creating developer tools and start working on VR with regards to the future web and licensing
 
I used visual studio to open large files. it'll open gigs in a second
 
nwp
@VermillionAzure analytics aka spyware, people didn't like it much
 
11:13 PM
@nwp It's insanely valuable in the right fields and in the right hands
 
To people collecting, not to people being collected
 
@doug65536 I guess I'll take your word for that.
 
nwp
yeah, I imagine that is the reason why they did it anyways
 
It doesn't need to be private data anyways
If Microsoft funnels a lot of time and effort into marrying the scientific community's use of data with business, things become a lot easier for them
 
All data is private
 
11:14 PM
@JerryCoffin sure, lots of people had better setups... but trust me, lots of people hilariously opened notepad, and while we waited, everyone had to listen to me say you can open it fast with something else
 
For example, say Microsoft gets into genetic data analysis
 
@CatPlusPlus Also to the people being collected in most cases. Oh, the collection itself doesn't provide them value directly, but it pays for things they do want.
 
They develop some analytics platform that has great support for it. They license it out, and when the supposed genetics boom comes around, Microsoft is there to reap benefit
 
@VermillionAzure Little late to that party. research.microsoft.com/apps/pubs/default.aspx?id=144178
 
One of the people from my group had a stay at Microsoft Research
I know they're doing a lot of different things
@JerryCoffin Ooooh genomics paper
 
11:17 PM
I was working on stress and performance so I was constantly dealing with extremely large traces and sharing findings with devs and looking at traces with them
 
They are actually using a ton of data right now - they even have a set of people doing image captioning using some data they got
 
@doug65536 What school did you go to?
 
Some convolutional neural network/recurrent neural network stuff - recently they made an image classifier that was miles ahead of every other currently discovered neural network
 
@OneRaynyDay They won a competition for that with deep learning, yeah
 
yupyup - MSCOCO dataset - I used the db with python tensorflow to get some pretty crazy results too
 
11:19 PM
@VermillionAzure local college
 
@doug65536 Yes, but you seem very accomplished and learned
 
programming since 1988, if you can stand 1988 computers, you must be half decent, right?
 
Are you academic or is that mostly experienced?
 
mostly experience but like everyone I also continuously learn
work is easy compared to fun stuff
I expect that is the same for most people
 
...sometimes I don't know if I'm just dumb or other people are just smart
 
11:23 PM
the problem is, fun jobs have impossibly high expectations, even when they have full blown incompetence on their team already
 
nwp
@VermillionAzure people tell you about their 1 accomplishment, not about their 100 failures
 
@nwp 100 learned failures even outweigh the accomplishment. So it's twice worse now.
 
they want one handed handstand pushups when their existing team can't do squats
 
@doug65536 Because never skip leg day
 
Hey guys - reading/writing bytes from a file using this method isn't wrong/terrible, right? pastebin.com/HRE6sneU
For some reason file reading unit tests didn't pass when I read something into my vector, then spat back out into another file.
 
11:31 PM
I have a bad feeling about that
 
@OneRaynyDay if you make it a bit more generic so it takes arbitrary iterators, then sure it is legitimate to pass in stream iterators
 
blob.insert(blob.begin(), std::istream_iterator<uint8_t>(f), std::istream_iterator<uint8_t>());
What does that last argument do?
 
it inserts from the start of the file iterator to the EOF marker iterator, into the start of blob which is empty right now
 
@doug65536 what did you mean arbitrary iterators?
 
11:33 PM
@OneRaynyDay templates
 
Ah. Gotcha. Not sure if that will make too big of a difference - same trick different style right
 
@OneRaynyDay I have a feeling that what you're trying to do doesn't work idiomatically?
 
hmm.. Then any suggestions? (Trying to extract out a vector of uint_8's as the final result)
 
Well... you're working with streams, right?
 
yep :)
 
11:35 PM
Streams are not supposed to be copied as a whole, right?
 
Right
 
A stream is, theoretically, a one-way flow of data, not necessarily the complete collection of data
So copying the stream's data into a vector may not necessarily mean you get everything
Suppose you were to redirect a socket or pipe to the istream. Then copying it at one point isn't ideal
 
erm, perhaps? I thought it would - there's no loss of data from reading a file here
If I don't use a ifstream/ofstream to read files, what else can I do?
 
you want to read a binary block? use .read
 
@OneRaynyDay I mean, don't use copy.
@OneRaynyDay Use more of a polling-like function like getstring or istream::get
 
11:39 PM
@VermillionAzure Ah okay. Gotcha - that makes more sense
 
@OneRaynyDay Additionally, it might be not so good to just simply copy the entire file into memory. You could do that, but I don't think that's memory-efficient or needed
@OneRaynyDay e.g. Parsers usually look at portions of a file, and then build a data structure slowly.
 
@VermillionAzure Youre right, but this is just a unit test - I'm testing the equality of the file so this isn't going to be done in the actual impl
 
But then again there are probably parallel methods that break up a file into pieces and distribute it
@OneRaynyDay Why not use diff?
 
@VermillionAzure I mean - I could either copy entire file into memory and check vector value, or write entire buffer into memory and use diff
 
@OneRaynyDay you just want to compare two files? first compare their sizes, and if equal, use .read to read whole thing, then std::equal them
if you know they are small
 
11:42 PM
@doug65536 Not two files - one is a "MemoryOutputStream", basically a stringstream
 
@OneRaynyDay Same concept
 
I get the vector blob from that and the vector blob from the file and compare
Yeah - pretty much same concept which is what I'm trying to do here :o Testing using vector blob comparison (One downside of having to use diff is that I don't even know if windows has diff, and also don't want to connect c++ into bash and do it
 
@OneRaynyDay A better algorithm would be to read a block of binary from each file using the same "coordinates" and then compare the blocks repeatedly
 
I would buf.resize a vector<char> to size of file, then stream.read(buf.data(), buf.size()) then std::equal(buf.begin(), buf.end(), other.begin())
 
@doug65536 Should he try and read in smaller sized blocks so it still works for larger files?
 
11:44 PM
but check that they are equal size at first
 
Oh, guys don't worry about the big O, this is just a unit test - 100 kb's at most
 
yeah, but it might be guaranteed to be small in a unit test, and YAGNI is strong in unit tests
 
@doug65536 I see - I'll try to do that. Thanks!
 
@OneRaynyDay I'd initialize blob from the iterators.
std::vector<uint8_t> blob{ std::istream_iterator<uint8_t>(f), std::istream_iterator<uint8_t>() };
 
@OneRaynyDay and btw, c++14 has std::equal(astart, aend, bstart, bend) and it will check the length first for you
but that would be after the resize, so it is not that good
 
11:47 PM
@JerryCoffin Ah, you missed an equal sign, but good idea :) EDIT: nevermind!
@JerryCoffin Darn - VS12 wouldn't let me. another reason why I dislike MS
 
@OneRaynyDay It certainly will (though you'll probably need to use () instead of {}. To prevent MVP, you'll have to add extra parens around the first argument.
std::vector<uint8_t> blob((std::istream_iterator<uint8_t>(f)), std::istream_iterator<uint8_t>() );
 
@JerryCoffin Ah, never knew this - thanks!
Wow this was overwhelming amount of information - thanks guys.
 
@OneRaynyDay windows has fc.exe /b file file, returns 0 exitcode if equal
if you ever need to quickly check at command line or something, follow it with && echo equal
 
@OneRaynyDay But wait, there's another completely different way too. coliru.stacked-crooked.com/a/698b9572b6f4e7e4
Gets slow on large files, but works great for small ones (up to a few megabytes or so).
 

« first day (2047 days earlier)      last day (2902 days later) »