« first day (774 days earlier)      last day (4191 days later) » 

6:00 PM
@Cicada Well, I thought that continuous copying and looping (I have to call DeviceIOControl multiple times because it only copies out as much data as is available in your buffer, and you have to 'continue' through each time) would be the major slowing factor.
 
@ThePhD AFAIK Windows uses 8KB buffers for most of its operations. Anything under or above that will get you suboptimal performance.
 
Guess it's 8192 then.
 
Yeah, 8192 = 8KB
 
I don't know how else to make this faster...
Damn yoooou, windoooows.
 
Don't allocate stuff
 
6:02 PM
@ThePhD Just tell your customers that the only supported configuration uses only SSDs.
 
Take everything out of loops as much as possible
 
2.7 seconds with 8 KB buffer... hm. And I've already set the std::unordered_map::resize (212624) to allocate enough space for an absurdly large directory count.
Maybe something can be done with how I create each directory entry?
 
@ThePhD run a profiler.
 
My cristal ball is non functional.
 
it will tell you exactly what function is taking all the time. Optimize that. Repeat
 
6:05 PM
Yeah....
unordered_map::insert is killing me.
Despite reserving it.
More than half my time is spent there.
 
@Cicada I was always told 4KB was optimal, due to page size
 
@MooingDuck That's Linux
 
The other time is spent creating the name string for this thing, despite using an in-place set for the string.
And that's literally what's taking all the time. Jesus, string.
 
@MooingDuck default ntfs cluster size is 4K, I think?
 
@ThePhD std::string is uber slow. Raymond Chen had a blog post about that.
@MooingDuck Ohwell I may be wrong actually. It's either 4 or 8. I'm not too sure now. Linux is certainly 4 but I have a doubt for Windows.
 
6:07 PM
@melak47 Now it is. It used to be smaller. Especially during the XP days.
 
Maybe I can just use char* then, and keep a size variable myself?
I'd have to dynamically allocate a char* for each entry, though. Which again sounds suboptimal..
 
@Cicada yea well you would think the library stuff is eventually fixed. I mean... unless people really want to encourage multiple string implementations x)
 
@Cicada Wikipedia implies Windows uses 4KB. Says the only systems with 8KB are ia64 and sparc.
 
We're on the scale of 160,420 entries... blarghaghabha fuck you, string.
 
@ThePhD what is your average string size?
 
6:09 PM
@Cicada researching...
 
@nixeagle Varies, from 162 bytes to 0 bytes.
 
average
 
96.
 
@nixeagle what do you mean, multiple implementations. what's wrong with cstring? :)
 
@ThePhD half your strings are under what length. :P
 
6:10 PM
96.
 
3*2^5
 
Well, there's also another way I could do this.
 
@MooingDuck There
 
Technically, the strings I want and the data I want are all being copied into this fat buffer I must have on the stack.
... No, no that wouldn't work. I'd hit 100K in Memory usage if I just copy over the whole buffer.
 
@ThePhD slow hash? suboptimal hash? check load factor. Use custom allocator. Use a classical map with custom allocator. Make it memory mapped. ideas ideas
 
6:12 PM
I'm hashing on a unsigned long long, so I don't think hashing is the problem.
 
@ThePhD You must have on the stack? I doubt it
 
@ThePhD i bet you explained that earlier. Because, you know, reserving large chunks of memory on the stack is just insane.
@ThePhD agreeing
 
@Cicada I went through this yesterday with DeadMG and a few others. The function doesn't seem to want to have the buffer anywhere else but on teh same stack frame as where the function call is.
 
@ThePhD Give a link to the goddamn function
 
I tried to have the buffer larger and elsewhere using _aligned_malloc, new, or just having a big buffer on the class definition itself.
And, okay.
 
6:14 PM
@MooingDuck assert(errno == ENOFUCKS); // postcondition ?
@Cicada Or perhaps, just to the function
 
@ThePhD I was going to mention something about pre-allocating memory, but I like @Cicada's idea.
 
@Cicada I looked at the function, it doens't officially require it. However, his code failed if the buffer was a class member, or allocated seperately via various methods, or any number of other things we tried. Worked fine when the buffer was on the stack.
 
@JerryCoffin What do you mean?
 
@sehe no >:
 
6:15 PM
@R.MartinhoFernandes I don't remember any more.
 
@sehe DeviceIoControl, near the bottom of his code
 
@Cicada Just star my wittiness, and I'll grant you the lenience :)
 
Damn you old men and your crappy memory.
 
@MooingDuck That makes no sense
I'll try something after dinner
 
@Cicada I'm aware of that
 
6:17 PM
@sehe LoadFactor isn't an issue, because I'm reserving 262164 spots, and getting only 162,*** items out. So even if allocation was bad, it's not re-allocating (or it shouldn't).
 
@ThePhD that is over 50% utilization. Try reserving more spots?
 
Drilling down into the std::unordered_map functions, _Buynode is takin the most time.
I don't even know what it does.
But it's part of ::insert
 
which means it is taking time to insert an item into the hash
which means you have lots of collisions
Why don't you try timing with std::map?
Just curious if an alternate data structure would perform better.
 
... You know what
Just for the lulz
 
6:21 PM
"believed", huh?
 
I'm going to see how well my own Dictionary<> class does.
 
it sounds like "believe in science"
 
This is going to be exciting. :3c
 
@ThePhD well my suspect here is your hashing function is suboptimal.
Which is why I suspect std::map may actually be faster.
 
@nixeagle my hashing function Std::Hash<unsigned long long> is suboptimal.
 
6:23 PM
Hashing does no good if you end up sticking most of the elements in the same bucket.
@ThePhD it may be for your data set
 
I'll just change it to MurmurHash.
 
If your hashing function puts all the elements in the same bucket, you wind up with a glorified list. x)
 
Well, I have a whole gamut of hashing functions I wrote / learned when I was making my dictionary.
 
so try some out
 
So I'll cycle through FowlerNoVoll, FowlerNoVoll1a, FirefoxCache, and MurmurHash3, and see what happens.
 
6:25 PM
unordered_map has a configurable hash.
 
@R.MartinhoFernandes which is what I was getting at.
 
@R.MartinhoFernandes I'm going to try that before I try my Dictionary class.
 
@Cicada The only "slowness" he mentions is that when the dictionary resized it did a deep copy of the strings. In C++11 that is a non-issue.
 
Blurgh, I'm so used to hashing statically.
I need to get used to this idea of function objects.
 
@Abyx that's about right, i think
 
6:28 PM
Goddammit the IE team is full of idiots
I just spent 45 minutes trying to figure out why my web page doesn't work in IE9, and why IE9 seems insistent on dropping to IE7 quirks mode when it has an HTML5 doctype
 
@Collin it's quite some years since i've used IE. i think there are good alternatives to IE these days. even window update is done without IE
 
@Cheersandhth.-Alf Oh me too, and while many people use things other than IE here at work, the default installed one is IE9
 
the #1 browser worldwide is google chrome. followed by firefox, i believe.
 
Anyway, there's a ridiculous default set in IE8 and 9 that causes pages on your intranet to be loaded in IE7 quirks mode by default, even if you have a good HTML5 doctype declaration. You have to put in some nonsense meta tag to get it to believe you really want to use standards mode
@Cheersandhth.-Alf I like chrome a lot
 
@Cheersandhth.-Alf I've never gotten windows update to work without IE
@Collin my work officially stopped supporting IE :D
 
6:33 PM
@MooingDuck even on win7?
 
@Collin I still use XP, probably related
 
@MooingDuck ah yeah, win 7 (and vista?) uses a standalone updater
 
Has anyone used trac (trac.edgewall) with subversion? I'm curious how the commit references "Refs #id" are supposed to look in the ticket view.
 
Hm. I need to figure out if I'm compiling in 64 bit or 32 bit...
THere's gotta be macros for that.
 
@CaptainGiraffe That would require that I use svn -- which I haven't in a while now...
 
6:37 PM
@ThePhD Are you sure? I'm not sure you need to know that
 
@ThePhD sizeof(size_t) == 64/8 :P
 
@nixeagle That would be awesome
 
@ThePhD Memory allocation.
 
6:38 PM
@R.MartinhoFernandes you caught me, fixed :P
 
@ThePhD Usually you want to use types that let the code work correctly either way, and ignore the size.
 
@JerryCoffin It's just for the hash function. RIght now MurmurHash is fixed to work on 32 bits right now.
I'm pretty sure tehre's a 64 bit one out there, I just wanted to bake the switch into a compile-time #ifdef .
 
uint32_t?
 
@Collin T* I presume
 
@ThePhD Bake it into a template?
 
6:41 PM
hmm
 
@R.MartinhoFernandes Or that, but my std::enable_if is still rocky.
 
@MooingDuck hashing from, but the hash value is still some 32-bit thing?
 
what are the rules for uniform initialization, anyway?
 
maybe I'm confused about this hash
 
@ThePhD Partial specializations, silly.
@Collin The hash values are size_ts.
 
6:42 PM
@JerryCoffin Oh good I thought you did that by hand :P
 
then use uint32_t if it actually has to be 32 bits?
 
@DeadMG What rules?
 
@R.MartinhoFernandes For when it can and cannot be used.
 
@Borgleader I realize I may seem a little crazy, but do I honestly seem that bad?
 
size_t can be 32 or 64, so maybe I can do template <T> struct MurmurHash {};, and then template<> struct MurmurHash<uint32> {};
 
6:43 PM
I have
 
And one for uint64 t
 
@JerryCoffin Hahahaha no
 
struct item {
    std::pair<const K, V> data;
    std::size_t cached_hash;
};
 
Since size_t is gaurunteed to be unsigned.
 
@DeadMG Is the cashed hash lazy or eager?
 
6:43 PM
@FredOverflow ... Wat.
 
and want to do like item { { blah1, blah2}, blah3 }
 
Lazy or eager?
 
@ThePhD Well, do you compute it on demand, or do you pre-compute it.
 
well, it's used right away, so lazy would be quite irrelevant
 
@ThePhD There is a 64-bit version -- two of them, AAMMOF (one for 64-bit platform, one for 32-bit platform). Instead of #ifdef, I'd put each in a separate file, and choose which to compile in the makefile.
 
6:44 PM
you've gotta use the hash right away to insert into the table
 
AAMMOF?
 
@ThePhD how about first time it to see if it even works faster or not :P
 
@ThePhD As A Matter Of Fact.
 
If it works faster, than worry about portability.
 
@JerryCoffin Why bother with the build system if the compiler can accurately tell those apart? Isn't the switch here to be made on sizeof(size_t)?
@DeadMG Looks fine to me.
 
6:46 PM
@R.MartinhoFernandes Let's see if Nov 12 CTP agrees
 
@R.MartinhoFernandes Why make the code less readable to handle something make can do just fine?
 
@JerryCoffin I'd rather deal with the preprocessor than deal with make.
 
@DeadMG The preprocessor cannot detect the size of size_t, silly.
 
@DeadMG Preprocessor can't deal with sizeof, so it'd have to be a template, or something on that order.
 
I know
it's just for comparison
 
6:47 PM
I think the template is fine enough.
If it's 64 bit though I can ditch MurmurHash altogether and just use CityHash
... Cross-eyed
 
The problem with the preprocesor is that it gets ugly in a hurry if you support multiple compilers (i.e., each is likely to define its own macro to signal 64-bit compilation).
 
128-bits?
When did MurmurHash go that far?
@JerryCoffin I'm never going to use anything other than MinGW GCC or straight GCC, or VC++. I don't see myself getting my hands on Borland or CodeWarrior or whatever those other compilers are, unless I'm doing embedded work.
 
hmmmm
 
@ThePhD I don't think I'd ignore clang at this point.
 
6:50 PM
for a random iterator like std::list::iterator, is it default-constructible?
 
Well, Clang for Mac, but that's honestly the last platform I'm going to be worried about.
 
@JerryCoffin I agree. Clang is not a minor implementation, not anymore.
 
Clang and GCC are fairly close buddies.
 
@ThePhD Clang is also viable for Unix, and technically, it might at some point in the very vague future, be viable for Windows too.
 
@DeadMG Clang is a GCC 4.2.
 
6:52 PM
Does GCC work by itself on Macs?
Or is Clang still the only option?
 
Hi all!
 
@R.MartinhoFernandes It pretends to be that to GCC-specific extensions, but it's a lot more than that if you're explicitly targetting Clang.
 
@DeadMG The macro crap you need to test for for this kind of thing is the same.
 
@DeadMG I don't think you can count on it.
 
6:54 PM
hiberfile.sys, pagefile.sys
Really big files. Tempted to delete.
Maybe my system will explode. :3c
 
Xeo
@DeadMG No word on that in the standard, IIRC.
@ThePhD Can't guess what they do from the name?
 
@ThePhD Disable hibernation, the first will go away.
I wouldn't touch the other one
 
@ThePhD You can delete hiberfil.sys, if you ask Windows to disable hibernation.
 
Xeo
Hibernation's awesome, though.
 
@ThePhD pagefile, maybe, but it won't let you delete that, hiberfile is your memory dump from the last time you hibernated the machine
 
6:55 PM
I know what they do. I'm just wondering if it'll actually explode my system.
 
pagefile, though, you'd want to grow giant balls before fucking with that shit
 
You can also decrease your pagefile size
 
Xeo
hiberfile.sys won't do anything to the running system.
 
@JerryCoffin That's actually rather irritating.
 
Xeo
For pagefile.sys, there's a good chance you can't even delete it since it's locked by the OS.
@DeadMG What'd you need it for?
 
6:56 PM
@Xeo Implementing a hash map.
 
@Xeo Damn -- you beat me...
 
@Xeo Hibernation on my computer is a lie. It says it's going into hibernation, but what it does is the screen blacks out, but if I have music player, the same song will repeat on loop for another 30 minutes (at least 3 minutes), while the computer becomes absolutely unresponsive.
 
...I had expected chrono to give me some basic IO like boost ptime and friends.. But it seems the only thing we are left with is std::localtime, 'may not be thread safe' ... Are my only options to either use boost or find all uses of localtime in all libraries and protect it? Not that boost time classes are not insanely cool
 
It's pretty sweet.
 
@ThePhD Since you can't use it anyway, just disable hibernation.
 
damn
my left headphone is broken
No need to double up.
 
Xeo
@DeadMG The right speaker on mine is gone (i.e., I cut it off when the frame broke)
 
Nuked hibernation, time to blow up the file.
 
@DeadMG You wanted an upgrade anyway.
 
6:58 PM
@Abyx why, is that... forbidden?
 
@ThePhD no touchie
 
rofl
 
Xeo
@JerryCoffin Fucking Sennheiser, man.
 
"Left 2 Die, 3 players (Real American's)"
who dafuq made that
 
Xeo
So stupidly expensive.
 
6:59 PM
@JerryCoffin To be quite fucking honest, I need robustness over sound quality.
I've only had these seven months and they're one of my longest-lasting pairs
 
@sehe it's not "forbidden", you just can't apply the word "believe" to science.
 
@DeadMG Oh, well that would indicate a rather different solution then.
 

« first day (774 days earlier)      last day (4191 days later) »