Conversation started Apr 4, 2017 at 16:45.
Apr 4, 2017 16:45
Hey sehe'
Ohai!
So, you say you have warnings regarding the usage of segment managers vs shared memory. I'm not sure I follow, what od you mean?
@sehe I'm probably not in a good position to evaluate him objectively--but my weirdness probably resulted from his, to at least some degree. Honestly, I don't know any more about the book than what I just said--but he was a young man during WWII, so reading rebellious propaganda at the time wouldn't be out of the question.
@LouisDionne Nothing spectacular, but just one or two things I keep thinking of whenever I think about using BIP
Let's see.
@LouisDionne About BIP's segment managers, I have one basic gripe. The ODF (or memory layout if you will) is inherently architecture/toolchain dependent. No biggie.
user1804599
yummy crisps
Apr 4, 2017 16:51
However, there is no way to find out about this. You cannot safely 'probe' a file to see whether it is valid for consumption with the binary you're using.
@LouisDionne That's sad, because it implies you cannot actually use e.g. a managed_mapped_file to just persist data. A recompile or some other change could mean the program crashes while opening that file.
The only real workaround I have thought of is rather clumsy: it'd be to use a raw shared_memory_object or mapped_file_region that starts at some physical offset into the "real" shared/persistent thing, and put the detection checksums/fingerprint before that. Next up, you'd have to layer a basic_managed_external_buffer<> on top of that "raw" object to get a segment-manager working.
@LouisDionne So, the gripe, to me, is versionability and error-handling on corrupt/incompatible data (right now, there's just Undefined Behaviour).
That's very annoying.
Any way we could get the author to improve the behavior of the library? It seems like an insanely useful library, but this is a big limitation IMO.
Next up, the other thing I remember as being painful is: creating/opening shared memory/mapped files is racy. In order to actually do it safely you would need to synchronize access, so you need to use e.g. named mutexes. I don't remember the specifics but there might be platforms (Windows?) where even the use/creation of named mutexes itself was not easy to get right.
Gosh. I wish I had better memory. Does Boost Interprocess have that weird singleton class in its implementation? I believe it exists for this very purpose, and I'm not sure whether it's clearly documented what the limitations are, if there are any still left.
For now, let me say, at least most (all?) examples fail to include the necessary IPC synchronization.
OK. Opening mapped files is not a big deal for me, I do it once at the beginning and I can afford to use any expensive synchronization mechanism,
@LouisDionne Just check that the synchronization mechanism works on your system :) You might need madvice/flocks :)
Apr 4, 2017 17:00
@LouisDionne Yeah by all means. I've always been able to postpone/skirt the issue. But I'd be happy if the library removed that threshold :)
@LouisDionne It is insanely useful and I've used it for many many ad-hoc tasks. It just abstracts things away so nicely.
Regarding your answer to my question: I'm not sure I understand why it wouldn't work if the readers don't try to read beyond what's mapped in their own process.
It's much like Boost Multi-Index: a great productivity booster, but some rough edges for production use
(BMI is much more amenable; I'd not hesitate to use it in production. It's just that most often the performance equations favour a hand-built datastructure)
@LouisDionne The point is: they will. At the very least: you can't control that. The segment manager does. And it is expressly documented not to support this. So "it might read beyond the original size" should be "it probably does". To me, as a programmer that equates to "it does".
So basically, the issue is like: writer grows the map, readers don't. Then, I try to grow the map for the readers through the segment manager, but the segment manager sees that the map is larger than it really is (from the data stored at the beginning of the map, which has been updated by the reader), and things break. Is that the issue?
Ven
Ven
I finally understand DEFINE-METHOD-COMBINATION. CLOS is a strange, strange beast...
@LouisDionne Yes, except for the I try to grow the map for the readers through the segment manager step, which is unnecessary for the breakage
user1804599
Apr 4, 2017 17:08
@Ven Looks like COBOL.
@LouisDionne If the client still has the segment mapped, at all, and accesses it, it will see control structures referring to things beyond the extend extent mapped. Leading to, undefined behaviour since BIP doesn't need to check (it's clearly documented as unsupported)
@sehe But it's not like the segment manager will try to access those things that are out of bounds just for the sake of it, right?
I mean, if I try to access them because I think they're in bounds, then that's my problem, but the segment manager has no business going into the heap it's managing (which is full of my own objects). No?
Ven
Ven
@rightfold except it has a MOP
I don't think COBOL 2012 has generic methods.
(/ multimethods)
@LouisDionne You can't know how the heap is implemented. If it's some kind of list, it might address from the end backwards, for whatever reason. So a simple readonly operation (lookup of previously existing data e.g.) could go outside the old bounds.
Note, I'm speaking to the documented functionality, and explaining how it makes sense. If you want to know whether it's easy to "lift" these limitations (say to keep read-only access to the "old" data) that's a question to ask the maintainers, I think.
@sehe OK, thanks. Do you see any workarounds for what I'm trying to do?
I mean, this has to be a common use case, no?
Apr 4, 2017 17:15
@LouisDionne I also think you should be safe iff you avoid all use of the segment manager, at all. So, iff the "server"/"producer" doesn't touch the existing data and the client only indirects process-local pointers into the already mapped area, that it already had before the resize, then I agree that you should probably expect everything to work.
@LouisDionne The workaround I've used is to communicate a "please-remap-your-shared-realms" between processes.
Not much of a workaround. More like "acceptance" :)
@sehe So you actually call grow in the readers, then, or you kill the current map and remap completely?
The latter seems inefficient?
And: overdimension and never grow. It's good that virtual memory is "free" :)
@LouisDionne No, only one party can grow, as documented. Nobody else should even have the map open (otherwise at least, add synchronization). After reopening, all parties will automatically see the new size.
@LouisDionne Yeah. remap completely might be inefficient (however, grow in-place might not always be possible, if at all on certain arch; after all, what if the adjacent address space is not free)
I see. So the writer says "shit, I'm about to grow". It notifies the readers. The readers close their maps. The writer grows its segment and notifies the readers. The readers re-open their maps.
Yup. Like I said:
2 mins ago, by sehe
Not much of a workaround. More like "acceptance" :)
Yeah.. That's not great since it requires a downtime where all the readers have their stuff unmapped (so they can't do useful work).
Apr 4, 2017 17:20
@LouisDionne Consider simply overdimensioning. Since it's virtual memory, you might get away with just "allocating" a TB. If it's a mapped file, just make it a sparse one (I've done that. No problem creating multi-terabyte mappedfiles in tmpfs)
user1804599
@Ven mop is honey in dutch
Not where I live.
Honing is honey.
"mop" could be "schatje" or "liefste"
@rightfold Also. Whatever happened to orthography? Capitalize "the greatest language on earth" and end your sentences!
@sehe oh honey
@sehe So I just go like ipc::managed_mapped_file shmem(ipc::open_or_create, "foo.bin", 1 TERABYTE);?
user1804599
@sehe Yes, honey.
Apr 4, 2017 17:22
@LucDanton Yeah, I was confused too
that TERABYTE
looks like a shitty macro
placeholder
ok
carry on then
@LouisDionne Mmm. Let me think now, how I made sure the file was sparse. Damn. I don't remember how I did. Must have been pre-created (probably just ::seek(fd, ..., 1ul<<40) and write 0 bytes.
Maybe I've not done that for managed_segments, but just raw mapped_file. Sorry if I gave you false hope there.
You could do a two-step boostrap:
1. create managed_mapped_file as usual (small)
2. grow it as a sparse file (using DD, fallocate64 special flags,... ?)
3. `grow()` the managed mapped file (cross fingers)
4. ????
5. hope the file is still sparse
That's sketchy too
The main point was about virtual memory not being real. So you can get away with mapping large segments even if largely unused.
I have to run. Be back in a few hours.
Ok. Thanks a lot for the help anyway.
Apr 4, 2017 17:28
Cheers :)
 
Conversation ended Apr 4, 2017 at 17:28.