« first day (2043 days earlier)      last day (2896 days later) » 

11:00 AM
Oh, that reminds me, I was supposed to read your bootstrap code :D
 
Oh god yes
hang on
let me locate big_boot_barrier
Hartmut hates the name, he may have changed it :p
 
> struct HPX_EXPORT big_boot_barrier
 
Apparently not.
B^3 lives
 
> boost::lockfree::queue<util::unique_function_nonser<void()>* > thunks;
oh lol
 
yup
search for stage :p
It used to be oneish functions
I can probably remember the sync points
 
11:04 AM
> init_tss("main-thread", 0, "", false);
This either inits a task state segment of some sort or is terribly named.
 
could be either
stage 1 boots up local runtime services, thread pools, the thread scheduler, and initiates AGAS bootstrap.
 
Wait, wheres stage 3? :D
 
we're getting there
 
There's no print for stage 3 :D
 
user1804599
 
user1804599
11:05 AM
facebook office
 
user1804599
how can you even concentrate in there
 
stage 1 is executed before the thread scheduler is online
 
user1804599
so many people
 
user1804599
it's like herrings in a can
 
the distinction between stage 1 and stage 2 is that stage 2 uses the functionality of the thread scheduler to do stuff. Stage 1 does only the things necessary to get to stage 2
 
stage 3 /should/ have been an synchronization to assume that all pre-start-up-functions are called on all localities. Hartmut may have broken it.
 
Ven
@Griwes ssh, let the ICE melt in you
 
Lobster obviously has not worked at many places ...
 
Ven
@rightfold you can't. that's why it's shit
 
After stage 3, all pre-start up functions have run, and then we can enter stage 4.
where user code gets run.
 
11:07 AM
@blelbach The comment with MS connect link in it doesn't work: "The content that you requested cannot be found or you do not have permission to view it."
Or is it a private thing?
 
user1804599
([^\/]+)(\/[^?]*)?(\?(.*))? oh god this regex
 
which ms connect link?
wait did hartmut break startup, hm.
 
Ell
@rightfold so stupid
 
connect.microsoft.com/VisualStudio/feedback/…, in the comment linked to by my "internally screaming" message
 
Ell
(.+)\/(.*)\?(.*) is equivalent, right?
kinda
 
11:09 AM
Oh. Obscure MS stuff in HPX is hartmut's fault, not mine
But that is probably correct.
Shit, HPX programs can't return from main on linux systems :p
we have to call _Exit()
 
lol
 
because we have to avoid calling exit functions that dumb libraries install
because those exit functions cause things to segfault and die.
 
Isn't that the user's problem, though?
 
user1804599
 
Ell
what are you guys talking about?
 
Ven
11:11 AM
@rightfold XD
 
Ell
@rightfold looks awful
 
  where messages = [ ("ocharles", "hackage is great")
                   , ("edwardk", "I love Simon Peyton Jones")
                   , ("spj", "We all love lazy evaluation")
lol
gotta have quality examples
 
@griwes: Yes, but what's easier - getting some dumb IO library to accept a patch that lets us say "PLEASE GOD DON'T INSTALL atexit() FUNCTIONS"
or just making our system call _Exit()
@griwes: github.com/STEllAR-GROUP/hpx/blob/… is where stage 3 happens
 
Ven
"all values have same type" that's wrong tho
they're bad at APL
 
@blelbach Wait, all atexit handlers segfault you?
 
11:12 AM
@griwes: No, only bad ones from HDF5.
...that said, never install an atexit() handler in an HPX program, bad things happen.
 
@blelbach So... again, that isn't really HPX's problem...
@blelbach ...*that* might be HPX's problem - atexit is useful!
 
@griwes: You know what the definition of "runtime system" is?
 
user1804599
@Ven no it's correct
 
user1804599
APL has only one type
 
A runtime system is just a library that's REALLY intrusive.
Your problems are HPX's problems.
 
Ven
11:14 AM
@rightfold kek
 
int pre_main(runtime_mode mode);
int pre_main(runtime_mode mode)
{
What.
 
Ell
what is HPX? :V
 
user1804599
@BartekBanachewicz SPJ is the unit of enthusiasm.
 
nwp
@Ell a parallelism library
 
@Ell: A 500k line parallel and distributed C++ programming framework built on top of Boost by me and two other madmen.
 
Ell
11:15 AM
nice
 
It implements the C++ threading library and extends it for distributed programming.
@griwes: Anyways, yah, 4 stages: Stage 1, boot up the thread scheduler, stage 2, use the thread scheduler to do the rest of setup. Stage 3, run pre-startup-functions, stage 4, user code, stage 5, shutdown-aka-crash
 
I kinda hoped for code that's more insane. :P
 
the big boot barrier gets used to sync stage 2-4 I beieve.
 
The BBB thing is funny, but I guess that's mostly that :P
 
11:17 AM
> the big boob barrier
 
There's insantiy for you.
 
nah
 
stellar.cct.lsu.edu/files/hpx_0.9.5/html/code/hpx/util/… <- this is one of my favorite things, although hartmut hated it and killed it.
 
user1804599
@blelbach awful
 
@rightfold Sometimes you just gotta #define main
 
user1804599
11:18 AM
no
 
user1804599
never
 
Someone suggested that we do the following
 
@blelbach heh
 
user1804599
As a library author you must WTFM, and as a library user you must RTFM.
 
user1804599
And TFM must document how to initialize your library properly if your library needs initialization.
 
11:19 AM
#define main __foo() {}; auto hpx_startup::user_main
^ Quiz time, tell me what that does :D
@rightfold It's not a library. It's a runtime system.
 
user1804599
Don't do horrible hacks to make it "easier".
 
Ell
@blelbach looks terrible :P
 
user1804599
Because it isn't. It'll bite you.
 
Ell
I haven't looked at HPX at all, but why does it require doing weird main() stuff?
 
@rightfold It's an optional header, for the record. But I'd stand by it even if that was the only way to activate it.
@Ell: It's one of the ways of doing initialization, if you include <hpx/hpx_main.hpp> then you don't have to initialize HPX.
 
11:21 AM
@blelbach Escapes nonsense like foo::main()?
 
so _Exit() is like MPI_Abort?
 
Good for making slides.
 
@Griwes it allows us to #define main AND support the user not returning from main.
 
Ell
@blelbach but I mean usually if it's a static lib you'd do like hpx_init() or if it's a shared lib then it'd be automatically initialised
what kind of magic do you need beyond this and how come?
 
11:22 AM
@blelbach eh
 
@Ell You can start it up by calling hpx::init()
 
user1804599
If you want your own main function, don't have the user write their own.
 
@Ell Some people don't want to do that, and asked for the header.
 
user1804599
Document in TFM that the user (who RTFMs) must not define a main function.
 
Ell
I can't imagine why they don't want to do that :V
 
11:23 AM
@griwes: You don't have to return from main. This was a problem for us. People didn't like including <hpx/hpx_main.hpp> and getting errors because they didn't return 0 from main.
 
@gnzlbg _Exit is literally a syscall that murders the process. With nothing specific to C++ runtime called.
 
The macro I quoted above uses return type deduction to "solve" this problem.
 
@blelbach ahhhhh, the implicit return 0 nonsense. Gotcha.
 
@Griwes lol, how did i miss this for so long
@blelbach so why does at_exit functions possibly "break" HPX?
 
@gnzlbg Probably you didn't bother reading through the standard? :D
 
11:24 AM
what does hdf5 try to do there that is evil?
 
@gnzlbg A library called HDF5.
 
I found it kinda accidentally.
 
I can't remember what horrible thing it did.
I believe it was trying to free or operate a resource that was no longer available.
 
@blelbach seems weird, hdf5 error handling is too C like, you have to pull the errors out, so maybe its trying to do something at the end in case the user didn't
 
The fact that their library installed atexit() handlers and didn't give me a way to not install them annoyed me.
So I next leveled them.
@gnzlbg yah.
@gnzlbg I felt they were doing something bad, so I did something worse to stop them.
 
Ell
11:25 AM
@blelbach which surely makes you as bad as them ;)
 
@blelbach did you write a bug report?
like believe it or not these people are reasonable
 
Ell
your library has a problem with another library try to be smart with main/atexit, your solution is a new main and new exit handlers :P
 
they might just commit a macro to disable at exit in 1 day
 
@Ell Ah, but I'm writing a runtime system, not a library. I get to break the rules.
 
@blelbach you are writting a C++ library
 
11:26 AM
@Ell The new main is a separate issue.
 
you can call it a run-time system if you want
 
Ell
@blelbach Whatever let's you sleep at night ;)
 
but it is still a C++ library
 
@Ell No, for that issue, it really is. Having a separate main has no relation to the hdf5 issue. Just the atexit() stuff.
 
Ell
11:27 AM
I see :)
 
@gnzlbg It's about the guarantee to the user, end of the day. HPX provides runtime services, similar to the C/C++ runtimes.
 
so does std::thread
and boost::thread
they are still just c++ libraries
 
If it wasn't a pain in the ass, we would actually replace the C and C++ runtime with an HPX clone - we'd actually replace the C startup routines and start up HPX there.
 
i'm not arguing, i'm just saying play nice
you are not the only run-time that a program might have
 
@gnzlbg boost::thread and std::thread do not have implicit resources or services.
 
11:29 AM
the C++ runtime is also there
MPI might be also there
 
In HPX, the system just provides you with thread pools.
 
OpenMP and Cuda as well
 
In Boost::thread or std::thread, you need to explicitly start up threads.
 
I might have to link library A using HPX with library B using OpenMP to do something in a program that uses TBB
 
In HPX, those services are available after runtime initialization
 
Ell
11:30 AM
@gnzlbg what about the intel thing?
 
@blelbach You made me look through some of my old kernel code... and I found the scheduling and IPI nonsense...
 
@gnzlbg That's why we have our own implementation of OpenMP
 
@blelbach yeah well you shouldn't probably
 
We don't support compatability with TBB, but it's never been an issue.
 
@gnzlbg Possibly :p
 
Especially at the bottom... ;(
 
your own implementation of openMP might crash against my program linking with a dynamic library that links against a different OpenMP runtime
 
@griwes: YOU ARE USING MY IDENTIFIERS
 
where
Also this is my kernel, I am the runtime system :D
 
11:31 AM
as long as I can initialize and tear down hpx at will @blelbach everything should be fine
and as long as you can detect whether hpx has been initialized hpx libraries should be composable
 
@gnzlbg Yah, we support that use case.
 
i've had to fix some MPI libraries that called initialize without checking if MPI was already initialized
 
@gnzlbg We don't support the different OMP implementations one. You should be compiling all your code with the same flags you built HPX with, and linking against it too. We do some amount of checking for that.
@GRIWES: _.* IS MINE
 
@blelbach not in my kernel :D
 
Do not use single underscore identifiers. Those are reserved for the implementation.
 
11:33 AM
@blelbach but you say I can only use your OpenMP implementation
I guess that HPX brings its own OpenMP-RT
 
@blelbach I am the implementation.
 
@gnzlbg HPX is the OpenMP RT
@Griwes Implementation means the C++ implementation. Not you
 
so if I compile my software with clang I have to pass it a flag -fopenmp=hpx ?
 
@gnzlbg No.
 
so?
 
11:34 AM
@blelbach It's a kernel, compiled without exceptions and RTTI, there's no C++ anymore.
 
Clang uses the Intel OpenMP runtime API, just like the Intel compiler.
We implement the same OpenMP runtime interface
 
clang can use multiple openmp-rt
it can also use gnulibomp
 
user1804599
@blelbach only in the global namespace
 
how does HPX understand openmp pragmas?
does it has a preprocessor?
 
@gnzlbg No, as I said
We don't do that.
We implement the Intel OpenMP RT API
 
11:35 AM
Eh, I really need to get my build system thingy up to a working condition, then I can go back to actually trying to start another iteration of the OS...
 
if your compiler supports the Intel OpenMP runtime
then you build against that and link against HPX and it magically works.
 
i just link against hpx
i see
 
@gnzlbg Pragmas are handled by the compiler; HPX only implements the runtime that's used by code generated from the pragmas.
 
@rightfold Fair point.
 
so i have to tell my compiler "compile for Intel OpenMP Rt but not link against it"
 
11:35 AM
@gnzlbg Nope
 
user1804599
Fixedsys Excelsior is the best font ever.
 
The Intel OpenMP runtime uses weak symbols
You can physically link against it.
same mechanism that allows tcmalloc, etc to override malloc
 
so hpx overrides those at run-time somehow?
 
@gnzlbg At link time. ELF linkers support weak symbols
 
i don't know anything about weak symbols so gotta learn that
 
11:37 AM
yesterday, by blelbach
what am I doing with my life? >.<
 
For example, all the symbols in the C library are weak. You can reimplement your own malloc, if you wanted to
 
Enjoying it, apparently
 
@gnzlbg Weak symbol are a really cool feature.
 
last thing
 
Declaring a function as weak basically says "if someone else defines this function later, use their definition".
 
11:38 AM
i have a parallel distributed octree and have been thinking about trying using hpx instead of MPI
 
@sehe Yep.
 
but I fail to find big scalings,
 
@gnzlbg I actually worked on parallel distributed octrees.
 
do you happen to have any resources about HPX scalability?
 
Btw., @blelbach, how many submissions are there right now?
 
11:39 AM
like ours goes up-to 500k cores with >90% efficiency
 
@Griwes Not enough. 10. 100. 50!
 
50!?
That's a lot. :D
 
and the ones from some other groups up to 1,500,000 cores AFAIK
 
@gnzlbg Just flat MPI, no OpenMP or anything else?
 
3.04140932e64 submissions. Nice.
 
11:40 AM
but the largest scalings I've seen for HPX are for < 60k cores
@blelbach just MPI yes
 
@gnzlbg Why are you looking at HPX, then?
 
not hybrid
@blelbach because of AMR and dynamic load balancing complicate things a lot
 
If your code's scaling today to those levels with MPI - then your problem is likely well suited to CSP.
@gnzlbg Gotcha.
There's not a lot of AMR codes that reach up to those scales with that level of efficiency.
 
@blelbach but one of the reasons we scale is that we get to control exactly where the data is
 
@gnzlbg HPX could probably scale a static problem up to those sorts of scales, likely with a little bit less efficiency.
 
11:42 AM
@blelbach there are some groups that are able to scale way better
 
@gnzlbg For AMR, you mean?
 
@blelbach with tree codes, it's not clear from the papers whether they used AMR or just static partitioned and a static tree
 
nods
 
@blelbach If I had to bet they probably used a static mesh in a cube for the scalings
 
@gnzlbg Yah
 
11:43 AM
but even then they still do scale better than us even with a static mesh
 
The reason we don't have results for that type of problem is that they would not be publishable, for the most part. E.g. HPX solving static problems is very boring.
 
and well we are genuinly interested in assesing the potential of new technologies
 
Pushing a static problem from 60k to 150k cores is a lot of work, and for us it's not valuable (because MPI already does it).
so we are mostly looking at pushing fully dynamic problems.
 
@blelbach those are the only one that matters for most applications anyways
 
yah
@gnzlbg HPX is a pretty good proxy for asynchronous runtimes. I work on a number of runtimes, I advise people who are looking to evaluate to use HPX.
One of the reasons is that Hartmut, the main HPX author, is interested in making HPX a product.
 
11:46 AM
there is a difference between what different groups do for showing off in a computing time application and how their software is used in practice
 
Some of the other async runtimes out there are still in "research" mode.
E.g. I think you are more likely to get support from the HPX developers than from some of the other groups.
 
i don't know about the async part of hpx
it looks interesting, but expensive
all those futures
maybe with heap allocation and type erasure
i don't know
 
they're pretty cheap :p. I spent a lot of time making them cheap
 
do they get inlined? do they enable vectorization?
 
The thing to understand is that the cost of the synchronization primitives /doesn't matter/
@gnzlbg Yes and yes
 
11:48 AM
don't misunderstand me but I'm very skeptic about that
 
Well, of course, for vectorization in C++, you need to #pragma simd and __assume_aligned()
Vectorization is another story because we don't have an aliasing solution in the C++ standard
 
yep but in a say stencil code over a grid
 
void f(double* restrict p, double const& speed_of_sound)
 
do you launch asynchronous work per stencil?
or per blocks of stencils?
 
^ speed_of_sound aliases p
@gnzlbg That's up to the developer
But doing it per stencil is almost always wrong
 
11:49 AM
@blelbach so that's what I meant
 
@gnzlbg The key to a framework like HPX is to identify and control the grain size.
 
but can I?
I can write loops over blocks of cells/stencils whatever
parallelize those loops over blocks with openmp
 
@gnzlbg The answer to that question is really "neither - you can control the grain size, so you can change a runtime parameter and have it run at a granularity of one stencil, or a block/tile of stencils"
 
and then layer MPI on top of that
with HPX I switch the two top layers with futures and rely on the run-time for the partitioning (IIUC)
 
@gnzlbg No, not exactly
 
11:51 AM
@blelbach that's interesting
 
You, the application programmer, need to express granularity in your HPX program
 
do you maybe have example applications where it is shown?
 
If you spawn off a future for every stencil, things will be slow. You need to spawn off a future for a tile, and have the tile size be controllabe
yah, one sec
 
like do i have to spawn futures over "blocks" of variable size to be able to control the block size
 
@BartekBanachewicz excuse me, but I don't see that rectangle you say, and neither the height. I see that in that formula uses vectors between vertices and x points, but not the height.
 
11:52 AM
or do i just write one future per stencil and am somehow able to tell the run-time how to group them?
 
@gnzlbg The runtime has no way to group them together.
Also, you'd miss out on vectorization that way
the amount of work that each future does should be (roughly) 10-100 microseconds
 
@blelbach I see, thanks
 
there's a slide deck explaining the example somewhere
That slide deck starts off with the naive approach (one stencil per iteration)
and shows how to get to the configurable approach
this is also how we nullify the cost of futures
If futures cost 2 microseconds (roughly how much they cost on Xeon server chips), then your tasks should be like 50 microseconds to 100 microseconds to amortize the cost
 
@blelbach github.com/STEllAR-GROUP/hpx/blob/master/examples/1d_stencil/… is it here where the granularity happens?
 
Assuming that task size stays in the magical "Saturating L2, not falling out into L3" territory
 
11:56 AM
@blelbach makes sense
 
Yah, that loop there is a tile loop
if that example was vectorized
there would be code ensuring the tiles were allocated at 64 byte alignments
and compiler hints indicating that the pointers are aligned
and #pragma simd to defeat aliasing evils
 
is there a good description of the distributed memory system of HPX somewhere?
like maybe for somebody coming from MPI
another reason I am skeptic is that without knowing it in detail it feels to magical
the whole move work to data idea is just completely different than the MPI way of thinking
 
@gnzlbg That is understandable :p
 
in which you just focus on where the data is
 
@gnzlbg Yah, it is a paradigm shift and it's difficult to make the transition
hang on a sec
 
11:59 AM
it might feel very manual for somebody used to a higher level, but... well to me its simple cause im used to it, but I guess for new people it isn't that hard to understand either
 

« first day (2043 days earlier)      last day (2896 days later) »