« first day (1670 days earlier)      last day (2110 days later) » 

1:24 AM
@HostileFork We could always split it into a safe native and an unsafe native, then wrap them together in the mezzanine with a /safe refinement.
compose/safe [some ridiculously large block]
 
@iceflow19 Leaks too much implementation detail I'd say; although I asked the question of how much /INTO is burdening users with concerns the system might "invisibly" take care of.
 
Though that begs the question would we want safe refinements on other functions too.
 
Everything is supposed to present a certain layer of abstraction. As per "The essence of architecture is the suppression of information not necessary to the task at hand, and so it is somehow fitting that the very nature of architecture is such that it never presents its whole self to us but only a facet or two at a time."
In the "misconceptions about C" department, there are some things in Rebol like:
value = -value;
if (value < 0) {
    // must have been an overflow...
    // do handling here
}
 
@HostileFork Yes but the amount of abstraction is arbitrary. Take C or any other low level language for example, most depend on some level by even lower abstractions (down to the machine level) and have some leakage higher up. Certainly Rebol is higher level, but by the same token a runtime can also be looked at as an idealized machine. So hmm.
 
@iceflow19 Well, still thinking the idealized machine and philosophy has to be hammered before going and optimizing. COMPOSE is done the way it is for performance...but I feel that before worrying so much about that a very clear story about the stack should have been told.
 
1:38 AM
@HostileFork I wont argue with that.
 
Funny because as long as you invoke a Do_Next, there's a little stack expansion that goes on. A kind of nonspecific "bump it by 100 if it's too close to running out"
Problem is that COMPOSE is not like REDUCE, in that it only calls Do_Next on parens.
So if you compose a bunch of non-paren things, like integers, it will just keep bumping integers onto the stack without going through the little "er, how 'bout 100-ish more stack if you run out?" tweak.
 
As I don't usually mess with stuff like this, when the managed stack expands does it do an allocation and then copy the previous stack over to the new expanded one?
Thus invalidating the pointers into the stack, as you said previously?
 
@iceflow19 Yup. Which means if you were trusting pointer-based (vs. offset based) value cells you were holding to be safe... u r screwed. You need to keep offsets.
Rebol internal code had the occasional instance of being protected against this; so some things that called Do_Next didn't try and cache into the stack.
Because the execution of arbitrary code probably tends to trip problems beyond the 100 "freebie" stack you got...
Making things difficult was that a large stack was allocated at the beginning, so you really only start seeing these problems if you run code for a while or deeply.
 
Would it be possible to have a chunked stack? A higher level example... like a linked list of fixed size arrays.
 
I was thinking it might be interesting in general to see how the system and its behavior would be on a chunked deque for underlying series mechanic. en.cppreference.com/w/cpp/container/deque
Pretty invasive change when a lot of code expects stable pointers
Anyway, the stack thing isn't any kind of serious issue relatively speaking.
I do believe--however--it represents the kind of thing that the first time it is realized that there is a design problem, then instead of bumping some number by 100 or whatever, that it be addressed and vetted. OTOH, if that's not what you want to work on right then and you're exploring some other idea, you might think it's "good enough" to let you think about what you wanted.
 
1:59 AM
@HostileFork We'd have to weigh the performance gains and losses.
@HostileFork True.
 
With the ability to build Rebol under C++ then it's possible to do some abstractions which augment the testing as the C++ build; so what would be a simple pointer in the C build can be a pointer-plus-sequence-number in the C++ debug build; and you can knock up the sequence number each time you push to the stack... then catch when someone is caching a pointer when it might cross an expand
 
That would pretty interesting actually.
 
Already got some tricks going on like that
I think the question about the stack goes back to the scope of application and what the user wants.
 
Btw, anyone ever try running multiple instances of Cheyenne at the same time? I noticed that Doc hardcoded some ports in Uniserve.
 
Red/System will presumably follow C and you pick a stack size and that's that; no relocation.
 
2:05 AM
@HostileFork That would be the expectation. Its meant to be really low level.
 
Run out of stack? Ask the OS for more with ulimit... stackoverflow.com/questions/13245019/…
But for Red and Rebol and issues like the compose here, I feel it should be higher level
I guess in a sense especially with compose; it doesn't look like something that should overflow the stack
 
@HostileFork Yes if anything compose should be a stack safe operation.
2
 
Whereas a user can grasp why do a: does [a] won't fly...
If you don't get an error on that, what... should happen? :-/
 
2:56 AM
>> shift -1 1
 
; Brought to you by: try.rebol.nl
== -2
 
21
Q: Why does left shift operation invoke Undefined Behaviour when the left side operand has negative value?

Prasoon SauravIn C bitwise left shift operation invokes Undefined Behaviour when the left side operand has negative value. Relevant quote from ISO C99 (6.5.7/4) The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with zeros. If E1 has an unsigned type, the value of ...

Doing left shift on a negative number is undefined behavior in C. So Rebol wants to support it?
It will have to be implemented differently, says master ub-san: ../src/core/n-math.c:282:19: runtime error: left shift of negative value -1
 
 
2 hours later…
5:06 AM
Sigh, well, there has to be something done about the stack problem if I am to meet the "robustness" goal. One simply can't turn up the heat on all the testing unless cached pointers into the stack have some kind of safety... it's a flat out systemic bug
I think that using the same underlying structure that is used for a BLOCK! for the stack may be the problem. The stack has different needs, and the need for contiguity in memory is perhaps not as important as having pointer stability while something is on the stack.
@iceflow19 So maybe the chunked stack is worth a try; I think defending against the pointer volatility is perhaps a slippery slope.
 
5:23 AM
Function thought: might be an idea to give a refinement a number instead of true to reflect the order the refinements were called. e.g. get-messages/by-date/by-person would be distinct from get-messages/by-person/by-date.
 
@rgchris My idea that I'm still fond of is that the refinement comes in as either NONE! or the word of the refinement itself.
It was for chaining purposes... so foo: func [a /b] [bar/:b ...] could be used, and if bar were a function then the invocation would pass the /b refinement and if it were none it would be ignored.
So append/#[none!]/#[none!] would be the same as append in this format (and in newpath, would be rendered as append//
 
Could see that being useful too, though wouldn't distinguish the two calls in my example.
 
Having a distinction brings about complexity not currently covered, which might add an unwanted degree of freedom
 
Too much freedom, arghh!
2
 
Anyway, status report is... I had a few down days of not feeling like programming, but got back to it and have gotten pretty far in the undefined behavior sanitizer
May not be able to do all the nitpicky ones
(at this point in time)
Looking more into the big crashers
Some if it is about signed/unsigned stuff in the jpeg decoder and things like that.
The stack expansion and unstable pointers is the biggest thing on the radar ATM, it's def. crashy
 
5:56 AM
Hm. I wonder if natives should in their spec declare how many stack slots they intend to use, and default to zero? Then if Do_Next was twisted into something that could directly write its result into a native's return slot in the data stack frame...the only stack space a usual native would need would be the known stack for the arguments, accessed by pointer, and guaranteed sequential from start of frame.
 
Im back. I fell asleep without realizing lol.
 
6:15 AM
@iceflow19 Welp, yup, I think the stack pointers need to be stable. Probably worth it to chunk it instead of trying to reallocate and shake everything up. Going to require some other changes. I guess the biggest question I have is whether the investigation be into "a new kind of block! series capability" that could be used for non-stacks, or if that's likely to just be a mire
 
I think perhaps one thing at a time. Doing it for the stack has some clear benefits. For blocks its not as clear at this point. I would also argue it may not be good for other design reasons to tie the Rebol stack to the implementation of block! solely due to them having different needs/focuses.
 
Probably. It was already rather weird, in that the tail of the block wasn't kept up to date due to the indirection... the "DSP" (data stack pointer) was a global variable that you didn't need to dereference into a series to get to...
I don't know how much performance that added. But in any case, it meant that before you could expand the stack you had to update the size from the DSP because it had gotten out of sync (!)
So if that sort of thing is worth doing for the stack then yes.
Series in general have this notion of being able to have a bit of leading capacity, and a bit of trailing capacity, and keeping the memory contiguous.
So you can take a value off the head and it doesn't move everything down, it just increases what it calls the "bias"... and if you insert a value at the head it'll reuse that
The stack is a stack. So different.
I'm going to fall asleep while realizing, but do feel free to continue to think on the nature and expectations as they relate to the stack... :-)
 
6:34 AM
Will do, and good night! :)
 
 
1 hour later…
7:59 AM
https://github.com/red/red/pull/1185
GitHub
Red Pull Req—Compare vector
qtxie
1432778111
 
 
4 hours later…
12:20 PM
@Morwenn Feel like I've heard that before for some reason... not bad. In any case, the sanitization and stuff has led to the discussion above if you've noticed; it just is looking to be that Rebol's stack needs to use a different model. Because it's using the same implementation guts as block series, when it needs to expand all the pointers go bad...and making the code robust w.r.t. that by refreshing by stack index is not good.
 
12:54 PM
@HostileFork It seems that it will be some tough work rewriting a core component of the implementation.
Will sanitization drive you insane? :)
 
@Morwenn I'm already insane, so perhaps a more practical question should be asked about the outcomes...
Like "think you'll actually finish this?" or "how much longer?"
 
xD
 
I'm starting as usual with a quick test, not worrying about performance. Make the stack a doubly-linked list and do a study of the clients who rely on contiguity of the memory. Find them, define the interface, then see what the deal is.
 
That's hardly something I can help you with.
@HostileFork It would surprise me if you already knew that. The project is almost unknown. That's the most viewed track of the entire discography on YouTube.
 
Good news is that clients who were doing weird things with the stack have mostly been reigned in. There was a lot of decrementing the stack pointer but expecting some caller to still read the value in the old top of stack position, even though it was volatile... but read it quickly "before it went bad" due to another push.
 
1:02 PM
Ugh...
 
It was an attempt to "simplify" the interface so that it could pretend the stack hadn't grown, so that there didn't have to be accounting in the client... they'd just use the value...
It generally didn't simplify things IMO, and some cases really were better solved with an interface where the client provided the value slot where they wanted the item put...
Classic parallels to those C routines you would call and it would have some internal buffer but give you back a pointer and go "oh right, be careful with that pointer!"
"it's 'volatile'..."
 
I have to admit that I'm kind of lost already.
 
Oh, y'know, like strerror.
Rebol doesn't like passing "boxed" cells by value. Costs too much for an interpreter to be doing that; they're up to 32 bytes (on 64-bit platforms). So it is always trying to pass them around by pointer and avoid copies
(A boxed cell being like an INTEGER!, or whatever. Holds a header which has type bits and other things, then the data... a SERIES! holds not only the pointer to the underlying vector of data but also the offset. These are 4 pointer-sized things wide.)
By protocol they're usually proxied around through stacks by pointer. Problem is, where's the memory and when does it get freed? When you do some kind of evaluation the result winds up on the top of the Rebol stack.
So if someone wants to do an evaluation and give back a result to you, but the result lives on the stack, there's a bit of a problem if they leave it on the stack when the caller didn't expect it. That puts potentially random data in the middle which could get picked up by an evaluation in progress.
They have to either copy the data off the stack into some memory that the client passed them, or get the client to be aware that after they've run the result they are looking for is (by convention) on the top of the stack.
But what Rebol was doing in many places was trying to get around this by returning a pointer to the value slot on the top of the stack and then pop it. Then it would say "!!! DANGER! Volatile! This value is above top of stack and anything that pushes will overwrite it." That's potentially a lot of things.
I said "meh" and made the debug build write trash above the top of stack every time you pop and eliminated the practice.
 
Well, you deliberately made hacks fail if I understand correctly.
 
That's how I see it.
 
1:16 PM
Transformed rather undefined behaviour into a consistent failure.
 
In any case, with such changes the stack is under control. It's just this unstable pointer issue. So a non-sequential (or only with limited sequential guarantees) stack implementation isn't as out of reach as it would be otherwise.
 
That still sounds like there is a bit of tough work remaining.
 
Maybe.
There's actually not too much code (in the scheme of things) that at this point in time access the stack by anything but integer or just cares about the value on top.
The code that accesses by integer expects it to be a fast contiguous pick, but I can make it slow for testing and then look at it.
 
 
2 hours later…
3:13 PM
@ShixinZeng --^ see notes on stack above, in heavier testing I think it is shown that the system has too much dependency on the idea that putting values on the stack keeps them "safe from GC" without facing the fact that the pointer could go bad from stack expansion. Don't think the answer is to invest further in a stack model where pointers go bad just for the sake of using a block! for the stack...
 
@HostileFork Yep, I've started reading the discussion, and I will check the code to understand the problem
Thanks!
 
 
1 hour later…
4:18 PM
@RebolBot
apply :print [10]
apply :print [10 20]
apply :print [10 20 30]
apply :print [10 20 30 40]
 
; Brought to you by: try.rebol.nl
10
20
20
20
 
@redbot
apply :print [10]
apply :print [10 20]
apply :print [10 20 30]
apply :print [10 20 30 40]
 
; Brought to you by: try.rebol.nl
*** Script error: apply has no value
*** Where: do*
*** Stack: do-console all not unset? set do first head reduce do* _execute if all not unset? set do first head reduce do*
 
4:59 PM
@iceflow19 I actually converted the stack to a doubly linked list (one value per list item), deleted the series block for the stack, edited some code and it... compiled. :-/ Now the "how long until boot", "how long until building hostilefork.com", and "will it actually be able to make it through the tests without crashing under the sanitizer?"
If this works, it looks like a chunked solution won't be too bad.
 
@RebolBot
apply :print [10 20 30]
apply/only :print [10 20 30]
 
; Brought to you by: try.rebol.nl
20
10
 
Er ... what gives? Is that a bug? I don't understand the output.
 
@MarkI Apply is just buggy. Don't worry, locking it all up. Any thoughts on the observations about stack pointer stability vs. contiguity?
 
@HostileFork Yep I was reading over the above. Let's cross our fingers and hope for the best.
 
5:10 PM
@HostileFork I prefer a system that works, if that helps ... :)
 
@MarkI Sold! (to the clipboard in the front row...)
 
@HostileFork But I do want an implementation that traps C stack overflows via SIGILL. That may be a topic for later on ...
 
@MarkI Yup.
Looking at some data structures and came across this one, skiplists:
In computer science, a skip list is a data structure that allows fast search within an ordered sequence of elements. Fast search is made possible by maintaining a linked hierarchy of subsequences, each skipping over fewer elements. Searching starts in the sparsest subsequence until two consecutive elements have been found, one smaller and one larger than the element searched for. Via the linked hierarchy these two elements link to elements of the next sparsest subsequence where searching is continued until finally we are searching in the full sequence. The elements that are skipped over may be...
 
I am curious, though, did you resurrect the R2 "list" block implementation? Doubly-linked is what it was supposed to be IIRC.
 
@MarkI Nope, just simple and single purpose with the intent of optimizing to the interface, once I figure out what the interface is.
I'd already eliminated nearly all direct assignments to the stack pointer; instead with operations saying what you were doing, like DS_DROP_TO(dsp) so that you were explicitly saying you wanted to batch drop things from the stack back to a certain point... allowed for poisoning cells above
Due to establishing that finite vocabulary this new structure might (as I said) not be too hard to make
 
5:32 PM
Ack. I hate these preprocessor macros trying to be inline functions and then evaluating their arguments multiple times. Dumb bugs. :-( Hm...you could probably catch at least some problems with C++ debug builds, so in any such unsafe macros you would say {auto temp = (arg); (arg) = temp; thing1(arg); thing2(arg);} That way you'd catch cases where the expression passed in wasn't an lvalue.
Trying to think of ways to allow you to say something like SET_INTEGER(DS_TOP, 10); but not SET_INTEGER(Alloc_Value(), 10);, where you'd get several values allocated...each getting a part of the initialization...
 
I have to get back to work ttl.
 
L8r!
The idea of it existing out-of-band and still compiling on old systems is a kind of weird puzzle/challenge...
Of course, one could just use some kind of transpiler and write your code how you wanted to in the first place. But where's the fun in that?
 
 
3 hours later…
8:33 PM
And shockingly, the fully non-contiguous stack system just built hostilefork.com under valgrind w/o error. (Albeit veeery slowly.) Which serves to show how good my existing screw-tightening was, really, more than anything...
Were we placing bets, I would have bet against that happening today (were there an advantage to betting against something I could myself delay and hence be on the other side of the odds.)
 
8:50 PM
@HostileFork I bet you won't take this bet ... :)
I have divided up the bugs I have found so far into simple (22), line-count-changing (17), workaroundable (5), and not fully thought through yet (23).
 
@MarkI Looking at this one... every time I have to poke at Redo_Func I have to go "really, what's going on?" github.com/rebol/rebol/blob/…
 
@HostileFork Didn't we consign that one to history? Isn't it only used in return/redo?
 
You overwrite the existing function value in the stack frame and then, er... memmove stuff. I've in the past written some "here's why this routine is ungood", that now I need to look at.
@MarkI Different thing. This is how ports dispatch through a protototype which then is mapped into a specific implementing function.
 
@HostileFork Ooh, right, port actions.
bbl
 
Ciao
Anyway, each time I look at it I go "hrrrrm, what?"
 
 
3 hours later…
11:30 PM
@HostileFork Im surprised you got it working that quickly. At this point the next step would be to make it less contiguous, and see how much the performance goes up.
 

« first day (1670 days earlier)      last day (2110 days later) »