« first day (1613 days earlier)      last day (2167 days later) » 
00:00 - 20:0020:00 - 00:00

8:03 PM
There is a way to prevent this particular crash case that wouldn't be too inefficient. Keep a stack of the currently executing PARSE rules, and when Expand_Block runs...check that stack to see if each value pointer in that stack lives inside the expanding block. If so, throw an error and stop the parse. Of course that doesn't fix the general problem.
 
8:20 PM
0
Q: What are the rules about self-modifying parse rules in Rebol/Red?

HostileForkI am looking at self-modifying rules and wondering what exactly the semantics are and how they would work at all. That's a pretty broad question, but I'll use a specific "how would I do this" to turn it into a more focused one. (^(64) being the hex ascii for lowercase "d" so it doesn't get foun...

 
Ah, the joy of self-modifying rules ...
>> parse "abcde" rule: ["a" "b" (clear rule) "c" "d" "e"]
 
; Brought to you by: try.rebol.nl
== true
 
right.
 
What's the bot syntax for R2?
rebol2> parse "abcde" rule: ["a" "b" (clear rule) "c" "d" "e"]
 
; Brought to you by: try.rebol.nl
== true
 
8:26 PM
red> parse "abcde" rule: ["a" "b" (clear rule) "c" "d" "e"]
 
; Brought to you by: try.rebol.nl
== true
 
Enjoy universal failure ...
 
It's not so much a problem if you could catch it and say "modifying rule on stack", there's just no hook for that. You can't even do it by tracking all the REBSERs and modifications because many of the series operations are churning on pointers inside the data, and you can't get a series back from a value data pointer.
It suggests that what the REBVAL* based series routines do should actually be REBVALs of series + index passed around so that you have something you can actually put into a "do not modify list". Then all the modifying routines have to check that.
 
As you rightly noted, someone needs to define the desired semantics first.
 
Of course, being not written in C++, you can't safely throw and unwind if the case happens.
 
8:31 PM
I'll spare us the snark ...
 
So your longjmp will leak whatever you were doing if you notice the problem.
 
You could do all kinds of tracking, and I'm quite sure it's not the mutators that should check (unless it's PARSE's own mutators), but PARSE that'd have to check the world is still sane after executing executing actions.
 
It's not just parse, I'm sure DO has the same problem.
Hm, well not if it's going by series/index. Hmmm.
 
I somewhat doubt that, DO is very careful with raw REBVAL* arithmetic.
 
>> inner: [clear outer] do compose outer: [print "Universal" (inner) print "Failure"]
 
8:36 PM
; Brought to you by: try.rebol.nl
Universal
Failure
 
@earl You were saying?
red> inner: [clear outer] do compose outer: [print "Universal" (inner) print "Failure"]
 
; Brought to you by: try.rebol.nl
Universal
Failure
 
I don't want it to be a counsel of despair, as I say, it's possible to make it as safe as such things can be.
Just not in C or Red/System.
You don't define semantics for it, you just throw if it happens.
 
@HostileFork That's not the same.
You DO a different block than OUTER here.
>> do outer: [print "Universal" clear outer print "Failure"]
 
; Brought to you by: try.rebol.nl
Universal
== []
 
8:43 PM
@earl Protecting that one case is important why?
 
Your example clears a different block than what it is executing. It's not related in any way.
 
@earl Who cares?
You can still crash it
 
Huh?
 
Well... I guess an Expand_Block won't crash it. But it's a semantic problem of running values past the actual series end.
 
Again, your example never does that. You don't even touch the series that is executed.
 
8:47 PM
@HostileFork i never figured out what was causing that
 
>> outer: [print "Universal" (inner) print "Failure"] inner: [insert back back outer [print "Hmmm"] probe outer clear outer] do outer
 
; Brought to you by: try.rebol.nl
Universal
Failure
 
@earl DO doesn't copy the block. Where is it keeping the original data?
 
@HostileFork You copied the block.
@RebolBot
inner: [clear outer]
outer: [print "Universal" (inner) print "Studios"]
code: compose outer ;; This creates a copy of OUTER
do code
 
; Brought to you by: try.rebol.nl
Universal
Studios
 
8:51 PM
Oh, oops.
I guess PARSE could work the same way. Doesn't solve semantics. Do you think that's a reasonable solution, performance piggies aside?
 
Also note how Do_Next does not operate on a REBVAL* but on a REBSER*.
@HostileFork Too tired to comprehensively think about solutions right now, sorry.
Also, I'd have to think about desired semantics first.
I think PARSE either needs to more attentively protect executed rules, or it needs to be more careful about where it steps.
 
@earl Perhaps it could use PROTECT and UNPROTECT when it was done...
 
Wasn't meaning to imply actual PROTECT, sorry.
 
Well, couldn't that work?
 
Depends on what is expected of self-modification.
 
9:00 PM
Seems to me the answer is that nothing can be modified meaningfully if it's on the stack.
Would have to be a separate "system protect bit" if one didn't want users to be able to mess with it and overrule you
 
Again: tired. Will need more comprehensive research into what is actually expected of PARSE. I'm of the Ladislav school of "if you do self-modifying rules, abandon all hope, ye who enter here".
(Ok, the Dante-Ladislav school.)
 
Sleep then...the problems won't be going anywhere overnight... :-)
I'll make sure no one fixes them before morning.
 
Heh, please by all means do.
Be sure to also shoot Maxim (@moliad) an inquiry. I think he is the most heavy user of self-modifying PARSE rules in current existence.
PROTECTing rules would disallow any modification. Unfortunately, there are currently quite a few modifications which are "safe" (by accident, we could say, but then, I promised not to be snarky).
Don't trigger an expansion, don't truncate back before the current position, etc ...
And to be perfectly fair: I don't in fact know if Ladislav is of the school I attributed to him above; our shared dislike really is about modifying input from within the parse, not about self-modifying rules.
 
9:23 PM
Um ... did no-one notice I posted the fix ...
 
9:51 PM
@MarkI There's no REBSER* for the rules around inside parse, upon which to use BLK_TAIL.
That's (most likely, only following @HostileFork's diagnosis from a distance) part of the problem -- parse very early on retrieves the series payload (VAL_BLK_DATA) for the RULES argument and then just does pointer arithmetic on this REBVAL* (instead of the safer REBSER* + index dance), just assuming that no one else will ever touch it.
 
10:08 PM
>> parse [a w] rules: ['a 'w]
 
; Brought to you by: try.rebol.nl
== true
 
So far so good.
>> parse [a w] rules: ['a (remove back tail rules) 'w]
 
; Brought to you by: try.rebol.nl
== false
 
Most likely what is expected. Rule is modified from within, PARSE correctly continues with the modified rule.
If my model as described above is somewhat accurate, then this only works, because the "remove last" idiom REMOVE BACK TAIL properly inserts an internal END! pseudo-value in the last slot, which PARSE also correctly picks up on.
Now compare above remove back tail behaviour with the following:
>> parse [a w] rules: ['a (clear rules) 'w]
 
; Brought to you by: try.rebol.nl
== true
 
10:11 PM
Dubbed the universal failure case.
CLEAR truncates by replacing the first value with an END! and by setting the series tail correctly. PARSE has already moved beyond the first value and has an internal REBVAL pointer still pointing at the ['w] position. Because CLEAR didn't modify that at all, PARSE still happily churns along on the "old" series, processing the 'W and then bumping in the "old" END! after that.
And that's still only the "awkward" cases.
If now, by any chance, the payload memory gets garbage collected (for whatever reason, Lest's actual case seems to be because the rules series gets expanded and hence realloc'd) a crash is not far away.
Clearly, the CLEAR vs REMOVE BACK TAIL example shows an inconsistency. I'd say, that the behaviour exhibited by the REMOVE BACK TAIL example is generally expected -- i.e. that PARSE correctly acts on modified rules. So the CLEAR example shows a bug, which, when fixed comprehensively and correctly, will also fix a range of much nastier memory corruption issues, such as the one Lest seems to stumble over.
(Insert tiredness caveats here.)
 
10:37 PM
>> repeat i 100 [print i lest %index.lest]
1
** Script error: protected value or series - cannot modify
** Where: append unless add-rule if if load-plugin parse lest repeat
** Near: append rules '|
@rebolbot delete
@rebolek I've added protection on the currently running rule stack to catch the dangerous cases. So it tells you where you're modifying something you shouldn't. There's one.
'>> safe: ["a"] rule: ["c" (clear rule)] parse "abc" [safe (clear safe) "b" rule]
** Script error: protected value or series - cannot modify
** Where: clear parse
** Near: clear rule
So it's all right to modify parse rules, just not one that's on the parse stack, and this catches that.
 
Brian, just FYI, I'm not ignoring the loop stuff on your repo. Cloned and went to push, but can't, so need to fork and then PR, but should I clean it up then and name to .reb or should I set up my own for consistency and get json.r in there...and I have other things to do too. :-)
 
@GreggIrwin Cool, thanks!
 
11:20 PM
'>> safe: ["a"] rule: ["c" (unprotect rule)] parse "abc" [safe (unprotect safe) "b" rule]
** Script error: PARSE - executing rule block was UNPROTECTed ["c" (unprotect rule)]
** Where: parse
** Near: parse "abc" [safe (unprotect safe) "b" rule]
 
11:43 PM
@rebolek Here's a commit for it. It's based against Rebol master, but should presumably be something you can pull in against rebolsource too. github.com/rebol/rebol/commit/…
@earl --^ I actually don't have a rebol build environment handy right this moment, as I've got a strange 64-bit VM that has problems building 32-bit executables, so that was written against rebolsource. Mostly made it to help @rebolek find places that can trigger the crash. But something to look at when you're not tired.
Things like "is it okay to pass an arbitrarily large block to an error with Trap1". I don't know, but I don't know how else to help people know what the rule was that had the problem. Anyway, a little study that's taken up some time and now I'm tired...
 
00:00 - 20:0020:00 - 00:00

« first day (1613 days earlier)      last day (2167 days later) »