« first day (50 days earlier)      last day (256 days later) » 

user895378
00:39
@bwoebi And you're sure you're pointing your browser at http://127.0.0.1:1338/ (with the port number included)?
user895378
@bwoebi This is one that I had just started working on when you asked if I'd pushed yet and just didn't bother to git rm --cached :)
user895378
@bwoebi Ah, you're definitely right on the s/broker/endpoint/
Yay for helping! \o/
user895378
Much helps! So assist! Wow.
01:14
SocketException in Artax :)
user895378
01:47
Not surprising.
user895378
Sockets love to Except.
09:39
@rdlowrey yes, I do…
10:10
@rdlowrey you're aware that one could DOS your websocket implementation by sending first a large chunk of data and then tons of mini-frames (4KB, memory page size)? This is by nature when you append always to your buffer: the string is going to be realloc()'ed all the time internally and needs to be copied whenever there's an used memory block just adjacent…
You'd maybe have better to not append to the buffer, but make the buffer actually an array of reads… or fragment these reads in chunks of max(4KB, last TCP frame content) (fragment to not create a too massive memory needed/input ratio)
And then somehow allocate the large chunk of memory at once when you're finished reading and paste your single mini-buffers into the bug string to return to the application.
It should not consume too much CPU, but the whole process will block as long as you do the memory copying operations … and then it reads the next bad TCP frame … and blocks again.
It's a not an issue as long as the maximum frame size is very low… but maybe 1 MB would be sensible for certain applications…
and memory speeds are at ±2 Gigabyte/s … so, we just would need to upload first 500 KB and then send 125x4KB … and that 32 times per second.
that's for the attacker just 32 Megabytes/second which you can actually have for not too much…
I think there's always a zero missing… but still… take ten computers and same problem again.
It's a common issue in PHP to just append userinput…
The amplification factor is around 3/4 * $maxFrameSize / 4KB which is a lot…
memory speeds are at 20 GB/s, but still, it's a factor here.
Ayers should still be able to handle 40k frames/sec (as it'd then be only 10 clients), that's not the problem I think…
11:46
*Aerys … I tend to misspell that name :-D
user895378
12:44
@bwoebi I am aware of this ... it's an easy problem to fix with rate limiting as well. If you look here you'll see this line:
user895378
// @TODO add minimum average frame size rate threshold to prevent tiny-frame DoS
user895378
Websocket endpoints already track the number of bytes/frames/messages received on a given connection.
user895378
It should just be a matter of adding configurable leaky-bucket rate-limiting rules to prevent malicious behavior.
user895378
The other thing I considered is (and this I will almost certainly do) ...
user895378
Once an in-progress message size exceeds a certain threshold (e.g. 2MB) just use the global Amp dispatcher to write that buffer to a temporary file (in a worker thread). That way you can accept really large messages without blocking or having to keep everything in-memory.
user895378
12:52
As message chunks continue to arrive you keep appending them to the temporary file.
user895378
All that said, accepting large files is always a security risk in any networked application.
user895378
The vast majority of websocket use-cases happen in browsers. Almost all browsers split up messages into 32K frames (as far as I've seen).
user895378
It should be fairly straight-forward to detect malicious activity with simple rate-limit tracking. Combining that with periodically offloading message buffers to temporary files with pthreads should eliminate any issues here.
user895378
@bwoebi I need more information to help ... it "just works" on any system I've tested with. Do the other examples work?
14:10
@rdlowrey I'm talking about frames of 4KB … which surely would match that minimum and be by that way ineffective...
Actually in real websocket applications you often send frames of a few hundred bytes… and if you use fullblown frameworks of a few KB…
user895378
Would using a php://memory resource stream mitigate the buffering problem?
user895378
I'm a little fuzzy on how those things work
@rdlowrey you just can use one memory stream I thought? so it'd be useless anyway
user895378
The problem with an array of reads is that you have to parse the data as if it's one single buffer. I suppose I could rewrite the parsing to do that ... just makes parsing more complicated :)
user895378
I guess it won't be too difficult, though, because you always know the exact length of the frame.
user895378
14:22
After you've read the frame header you can just stack read buffers onto an array until you receive enough data that you know you have the full frame.
^ This.
@rdlowrey I'm on OS X here… Btw. the other examples just respond the same way: blank page
user895378
@bwoebi can you post the headers you're receiving?
user895378
Or better yet, post the raw output you get from something like telnet?
user895378
$ telnet 127.0.0.1 1338

GET / HTTP/1.1
Getting connection closed
user895378
14:27
With telnet?
I would say good morning but it seems there are issues o.0
root# telnet 127.0.0.1 1338
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Connection closed by foreign host.
user895378
BTW the default idle timeout is 5 seconds so if you don't send the request start line quick enough you're supposed to get disconnected :)
user895378
Add this line in the example file:
user895378
$__keepAliveTimeout = 9999;
user895378
14:28
will make your life much easier :)
oh, okay, that explains something in telnet^^
user895378
That prevents people from establishing tons of connections and not doing anything with them to DoS you.
Getting connection closed anyway… hmhm.
well, will try with browser and looking at tcp stream
user895378
This is what I get in CentOS:
user895378
GET / HTTP/1.1

HTTP/1.1 200 OK
Content-Length: 48
Connection: keep-alive
Keep-Alive: timeout=9999, max=149
Content-Type: text/html; charset=utf-8
Date: Sat, 12 Apr 2014 14:30:39 UTC

<html><body><h1>Hello, world.</h1></body></html>
user895378
14:33
@bwoebi the first place to start debugging is lib/Server.php ~line 400 in the onClient method. You can put some sort of output there to dump to your console.
user895378
then drop down about 20 lines to readClientSocketData($client)
user895378
to see what's actually being read from the client connection.
user895378
Then if you go down to line 692 you'll see the invokeHostApplication($cycle) function
user895378
That's where the responder should actually be invoked and return the html string.
user895378
If you're able to get that far and the responder callable actually returns the expected html string then the problem has to do with the client connection being closed (for some reason) before the response can be written.
14:46
okay, will do
@rdlowrey the whole header is read in $data. But I'm never getting to the invokeHostApplication method
user895378
What happens if you insert var_dump($parsedRequest); inside the parse loop at line 435?
string(350) "GET / HTTP/1.1
Host: localhost:1338
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.74.9 (KHTML, like Gecko) Version/7.0.2 Safari/537.74.9
Accept-Language: en-us
Cache-Control: max-age=0
Connection: keep-alive

"
wait
@rdlowrey nothing.
user895378
okay, there could be a parser problem. Lemme try sending that raw request in telnet ...
user895378
weird ... that works just fine for me in telnet. I'll keep looking.
yep, a var_dump(1) in line 434 actually works…
I think I've given you once access to my server… I didn't revoke it yet…
user895378
14:55
Yeah I remember that -- I don't seem to have the address on hand. What is it again?
bobweinand.no-ip.org
Search for that domain in your old mails…
user895378
okay, I'm connected.
@rdlowrey repository is in /var/root/Aerys
user895378
cd: /var/root/Aerys: Permission denied
user895378
I'll just reclone in my home directory to debug ...
14:57
then do sudo -i or so
@rdlowrey Could you reproduce it?
user895378
I'm still trying to get the submodules working correctly :)
hehe, I had to manually clone them^^
user895378
Yeah, same here.
user895378
Yeah, reproduced. Time for some debugging ...
good luck with that :-)
user895378
15:11
It's clearly a parsing problem ... that's a good sign because it means there's not some socket incompatibility or something that really makes MacOS problematic.
and why do these headers then work on centos?
user895378
I just need to see where my message parser is having trouble in your environment.
user895378
Well mac may report EOF when I don't expect it to. I definitely think it's a software problem on my end. I'll keep you posted ...
Maybe your CentOS isn't using the pecl parser?
or it has still version 1 of the parser and I have v2?
user895378
Ah found the problem.
user895378
15:23
@bwoebi and so did you
user895378
It's the pecl parser
user895378
I'm actually moving away from that.
user895378
I'm eventually going to remove the pecl/http parser altogether
Does the pecl parser have a bug?
or is it the interaction with it? @rdlowrey
user895378
I think my wrapper around it seems to have a bug. But I've found that my userland parser is more or less equivalent with the pecl version I've created (speed-wise) so I'm doing away with it.
user895378
15:25
So you won't need pecl/http going forward with aerys.
user895378
I'm going to commit a couple of lines of changes that will fix the bug (I've already tested it on your box).
user895378
I'll ping you after I push those so you can pull them in.
user895378
I've been working without pecl/http on my local dev box for awhile now so I never noticed the issue.
user895378
@bwoebi okay, those changes are pushed to the abstractions branch
@rdlowrey ah you just removed it completely
user895378
15:31
Yeah, I'm going to eventually remove it altogether.
user895378
And there won't be a separate Aerys\Parse subnamespace at all.
user895378
One parser to rule them all.
:-)
user895378
And in any case, if I ever use a compiled http parser it will be one I write myself. So the pecl thing isn't very useful. When you throw in that it's not significantly faster than my userland version it doesn't make a ton of sense to maintain it and add complexity to the repo or tell people to pull in another pecl extension.
Btw @rdlowrey please try running example 002 once and remove these notices…
@rdlowrey you might want to tell Mike that (m6w6)
user895378
15:36
I may at some point see if Mike would consider adding a stream-based parser (i.e. one that doesn't rely on having the entire message when called).
user895378
@bwoebi what notices?
user895378
Also, note that for example 002 to work you need to modify your hosts file.
@rdlowrey I know…
I'm getting a bunch of notices on startup and on every requets about constant redefinitions in Unkown on line 0
user895378
Hmm ... that doesn't have anything to do with aerys
user895378
That looks like a pecl/http problem
user895378
15:40
try disabling the extension and I bet you won't get the notices.
hmm… no I need that one in another application
I was just wondering why I'm getting that in 002, but not in 001…^^
user895378
No idea ... that's weird.
user895378
Never seen that before.
me neither
user895378
16:11

Why I'm killing the pecl/http aerys parser

48 mins ago, 10 minutes total – 19 messages, 2 users, 0 stars

Bookmarked 35 secs ago by rdlowrey

user895378
@NikiC ^^ See above.
@rdlowrey So effectively: I don't trust no HTTP parser I haven't written myself :)
user895378
Basically.
Good :)
On the other hand, heavy reliance on pthreads is not good ;)
Honestly I think as much code as possible should be in user-land.
A small, extensible core is the way to go.
This means less C/C++ code to maintain
user895378
16:21
@NikiC Well, I'm going to make sure everything works regardless of whether or not you have pthreads.
Plus it encourages the core to make the PHP engine faster.
user895378
BUT ... there's no way to serve static files in a non-blocking way without threads. So if you want to use aerys to serve static files in production you MUST have the pthreads extension.
@rdlowrey Is there no way or no way to make a way?
Yeah, a thread implementation (or some concurrency) would be excellent for core material.
user895378
@NikiC There is no way because this: remlab.net/op/nonblock.shtml
16:23
@LeviMorrison Then it first would need to be more stable… less ways to sigsegv…
@bwoebi Definitely ^^
user895378
There are things you can do ... there's libeio (won't work on windows) ... libuv (the php extension is not stable or consistently developed).
user895378
Both of those C libs delegate to worker threads under the hood to do filesystem IO.
user895378
So you need threads of some kind.
user895378
And pthreads gives me cross-OS compatibility for free.
16:25
k
as long as it works without, everything's good :)
user895378
Yeah I mean if you're doing really serious serving you should be using a CDN for your static files anyway.
I'm always a bit apprehensive of heavy pthreads reliance because I know that zts support is in no way a priority for php. Usually it's little more than an afterthought
user895378
Right.
@NikiC is it listed in things to do better in PHP6?
@bwoebi no
It's listed in "Things we might bother keeping compatible in PHP6" ;)
16:28
Speaking of things for PHP 6, we should go through this list from 2005 and see what hasn't made it and re-evaluate them: derickrethans.nl/files/meeting-notes.html
user895378
16:58
Doesn't pretty much every apache PHP instance depend on TSRM or is that my imagination?
user895378
Anything where the web SAPI is running inside threads would need it, right? Or am I just misunderstanding?
user895378
If that's correct then removing zts would break pretty much every PHP windows apache install.
17:27
@rdlowrey exactly… but who cares about windows? :-D (joking)
user895378
I certainly don't, but I'm thinking that's a pretty big reason why zts can't be dropped ...
So I imagine you are working on Aerys at the moment?
I ask because yesterday another department CSR asked us what we use to consume BYU's web services and I had to tell them I'm using a very old, outdated version of a still not 1.0 software.
^^
If anybody of you guys has time and even if it's german: Any suggestions to our documentation layout @ engine-alpha.org/wiki/Dokumentation/ea/Farbe ? It's planned to add a small class card to the sidebar.
user895378
@LeviMorrison Hmm ... well ... I suppose I could spend this week finalizing a 1.0 version of Artax :)
user895378
It wouldn't be that difficult.
user895378
17:42
I could probably do it in a day or so and just leave the proxy support for v2.0
I know you probably have at least a week and likely two before they can start programming.
They first have to go through a really basic security audit and sign some informational agreement paperwork.
The good news is that the old version has been pretty stable. We haven't found any bugs in it at all ^^
user895378
Okay, cool. I'll try to get 1.0 out the door this week.
And then I can have my web developer go through the upgrade process the next week and give you real feedback.
As much as possible I will try to have my web developer fix any issues he encounters too.
Rather than directly bothering you with them.
I'll double check with my boss on Monday but I'm pretty sure we can contribute to open source products we are using.
If the answer is no then I've been violating it unintentionally ^^
user895378
:)
user895378
@kelunik It looks fine to me. The only suggestion I would make is to use a bit more of your main div in the center since you have all that space on the right side (unless that's where you're talking about adding the "small class card to the sidebar").
17:52
There seems to be a lot of open white space that's just open; doesn't seem to have a reason.
Everything left of your center div before you hit the 'container' background, basically.
Also, it seems to be using font tags? What?
for what this room ?
I can ask about the software quality here ?
ok thanx
17:56
@rdlowrey Why did you tag both v0.5 and v0.6?
user895378
@NikiC Apparently I forgot to git push --tags when I did v0.5.0 a couple of days ago so it hadn't shown up yet.
@rdlowrey No idea really ... I though IIS is what ZTS is needed for
@rdlowrey Oh, not likely to be dropped. Just saying that it isn't a priority. Especially to those working on perf.
@kelunik Also: use a larger font size. Even going from 13px to 14px will make a difference.
Also, use code tags for code sections (both inline and block)
@rdlowrey engine-alpha.org/wiki/Dokumentation/ea/Game should give you and idea of what I'm planning to do with the sidebar
@LeviMorrison I'll try more things after I got something to eat
Also, the glow around engine alpha is hurting its appearance ^^ This is a much smaller issue than semantics and font size.
18:04
@LeviMorrison good eating
ah soryy is @Kelunik lol
 
2 hours later…
19:40
@fahdijbeli thanks
19:53
Getting… Fatal error: require(): Failed opening required '/private/var/root/Aerys/src/../lib/Watch/BinOptions.php' (in autoloader)
Just tried without pthreads…
okay, ftfy
oh, there are more such problems… sigh
20:11
I'm trying to use phpdbg on Aerys… github.com/rdlowrey/Aerys/blob/abstractions/lib/Watch/… … bääm… you invoke phpdbg via phpdbg, that can only go wrong… ^^
and phpdbg requires args to be passed after -- … so^^ And a -rr option isn't bad too then…
21:01
only one bit set: $x == ($x & -$x) ftw
It's useful when wanting to realloc in steps of times two :-)
instead of having a counter: size of memory chunk & actual pointer
github.com/engine-alpha/engine-alpha/commit/… < took me more than half an hour to find that bug. MediaWiki escaped some </div>...
@kelunik I don't find any bug, only a little imperfection.
@bwoebi Not sure I got that use
I.e. what does "only one bit set check" have to do with realloc in powers of two?
user895378
@bwoebi yeah that doesn't surprise me. I suspect the non-pcntl version is probably very broken right now. I haven't done anything without pcntl since the 7,000 line rewrite.
user895378
21:11
I'll work on it now ...
@bwoebi why doesn't zend_ast use the struct hack?
@NikiC what do you mean with struct hack?
@bwoebi zend_ast child[1], so you write ast->child[n] instead of (&ast->child)[n]
"struct hack" is the term for that pattern ^^
hack because it's technically UB to index outside the [1] bound and the zend_ast child[] syntax was only added in C99, so nobody uses it ^^
uh, no idea… probably because I didn't know that this is fine…
I think it was because compiler complained
llvm-gcc is very pedantic…
ah yes
@NikiC I know why: it'd be a self-referencing struct then
struct named { named child[1]; } does naturally not work…, so doesn't it work neither here…
21:21
@bwoebi Sorry, I meant zend_ast *child[1] of course ;)
@bwoebi Actually, dmitry did this: lxr.php.net/diff/PHP_TRUNK/Zend/…
And I reverted it, don't ask me why … github.com/php/php-src/commit/…
you just changed it back in the next commit, for some reason: lxr.php.net/diff/PHP_TRUNK/Zend/…
no idea...
feel free to revert…
no, I don't revert other people's changes ;)
@NikiC then it just will remain as is.
It's nothing that bothers me…
21:31
@LeviMorrison thanks for the logo tip!
 
2 hours later…
23:26
Errrrrm… would you mind using tabs @rdlowrey? :-D It's the major thing which annoyed me just now^^ I'm typing tabs all the time just to notice that it doesn't align with your 4-spaces format…
tabs look best with 4 spaces^^
The only good thing is that I use 8 chars wide tabs and so I notice the difference^^
user895378
23:49
@bwoebi BLASPHEMY.
@rdlowrey I agree. Using spaces is blasphemy.
user895378
Sorry, the spaces are something I can't compromise. I use spaces in PHP and tabs in C code.
user895378
What editor are you using?
/usr/bin/nano
I just can't work with vi(m)… I constantly forget to enter insert mode and then I type an r and am accidentally in replace mode etc. It sucks :x
user895378
25
A: How to make the tab character 4 spaces instead of 8 spaces in nano?

Sven RojekIf you use nano with a language like python (as in your example) it's also a good idea to convert tabs to spaces. Edit your ~/.nanorc file (or create it) and add: set tabsize 4 set tabstospaces If you already got a file with tabs and want to convert them to spaces i recommend the expandcomman...

user895378
23:54
:)
I know. But then it doesn't fit with my own projects :-P
But yes, I expected that reaction; just wanted to notice here that it annoys me^^
Btw. @rdlowrey I think you can move Parser.php and ParseException.php a directory higher (/lib)

« first day (50 days earlier)      last day (256 days later) »