« first day (4025 days earlier)      last day (906 days later) » 

10:39 AM
Hi, is there a way to convert Closure into AST? Without parsing the whole file
 
Morngins
 
10:58 AM
\o
@CanVural no
(by the time PHP is executing the AST is gone)
 
Hum, I thought I was on a branch... oh well that's a daring commit I made directly onto master
 
11:34 AM
I have just realized I have been using references to free up memory wrongly my whole life. 3v4l.org/AEeG0 In this example, I'd expect echo $x; to return empty/null
If I understood it correctly, unsetting a reference variable just removes the reference, but does not free up the original memory allocation
To free up memory I'd have to do $x = null; instead
3v4l.org/c2nSE I've been doing memory optimizations wrong my whole life :)
 
you have to have a really good understanding of PHP in order to optimize anything in user code ...
don't try ... it's not your problem or job, don't write obviously crappy code, but don't try to do magic with references and use non-standard patterns in the name of efficiency because you're probably wrong and either making it worse or doing nothing at all except making your code more complicated
for example
> To free up memory I'd have to do $x = null; instead
no, that would not free a single bit of memory ....
 
@JoeWatkins Hmm, the 3v4l.org snippet indicates that memory has been freed. Am I incorrectly interpreting it?
 
you think it's freeing the string "x" and it is not, the string "x" will not be freed for the life of the request and possibly beyond it ...
it remains, whether you have a variable pointing at it or not ... I'm talking about this specific code only ...
that's what I mean by you need a good understanding, "unset free's memory" is not a good understanding of what unset does, and it will not help you to optimize anything ...
 
I see. The refcount won't be decreased by assigning it to null. However, the "free up memory" comes from changing the value from a 1MB string to null, and there is a memory usage gain there even though there are active references to that value, as far as I understand
 
there's no 1mb string in the code I'm looking at
 
11:47 AM
I am sorry. I have pasted two 3v4l snippets, please check the second one
 
Lucas, you're talking to one of the people who knows the engine the best
Whatever you "see" from a 3v4l snippet can probably be explained by a shit load of other reasons
 
@LucasBustamante right, they aren't literal strings like in the first one
ref counting and memory management are not the domain of user code whatever ... in the real world this "optimization" breaks so easily that it's not worth doing ...
references should generally be avoided ... they might even disappear from the language one day (we can all hope) ...
 
That was a very good point. I'm honestly very surprised with this opinion. I always thought as references as good practice when dealing with huge amounts of data (I'm parsing a 60 MB array in my project, for instance).

I have played around with Go for a while, and Go passes values around as pointers quite often, for instance. I think I need to understand better the difference between pointers in C and references in PHP.

Either way, I have trust that you are speaking from a level of understand that I don't have, so I will think carefully about this
 
PHP is copy on write, so if you're only reading data from a variable there is no copy and it is "like" a reference, just a bazillion times better
 
this
 
12:02 PM
Again, don't use references except if you REALLY REALLY REALLY need them
 
and that
$a = [1,2,3];

function ($a) {

}
the thing that the function receives is what C/go/anything would call a pointer, it is "by reference" already ... when we say "PHP is copy on write", we mean that a copy of $a will not be created unless the function writes to $a
the terms by reference and by value are not very useful in php land, even when they're used in the manual they don't really do a lot to explain what is going on ...
 
I'm aware about COW, my idea with passing the 60 MB array as a reference was to unset while I parse it to free up memory to handle more concurrent usages.

I'm not sure if this makes sense at all because each PHP process has it's own pre-defined max memory limit.
 
what do you mean by concurrent ?
 
Multiple users using the application at the same time, each of them requiring a 60 MB array.

If I can decrease the size of this array during runtime, I was expecting the server to have more available free memory to handle more users
Honestly, I'm not sure if this makes sense.
 
well, we're not talking about simple code anymore and I don't like to guess
but I'll take a guess whatever ...
requiring a 60mb array is probably the root problem ...
is it the same array for all processes ?
 
12:12 PM
It's a trial project for a company, and the requirement is to fetch 300 e-mails using the Gmail API, parse them, and return them as JSON
And it should handle at least 6 concurrent users
To parse them, I have to transform them
To avoid using memory, I was unsetting each item of the original array as I parse them
If I fetch the last 300 "Stack Overflow Newsletter" e-mails in my inbox I get a 70MB array
 
do all users have to access the same data ?
 
Nope, they can search for whathever e-mails they would like
It's like a Gmail search box that returns formatted results
 
> the requirement is to fetch 300 e-mails using the Gmail API,
You misspelled fetch individual e-mails, store them locally in something like redis, and then fetch them really fast from that local storage. A common typo, the keys are like right next to each other.
> To avoid using memory
 
One of the requirements is that I can't store them
 
You should get your code working, and then get feedback about what it feels like to use it.
 
12:16 PM
I do the API call, parse and return
It's done already, I can't change it anymore actually
But I did the memory "optimization" wrong
 
Why do you think you need to optimize memory usage?
 
I used unset instead of null. And now, it seems that even that is not very desirable. Fragile, at least.
 
have you ever worked on anything that you could describe as distributed ?
 
I actually had to google what "distributed" means
 
ok
so
when a C/go or some other low level language engineer is looking for ways to scale, the first thing they will likely do is look for memory and CPU optimizations, squeeze everything they have from the hardware they currently have
in PHP, we don't care about that ... the way PHP scales is "add hardware", simple as that ... when you're writing your code, keep an eye on how easy it will be to add that hardware ... that is all that matters, if you can add hardware you can scale to the size of the universe given enough money ...
the entire domain of memory and CPU optimization belongs to internals ... you shouldn't write obviously crappy code, and you should do the normal things like profiling ... but chasing after micro-optimization in user land is a fools game ... you code needs to be simple, easy to understand, and easy to scale ... notice I didn't mention fast or efficient, not your problem ... code that is easy to understand, simple, and easy to scale will by happenstance perform well because of
internal optimizations that you get for free ...
this ought to go some way to explaining the position that references are bad ... they're not simple, and you can add them to your code thinking they are doing one thing, and they can be making it worse, or doing nothing, and all the while they have broken the first rule - code should be simple and easy to understand ...
 
12:37 PM
That's a very good point. I take from this conversation that micro-optimizations are hard and fragile, and should not be done if it compromises code readability.
"Simplicity is the ultimate sophistication"
 
I'll go further, more often than not, micro-optimizing PHP makes no sense whatsoever. PHP scales at the macro level by adding hardware, and this will always be cheaper than working at the micro level by expending human resources which tend to be much more expensive than electronic resources ...
 
22 mins ago, by Danack
Why do you think you need to optimize memory usage?
I'd still kind of like an answer to that.
 
the only PHP code worth micro-optimizing is your very hot code. which is executed hundreds of thousands times per second. Everything else is not… most applications do not have such hot code, so it's not worth it…
 
12:56 PM
/does awkward shuffles trying to reduce a per user request time by 0.5ms
 
@Danack Sorry guys, it's just too much context to try to squeeze into small comments. If you are really curious, I could explain
I have listened and will remember all feedback about micro-optimizations
The Gmail API returns me a HTTP response in the body that follows the multipart content-type RFC
 
Well, even without an answer, I'm still going to link this: en.wikipedia.org/wiki/Streetlight_effect
 
I have to explode this 60 MB string into an array, where each item is a response of the multipart
 
Also....you should probably investigate how to measure app bottlenecks....
 
Then I iterate over each item of the array, to convert each response where the body is a Gmail API Message Resource, and format it in a way to pass it to the FE, removing any clutter, sanitizing everything, removing malicious HTML, etc
 
1:00 PM
You're telling me what your code does. You're not saying why you think you need to optimize memory usage.
 
The input is huge, when a variable might occupy a lot of memory I start to worry about it
 
So you're just doing stuff at random? cool. have fun with that.
 
I didn't want to substr something or explode something and have a memory exhaust, or a memory peak of 100% the size of the original string
I found it's very easy to have a memory exhaust when you're dealing with a variable that is almost the size of your available memory
 
add more memory then =)
 
It's a trial project
They made it on purpose
 
1:03 PM
So it's an interview project?
 
Yes, something like that
A trial project for a job application
 
and do you have to "explode a 60mb string" or is that what you think you have to do?
 
There's literally no other way around it
I had to write a multipart parser that takes a multipart HTTP raw response string as input and convert it to an array, there's no way to parse the data inside without parsing it
I need to sanitize the data, for instance
 
So you can't for example stream the response?
 
I can't, because I need to format and sanitize it first
I can't just throw the raw response I get from the Gmail API into a JSON response
I must say, it was an ingeniously crafted trial
 
1:08 PM
I don't follow, I'm asking if you only have a 60mb string to work with, or if you can receive it via a stream (the response from the server)
 
Hmmm...
 
Because if you can receive it via stream, you just break it out in to individual files as you go along
 
I didn't think about it
Hmm... I don't think I could, actually... Maybe...?
The multipart content-type splits the parts by boundaries...
Well, I'd have to think about it, but it's good food for thought
 
Yes, thus you could pull down in chunks until you hit a boundary, then write what you have so far to your memory segment, then you can discard up to that point in your receive buffer, continue until you hit the next one, taking care in case you received 2 segments in the same receive chunk, repeat until EOF
Or just have the entire string, do your location finds in it, keep those in memory, write the string to disk, discard the original in memory, then use disk IO to pull back the particular ranges
But the real answer is "if you ever think you could blow the RAM buffer, you should be writing directly to disk in the first place" because if you've got a precious 60mb buffer and can't afford to go to 120 to hold 2 copies of it, what happens when someone sends a 120mb attachment
Kay, I'm done :P
 
That's a very good point. The attachments are not included in the response, though, only some basic metadata such as mimetypes and IDs that you can dispatch another API request to fetch.

However, the Gmail Body, if I researched correctly at the time, is allowed to have up to 25mb, so 300 of them will explode your memory no matter what. Streaming to a file seems the only sane way of handling it if this was a requirement.

I was given 40 hours to complete this project, so I had to make compromises with the best of my capabilities. At the time, I thought of asking from Gmail only for e-mails
 
1:28 PM
Must be nice to get people to do 40 hours free work :p
 
2:04 PM
Microsoft are apparently screwing up their open-source foundation: theregister.com/2021/10/11/dotnet_foundation_community_issues + theverge.com/2021/10/22/22740701/…
2
 
 
2 hours later…
4:10 PM
Can anyone see what am I doing wrong here? phpize.online/…
Why are the values NULL?
This works phpize.online/…
 
4:27 PM
I think this is a bug in PHP 3v4l.org/d68ni
Do I really need to do this? 3v4l.org/MAAGu
 
4:43 PM
@Dharman References should simply die.
 
maybe don't use references?
 
mysqli uses references, so I can't stop using them
 
 
4 hours later…
9:12 PM
Isn't there something that auto-assigns previous exceptions even if they were not present on construction in certain cases?
 
9:39 PM
Not familiar to me...
 
 
2 hours later…
11:12 PM
@Dharman array_fill is not aliasing, but a little foreach can do: phpize.online/…
you get the NULL values for free, because that is the default value of a variable.
 
@hakre Thanks, that's an interesting solution.
 
do you need the variables inside that method or outside?
 
I was trying to simplify this code github.com/doctrine/dbal/pull/4920/files
It feels like whichever solution we use there, it's still a hack
 
The comments distract me in that PR, so I don't get it yet.
 
Generally, there should be a simpler solution than a foreach loop, and if it was a normal array than a splat operator would do the job, but with properties you have to create a reference additionally
 
11:24 PM
91-93 left is basically having references in the array, it should have 93 1/2 and unset($value); at least thought.
so you want to get rid of the iteration?
 
yes
 
but foreach is pretty straight forward and array_* are all function calls.
 
yeah, your foreach solution is better than the current one
 
the array_fill is superflous here, $refs is duplicating it.
 
no, refs is just an array with references to the array created by array_fill
My solution is this:
$this->boundValues = array_fill(0, count($this->columnNames), null);

// The following is necessary as PHP cannot handle references to properties properly
$refs = &$this->boundValues;

if (! $this->statement->bind_result(...$refs)) {
throw StatementError::new($this->statement);
}
 
11:29 PM
yes, but it's possible to create that array directly from the names array like in the example.
php handles references on properties well....
 
not in this case, which is why I was stuck
if it did, I could do this:
$this->boundValues = array_fill(0, count($this->columnNames), null);

if (! $this->statement->bind_result(...$this->boundValues)) {
throw StatementError::new($this->statement);
}
but I need to create a reference to the property to have it remember the references to ZVALs inside
 
that's not the reference to the property.
 
this? $refs = &$this->boundValues;
 
that is, what I meant is that the references in the array are important, not to the array.
 
I know
 
11:32 PM
array_fill fills the array just with values.
 
I know
 
then you're binding values.
 
binding variables
or more accurately ZVALs as they don't even have to be variables
 
which ever mental model you prefer... ;)
so how to fill an array with n aliases?
 
Use array_fill
that should do it...
except if you are assigning the result to a property
then you need to additionally create dummy reference to the property
 
11:35 PM
can only do one I guess: array_fill(0, count($this->columnNames), &$ref);
 
null value is fine
 
you want a variable to be bound, not a null value, don't you?
 
$arr = array_fill(0, count($this->columnNames), null);
$stmt->bind_result(...$arr);
This works
This doesn't:
$this->arr = array_fill(0, count($this->columnNames), null);
$stmt->bind_result(...$this->arr);
To make the second one work you need one dummy line:
$this->arr = array_fill(0, count($this->columnNames), null);
$unused = &$this->arr;
$stmt->bind_result(...$this->arr);
Simple, right?
 
here is an example where it works, but it's IoC, so perhaps not fitting:
$listBinder = new class () {
    private $refs;
    function bind(&$p1, &$p2, &$p3) {
        $this->refs = [&$p1, &$p2, &$p3];
        [$p1, $p2, $p3] = range(4, 6);
    }
    function change() {
        $this->refs[0] = 42;
        $this->refs[1] = 'foo';
        $this->refs[2] = 'turtles all the way down';
    }
};
 
That's close, but not equivalent
 
11:40 PM
it's an example where you have the array_fill of null's array
it does not show how mysqli does it, so only userland.
 
Yeah, but you don't save a reference to ZVALs that make the array anywhere
 
it just shows that $this->refs is not aliased, only the array members.
 
yeah, I get it
See my example 3v4l.org/2WCiI
$a and $b are references to ZVALs that make up the array
they aren't just references
PHP has this weird concept of references that are not references
I don't understand it fully, so I can't explain it how it works. It's Nikita-level stuff
basically refcount has to be above 2 IIRC
I really have no idea how it works
 
That was optimized some time ago for scalars IIRC.
It was easier earlier.
 
References are garbage and I wish there was easier way to do it in mysqli
 
11:47 PM
but without references you would not be able to bind the variables.
 
good riddance
 
it really works with the reference. didn't knew.
perhaps a benefit of the splat operator, with call_user_func_array there is no way.
it needs the references in the array.
 
yup
 

« first day (4025 days earlier)      last day (906 days later) »