last day (16 days later) » 

4:58 PM
0
Q: Memory leak in program that uses LWP::UserAgent to download a file

gapvisionI am trying to revive a Perl script I was using a long time ago. It is for downloading files from cloud storage to my local client. I'm pretty sure it worked fine back then, but now I am having the issue that LWP::UserAgent downloads the file entirely into the memory before writing it to disk. Ex...

 
The Perl is less than ideal but superficially there's nothing wrong with it. 2**14 is 16*1024, so LWP will pass 16KB-ish chunks of data (in memory) to the didReceiveData subroutine. It is entirely down to that subroutine what happens thereafter and you don't show it. What makes you think "it loads it entirely into the memory bevor writing it to the disk"? There is far to much here that is irrelevant and not enough that is. Please read How to create a Minimal, Complete, and Verifiable example
 
@Borodin you are right of course. I could remove 99% of the code above and reduce it to that one line of code causing my problem. The obvious problem is that I do not know which that would be. So, while I do not know which line to keep I did add the didReceiveData to the code above. And why do I think that it "it loads it entirely into the memory bevor writing it to the disk", well because the memory consumption increases by several GBs before out of a sudden a file at the designated target location with the designated name appears having exactly the same size ;)
@SteffenUllrich $verbosity = 0 is the standard setting which already triggers all print statements in this method. It seems is it not getting called before the download has finished. (the output appears when the file is done)
@SteffenUllrich believe me, the output won't help. It just states that the download was successful and tries to print a progress bar
 
@gapvision: You're rejecting all suggested solutions and requests for more information on the basis that you know better. You won't get help with your problem that way. Is your program really so big that you can't just post all of it? I am sure you would get an answer quickly that way.
@gapvision: Is this code really your own? Have your really downloaded a file using it?
 
@Borodin the code ist originally from github.com/arrizer/PutIO-Perl-folder-sync, and I ensure you I downloaded files using it.
@Borodin so you ask me to post all my code while you are blaming me for posting to much code ?! ;)
 
There were too alternatives. The ideal would have been for you to reduce what you posted to a minimal program that still displayed the problems that you needed help with. Failing that, either because you refused or were unable to make that reduction, you could have posted the entire thing in the hope that someone would do it for you. You did neither except for begrudgingly posting the callback subroutine. There has never been a problem either with Perl or with the LWP library. Your program has probably never worked properly. Where do you want to go from there?
If you still have $agent->add_handler(request_prepare => \&prepareRequest) and $agent->add_handler(response_header => \&didReceiveResponse) from the original putiosync.pl then that makes a big difference.
 
4:58 PM
yes I have, I made no changes to the script
 
 
3 hours later…
7:42 PM
I added some output and further informations
if you still don't believe me I can make a video or add timestamps if you want...
 

  last day (16 days later) »