last day (15 days later) » 

6:45 PM
14
Q: Sudden delay while recording audio over long time periods inside the JVM

Fabian B.I'm implementing an application which records and analyzes audio in real time (or at least as close to real time as possible), using the JDK Version 8 Update 201. While performing a test which simulates typical use cases of the application, I noticed that after several hours of recording audio co...

 
Just a thought, but maybe the garbage collector is kicking in and pausing your application?
 
@DanW Good thought, however this doesn't quite explain why this sudden delay persists until i close and reopen the line. I think that if the garbage collector has something to do with this, the delay should be gone as soon as the garbage collector has finished its job, because without any timing the lines internal buffer will get drained as quickly as possible.
 
I would try to check whats going on with a profiler. Maybe there's some memory leak somewhere there.. It would also be good you to upload one sample project to github or gitlab (even with sample sound frames to perform a test) and add the link into your question.
 
If it is the garbage collector kicking in, you might improve things by reusing the ByteArray and (if possible) AudioData objects instead of creating new ones each time through. (For example, you could create them outside the while (!isInterrupted) loop, and pass them in as extra parameters to the read() function.) Generally speaking, the fewer objects you create, the less work the GC has to do!
Also: I don't know what AudioData class this is, but does it need to be closed or released or something after use? If so, make sure you're doing so, else it could leak non-heap resources.
 
@XtremeBiker I'm going to create a test project which contains only the parts that are needed for capturing audio (including the suggestion that @gidds made). Once that's done, I will test if the issue is resolved. If the delay still starts to occur, I will push it to Github and add a link to the repo. @gidds. Thanks for your suggestions. I will try to incorporate them in a small test project. For clarification: AudioData is a class created by me which stores the audio samples and the audio format and supports converting the byte array to an integer array as well as to a float array.
 
6:45 PM
I'm going to create a test project which contains only the parts that are needed for capturing audio (including the suggestion that @gidds made). Once that's done, I will test if the issue is resolved. If the delay still starts to occur, I will push it to Github and add a link to the repo. That's exactly the way to go ;-) Also please provide your JDK version.
 
Okay, so I've tried to implement your suggestions as best as I could. If I didn't miss anything, there should be no more array creation inside a loop. However, the issue is still not resolved. If you want to have a look and check if the issue is also appearing on your machine, you can find my very basic example application under this Github repo.
 
Maybe using some sort of parallel GC could do the trick - I don't know though, whether this is also helping with JVM version > 8 (but it should be easy to read up on parallel gc). That way, the pauses should be reduced significantly, depending on how many CPU cores you have.
 
@XtremeBiker Totally forgot to tell you my JDK version... I've updated my JDK and JRE to Version 8 Update 201 (which should be the latest Java 8 JDK as of now). The issue is still appearing though with this JDK.
@Frischling I'll try letting it run using parallel GC and tell you if this helped.
 
Note the -Xlog:gc for JDK > 9, that should align with your audio pauses.
 
@Frischling I tried executing it with parallel GC and enabled GC output. From time to time the garbage collector seems to be kicking in as I get some output which looks like this: 28686.646: [GC (Allocation Failure) 32116K->868K(206848K), 0.0003508 secs]. However, it doesn't seem like the GC is involved in this issue as I didn't get any GC output between 6 hours of runtime (no noticeable delay up until this point) and 7 hours of runtime (audio samples delayed by about one to two seconds).
Was anyone able to reproduce the issue using my example application?
 
6:45 PM
Have you tried calling flush every X minutes to see if that helps?
 
@DanW Not yet, but I'll try that out.
@DanW Unfortunately, flushing the TargetDataLines internal buffer every 10 minutes also did not resolve the issue. The more I investigate this problem, the more I think that it seems to be either some limitation or some bug in the API...
 
Use Profiler - Garbage collector hits the fan stopping app for a moment.
 
@Antoniossss I just finished letting it run with the Java VisualVM attached to it. However, I couldn't find anything suspicious in neither the garbage collector log file nor in the monitor tab of the Java VisualVM. You can find an annotated screenshot of my results here.
 
Good job on profiling. Looks like GC is not the problem. And you say that input flushing does not change a thing hmm. What is AudioInput? cannot find javadoc for that nor it is in inports
NVM about that AudioInput - different file.
You could check actual implementation of TargetDataLine that you aer using to see if there are no any clues in the source code.
 
Maybe it is some kind of integer overflow bug somewhere in the API? I've had a similar problem with jackd a while ago, where there is an internal sample counter that wasn't properly maintained and regularly went haywire after a certain amount of samples.
 
6:45 PM
@escitalopram As of now, that's what I think the cause could be. However, this should be confirmed or denied by a developer of the API.
@escitalopram Your comment got me thinking a bit more about a possible overflow. Assuming there is an internal (unsigned) 32-bit integer which counts the number of bytes read from the audio device, it would overflow after 2^32 byte / (2 byte * 2 *44100 Hertz) = 24348 seconds = ~6.76 hours using a 16-bit stereo audio format with a sample rate of 44100 Hertz. This is inside the interval in which I noticed the appearance of the delay. So an overflowing integer might explain the issue.
 
Now how would counter "reset" could cause delay?
Im not negating it, just wondering.
 
@Antoniossss I'm also not quite sure how an integer overflow could cause this delay, but the time for which I calculated that the overflow should take place seems to correspond to the time when the delay starts.
Small update: If I execute it in Linux, it still runs totally fine after 9 hours of continuous runtime. By now I'm sure that it has to do something with the Windows version of the JVM.
 
OK, just a passing thought, could it have something to do with the app being written in kotlin instead of java? Could the OP test the same code, but written and compiled in java instead. Also, if it is a JVM issue, could the OP test the code written and compiled on OpenJVM?
 
Have you tried to record audio with a non-Java application on Windows, to rule out the possibility that the audio driver is the problem?
 
@KartikSoneji Actually, the Kotlin version was a rewrite of the original Java version in which the issue was first seen. Because by then I thought that it must be some bug in my code, I decided to rewrite it from scratch in Kotlin since I had already started to switch from Java to Kotlin. So I think it is very unlikely that the issue is related to the use of Kotlin instead of plain old Java. But I'm going to give OpenJVM a try.
@escitalopram I have already tried this by recording audio using Audacity (but somehow forgot to mention it in the question). This worked totally fine without any noticeable delay for the whole 8 hours in which I tested it.
@KartikSoneji When using OpenJ9 the issue still appears on Windows... I'm going to test it using OpenJ9 on Linux tomorrow.
 
6:45 PM
Then I am out of ideas. Maybe Windows 10 is the problem? Can you test it on Windows 7 pc?
 
@KartikSoneji I could try it inside a Windows 7 VM. However, I think I might have already found an indication that using Windows 7 won't help, as I just found a bug report in the Java bug database which seems to be related to my problem.
 
OK. So, what are you planning on doing next? The bug does not seem to be patched. @FabianB.
 
@KartikSoneji I'm actually not quite sure how to continue with this... Someone has commented on the bug report that they're not able to reproduce the issue. Maybe that was a developer at Oracle? And if they can't reproduce it, it's hard if not impossible to fix it. Linking this question to the report may help them reproduce the issue and to find the underlying cause. If I could only find out how to comment on this bug report...
 
Have you considered using Java 9/10? If you are lucky, the bug will have been fixed. Or, you could also try Java 7, and the bug may not be present in there. @FabianB.
 
@KartikSoneji It seems like the bug is not fixed in the latest Java 12 version, as the issue is still occurring... I might have to think about some kind of workaround.
 
6:45 PM
What about Java 7? And for a workaround, every 3 hours or so, can you start a new recording in parallel to the current recording then end the current recording so that you don't have any gaps. @FabianB.
 
@KartikSoneji I have not tried it using Java 7 yet, but I will do that. Your suggested workaround is what I'll most likely end up implementing.
 

last day (15 days later) »