last day (14 days later) » 

9:40 AM
0
Q: How to prove that the error is not my Java agent or how to debug it?

nadavyOne of our clients have contact me complaining that he's getting the following JRE crash while using our Java Agent. According to the error (below) the crash is on the native code since the problematic frame is categorized as 'C'. I've did some googling and it seems like there are some open bug...

 
No. I've tried to use the same java version and the same linux machine (they use an instance from Amazon so I've created a machine from the same image).
In addition, in one of the bugs they've mentioned that it reproduce 'sometimes'. Fun for me :)
 
Are you re(transforming) any JDK classes from your agent?
 
Yes. We are re-transforming the URL class. Do you think that's related? If so, how did you came to this conclusion?
 
Some classes are rather essential to the JVM and are treated differently. For example, if you change the behavior of java.lang.Object or java.lang.String too much, then you will typically observe very erratic behavior and crashes are not too uncommon. What are you changing in the URL class? My gutt feeling tells me that you break the class loading mechanism (the built-in class loaders prior to Java 9 are based on URL connections) by changing the way URL behaves. Typically, you suffer some form of circularity where your instrumentation triggers class loading.
Also, look at sun.invoke.util.ValueConversions where methods are bound to call sites at the identity call which can trigger class loading. This is another indicator. The reproduction probably depends on the class loading order which is why it is difficult to reproduce and only happens "sometimes". It literally depends on what buttons are pressed in the application in what order.
 
Basically we are add headers on every outgoing http request
*adding
So, if I'm getting you right, you are saying that 'sun.invoke.util.ValueConversions' triggers additional class loading which in a specific scenario triggers the bug. Am I correct?
If so, I'll try to see what happens when I turn it off.
In addition, I've seen a post saying that running java with -XX:CompileCommand=exclude,java/lang/invoke/LambdaForm$*.invoke* does the trick and I do see that in premain LambdaForm class and CallSite are loaded.
From your understanding, do you think it worth the chance?
 
9:49 AM
There is a good chance for it, yes.
Are you instrumenting URLConnection or URL?
 
Mind if I'll send you the code so you could have a look?
This will allow you to get the bigger picture.
 
If you run the JVM with this additional command, it means that the JVM always interprets LambdaForm invocations
Sure, rafael.wth@gmail.com
 
We are instrumenting java.net.URL
 
About my comment, it might fix the problem because LambdaForms are never compiled, therefore no faul, pre-instrumentation offset gets hardcoded
But it severly reduces performance as all LambdaForms (i.e. lambda expressions in Java) are no longer treated as first class citizens.
 
Just sent you the code
I think that I should try to disable the instrumentation of the java.net.url and if that will not work, I'll try to use the flag mentioned above.
 
9:56 AM
Ok, the ASM Advice adapter shuffels quite a lot around.
I would always avoid using such intrusive instrumentations as they can cause problems, even with HotSpot
 
May ask you how did you became so knowledgeable in the inner workings of the JVM? I really want to deep dive to the inner workings but I can't find good material on it.
 
Long journey, most information I take from the HotSpot source code
Some advertisement, if you write a Java agent, have a look at my library Byte Buddy: github.com/raphw/byte-buddy
It takes care of a lot of those problematic things ASM requires you to handle yourself
 
Cool :)
 
And it is less intrusive than the ASM AdviceAdapter
 
I will have a look.
 
9:59 AM
Have a look at the Advice component, you basically write a single Java class with a static method annotated with @Advice.OnMethodExit
 
About the ASM AdviceAdapter - can I use it for instrumentation user code and avoid it when instrumenting java code? or in both cases?
 
The biggest problem comes when you run other agents or debugger APIs
you rearrange the local variable indices for all methods
in their usage order
If another agent expexts a variable for a certain method to be at index X but now its at index Y, things might break.
 
And that's indeed the case as they have two other agents in parallel
 
yes, that is then probably your problem
 
I see
 
10:03 AM
try Byte Buddy for this:

class MyAdvice {
@Advice.OnMethodExit
static void exit(@Advice.Return(readOnly=false) UrlConnection c) {
c = new URLConnectionWrapper(c);
}
}
 
The issue is that we are also collecting Code Coverage so that's not the only case
And we plan on adding support in threads
 
I meant more as a proof of concept. Just add this to your main method and check if the problem goes away:

new AgentBuilder.Default()
.type(URL.class).transform( (t,b,cl) -> b.visit(Advice.to(MyAdvice.class).on(someMethod))
.installOn(inst)
 
excuse me - tracing a flow across threads.
 
Are you writing an APM?
 
I see - I could try
Some sort. No exactly it.
 
10:08 AM
Its using Byte Buddy agents for APM
Of course, just as a suggestion, ASM-based agents become tricky
Ok, your issue is resolved? If you ever need help with agents, I can also help you as a consultant if you have more thorough questions than SO. Cheers!
 
Thanks Rafael - I really appreciate your help and I'll try to check the options you've suggested.
 
No problem! Have a great day.
 
I'll talk with my peers and we might use you as a consultant.
Have a good one!
 

last day (14 days later) »