JEXL3 Performance degradation compared to 2.1.1

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

JEXL3 Performance degradation compared to 2.1.1

Philippe Mouawad
Hello,
I integrated today jexl3 to JMeter trunk:

   -
   https://github.com/apache/jmeter/commit/e1bfb7bb01169278d57f4fea7bf0cfaeba44ac50


I made some rapid benchmark using JMeter.
My Test plan contains only 1 thread Group with a Debug Sampler as a child:
Debug Sampler has Name field set to: DS-${__jexl3(3+2,RESULT)}

Using Java 8u45 on a Mac OSX 10.9.5 / 2.7 GHz Intel Core i7 / 16 Gb 1600
MHz DD3

1 Thread:
JEXL2 : Generate Summary Results =  20000 in 00:00:00 = 75471.7/s Avg:
0 Min:     0 Max:     1 Err:     0 (0.00%)
JEXL3 : Generate Summary Results =  20000 in 00:00:01 = 32102.7/s Avg:
0 Min:     0 Max:     1 Err:     0 (0.00%)

4 Threads:
JEXL2 : Generate Summary Results =  80000 in 00:00:00 = 285714.3/s Avg:
0 Min:     0 Max:     1 Err:     0 (0.00%)
JEXL3 : Generate Summary Results =  80000 in 00:00:01 = 103092.8/s Avg:
0 Min:     0 Max:     1 Err:     0 (0.00%)

As you can see the throughput decreased:
1 Thread : -57%
4 Threads : -68%


My benchmark is not as clean as a Java Micro Benchmark , but I lack of time
:-)

I thought you might be interested.

Regards
Philippe M.
@philmdot
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

henrib
Hello Philippe; Interesting results indeed. I've been trying to interpret / replicate them but JMeter is a bit too complex for me to understand what the benchmark is achieving and what is measured. Is it 20000 loops of an expression that is '3 + 2' ? And the same is performed with 4 threads in parallel using the same engine ? Any (easy) way to extract just the evaluated expressions / scripts with minimal dependencies (unit, jexl2/3)? Thanks for the info, Cheers Henrib
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

henrib
In reply to this post by Philippe Mouawad
Hi again;
It seems you've found a real performance regression.
I've been playing some more and it seems the logic around the operator overloading resolution/caching is flawed in the simplest case, i.e. when there is no overload.
As a workaround, you may want to try using an overloading JexlArithmetic as is:
{code}

    public static class JMeterArithmetic extends JexlArithmetic {
        public JMeterArithmetic(boolean astrict) {
            super(astrict);
        }

        /**
         * A workaround to create an operator overload
         * @param jma an improbable parameter class
         * @return 1
         */
        public int size(JMeterArithmetic jma) {
            return 1;
        }
    }
{code}

And use it at engine creation time:
{code}

        JexlEngine jexl = new JexlBuilder()
                    .cache(512)
                    .silent(true)
                    .strict(true)
                    .arithmetic(new JMeterArithmetic(true))
                    .create();
{code}

Let us know how it goes (if/when you find the time).

Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

Philippe Mouawad
Hi,
Thanks for the suggestion.

I tested it but performance results are the same.
Note I don't understand the fix as size method doesn't have this signature
in ancestor and I don't understand neither why it would be called in this
context, even after changing the signature to match the parent, Debugging
it, it is not called.

Regards

On Friday, January 22, 2016, henrib <[hidden email]> wrote:

> Hi again;
> It seems you've found a real performance regression.
> I've been playing some more and it seems the logic around the operator
> overloading resolution/caching is flawed in the simplest case, i.e. when
> there is no overload.
> As a workaround, you may want to try using an overloading JexlArithmetic as
> is:
> {code}
>
>     public static class JMeterArithmetic extends JexlArithmetic {
>         public JMeterArithmetic(boolean astrict) {
>             super(astrict);
>         }
>
>         /**
>          * A workaround to create an operator overload
>          * @param jma an improbable parameter class
>          * @return 1
>          */
>         public int size(JMeterArithmetic jma) {
>             return 1;
>         }
>     }
> {code}
>
> And use it at engine creation time:
> {code}
>
>         JexlEngine jexl = new JexlBuilder()
>                     .cache(512)
>                     .silent(true)
>                     .strict(true)
>                     .arithmetic(new JMeterArithmetic(true))
>                     .create();
> {code}
>
> Let us know how it goes (if/when you find the time).
>
>
>
>
>
> --
> View this message in context:
> http://apache-commons.680414.n4.nabble.com/JEXL3-Performance-degradation-compared-to-2-1-1-tp4682086p4682110.html
> Sent from the Commons - User mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] <javascript:;>
> For additional commands, e-mail: [hidden email]
> <javascript:;>
>
>

--
Cordialement.
Philippe Mouawad.
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

henrib
Hi;
Since performance is the same with the workaround, we've got another/different performance issue. Is there any easy way to replicate your test - or at least its essence - ? What does your benchmark do ? What are the evaluated expressions / scripts ?

Back too your note: the 'size' method is discovered through introspection as an overload of the 'size' operator; this creates an entry in a cache for that arithmetic class avoiding to re-discover the operator overloads (Uberspect) on each execution. So, no, this method is not called; it is just meant as a workaround of the bug.

Regards
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

Philippe Mouawad
Hi Henri,
I attached a simple Main class to :
https://issues.apache.org/jira/browse/JEXL-186
and the results.

Regards
Philippe

On Fri, Jan 22, 2016 at 8:19 PM, henrib <[hidden email]> wrote:

> Hi;
> Since performance is the same with the workaround, we've got
> another/different performance issue. Is there any easy way to replicate
> your
> test - or at least its essence - ? What does your benchmark do ? What are
> the evaluated expressions / scripts ?
>
> Back too your note: the 'size' method is discovered through introspection
> as
> an overload of the 'size' operator; this creates an entry in a cache for
> that arithmetic class avoiding to re-discover the operator overloads
> (Uberspect) on each execution. So, no, this method is not called; it is
> just
> meant as a workaround of the bug.
>
> Regards
>
>
>
> --
> View this message in context:
> http://apache-commons.680414.n4.nabble.com/JEXL3-Performance-degradation-compared-to-2-1-1-tp4682086p4682115.html
> Sent from the Commons - User mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Cordialement.
Philippe Mouawad.
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

henrib
Hi Philippe;
Default debug mode in JEXL3 seems like the other big slowdown cause.
Check JEXL-186 for an updated engine startup sequence :-).
Cheers,
Henrib
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

Philippe Mouawad
Hi Henri,
I just answered in JIRA :-)
It's now ok with JMeterArithmetic and debug(false)
Bug shouldn't debug be false by default ?

Second question, what is the impact of keeping JMeterArithmetic ? Or do you
plan a new release of commons-jexl3 ?

Anyway, thanks for your very fast support ! Great !

Regards


On Fri, Jan 22, 2016 at 10:14 PM, henrib <[hidden email]> wrote:

> Hi Philippe;
> Default debug mode in JEXL3 seems like the other big slowdown cause.
> Check JEXL-186 for an updated engine startup sequence :-).
> Cheers,
> Henrib
>
>
>
> --
> View this message in context:
> http://apache-commons.680414.n4.nabble.com/JEXL3-Performance-degradation-compared-to-2-1-1-tp4682086p4682123.html
> Sent from the Commons - User mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Cordialement.
Philippe Mouawad.
Reply | Threaded
Open this post in threaded view
|

Re: JEXL3 Performance degradation compared to 2.1.1

henrib
Hi Philippe;

I think it's ok to leave the default as debug(true) since it is easy (and often necessary) to create a 'custom'  JEXL instance be it for contexts, namespaces and/or arithmetic. It is convenient to provide JexlInfo instances when creating scripts to track potential errors; in that case, there is no (little) cost to debug(true).

In the general case, overloading the JexlArithmetic (ala JMeterArithmetic) is a good and efficient workaround to JEXL-186; you might even want to customize it further (overloaded versions of add(...) for instance).

Re-publishing a version/patch for that one bug fix seems overkill imho, especially with a decent/easy workaround. But if you are willing to go through the release hoops, please do. :-)

Cheers,
Henrib