Blog

JMeter vs NeoLoad

29 Oct, 2008

Recently I was asked by a client to introduce performance testing into their development process.
They have 1 week iterations where they integrate the different components of their system and they wanted to performance test the end-result every other week.
Starting out small and simple I downloaded JMeter from apache, fired up their recording proxy, recorded a scenario, changed all the relevant form fields into regex variables and off I went.
It worked great, we had some interesting data with some nice graphs to show performance was okish.
But the happiness didn’t last long..

Things quickly went down-hill from there.

  • We had a couple of peaks in the performance data; with no hope of understanding why those happened.
  • We had a need for a slightly different graph; But no way to generate it, except to copy and paste all the raw data into excel and generate the graph from there. The latest version of course, because earlier versions of excel can not handle all the data.
  • We wanted to scale up the load to the point where 1 machine was not able to generate the load anymore; We divided up the data sets, started 3 separate JMeters at roughly the same time and integrated the XML files later. This took between half a day and a day to accomplish
  • We needed information on pageviews, not HTTP requests; There seems to be no way in JMeter to group a couple of HTTP request in one pageview.
  • We could not get the ramp up period to work in JMeter. Our servers were getting hammered in the first couple of minutes until everything settled down and we got a normal amount of traffic going
  • It was very hard to compare the data from one run to the data on a previous run. It was very hard to tell whether the performance had gotten better or worse, or if there was something else going on.

But the major issue did not surface until a couple of weeks later. When someone changed something in the flow of one of the scenarios we tested. We had a failing test with hardly a chance of figuring out what to do about it. After hours of trying to find the problem we gave up and rerecorded the entire thing, redid the regular expressions etc etc.
It was around this time I decided to look for another solution and was recommended NeoLoad by a fellow Xebian. It is a commercial product with what we would call reasonable prices. While definitively more expensive than free, it does solve all the problems we were having with JMeter. You can add what they call monitors that monitor your database statistics, webserver load etc. This gives you the tool to find the root of ‘strange’ behaviour in the problem areas. You can distribute the load over multiple machines just by installing an agent on another machine and checking a checkbox in the controller. It allows you to view data on HTTP request level *and* pageview level. And best of all, comparing a report with an earlier report is done in 30 seconds. And bestest of all is that you can compare current responses with the responses you got when you recorded them. This allows you to see if someone added/changed/removed form fields for example. If that happens you find the error quickly and do not have to rerecord and prepare the entire scenario.
So the bottom line is it took us almost 3 extra mandays to prepare JMeter and post-process the reports into graphs. With NeoLoad that time is down to about 30 minutes. We have better looking reports with more information than we did before, which we can easily compare with previous runs.
And we can now distribute the load over more machines than we did before.
Now I am not saying JMeter is useless, far from it. It allowed me to quickly get started and get me results really quickly. It supports a lot more different type of requests than NeoLoad does. It can do JMS messages, database calls etc. I have successfully used JMeter to performance tune applications. But for big, serious performance tests, that are part of acceptance testing for example, I really think you should consider NeoLoad.
As a disclaimer, I or Xebia is in no way affiliated with Neotys in any shape or form, except that we got a trial license when I asked for one. Something you can and should do as well.

guest
16 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
anjan bacchu
13 years ago

hi there,
good to know about this tool.
Have you heard about sun’s open source faban ? Here’s Scott Oaks(java security book fame) talking about a benchmarking scenario : http://weblogs.java.net/blog/sdo/archive/2007/05/how_to_test_con.html — very cool.
Good to know about NeoLoad — it sure is interesting to know that it can save you good amount of time for something like Acceptance testing.
BR,
~A

Paul Bakker
13 years ago

Looks really good, thanks for the post. It looks much easier to use than JMeter. I will definitely try it out.

Kai
Kai
13 years ago

Hi!
I’ve been using JMeter with success and can say it’s surely possible. If you want to group a bunch of HTTP requests (samplers) together, just use a transaction controller. This way you can get more meaningful results.
Cheers,
Kai

LV2k
LV2k
13 years ago

I would like to do Performance testing for 250 users. My application is built using applets, basically a gaming application. Can anyone pls suggest some open source tool that supports java applets. I need only in Open source

Oliver
Oliver
13 years ago

Hi
The report above is partly right. The issues you have had with JMeter are partly due to a lack of know-how around JMeter. JMeter’s graphing functions are abysmal. I’ve written my own graphing functions.
And a little hint, never use Excel to plot anything. It can only show 32000 resultsets. Use OpenOffice. That doesn’t seem to have a limit.
As for scalability and rampup, both work excellently. Don’t know what happened with your rampup.
I think if you want an Open Source performance testing tool JMeter is the way to go (there are other good ones but they are generally more complex to use). NeoLoad looks good for a commercial tool and is definitely cheaper than most. Personally I’d still think it’s money down the drain, that is spent better in manpower to learn to use your tool.
All commercisl performance testing tools have one drawback. You cannot discredit the tool from your time measurements. How do you know the tool is giving you the right response times? We have analysed JMeter on a code level to check that it does what we expect. then you can guarantee that it’s the program under test and not your tool. Don’t underestimate this fact!
Cheers
Oliver

Ravi Gill
Ravi Gill
13 years ago

Hi Erwin,
I have used neoload, load runner, siege etc.
And i have been using jmeter for about 2 years now.
There is absolutely nothing that you cant do with jmeter that you can with neoload.
some of the points:
* We had a couple of peaks in the performance data; with no hope of understanding why those happened.
– Did you monitor the targeted servers, and looked in the log files, i am sure you can sync the times and see what happend during those peaks.
* We had a need for a slightly different graph; But no way to generate it, except to copy and paste all the raw data into excel and generate the graph from there. The latest version of course, because earlier versions of excel can not handle all the data.
-An simple perl script (available on the net) and or a java class(also available on the net) can be used to parse those results and gnu-plot and or any other plotting tool can be used to generate graphs. once automated it makes life pretty eay :).
* We wanted to scale up the load to the point where 1 machine was not able to generate the load anymore; We divided up the data sets, started 3 separate JMeter instances at roughly the same time and integrated the XML files later. This took between half a day and a day to accomplish
Jmeter can be used for distributed testing, which means the master jmeter gui(even non-gui) can use extra servers to generate load and save the results from all the slave+master server on the master server itself. starting 3 different instances of jmeter and then combining the xml files is not advised.
* We needed information on pageviews, not HTTP requests; There seems to be no way in JMeter to group a couple of HTTP request in one pageview.
– you can easily accomplish this with “simple controller”, indi Http requests can be stacked as child under simple controllers.
* We could not get the ramp up period to work in JMeter. Our servers were getting hammered in the first couple of minutes until everything settled and we got a normal amount of traffic going
-this is odd behaviour, i hope all the givne variables were correct.
* It was very hard to compare the data from one run to the data on a previous run. It was very hard to tell whether the performance had gotten better or worse, or if there was something else going on.
-Are you sure that the servers were not shared with any other parties/apps that were generating load of their own and or the testdata was same for all the tests?, different variables can also interfere here like available network bandwidth etc, the jmeterclient should ideally be under the same subnet and close to the targeted environment.
For any questions i would like to forward you to your fellow colleague from the netherlandse called Levent, he seems to be an expert at jmeter aswell :).
with kind regards,
Ravi Gill,

Ravi gill
Ravi gill
13 years ago

Hi Erwin,
I Completely agree with you here that performance testing with jmeter can be really time consuming as the inbuilt jmeter reporting is not very reliable.
Neoload is a pretty impressive tool and definitely competes with the market lead loadrunner in both pricing and value.
Regarding Jmeter i remember that in the previous versions the distributed testing was not working and the simple controller was not available either, but both of these problems have been resolved in the latest release version 2.3.2, you should definitely check it out :).
Please do give my greetings to levent the next time you see him.
With kind regards,
Ravi.

Shiva Kumar
Shiva Kumar
12 years ago

Hi,
Please help me comparision between loadrunner,Jmeter,Webload

Gerind
Gerind
9 years ago

This is quite an interesting article, even though i put – “i have been there and done that” across JMeter and Neoload for a very high volume based testing using AWS, as part of our geographical load distribution.
Here are my 2 cents of info:
a. JMeter can be used for small scale load testing with limited features and if you can live with it – then its fine. If anybody wants to know the reliability or scalabilit of the applicaiton – then this isnt the best choice. what else can you expect of a ‘free-tool’!!!
b. i have tried to conduct a high volume testing using JMeter for more than 50,000 vus – but, couldnt scale more than 20,000 users – even on a 64-bit-Linux basd OS
c. usually perf testing / scalability / reliability / stress testing are conducted within a shorter time frame, to unearth performance issues / bottleneck; the team should be more judicious in conducting these tests rather than building utilities aroung JMeter and maintaining the same.
d. at the end of the perf testing – all eyes are on the issues and a good analytical report – to culmincate all the metrics of back end servers, infra, apps metrics. so need a tool which shudnt burn the budget and at the same time to be more flexible and stable to scale upto high volumes….
being this, when we tried with Neoload – it was quite ok adopt, scale up to 1 Million user session with 75,000 vus over AWS without any glitches and at the same time, we spent very few minutes in extracting data and put it into a good graphicall and easy-to-understand-for-non-techines-and-managment-team report, with a small spend.
Anecdote: About 1.5 yrs back, we had done an extensive experiment of conducting a load of 1 Million Virtual Users PER SECOND, just by using 4 OPEN SOURCE tools in our lab on a very sample applicaiton with tomcat and MySQL, using a EC2 environment – dont ask me about the consolidation of reports from all these open source tools….it took days….!!!!with manual co-relation.

kirti
kirti
8 years ago

Can I use JMeter for testing PHP based applications.

RUDRA GANESH SUBBARAYULU
RUDRA GANESH SUBBARAYULU
6 years ago

I really don’t get how the functionality of TCP Sampler,BSF Sampler in jmeter is used in Neoload. Neoload functionality is hard to define and understand.

Philippe Mouawad
4 years ago

This comparison is very old and should maybe be updated or removed as it is very unfair:
– All issues are potentially related to using JMeter in GUI mode (bad practice)
– to group a couple of HTTP request, use Transaction Controller
– JMeter 3.X has a nice Web Report with lot of graphs and tables
– Unmaintanable test due to missing assertions
– distributed testing has been largely optimized since this version
Regards
Philippe M.
Apache JMeter Team

Explore related posts