JMeter vs NeoLoad

Recently I was asked by a client to introduce performance testing into their development process.
They have 1 week iterations where they integrate the different components of their system and they wanted to performance test the end-result every other week.
Starting out small and simple I downloaded JMeter from apache, fired up their recording proxy, recorded a scenario, changed all the relevant form fields into regex variables and off I went.
It worked great, we had some interesting data with some nice graphs to show performance was okish.
But the happiness didn’t last long..

Things quickly went down-hill from there.

• We had a couple of peaks in the performance data; with no hope of understanding why those happened.
• We had a need for a slightly different graph; But no way to generate it, except to copy and paste all the raw data into excel and generate the graph from there. The latest version of course, because earlier versions of excel can not handle all the data.
• We wanted to scale up the load to the point where 1 machine was not able to generate the load anymore; We divided up the data sets, started 3 separate JMeters at roughly the same time and integrated the XML files later. This took between half a day and a day to accomplish
• We needed information on pageviews, not HTTP requests; There seems to be no way in JMeter to group a couple of HTTP request in one pageview.
• We could not get the ramp up period to work in JMeter. Our servers were getting hammered in the first couple of minutes until everything settled down and we got a normal amount of traffic going
• It was very hard to compare the data from one run to the data on a previous run. It was very hard to tell whether the performance had gotten better or worse, or if there was something else going on.

But the major issue did not surface until a couple of weeks later. When someone changed something in the flow of one of the scenarios we tested. We had a failing test with hardly a chance of figuring out what to do about it. After hours of trying to find the problem we gave up and rerecorded the entire thing, redid the regular expressions etc etc.
It was around this time I decided to look for another solution and was recommended NeoLoad by a fellow Xebian. It is a commercial product with what we would call reasonable prices. While definitively more expensive than free, it does solve all the problems we were having with JMeter. You can add what they call monitors that monitor your database statistics, webserver load etc. This gives you the tool to find the root of ‘strange’ behaviour in the problem areas. You can distribute the load over multiple machines just by installing an agent on another machine and checking a checkbox in the controller. It allows you to view data on HTTP request level and pageview level. And best of all, comparing a report with an earlier report is done in 30 seconds. And bestest of all is that you can compare current responses with the responses you got when you recorded them. This allows you to see if someone added/changed/removed form fields for example. If that happens you find the error quickly and do not have to rerecord and prepare the entire scenario.
So the bottom line is it took us almost 3 extra mandays to prepare JMeter and post-process the reports into graphs. With NeoLoad that time is down to about 30 minutes. We have better looking reports with more information than we did before, which we can easily compare with previous runs.
And we can now distribute the load over more machines than we did before.
Now I am not saying JMeter is useless, far from it. It allowed me to quickly get started and get me results really quickly. It supports a lot more different type of requests than NeoLoad does. It can do JMS messages, database calls etc. I have successfully used JMeter to performance tune applications. But for big, serious performance tests, that are part of acceptance testing for example, I really think you should consider NeoLoad.
As a disclaimer, I or Xebia is in no way affiliated with Neotys in any shape or form, except that we got a trial license when I asked for one. Something you can and should do as well.