On the 5th of June, we (Lammert Westerhoff and Freek Wielstra) presented the State of Web Development 2014 at our yearly conference, XebiCon on the SS Rotterdam. It was a great success and we had some great feedback. We’d like to share the same presentation we gave there with everyone who couldn’t make it to the conference in this blog post.
We started the presentation by showing an overwhelming tag cloud:
When you’re new to web development or when you haven’t done any web development for a long time, you’re likely to get lost in this jungle of web technologies. Don’t worry if you are overwhelmed with this, because that’s intentional. This tag cloud contains many languages and web frameworks that modern web applications consist of. But it also contains a number of development tools that are used during development. The goal of this post is to guide you through this jungle by explaining to you which technologies you can use to build a modern web application and which tools you need during the full development lifecycle.
The get a better understanding of which technologies and tools we should use, we should first look at bit at the evolution of a number of important technologies. Then we will understand why web development is quite complex these days and why we need tooling. Once that’s clear, we can have a look at the available tooling.
The evolution of web applications
A traditional website consists of HTML, CSS and sometimes a little javascript. The HTML tells the browser the content that needs to be displayed. The CSS tells it how that content should be displayed and what it needs to look like. And JavaScript was mainly used to add small behaviour, like displaying a popup or showing an image carousel.
Of course, these static websites were hard to maintain. A change of content needed a change in the HTML file which then needed to be uploaded to the web server with FTP.
Luckily, there were a couple of technologies that solved this problem for us. Using a technology like PHP, JSP or ASP we could dynamically generate the websites at runtime, which were then sent to the browser. These websites usually were connected to a database. The database then stored the content instead of storing it directly in the HTML files. That way we could make changes to the website by changing the content stored in the database. This allowed us to already build more complex web applications that also stored user created content, like blogs or even entire web stores.
While we had the server side technologies that we needed to generate our websites, we needed new innovations on the client side as well. Especially because we entered the era of mobile devices, we needed new browser capabilities, like location services, browser storage and touch events. Therefore, new HTML and CSS standards were being developed that contain many features that make the web browsers much more powerful and richer, which allows us developers to build richer web applications for both mobile and desktop.
To make this a lot easier for everyone, Twitter released Bootstrap, which handles all the responsiveness automatically. It’s still being used by many responsive websites today.
With all the new powerful features of CSS3, the style sheets also became more complex and tedious. To overcome that problem, technologies like Sass and Less were developed. These are CSS preprocessors that offer new language features that make some of the more tedious styling easier to do. Both will produce CSS and both solve the same problems. Which one you should choose depends on your circumstances and personal flavour.
Users don’t only want web sites to work and look good on every device, they also want web sites to be more interactive. They want the web site to respond to their actions. For example, zoom into an image or drag an item into a shopping basket. Even though all this was already possible in JavaScript for a long time, jQuery made such things a lot easier. jQuery is a JavaScript library that makes Dom manipulation easier, especially for multiple browsers. With the popularity of jQuery, the web became overwhelmed with jQuery plugins that did all sort of things. Below we see a screenshot of the plugin registry. And of course a library for mobile had to be released.
One of the very important features of jQuery is that it makes it easy to make AJAX requests. AJAX stands for Asynchronous Javascript And XML. It allows the web browser to retrieve data from the backend without leaving the page or doing a page refresh. And only the data is transferred from the backend to the client, instead of entire generated pages. Once the data is retrieved, it is displayed dynamically using the DOM manipulation features of jQuery. And even though XML is in the name, nowadays JSON is often used instead of XML, which is a more lightweight and more flexible data format.
AJAX requests are also at the core of single web page applications (SPA).
These web applications load all the HTML, CSS and Javascript resources at once. Once the page is loaded, you won’t have any other pages loading or refreshing anymore. All the rendering and navigation is done at the client side without leaving the actual page. Of course the user might perceive that he’s navigating between different pages, but this is all handled on the client by Javascript.
The single page web applications are becoming more and more popular since they usually provide a better user experience. They rely heavily on the AJAX requests that we saw earlier to retrieve data from a server. The data that is retrieved by these request are one of the only types of communication between the client and server once the page is initially loaded.
Since with SPA’s, we move a lot of logic and complexity from the server to the client, it really comes in handy to use the framework that handles most of the complexity. Three of the most popular frameworks are Backbone.js, Ember.js and AngularJS. Which one you should choose depends a lot on your requirements, what you’re used to and personal preference. Though it is clear that AngularJS is the most popular one at this moment, has the most activity and the largest community. People seem to be moving from Backbone to Angular and Ember. We also use Angular ourselves and we’re happy with it. The Google Angular team is working on the next major version which should even be a big improvement. That’s why we think Angular is a good and safe choice.
From what we’ve seen in the previous paragraphs, it’s obvious that more and more functionality and complexity is moved from the server side to the client side. And by using frameworks and libraries such as Bootstrap, jQuery and Angular, we also increase our client side dependencies. This also means that our development focus and efforts are moving from the server to the client. It’s time to take front end development serious and to do it right. So let’s have a look at what this means for our development process.
Web Application Development
As we saw earlier, we start simple by creating a couple of HTML, CSS and Javascript files. Then we would like to take advantage of the power of Less or Sass for our stylesheets. And since we’re making a rich or single page web application, we’ll be making a lot of javascript files. And perhaps some CoffeeScript or Dart instead of just Javascript. For those who don’t know coffee script or Dart, these are alternative languages that both compile to Javascript. CoffeeScript is almost the same language as javascript but just with a slightly different syntax. Dart however is quite different is definitely worth checking out when you have a bit of spare time.
Of course we want to use many of the great third party libraries that we discussed earlier. And since those libraries often depend on other libraries, we’ll end up with a lot of external libraries.
With this picture we kind of get an idea which resources we have for our web application. But to just create these files, download the libraries and upload them to our web server is not enough. First of all, web browsers don’t understand Less, Sass, CoffeeScript or Dart. We need to produce something that the browser does understand. For Less and Sass that’s CSS and Coffee script and Dart need to be compiled to normal Javascript.
Now we’re at a stage that we could deploy these files to our web server, but still that’s not really the way we like to develop our apps. So we’re going to do a couple of extra steps. We will minify almost all of our files. This will grab all those files, do some smart tricks, and we’ll end up with much smaller files. This will safe the end user some download time, which is nice.
We also want to make sure that all of the files we made don’t have any syntax mistakes. That’s why we’ll run a syntax validator on all of those files.
And of course we need to test. So we write a lot of unit tests to cover all our JavaScript functionality. But we also write view tests to make sure that our views display the correct information to the user. With good decoupling we can run our unit tests and view tests in isolation, which is good since we can easily spot problems when one of our test fails and it makes it easy to maintain and refactor our code. But we also want to make sure everything works well together, which is why we also write a couple of end to end tests.
With all these steps in place we’re pretty confident that we have everything we need to make sure that we write quality code. But there is one more thing we need to worry about during development. And that’s dependency management. We want automatic downloads of the third party libraries that we use and their dependencies. And we want to specify their version as well, or specify a version range and get automatic updates within our range.
Now we’re completely happy with all the steps in our development process. But how do we do all this? Or better said, how do we automate all this? We need a build tool.
Let’s quickly recap the things we want our build tool to do. It needs to compile and minify our style sheets and scripts. After that it will take these files and together with the HTML build a distribution package that we can deploy to our production server.
And with every change in any of our files, we want to make sure that everything still works as we would expect. So we validate our files and run our tests. These are our quality assurance steps.
Like we just mentioned, we also want all of our dependencies to be managed automatically.
But there are even more things that we can do with modern build tools that we couldn’t even think of but are great for development. How about live watch and reload of our browser during development. We launch our app in a browser, change some logic or styling, and the web browser will refresh the page automatically to show our changes.
The web development community has been very innovative and came up with many more cool thing we can do with these tools. And you can even write your own build actions.
At this point we have a full understanding of what the web development process looks like. And by seeing this, we can say that front end development has become a first class citizen of software development. Now that we recognise the complexity of front end development and that we need tooling to manage this complexity, it time to have a deeper look at some of the most popular tools of this moment.
Modern tools
Most of the web development tools that are built these days are part of the Node.js ecosystem.
Node.js is a JavaScript engine to run JS outside a browser environment. It has powerful system libraries for basics like file and network access, all with a minimal footprint These characteristics make NodeJS a perfect platform to build server side applications and terminal applications.
Since it’s JavaScript, a language that web developers already know, it’s the perfect environment to build web development tools with.
The Node Package Manager makes it easy to install any registered Node.js tool or application with just a single command. It keeps an online registry of all available packages, and anyone can publish their own. The Node.js and NPM ecosystem have developed into an active and healthy developer community.
One of the most popular tools written in Node.js is Grunt. It is the standard build tool for front-end web projects. It’s a task based build tool, which means you can define a set of tasks that need to run for a production build, development build or a test run. Sounds pretty much what we need for our development process.
Since it’s plugin based, it has plugins for pretty much any task you want to run. That includes plugins for our compilation, minification, validation and testing steps, as well as dependency management. That makes it perfect for us.
Once you start grunt in development mode, it will launch its built-in webserver and host your web application. And with the right plugins, we get our live reload. The Grunt plugin will watch for any changes in our resources, perform necessary tasks and refresh the open browser page for us, so we can see the results of our work right away – drastically shortening the feedback loop compared to traditional web development.
Since Grunt is the standard and most popular build tool for web projects at this moment, it’s widely adopted and has a large and mature developer ecosystem.
But Grunt also has it’s drawbacks. Let’s have a look at Grunt’s configuration file, the Gruntfile.js. This file contains configuration for our tasks and the plugins that perform them.
With a lot of tasks, the Gruntfile really becomes a lot of configuration. The example below taken from our own project and it’s only about half of our configuration. No need to read what it says exactly, it’s just to give you an idea about the amount.
But the amount of configuration is not the only downside of Grunt. The other problem is that it works with temporary files and directories. For each task (depending on the task, of course), it will read a couple of files, transform or convert them into something else and write the output back to disk. The the next task will then use these new files as their input. This creates a chain of temporary files and directories. It’s not a huge problem, especially but it certainly is not the fastest way of doing things.
That’s why Gulp.js, the streaming build system, was made. It’s a newer, alternative build tool that’s quickly gaining popularity. Gulp can do pretty much the same things as Grunt, so we don’t need to worry too much that we lose essential features of our development process. However, since Gulp is newer than Grunt, it doesn’t have as many plugins as Grunt yet. But since Gulp does the same as Grunt, but then better, it seems that many developers make the switch to Gulp and more and more plugins are made for Gulp as well.
So what’s the difference? Instead of requiring a lot of configuration for it’s tasks, Gulp instead uses “Code over configuration”. That means that you just tell the tool to do something in a certain way instead of configuring both what it should do and how it should do it. Much easier and much more direct. And resulting in far less code than Grunt’s configuration for doing the same.
It also solves Grunt’s problem of temporary files and directories since it’s stream based. Tasks will directly stream their output to the next task. This results in much faster build times.
So it seems that Gulp solves exactly the two problems we had with Grunt. So it’s to be expected that it will gain popularity and eventually pass Grunt. It’s just a matter of time until it reaches the same maturity that Grunt currently has.
We talked earlier about dependency management to manage our third party libraries. While Grunt and Gulp use the Node Package Manager directly to download it’s plugins and their dependencies, we still need to manage the dependencies for our app. To do that, we use Bower, a package manager for the web, by Twitter. Bower works similar as NPM in that you specify all your dependencies in a single configuration file. Bower will then download these libraries from the Internet. Since Grunt and Gulp have plugins for Bower, you don’t even need to worry about this, since it will make sure to download the files when you build your project or to download updates when needed – if your build system is properly configured.
So many tools and frameworks! Isn’t it too much work to setup and configure?
Yes; setting up a full web app development stack, configuring each plugin for your project, is a lot of development work which can take up days. Luckily there is a tool to solve that problem: Yeoman.
Yeoman is a project scaffolding tool that generates a basis for your project, using templates called Generators. Some of those generators will set up a web application project with a preconfigured build strait, containing everything we need for compilation, minification, validation, testing, dependency management and much more. It will also ask us if we want to use frameworks such as Bootstrap and AngularJS. This makes it an ideal solution to setup a new project and automated development process.
More recent generators will set up Gulp as the build tool, providing an alternative to Bower. This should help developers that aren’t familiar with Gulp yet to get up to speed quickly.
Let’s recap what we talked about in this post in a summary.
More
The full presentation slides will soon be available from the XebiCon website. Feel free to comment below with your feedback or reach out to us on Twitter.
Lammert Westerhoff @lwesterhoff
Freek Wielstra @frwielstra