On the 5th of June, we (Lammert Westerhoff and Freek Wielstra) presented the State of Web Development 2014 at our yearly conference, XebiCon on the SS Rotterdam. It was a great success and we had some great feedback. We'd like to share the same presentation we gave there with everyone who couldn't make it to the conference in this blog post. We started the presentation by showing an overwhelming tag cloud:
When you’re new to web development or when you haven’t done any web development for a long time, you’re likely to get lost in this jungle of web technologies. Don’t worry if you are overwhelmed with this, because that’s intentional. This tag cloud contains many languages and web frameworks that modern web applications consist of. But it also contains a number of development tools that are used during development. The goal of this post is to guide you through this jungle by explaining to you which technologies you can use to build a modern web application and which tools you need during the full development lifecycle.The get a better understanding of which technologies and tools we should use, we should first look at bit at the evolution of a number of important technologies. Then we will understand why web development is quite complex these days and why we need tooling. Once that's clear, we can have a look at the available tooling.
The evolution of web applications
Of course, these static websites were hard to maintain. A change of content needed a change in the HTML file which then needed to be uploaded to the web server with FTP.
Luckily, there were a couple of technologies that solved this problem for us. Using a technology like PHP, JSP or ASP we could dynamically generate the websites at runtime, which were then sent to the browser. These websites usually were connected to a database. The database then stored the content instead of storing it directly in the HTML files. That way we could make changes to the website by changing the content stored in the database. This allowed us to already build more complex web applications that also stored user created content, like blogs or even entire web stores.
While we had the server side technologies that we needed to generate our websites, we needed new innovations on the client side as well. Especially because we entered the era of mobile devices, we needed new browser capabilities, like location services, browser storage and touch events. Therefore, new HTML and CSS standards were being developed that contain many features that make the web browsers much more powerful and richer, which allows us developers to build richer web applications for both mobile and desktop.
To make this a lot easier for everyone, Twitter released Bootstrap, which handles all the responsiveness automatically. It’s still being used by many responsive websites today.
With all the new powerful features of CSS3, the style sheets also became more complex and tedious. To overcome that problem, technologies like Sass and Less were developed. These are CSS preprocessors that offer new language features that make some of the more tedious styling easier to do. Both will produce CSS and both solve the same problems. Which one you should choose depends on your circumstances and personal flavour.
AJAX requests are also at the core of single web page applications (SPA).
The single page web applications are becoming more and more popular since they usually provide a better user experience. They rely heavily on the AJAX requests that we saw earlier to retrieve data from a server. The data that is retrieved by these request are one of the only types of communication between the client and server once the page is initially loaded.
Since with SPA’s, we move a lot of logic and complexity from the server to the client, it really comes in handy to use the framework that handles most of the complexity. Three of the most popular frameworks are Backbone.js, Ember.js and AngularJS. Which one you should choose depends a lot on your requirements, what you’re used to and personal preference. Though it is clear that AngularJS is the most popular one at this moment, has the most activity and the largest community. People seem to be moving from Backbone to Angular and Ember. We also use Angular ourselves and we’re happy with it. The Google Angular team is working on the next major version which should even be a big improvement. That’s why we think Angular is a good and safe choice.
From what we've seen in the previous paragraphs, it’s obvious that more and more functionality and complexity is moved from the server side to the client side. And by using frameworks and libraries such as Bootstrap, jQuery and Angular, we also increase our client side dependencies. This also means that our development focus and efforts are moving from the server to the client. It’s time to take front end development serious and to do it right. So let’s have a look at what this means for our development process.
Web Application Development
Of course we want to use many of the great third party libraries that we discussed earlier. And since those libraries often depend on other libraries, we’ll end up with a lot of external libraries.
Now we’re at a stage that we could deploy these files to our web server, but still that’s not really the way we like to develop our apps. So we’re going to do a couple of extra steps. We will minify almost all of our files. This will grab all those files, do some smart tricks, and we’ll end up with much smaller files. This will safe the end user some download time, which is nice.
We also want to make sure that all of the files we made don’t have any syntax mistakes. That’s why we’ll run a syntax validator on all of those files.
With all these steps in place we’re pretty confident that we have everything we need to make sure that we write quality code. But there is one more thing we need to worry about during development. And that’s dependency management. We want automatic downloads of the third party libraries that we use and their dependencies. And we want to specify their version as well, or specify a version range and get automatic updates within our range.
Now we’re completely happy with all the steps in our development process. But how do we do all this? Or better said, how do we automate all this? We need a build tool.
Let’s quickly recap the things we want our build tool to do. It needs to compile and minify our style sheets and scripts. After that it will take these files and together with the HTML build a distribution package that we can deploy to our production server.
And with every change in any of our files, we want to make sure that everything still works as we would expect. So we validate our files and run our tests. These are our quality assurance steps.
Like we just mentioned, we also want all of our dependencies to be managed automatically.
But there are even more things that we can do with modern build tools that we couldn’t even think of but are great for development. How about live watch and reload of our browser during development. We launch our app in a browser, change some logic or styling, and the web browser will refresh the page automatically to show our changes.
The web development community has been very innovative and came up with many more cool thing we can do with these tools. And you can even write your own build actions.
At this point we have a full understanding of what the web development process looks like. And by seeing this, we can say that front end development has become a first class citizen of software development. Now that we recognise the complexity of front end development and that we need tooling to manage this complexity, it time to have a deeper look at some of the most popular tools of this moment.
Most of the web development tools that are built these days are part of the Node.js ecosystem.
The Node Package Manager makes it easy to install any registered Node.js tool or application with just a single command. It keeps an online registry of all available packages, and anyone can publish their own. The Node.js and NPM ecosystem have developed into an active and healthy developer community.
One of the most popular tools written in Node.js is Grunt. It is the standard build tool for front-end web projects. It’s a task based build tool, which means you can define a set of tasks that need to run for a production build, development build or a test run. Sounds pretty much what we need for our development process.
Since it’s plugin based, it has plugins for pretty much any task you want to run. That includes plugins for our compilation, minification, validation and testing steps, as well as dependency management. That makes it perfect for us.
Once you start grunt in development mode, it will launch its built-in webserver and host your web application. And with the right plugins, we get our live reload. The Grunt plugin will watch for any changes in our resources, perform necessary tasks and refresh the open browser page for us, so we can see the results of our work right away - drastically shortening the feedback loop compared to traditional web development.
Since Grunt is the standard and most popular build tool for web projects at this moment, it’s widely adopted and has a large and mature developer ecosystem.
But Grunt also has it’s drawbacks. Let’s have a look at Grunt’s configuration file, the Gruntfile.js. This file contains configuration for our tasks and the plugins that perform them.
With a lot of tasks, the Gruntfile really becomes a lot of configuration. The example below taken from our own project and it’s only about half of our configuration. No need to read what it says exactly, it’s just to give you an idea about the amount.
But the amount of configuration is not the only downside of Grunt. The other problem is that it works with temporary files and directories. For each task (depending on the task, of course), it will read a couple of files, transform or convert them into something else and write the output back to disk. The the next task will then use these new files as their input. This creates a chain of temporary files and directories. It’s not a huge problem, especially but it certainly is not the fastest way of doing things.
That’s why Gulp.js, the streaming build system, was made. It’s a newer, alternative build tool that’s quickly gaining popularity. Gulp can do pretty much the same things as Grunt, so we don’t need to worry too much that we lose essential features of our development process. However, since Gulp is newer than Grunt, it doesn’t have as many plugins as Grunt yet. But since Gulp does the same as Grunt, but then better, it seems that many developers make the switch to Gulp and more and more plugins are made for Gulp as well.
So what’s the difference? Instead of requiring a lot of configuration for it’s tasks, Gulp instead uses “Code over configuration”. That means that you just tell the tool to do something in a certain way instead of configuring both what it should do and how it should do it. Much easier and much more direct. And resulting in far less code than Grunt’s configuration for doing the same.
It also solves Grunt’s problem of temporary files and directories since it’s stream based. Tasks will directly stream their output to the next task. This results in much faster build times.
So it seems that Gulp solves exactly the two problems we had with Grunt. So it’s to be expected that it will gain popularity and eventually pass Grunt. It’s just a matter of time until it reaches the same maturity that Grunt currently has.
We talked earlier about dependency management to manage our third party libraries. While Grunt and Gulp use the Node Package Manager directly to download it’s plugins and their dependencies, we still need to manage the dependencies for our app. To do that, we use Bower, a package manager for the web, by Twitter. Bower works similar as NPM in that you specify all your dependencies in a single configuration file. Bower will then download these libraries from the Internet. Since Grunt and Gulp have plugins for Bower, you don’t even need to worry about this, since it will make sure to download the files when you build your project or to download updates when needed - if your build system is properly configured.
So many tools and frameworks! Isn’t it too much work to setup and configure? Yes; setting up a full web app development stack, configuring each plugin for your project, is a lot of development work which can take up days. Luckily there is a tool to solve that problem: Yeoman.
Yeoman is a project scaffolding tool that generates a basis for your project, using templates called Generators. Some of those generators will set up a web application project with a preconfigured build strait, containing everything we need for compilation, minification, validation, testing, dependency management and much more. It will also ask us if we want to use frameworks such as Bootstrap and AngularJS. This makes it an ideal solution to setup a new project and automated development process.
More recent generators will set up Gulp as the build tool, providing an alternative to Bower. This should help developers that aren’t familiar with Gulp yet to get up to speed quickly.
Let’s recap what we talked about in this post in a summary.