Decreasing AJAX web app loa...

When developing Doculicious, one of my main concerns was how fast the heavy JavaScript parts would load and run in the browser. Doculicious uses a fairly customised version of the Dojo Toolkit, and the raw files that were being loaded for some pages added up to over 550KB - made up of about 50 individual files, all being loaded by Dojo with its packaging system. This was fine for development and great for debugging, but I knew I had to do something before this went into production. In this post I'll talk about some of the things we use in Doculicious to make our JavaScript smaller and load faster, but still easy to develop on. Hopefully it should be general enough to help anyone who has a site using JavaScript, but we use Dojo so is a bit focused on its packaging system which includes multiple files at run-time. If your app doesn't use Dojo but has multiple JavaScript files, you'll probably still get some benefit out of this.

There are 2 main ways to speed up a JavaScript application - Make the JavaScript load faster into the browser, and make it run quicker. There's a great article at IBM developerWorks called AJAX performance analysis that goes into both of these. The first section talks about the tools to use in Web application development - Firebug and YSlow, which are both required tools on any development box I use. The next goes into reducing the network transfer time, and this is where I'll focus on what we did with doculicious.

Reducing File Size and HTTP requests

The Dojo Toolkit uses a packaging system similar to Java, and it's a great way to separate out all the files and keep them organised and easy to develop. We've created our own package for doculicious and one of our JavaScript heavy pages may require up to 50 individual files to be included. Because of current browser connection limits, this is way too much, and it's much better to limit the number of JavaScript files to one or two instead of loading many.

What I wanted to do for doculicious was to create individual files for each JS heavy page, with each file including just the JavaScript code it needed. This individual file could then be used with the dojo.js file so that all the required JavaScript for a page would be in 2 files. Because Dojo automatically includes JS files depending on which widgets and functions are used, I needed to define which ones were used on which page so that I could join them all together. Firebug is great for this, and using the Net tab quickly lets you see all the included JS files:

My initial goal was to create a range of "base" files that could be concatenated together in an Ant script to create a JavaScript file for any doculicious page. Using Firebug to find the common files across each of our pages, I made a list of all the Dojo specific files that were needed, and also of our custom doculicious files. At this point I realised that each of our pages used very similar Dojo functions and widgets, so I made the decision to manually join these together into one file that could be included across all the files I needed. This is probably not the best thing to do if you are using the latest version of Dojo or another packaging system, as when changes are made it will be harder to integrate them, but our base Dojo is version 0.4.3 and I do not plan on upgrading it. We've only found a couple of bugs over the past 18 months, so I'm comfortable that this base is stable.

Concatenating your JavaScript into a single file is great for production code, but you still need to develop on it. So when doing this we need to keep our individual files available for development, while having a simple process to make the production code. I use Eclipse with the MyEclipse plugin as my IDE. The JavaScript project is structured like below, with the js folder being included in the web project so that it is deployed along with it:

JavaScript Project
   - buildscripts
      - build
         - functions
         - widgets
      - lib
      - build.xml
   - js
      - digitalcarpenter
         - system
            - Command.js
            - DocumentState.js
            - State.js
            - ...
         - widgets
            - Canvas.js
            - Elements.js
            - ...
      - dojo
         - dojo_functions_widgets_minimal.js
         - dojo.js

1. At the beginning we define the paths that are used and the files we are going to create.
2. We then build the directory structure for the build
3. Then in copy we move all the files we are going to use into the build/functions and build/widgets. Taking care to not move stuff that isn't needed, such as testing files, images and css.
4. Next we define a regular expression replace to remove all lines with "dojo.require" on them. This line is the way Dojo knows which files to include, and it's not needed in our final file because all the code is together. In fact things break if these are left in. Depending on which JS library you use, you may not need to do something similar when cancatenating your files together.
5. The next area is where we actually create our specific file. First we copy the base file to our working directory, then we concat all the individual files together, appending them to this file. Order may be important. Some of the widgets in our code include other widgets, so they need to be added before-hand.
6. The last step is calling ./lib/custom_rhino.jar. This is the Dojo ShrinkSafe application which removes comments and shrinks down the code. I highly recommend that you use this. ShrinkSafe has worked great for all of my JS, and I've never had errors. Because it parses the JavaScript it does a great job. It will even tell me if I've left some horrible bug in the code.
7. At the end is the create and clean methods you use to run the whole thing.

<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="DigitalCarpenter_JavaScript" default="create"
	basedir=".">
	<description>
		Creates the javascript files needed for Doculicious.
	</description>
	<!-- set global properties for this build -->
	<property name="build" location="build" />
	<property name="widgets" location="${build}/widgets" />
	<property name="functions" location="${build}/functions" />
	<property name="lib" location="lib" />
	<property name="dojo_src" location="${basedir}/../js/dojo" />
	<property name="dc_src" location="${basedir}/../js/digitalcarpenter" />
	
	<property name="dojo_base" value="dojo_base" />
	<property name="dojo_minimal" value="dojo_functions_widgets_minimal" />
	<property name="base_workshop_dojo_file" value="workshop_dojo" />
	<property name="document_workshop_js" value="document_workshop" />

	<target name="init">
		<!-- Create the build directory structure used to create the JS files -->
		<mkdir dir="${build}" />
		<mkdir dir="${widgets}" />
		<mkdir dir="${functions}" />
	</target>
	
	<target name="copy" depends="init">
		<copy file="${dojo_src}/${base_workshop_dojo_file}.js" todir="${build}" />
		<copy file="${dojo_src}/${dojo_minimal}.js" todir="${build}" />
		<copy todir="${functions}">
			<fileset dir="${dc_src}/system" />
		</copy>
		<copy todir="${widgets}" includeEmptyDirs="false">
			<fileset dir="${dc_src}/widget">
				<exclude name="_package_.js" />
				<exclude name="ButtonField.js" />
				<exclude name="InlineEditor.js" />
				<exclude name="RichEditor.js" />
				<exclude name="TestWidget.js" />
				<exclude name="Popup.js" />
				<exclude name="templates/*" />
				<exclude name="templates/buttons/*" />
				<exclude name="templates/images/*" />
			</fileset>
		</copy>
	</target>
	
	<target name="remove_dojorequires" depends="copy">
		<replaceregexp match='dojo.require' replace="//dojo.require" byline="true">
			<fileset dir="${widgets}"/>
		</replaceregexp>
		
		<replaceregexp match='dojo.require' replace="//dojo.require" byline="true">
			<fileset dir="${functions}"/>
		</replaceregexp>
	</target>

	<target name="document_workshop" depends="remove_dojorequires">
		<copy file="${build}/${base_workshop_dojo_file}.js" 
                               tofile="${build}/${document_workshop_js}.js" />
		<!-- NOTE: The order of the below files is important -
		            later code expects the previous code to be available -->
		<concat destfile="${build}/${document_workshop_js}.js" append="true">
	    	<filelist id="dw_functions" dir="${functions}">
			    <file name="Utils.js"/>
			    <file name="SHA1.js"/>
			    <file name="Constants.js"/>
			    <file name="Command.js"/>
			    <file name="DocState.js"/>
			</filelist>
			<filelist id="dw_widgets" dir="${widgets}">
				<!-- file name="Resizer.js"/ -->
			    <file name="ImagePicker.js"/>
			    <file name="FloatingImagePicker.js"/>
			    <file name="TextField.js"/>
			    <file name="Element.js"/>
			    <file name="Canvas.js"/>
			    <file name="ColorPalette.js"/>
			    <file name="FontSizePicker.js"/>
			    <file name="FontPicker.js"/>
			    <file name="RichTextToolbar.js"/>
			    <file name="Dialog.js"/>
			</filelist>
	  	</concat>
	  	
	  	<antcall target="-rhino-compress">
			<param name="srcFile" value="${build}/${document_workshop_js}" />
		</antcall>
		<!-- Copy the completed file to the js/dojo directory - 
                           will overwrite the one already there -->
		<copy overwrite="true" 
                               file="${build}/${document_workshop_js}.js" 
                               tofile="${dojo_src}/${document_workshop_js}.js" />
	</target>
	
	<target name="-rhino-compress" unless="nostrip">
		<copy overwrite="true" file="${srcFile}.js" tofile="${srcFile}.uncompressed.js" />
		<java jar="./lib/custom_rhino.jar" 
                              failonerror="true" 
                              fork="true" 
                              logerror="true" 
                              output="${srcFile}.js">
			<arg value="-strict" />
			<arg value="-opt"/>
			<arg value="-1" />
			<arg value="-c" />
			<arg value="${srcFile}.uncompressed.js" />
		</java>
	</target>
	
	<target name="create" depends="clean, init, copy, remove_dojorequires, document_workshop" 
                          description="creates the javascript files">
		<echo message="All done! :)" />
	</target>

	<target name="clean" description="clean up">
		<!-- Clean up the files used -->
		<delete dir="${build}" />
		<delete dir="${widgets}" />
		<delete dir="${functions}" />
	</target>
</project>

Cool, we've now got our Ant file ready to create the individual files. The example above creates only one file. Our production one creates a couple of different files. Because this allows you to pick and choose individual files you can customise the generated JS very specifically. Doculicious has two main "applications". One is the front end web form and the other is the template/style creator. They use similar functions and widgets, but doing it this way I can save about 30kb on the uncompressed files by only including the stuff each page needs.

Now when I'm developing on my code I can just comment out the single file call at the top of the HTML page and uncomment the dojo.require lines to use the individual files, allowing us to program on nicely formatted files that have comments and line breaks. When everything's tested and ready for prod we just re-comment the dojo.requires, uncomment the individual line, run ant create and increment the jsVersion variable so that clients get the new version.

<!-- Comment out the following line to test the individual js files
<script type="text/javascript" src="/js/dojo/document_workshop.js?v${jsVersion }"></script>		 
-->
<script type="text/javascript">
	<!-- Uncomment these to test the individual js files-->
	dojo.require("digitalcarpenter.widget.Element");
	dojo.require("digitalcarpenter.widget.Canvas");
	dojo.require("digitalcarpenter.system.DocState");

Now our production code is all nicely together in one file. We still have commented and formatted code to develop on, and running it all through ShrinkSafe has also lowered the file size. Our uncompressed file has gone from 370KB to 262KB, but most importantly we've gone from about 50 individual file calls to 2. The file size still seems large, but with mod_deflate on Apache2 it comes down to 68KB and with aggressive caching, clients should only need to download it when we change it.

Tags: AJAX, Web development, web 2.0, JavaScript,