SWAT Blog

Posts Tagged ‘virtual scrolling’

Virtual Scrolling with dGrid – dojo, dGrid, node.js and mongodb

Wednesday, May 9th, 2012

Recently Sitepen has released a beta of light-weight feature-rich data grid component (compatible with dojo 1.7) called dgrid. I am working on upgrading an application that uses dojo 1.4 to dojo 1.7 and was looking for replacing the custom grid solution that I wrote with a better grid systems. My requirements were to replace the pagination with virtual scrolling and server side sorting – dgrid fits the bill. Though I couldn’t find any example of such an implementation and as such I thought of writing an example my self and share with others in this blog post.

The example shows the SnP 500 company list (data taken from wikipedia) in dgrid setup for virtual scrolling and server side sorting.

Technologies I have used are as follows:

  1. Frontend – Dojo 1.7/HTML/JavaScript (dgrid, put-selector and xstlye AMD based modular components)
  2. Services – node.js (Express – for exposing/routing REST based services and Mongoskin – as mongodb driver)
  3. Database – mongodb

The main objective of this post is to highlight how dgrid can be used for virtual scrolling. I have used node.js and mongodb as I find these appropriate for quick prototyping and as such I will not cover details of these technologies in this post. Once the mechanism of REST calls is clear any technology stack can be used to provide REST services.

Project Setup

  1. Create project directory (PROJECT_ROOT)- In this example project directory is ‘dgrid_virtual_scroll’
  2. Create subdirectory ‘PROJECT_ROOT/public/resources/js/lib’ under project directory
  3. Download and extract following under ‘PROJECT_ROOT/public/resources/js/lib':
    1. dojo toolkit 1.7 – http://dojotoolkit.org/download/
    2. dgrid – http://dojofoundation.org/packages/dgrid/#demos
    3. xstlye – https://github.com/kriszyp/xstyle
    4. put-selector – https://github.com/kriszyp/put-selector
  4. Make sure node.js and npm are installed – Installation can be found at http://nodejs.org/#download
  5. Install express and mongoskin packages using npm
    1. navigate to project directory (PROJECT_ROOT)
    2. Run following commands:
      npm install express
      npm install mongoskin
      
  6. Create blank index.html file under PROJECT_ROOT/public directory
  7. Create blank app.js file under PROJECT_ROOT directory

The directory structure should look like the figure below:

Database setup

  1. Install mongodb – http://www.mongodb.org/downloads
  2. Insert data:
    1. download test data file – companies.txt
    2. Start mongodb server by running command:
      mongod
      
    3. import test data file by running following command:
      mongoimport -d snp -c companies companies.txt
      

      The command will import the data in a collection called companies in a database called snp in mongodb.

Create Node.js service application
Open file PROJECT_ROOT/app.js and copy following code:

var http = require('http'),
	express = require('express'),
    mongo = require('mongoskin'),
    server = new mongo.Server('localhost', 27017, {auto_reconnect: true}),
    db = new mongo.Db('snp', server);


 var app = module.exports = express.createServer();
app.configure(function() {
  app.set('views', __dirname + '/views');
  app.use(express.favicon());
  app.use(express.bodyParser());
  app.use(express.cookieParser());
  app.use(express.logger({ format: 'x1b[1m:methodx1b[0m x1b[33m:urlx1b[0m :response-time ms' }))
  app.use(express.methodOverride());
  app.use(express.static(__dirname + '/public'));
});
 

app.get('/rest',  function(req, res) {
	var urlpart = req.url;
	var sortObj ={};
	if(urlpart){
		if(urlpart.indexOf('(')>-1){
			var sortpart=urlpart.split('(')[1].slice(0, -1);
			var sortfield=sortpart.substring(1);
			var sortsign=sortpart.substring(0,1);
			var sortdirection=1;
			if (sortsign==='-') sortdirection=-1;
			sortObj = JSON.parse('{"'+sortfield+'":'+sortdirection+'}');
		}
	}
	console.log(sortObj)
	var start=0;
	var end=10000;	
	var range=req.header('Range');
	if(range!==null && range !==undefined){
		var m=range.split('=')[1].split('-');
	 	start=parseInt(m[0]);
		end=parseInt(m[1]);
	}
	var pagesize=end-start+1;
	db.open(function(dberr, db) {
		if(!dberr){
			db.collection('companies').count(function(err, data){
				res.header('Content-Range',start+'-'+end+'/'+data);
				db.collection('companies').find().sort(sortObj).skip(start).limit(pagesize).toArray(function(err, items){
					res.send(JSON.stringify(items));
					db.close();
				}) ;

			}) ;
		}	
	});
	
	
});
app.get('/',  function(req, res) {
		res.redirect('/index.html');
});
  
if (!module.parent) {
  app.listen(3000);
  console.log('Express server listening on port %d, environment: %s', app.address().port, app.settings.env)
}

Line 55 in above code is the default routing to the index.html page (we’ll come to it shortly) and Line 18 is routing of REST service call from index.html which sends the JSON back to the index.html. The documentation for REST API for JsonRest can be found at http://dojotoolkit.org/reference-guide/1.7/quickstart/rest.html. Thanks are due to Ken Franqueiro for pointing me to the API doc – which I found very useful.

Create index.html Application frontend
Open file PROJECT_ROOT/public/index.html and copy the following code:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
	<head>
		<title>Test JsonRest store</title>
		<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
		<style type="text/css">
			@import "/resources/js/lib/dojo/resources/dojo.css";
			@import "/resources/js/lib/dgrid/css/skins/claro.css";
			@import "/resources/js/lib/dijit/themes/claro/claro.css";		
			#container { width: 960px; margin: auto; }
			#grid{ height:500px; }
		</style>
		<script type="text/javascript" src="/resources/js/lib/dojo/dojo.js" data-dojo-config="async: true"></script>
	</head>
	<body class="claro">
		<div id="container"></div>
	</body>
	<script type="text/javascript">
		require(["dojo/dom", "dgrid/OnDemandGrid", "dgrid/Selection","put-selector/put", 
			"dgrid/Keyboard", "dojo/_base/declare", "dojo/store/JsonRest", 
			"dojo/store/Observable", "dojo/domReady!" ], 
			function(dom, Grid, Selection, put, Keyboard, declare, JsonRest, Observable) {
				function initView(){
					put(dom.byId("container"),"div#grid");
					
					var companyStore = Observable(JsonRest({
						target : "/rest",
						idProperty : "_id"
					}));
					
					var columns = [ {
						label : 'S&P Ticker',
						field : 'ticker',
						sortable : true
					}, {
						label : 'Company Name',
						field : 'company',
						sortable : true
					}, {
						label : 'Industry',
						field : 'industry',
						sortable : true
					}, {
						label : 'Head Office',
						field : 'headoffice',
						sortable : true
					}];
			
					var grid = new (declare([ Grid, Selection, Keyboard ]))({
						store : companyStore,
						getBeforePut : false,
						columns : columns,
						minRowsPerPage : 25,
						loadingMessage: 'Loading data...',
						noDataMessage: 'No data found'
					}, "grid");
			}   		
			initView();
		});
	</script>
</html>

Run application

  1. Make sure mongodb is up and running
  2. Startup node application by running following command:
    cd PROJECT_ROOT
    node app.js
    npm install mongoskin
    
  3. Open http://localhost:3000/ you should see the view similar to following:
  4. Try scrolling down and sorting by clicking on column headers and notice the Mongodb and Node.js console windows how the calls are being made and the results on the browser window

If you have firebug installed you can open firebug and can see various header variables and request parameters that are used during rest calls:

In nutshell The Virtual scrolling and pagination request and response parameter looks like following:

  1. REST request – Range of data is sent in Request Header under Range variable with value like ‘items=24-68′ and sort field name and direction is sent as the part of URL like http://localhost:3000/rest?sort(+ticker) under the braces +/- sign signifies the sort direction and the text is the column name that needs to be sorted
  2. REST response – The data range for the requested range is sent under header variable ‘Content-Range’ the value follows he patter ‘fromIndex-toIndex/totalNumberOfrecords’ in the screenshot you can see 24-68/500 where 24 is start index, 68 is end index and 500 is the total number of records.
  3. I have tried this with over million records it works like a charm.
    DONE!!!

  • © 2004-2015 Special Work & Technology Limited