Archive for the ‘Software Development’ Category

Virtual Scrolling with dGrid – dojo, dGrid, node.js and mongodb

Wednesday, May 9th, 2012

Recently Sitepen has released a beta of light-weight feature-rich data grid component (compatible with dojo 1.7) called dgrid. I am working on upgrading an application that uses dojo 1.4 to dojo 1.7 and was looking for replacing the custom grid solution that I wrote with a better grid systems. My requirements were to replace the pagination with virtual scrolling and server side sorting – dgrid fits the bill. Though I couldn’t find any example of such an implementation and as such I thought of writing an example my self and share with others in this blog post.

The example shows the SnP 500 company list (data taken from wikipedia) in dgrid setup for virtual scrolling and server side sorting.

Technologies I have used are as follows:

  1. Frontend – Dojo 1.7/HTML/JavaScript (dgrid, put-selector and xstlye AMD based modular components)
  2. Services – node.js (Express – for exposing/routing REST based services and Mongoskin – as mongodb driver)
  3. Database – mongodb

The main objective of this post is to highlight how dgrid can be used for virtual scrolling. I have used node.js and mongodb as I find these appropriate for quick prototyping and as such I will not cover details of these technologies in this post. Once the mechanism of REST calls is clear any technology stack can be used to provide REST services.

Project Setup

  1. Create project directory (PROJECT_ROOT)- In this example project directory is ‘dgrid_virtual_scroll’
  2. Create subdirectory ‘PROJECT_ROOT/public/resources/js/lib’ under project directory
  3. Download and extract following under ‘PROJECT_ROOT/public/resources/js/lib':
    1. dojo toolkit 1.7 – http://dojotoolkit.org/download/
    2. dgrid – http://dojofoundation.org/packages/dgrid/#demos
    3. xstlye – https://github.com/kriszyp/xstyle
    4. put-selector – https://github.com/kriszyp/put-selector
  4. Make sure node.js and npm are installed – Installation can be found at http://nodejs.org/#download
  5. Install express and mongoskin packages using npm
    1. navigate to project directory (PROJECT_ROOT)
    2. Run following commands:
      npm install express
      npm install mongoskin
  6. Create blank index.html file under PROJECT_ROOT/public directory
  7. Create blank app.js file under PROJECT_ROOT directory

The directory structure should look like the figure below:

Database setup

  1. Install mongodb – http://www.mongodb.org/downloads
  2. Insert data:
    1. download test data file – companies.txt
    2. Start mongodb server by running command:
    3. import test data file by running following command:
      mongoimport -d snp -c companies companies.txt

      The command will import the data in a collection called companies in a database called snp in mongodb.

Create Node.js service application
Open file PROJECT_ROOT/app.js and copy following code:

var http = require('http'),
	express = require('express'),
    mongo = require('mongoskin'),
    server = new mongo.Server('localhost', 27017, {auto_reconnect: true}),
    db = new mongo.Db('snp', server);

 var app = module.exports = express.createServer();
app.configure(function() {
  app.set('views', __dirname + '/views');
  app.use(express.logger({ format: 'x1b[1m:methodx1b[0m x1b[33m:urlx1b[0m :response-time ms' }))
  app.use(express.static(__dirname + '/public'));

app.get('/rest',  function(req, res) {
	var urlpart = req.url;
	var sortObj ={};
			var sortpart=urlpart.split('(')[1].slice(0, -1);
			var sortfield=sortpart.substring(1);
			var sortsign=sortpart.substring(0,1);
			var sortdirection=1;
			if (sortsign==='-') sortdirection=-1;
			sortObj = JSON.parse('{"'+sortfield+'":'+sortdirection+'}');
	var start=0;
	var end=10000;	
	var range=req.header('Range');
	if(range!==null && range !==undefined){
		var m=range.split('=')[1].split('-');
	var pagesize=end-start+1;
	db.open(function(dberr, db) {
			db.collection('companies').count(function(err, data){
				db.collection('companies').find().sort(sortObj).skip(start).limit(pagesize).toArray(function(err, items){
				}) ;

			}) ;
app.get('/',  function(req, res) {
if (!module.parent) {
  console.log('Express server listening on port %d, environment: %s', app.address().port, app.settings.env)

Line 55 in above code is the default routing to the index.html page (we’ll come to it shortly) and Line 18 is routing of REST service call from index.html which sends the JSON back to the index.html. The documentation for REST API for JsonRest can be found at http://dojotoolkit.org/reference-guide/1.7/quickstart/rest.html. Thanks are due to Ken Franqueiro for pointing me to the API doc – which I found very useful.

Create index.html Application frontend
Open file PROJECT_ROOT/public/index.html and copy the following code:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
		<title>Test JsonRest store</title>
		<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
		<style type="text/css">
			@import "/resources/js/lib/dojo/resources/dojo.css";
			@import "/resources/js/lib/dgrid/css/skins/claro.css";
			@import "/resources/js/lib/dijit/themes/claro/claro.css";		
			#container { width: 960px; margin: auto; }
			#grid{ height:500px; }
		<script type="text/javascript" src="/resources/js/lib/dojo/dojo.js" data-dojo-config="async: true"></script>
	<body class="claro">
		<div id="container"></div>
	<script type="text/javascript">
		require(["dojo/dom", "dgrid/OnDemandGrid", "dgrid/Selection","put-selector/put", 
			"dgrid/Keyboard", "dojo/_base/declare", "dojo/store/JsonRest", 
			"dojo/store/Observable", "dojo/domReady!" ], 
			function(dom, Grid, Selection, put, Keyboard, declare, JsonRest, Observable) {
				function initView(){
					var companyStore = Observable(JsonRest({
						target : "/rest",
						idProperty : "_id"
					var columns = [ {
						label : 'S&P Ticker',
						field : 'ticker',
						sortable : true
					}, {
						label : 'Company Name',
						field : 'company',
						sortable : true
					}, {
						label : 'Industry',
						field : 'industry',
						sortable : true
					}, {
						label : 'Head Office',
						field : 'headoffice',
						sortable : true
					var grid = new (declare([ Grid, Selection, Keyboard ]))({
						store : companyStore,
						getBeforePut : false,
						columns : columns,
						minRowsPerPage : 25,
						loadingMessage: 'Loading data...',
						noDataMessage: 'No data found'
					}, "grid");

Run application

  1. Make sure mongodb is up and running
  2. Startup node application by running following command:
    node app.js
    npm install mongoskin
  3. Open http://localhost:3000/ you should see the view similar to following:
  4. Try scrolling down and sorting by clicking on column headers and notice the Mongodb and Node.js console windows how the calls are being made and the results on the browser window

If you have firebug installed you can open firebug and can see various header variables and request parameters that are used during rest calls:

In nutshell The Virtual scrolling and pagination request and response parameter looks like following:

  1. REST request – Range of data is sent in Request Header under Range variable with value like ‘items=24-68′ and sort field name and direction is sent as the part of URL like http://localhost:3000/rest?sort(+ticker) under the braces +/- sign signifies the sort direction and the text is the column name that needs to be sorted
  2. REST response – The data range for the requested range is sent under header variable ‘Content-Range’ the value follows he patter ‘fromIndex-toIndex/totalNumberOfrecords’ in the screenshot you can see 24-68/500 where 24 is start index, 68 is end index and 500 is the total number of records.
  3. I have tried this with over million records it works like a charm.

popen() – execute shell command from C/C++

Wednesday, March 23rd, 2011

The popen() function executes a command and pipes the executes command to the calling program, returning a pointer to the stream which can be used by calling program to read/write to the pipe.
Below are the C/C++ snippets to run a simple command, read the stream from pipe and then write to console.

C Implementation

#include <stdio.h>

int main(void) {
	FILE *in;
	extern FILE *popen();
	char buff[512];

	if(!(in = popen("ls -sail", "r"))){

	while(fgets(buff, sizeof(buff), in)!=NULL){
		printf("%s", buff);


C++ Implementation

#include <iostream>
#include <stdio.h>

using namespace std;

int main() {
	FILE *in;
	char buff[512];

	if(!(in = popen("ls -sail", "r"))){
		return 1;

	while(fgets(buff, sizeof(buff), in)!=NULL){
		cout << buff;

	return 0;

popen() is included in SUS version 2. More details can be found at http://pubs.opengroup.org/onlinepubs/007908799/xsh/popen.html

Show Chinese characters on JSP pages using property files in web application

Friday, February 4th, 2011

Below is procedure to show Chinese characters on JSP pages using property files in web application.

  1. Create new property file for Chinese language of extension _zh.properties like ABC_zh.properties.
  2. Open the newly created property file in UTF-8 enabled editor like notepad or Edit+.
  3. Put the relevant key values in Chinese language. Like. customer.number=客户号码
  4. After setting all key values in property file, save file as encoding as UTF-8.
  5. Use tool native2ascii from JDK bin directory to convert Chinese characters to Unicode using following command by ensuring the encoding to UTF-8.
  6. Convert native to ASCII using following command:
    $JAVA_JOMEbin>native2ascii.exe -encoding UTF-8 ABC_zh.properties ABC_zh_1.properties
  7. Copy newly created ABC_zh_1.properties to original location of property files and renamed to ABC_zh.properties
  8. Add the following line in starting the weblogic server like


  9. Add following java options:
    set JAVA_OPTIONS=%JAVA_OPTIONS% %JAVA_PROPERTIES% -Dwlw.iterativeDev=%iterativeDevFlag% -Dwlw.testConsole=%testConsoleFlag% - Dwlw.logErrorsToConsole=%logErrorsToConsoleFlag% -Dfile.encoding=UTF-8
  10. Add the content type as UTF-8 in displaying JSP pages by adding following line at start of JSP file.
  11. Thats It !!!

Fix large Eclipse tabs and menu bar in Ubuntu

Sunday, June 6th, 2010

Run following command in terminal to edit .gtkrc-2.0 file in your home directory:

$ gedit ~/.gtkrc-2.0

Copy paste below in .gtkrc file and save :

style "gtkcompact" {
class "GtkWidget" style "gtkcompact"

Restart Eclipse – tab size and menu bar height should now be fine.

I have tried this in Ubuntu 9.10 and 10.04 – should work for other distros as well.

For installing Weblogic 10.3 on Snow Leopard

Saturday, June 5th, 2010

Hack to validate JVM while installing Weblogic 10.3 on Snow Leopard:

cd /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/sudo mkdir jre
cd jresudo ln -s ../lib libsudo ln -s ../bin bin

Install MySQL gem on 64 bit Snow Leopard

Wednesday, March 3rd, 2010

Install 64 bit MySQL then run following commands in terminal:

export ARCHFLAGS="-arch i386 -arch x86_64"</pre>
gem install --no-rdoc --no-ri mysql -- --with-mysql-dir=/usr/local --with-mysql-config=/usr/local/mysql/bin/mysql_config


Snow Leopard – Eclipse CDT – Launch Failed. Binary Not Found

Thursday, November 5th, 2009

Go to project Properties, C/C++ Build, Settings then click on Miscellaneous under Mac OS X C++ Linker and put “-arch i386″ in the Linker Flag Box and then Miscellaneous under gcc c++ compiler and put the same: “-arch i386″ in the Other Flags box.

I can now Compile and run my programs in Eclipse again :).

  • © 2004-2015 Special Work & Technology Limited