How to convert CSV into XML and XML into CSV using Java

I've put up a new article on the Ricebridge site: How to convert CSV into XML and XML into CSV using Java. It does exactly what is says on the tin.

I explain how to get XML Manager and CSV Manager working together to solve a common data-processing problem. Not the most exiting sort of thing I know, but it's just the sort of annoying job you'd like to have solved without too much coding. Oh, and the example code is free for commercial use. And did I mention that you get a free CSV Manager license with XML Manager? You sure do. So what are you waiting for? Stop writing that dreadful DOM API code this minute! And don't even think about using StringTokenizer for CSV! I see you! Bad programmer, bad!




Posted in Java | Leave a comment

A Big Little Idea

Send your customers a hand-written thank you note. It's as simple as that. I'm a regular reader of Duct Tape Marketing, and I think John Jantsch is really on the ball. If you run a business, go check him out. Nearly every post has a great idea in it.

I got this idea from one of his recent posts: Hand-Written Notes Have Become A Power Tool. It's so obvious when you think about it — email is too easy. Handwriting is a lost art and takes far too much time. Hence, it shows respect. I've been doing this for all my customers so far this year and the response has been great — people love it.

You can't just fire off any old thing on some office paper though – take the time to prepare some nice heavy note paper for writing. And make sure that your letters are from you, not some hired hand (ha ha). It has to be from the top for this to count.

How much effort is it? Actually, it's enjoyable. Once you get into it, you look forward to making a connection with each new customer. You'll feel good, and they'll feel good. Email does nothing in comparison.

So, you want to put this plan into action? Go out now and order some headed notepaper (not the stuff you use for invoices!). Right now. That's the first thing you have to do to make it happen. Oh, and stamps! Ah, snailmail. It feels good.




Posted in Business | Leave a comment

Dojo: Progress!

NOTE: Please see Bob Leurck's example for some updates on this code.

The last time I posted about dojo we looked at creating a simple dojo widget. The final aim was to produce a web-oriented progress bar.

Well, now it's finished. Head over to the demo page for XML Manager (my latest product) to try it out. It's a web progress bar because it adapts to the amount of time a server request is expected to take. Each little box takes a little bit longer than the last one, so that it can cover short wait periods just as well as long wait periods.

This post will take a look at the JavaScript, HTML and CSS code to implement this progress bar as a dojo widget. In the last post the code I gave was actually for dojo 0.1. Since dojo 0.2 is the latest version, I've moved to that for the final implementation. There aren't really any big changes.

On thing I did learn about dojo &mdash don't use your own onload method. This really screws up dojo and prevents it from running properly. Instead you need to go native:

dojo.event.connect(window, "onload", yourFunction);

This actually makes your HTML cleaner and you get the benefits of dojo's memory-leak-avoiding event-system (that would be one of the big reasons for using dojo in the first place).

OK, let's look at some JavaScript first. Here's the JavaScript you use on the page where you want a widget to appear. This is just normal JavaScript code appearing in the script element or in an external JavaScript file referenced using the src attribute of the script element. Personally I prefer using exernal JavaScript files as it keeps everything nicely separated. So, here are the relevant bits from the demo.js external JavaScript file, as used by the XML Manager demo mentioned above.

demo.js

dojo.require("dojo.widget.*");
dojo.hostenv.setModulePrefix('ricebridge', 'ricebridge');
dojo.widget.manager.registerWidgetPackage('ricebridge.widget');
dojo.require("ricebridge.widget.RicebridgeProgress");


function startProgress() {
  var progress = dojo.widget.manager.getWidgetById("progress");
  if( progress ) {
    progress.startProgress();
  }
}

function stopProgress() {
  var progress = dojo.widget.manager.getWidgetById("progress");
  if( progress ) {
    progress.stopProgress();
  }
}

Argh! What's all that dojo gunk at the top? Well the dojo documentation about creating your own widgets only shows how you to create them in the dojo.widget package. For production use, you'll really want to put them in your own package. In this case, I've chosen the package ricebridge.widget. The first line at the top is a standard dojo incantation to include the dojo.widget package. The next two lines set up the ricebridge.widget package, so that dojo will look for it in a ricebridge folder in the same folder as the dojo.js file. For the moment, I'm just accepting that this little invocation works and getting on with it. The last line just tells dojo to include our new package. It's the same idea as the first line.

The two functions startProgress and stopProgress are the main control points for the widget. They use the dojo widget manager to find the progress bar wiget object, and start and stop it, by calling methods that we will define in the progress widget itself. In the XML Manager demo, these methods are called when a server request is sent, and then when it returns, to start and stop the progress bar display.

Here's how you use the widget in your HTML code:

demo.htm

<div dojoType="RicebridgeProgress" widgetId="progress"
  numboxes="30" width="300" height="20" multiplier="10"
  basecolor="#ccc" oldcolor="#666" hicolor="#00f">
</div>

That's nice and easy, isn't it? The dojoType attribute places this div under the control of the dojo widget system, which will turn it into our progress bar using an HTML template (coming up!). We've defined the name RicebridgeProgress to point to our progress bar widget. We also use the dojo-specific widgetId attribute to identify the widget object. This widget identifier is used by the getWidgetById method of the widget manager to find the widget. We used this method in the demo.js file above. You'll notice that there are a lot of custom attributes on this div as well. This is how dojo enables a widget to have a custom set of parameters. For our web progress bar, we use the following attributes to set these parameters:

Attribute What it does
numboxes Number of indicator boxes in the progress bar.
width width of the bar in pixels
height height of the bar in pixels
multiplier time in millis to increase the wait period of the next box by
basecolor color of inactive box
oldcolor color of box reached in previous operation
hicolor color of box reached in current operation

So how does dojo know what to show for a widget? Where is the widget HTML defined? dojo uses an HTML and CSS template system for this purpose. Let's look at those files now. In the case of the progress bar they're pretty simple as we build part of the widget HTML dynamically use the JavaScript DOM API.

The other question is where to put these files. They go in the ricebridge/widget folder, corresponding to the ricebridge.widget dojo package. This folder structure is placed in the same root folder as the dojo.js file as per the dojo package discovery convention.

Here's the widget HTML:

progress.htm

<div dojoAttachPoint="main" class="rbprogress">
<table border="0"><tr dojoAttachPoint="bar"></tr></table>
</div>

We're going to build the boxes using html td elements. I'm using a table because it makes it easy to get the boxes all the same size. You could use different elements, such as an HTML list. The td elements will be created dynamically, based on the numboxes parameter.

The other thing to notice are the dojoAttachPoint attributes. These will be used by the widget JavaScript object to access parts of the widget HTML. We'll see this in action in a minute.

Each widget can also have some CSS associated with it. Here's the CSS for the progress bar:

progress.css

div.rbprogress {
  border:  1px solid #666;
  margin:  0px;
  padding: 0px;
}
div.rbprogress table {
  width:    100%;
  height:   100%;
  border:   0px;
  margin:   0px;
  padding:  0px;
}
div.rbprogress td {
  border: 1px solid #666;
  margin: 2px;
}

This is fairly standard stuff. At the moment I am not sure how to customise this for each widget instance on an HTML page. It seems one would have to set the required styles dynamically using JavaScript. It is something I will be looking at in the future.

OK. That's all that out of the way. Let's get to the meat. Here is the JavaScript code that defines the widget itself. There's a good bit here, so we'll go through it one step at a time. This JavaScript file is placed in the same folder as the progress.htm and progress.css files.

At the top of the file, we inform the dojo package system of the objects we are “providing”. Notice that we provide two objects, HtmlRicebridgeProgress and RicebridgeProgress. Based on the dojo widget example this is just the way that you do it, so cut, paste and don't ask too many questions! After the “provides”, we “require” any additional dojo stuff that we need to implement the widget. Obviously we'll need the dojo.widget package to get at the dojo widget support (our widget inherits from the generic dojo widget object), and we need dojo.lang for the dojo.inherits method, used at the end of the file.

RicebridgeProgress.js

dojo.provide("ricebridge.widget.HtmlRicebridgeProgress");
dojo.provide("ricebridge.widget.RicebridgeProgress");

dojo.require("dojo.widget.*");
dojo.require("dojo.lang.*");

ricebridge.widget.HtmlRicebridgeProgress = function() {
  dojo.widget.HtmlWidget.call(this);

  this.templatePath    = dojo.uri.dojoUri("ricebridge/widget/progress.htm");
  this.templateCssPath = dojo.uri.dojoUri("ricebridge/widget/progress.css");
  this.widgetType      = "RicebridgeProgress";

  this.numboxes        = 10;
  this.multiplier      = 100;
  this.width           = 500;
  this.height          = 50;
  this.basecolor       = '#ccc';
  this.hicolor         = '#00f';
  this.oldcolor        = '#666';

  var running = false;
  var box     = 0;

  this.fillInTemplate = function() {
    this.main.style.width  = this.width+'px';
    this.main.style.height = this.height+'px';
    for( var bI = 0; bI < this.numboxes; bI++ ) {
      var td = document.createElement('td');
      td.style.backgroundColor = this.basecolor;
      this.bar.appendChild(td);
    }
  }

  this.startProgress = function() {
    running = true;
    this.markNext();
  }

  this.markNext = function() {
    if( running && box < this.numboxes ) {
      this.bar.childNodes[box].style.backgroundColor = this.hicolor;
      var _this = this;
      dojo.lang.setTimeout(function(){_this.markNext()},this.multiplier*(1+box));
      box++;
    }
  }

  this.stopProgress = function() {
    running = false;
    for( var bI = 0; bI < box; bI++ ) {
      this.bar.childNodes[bI].style.backgroundColor = this.oldcolor;
    }
    for( var bI = box; bI < this.numboxes; bI++ ) {
      this.bar.childNodes[bI].style.backgroundColor = this.basecolor;
    }
    box = 0;
  }
}
dojo.inherits(ricebridge.widget.HtmlRicebridgeProgress, dojo.widget.HtmlWidget);
dojo.widget.tags.addParseTreeHandler("dojo:RicebridgeProgress");

So let's start by defining an object for our widget. This object will be called ricebridge.widget.HtmlRicebridgeProgress. The Html prefix seems to be required by dojo - I have not tried to do this without it. The Ricebridge prefix is there because we need a way to avoid conflicts with the names of existing widgets when we use the dojoType attribute. Um, isn't that what the ricebridge.widget package is for? Yeah, I thought so too, but apparently not. Again, this seems the safest bet at the moment until I find a better way.

The first few lines of the widget definition are taken from the dojo widget article. We call the dojo HtmlWidget constructor, as the first step in inheriting from it (the second step occurs at the end of the file, using the dojo.inherits method). The next three lines set up some standard dojo widget properties, indicating the location of the HTML and CSS template files, and the widget type name, as used by the dojoType attribute.

These lines are pretty much boilerplate. Next we have the object-specific properties. Dojo checks these names against the attributes of the widget div in your HTML page and sets any that match. This is how the parameterisation discussed above works. It's a nice bit of dojo magic.

After these public fields, we define two private fields, running and box, to hold the current progress bar state. I'm using Douglas Crockford's suggestions for JavaScript field encapsulation.

Next comes the fillInTemplate method. This is a dojo interface method that should be in every widget object. It does the final work of building the widget DOM. In our case we create the correct number of td elements, corresponding to the number of boxes requested in the widget div numboxes parameter. Notice that dojo has converted the dojoAttachPoint references into widget object properties of the same name. Thus, this.main and this.bar refer to the correct elements in the HTML template. Another handy dojoism.

The startProgress and stopProgress methods do exactly what they say. These are the methods called by our main JavaScript file, as mentioned above. All they do is update the color of each table cell as time passes. We use a utility method, markNext, to handle the setTimeout callback. Notice that this uses the dojo.lang.setTimeout method, and not the normal JavaScript setTimeout method (doing things the "dojo way" keeps you out of trouble). Also, we use the _this variable to maintain a reference to the widget object in the anonymous function passed as an argument to the setTimeout method.

And that's pretty much it. You can check out the web progress bar in action over on the XML Manager demo page.

So should you use dojo? Technically it must be one of the most advanced JavaScript frameworks out there. It's a bit weak on the documentation side, but you can always read the source, which is nice and neat and pretty easy to follow. After this little evaluation exercise, I'm going to go with it. I have an opportunity at the moment to put dojo through it's paces in a client project and I'm going to work it hard. I'll let you guys know how it goes.

One more thing, Alex Russell, the dojo maintainer, was nice enough to comment on my previous dojo post. That indicates that the project leaders are pretty committed and that the dojo developers actually care about their users. Not a bad sign at all!




Posted in Web | Leave a comment

Friday Fun: Math Nerdism

WARNING: danger of mathematically-induced mind-body separation.

The Number Spiral. No. Seriously. You keep reading because you want to know “why! why?”




Posted in Fun | Leave a comment

Tragic Typing

Since starting this blog, I've realised that my abysmal typing is holding me back. I've come to think that you need to type well to write well. I just have to hit the delete key way to much. I'm fast, but I'm not accurate.

Remington Typewriter I never really learned to type properly. As a programmer you never have to — too much auto-completion. And programmers have to type so many squigqly characters anyway it's hardly worth learning to type real text. I did try a few typing courses from books over the years. In fact, I first tried to learn typing on this thing. My grandfather gave it to me. I wrote my first C.V. on it!

The other reason that I really need to get my typing into shape is that when you start a company you have to do a lot more writing than coding. If you want to market your stuff, you've got to write about it.

I actually realised that the typing thing was a big gap in my skillset a couple of years ago. I was working for this great boss (Hi Ralph!) who really knew his stuff. One day I went into his office and he was completely killing the keyboard. Smoke was rising from his fingers. I was a bit blown anyway. “Wow,” I said. So then he says, “You mean, you can't touch-type?” Ouch. I knew I had to do something about it.

Of course, being the great procrastinator that I am, I'm still “doing something about it” several years later. Well, over the break I decided to really do something about it. Really Really. And since we live in the era of good old Web 2.0, I thought to myself, “Hey, there must be some cool JavaScript-ified online typing courses out there.” Oh Yeah. There sure are! Just try Google. Tons of 'em.

So what's my new learn-typing strategy? To work through all the free online typing courses. Something has to wear off. Right? I'm starting with Good Typing which has a clean, simple, but slightly dated interface. Adsense supported of course, but isn't everything these days.

So far I've got to lesson 5 and it's starting to get hard. No pain no gain I guess…




Posted in General | Leave a comment

Friday Fun: Song Tapper

Got some stupid melody stuck in your head?

Well go find out what the song is!




Posted in Fun | Leave a comment

Document Delight

I've been following this blog called Creating Passionate Users for a while. It's by the people who write the “Head First” books. Now I've never actually read any of their books, but I did use the amazon preview to check them out. Wow. Wish I'd had textbooks like that in school. The books are amazing at presenting information. It's hard to explain, but the format and style of the presentation just plugs right into your brain.

Anyway, I started following the blog and they spend a lot of time describing the psychology behind their text production. Well I was really taken with it and wanted to try it out. I used some of the ideas for a tutorial/article about CSV and IP address location. Of course, my attempts are pretty woeful in comparison, but here's the interesting thing – it was a lot more fun to write than most of my documentation. I've posted before about the real difficulties of producing decent API documentation. And this “Head First” stuff looks like a good way to tackle the problem.

Today, they have just posted a great little article summarising the main points of their approach. If you read nothing else in the blogosphere today, read this, it's worth it.




Posted in General | Leave a comment

Dojo: Learn By Doing

Do you dojo? Dojo is the JavaScript toolkit that I'm going to use from now on. Why? Well I did a quick eval of most of the toolkits out there, and it made the best impression. I wasn't looking for database-oriented features. I was looking more for something to help me put together DHTML interfaces.

The thing is, when you enter a new programming space you are faced with a real challenge to your time. You can't properly evaluate each option, because that means learning each system completely. And for programmers, since most learning is done by doing, that would just take too much time. If you can find some neutral reviews of the options in your chosen area, great. But what you find most often are opposing position statements – heartfelt defences of chosen frameworks and thinly disguised contempt for the other options.

My approach in this case is to check out each site, check the level of activity, the state of the docs, the overall approach, the background of the main committers, all that stuff. But not exhaustively. You want to trust your intuition. After all, the right framework for you might not be the “best” framework, the one to rule them all. So, Blink.

I opened my eyes, and I saw dojo. Whojo? It's the JavaScript stuff behind JotSpot, apparently. Technically it's a merger of several extant frameworks into a sort of JavaScript Gestalt. In other words, a great big grab-bag of JavaScript goodies! And they have a pretty website!

The real biggie for any JavaScript toolkit is HTTP request handling. Hiding all that dodgy XMLHttpRequest crap. Such a humane interface (NOT). And dojo sure does it nicely. I decided to use dojo for the online demo of my new product: XML Manager Demo. This demo submits an XML document for server-side parsing, and includes a set of XPath expressions defining what to parse out of the document. If you use dojo together with the JSON data format, the whole thing is a cinch. Here's the request code:

dojo.io.bind({
  url: 'demo-handler.htm',
  method: 'post',
  postContent: jsonquerystr,
  handle: function(type, data, evt){ 
    if( type == 'load' ){
       jso = JSON.parse(data);
       if( jso.data ) {
         setTableData( jso.data );
       }
       else if( jso.xml ) {
         setXmlData( jso.xml );
       }
       setMessage( jso.stats+
                   (jso.msg?jso.msg:'') );
    } 
    else {
      setMessage( "Sorry, "+
        "there's been a server error." );
    }
  },
  mimetype: 'text/plain'
});

Pretty nifty huh? If you want to see the whole thing, here's the code: <a href="http://www.ricebridge.com/products/xmlman/demo.js">demo.js</a> (Yeah, I know, that code needs some cleanup – hey, it's on the to-do list…).

The only little problem is that there is no user feedback when the demo is on the server-side doing the XML parsing. The user just has to wait. Not so good.

Ages ago I had a similar problem with the back office admin site of a client. Now I was using a hacked together pre-AJAX solution – an invisible iframe with a hard-coded form. But it worked pretty like the modern AJAX model. Anyway, I had the same problem: how to provide feedback to the user when a request has been submitted behind the scenes.

What I wanted was a progress bar. It would start with a set of grey boxes and change them blue one by one until the request was done. Of course, we have the little issue of not knowing how long the server is going to take. So the idea I came up with was to make it an exponential progress bar. Each little box takes longer than the last one, rather than all boxes dividing the time equally. So you never actually get to the final box. Instead, when the server returns, you mark the boxes that have turned blue a darker gray. What you are left with is an indication of how long the request will take the next time (based on the principla that, as a general rule of thumb, tomorrow's weather has a 70% probability of being the same as today's). This turns out to work rather well, and users pick it up right away without explanation. The whole thing is completely visual.

So this what I want for my demo, a web progress bar, not a desktop progress bar. So at first I was just going to cut-and-paste (ah the joys of being a professional!) the old code over to my site. But then I thought: hold on, what if I write a dojo widget to do this? Coolaboola says I, we'll do this properly!

Well, it turns out that dojo is, um, challenged, in the documentation domain. But they do have starter article on widget writing. It does tell you pretty much all you need to know to write a dojo widget. And that's the problem. I don't want to write a dojo widget, I want to write my own widget. Hey, I might submit it to the project later, but for the moment, as a good engineer, I want to keep my custom stuff from polluting the dojo framework. Hmm.

So here then is my first utterly simple and completely pointless widget, to be morphed into a progress bar live on this blog! All it does is display the word “Foo” with a border. But it is a proper dojo widget, and it is as separate as possible from the main dojo code. I say, “as possible”, because complete separation turns out to be, well, difficult. I am just hacking about here, so bear with me.

Dojo widgets have their own HTML and CSS files. I placed the files for my Foo widget in the same folder as the test HTML page. Here they are:

foo.html

<div class="foo"><b>Foo</b></div>

foo.css

div.foo {
  border: 1px solid black;
}

Whoopy-doo, says you. Indeed. But hey, it's a start.

So the main page, the page with the user interface, looks like this:

usefoo.htm

<html>
<head>

<script type="text/javascript">
	var djConfig = {isDebug: true};
</script>
<script src="dojo.js"></script>
<script>
dojo.require("dojo.widget.*");
dojo.require("dojo.widget.Foo");
</script>

</head>
<body>

<div dojoType="foo">
</div>

</body>
</html>

Notice the isDebug declaration. That's real handy. Use it.

The dojo require statements are a dojoism that tells the dojo system about the widget support code that we need. It's standard dojo stuff and covered in their docs. The div with the dojoType attribute is where the magic happens. This makes our Foo widget appear.

Now for the widget code itself. This pretty much a direct copy from their example.

src/widget/Foo.js

dojo.provide("dojo.widget.HtmlFoo");
dojo.provide("dojo.widget.Foo");

dojo.require("dojo.widget.*");

dojo.widget.HtmlFoo = function(){
    dojo.widget.HtmlWidget.call(this);

    this.templatePath = dojo.uri.dojoUri("foo.html");
    this.templateCssPath = dojo.uri.dojoUri("foo.css");
  
    this.widgetType = "Foo";
}
dj_inherits(dojo.widget.HtmlFoo, dojo.widget.HtmlWidget);
dojo.widget.tags.addParseTreeHandler("dojo:Foo");

This is just the object-oriented JavaScript that defines the Foo widget. I still have a pretty weak grasp of all this, so let's just leave it as an artefact and move on. Suffice to say that you need the Html prefix on your widget name, otherwise the voodoo won't work.

The tricky bit is where you put this file. You have to put it in a folder structure where dojo can find it. It needs to be under src/widget, relative to your main HTML user interface file. Why? Dunno, just does.

And amazingly, the whole thing works. Next time I'll be blogging about my attempts to get this toy code to actually do something. I still want that web progress bar. Hopefully this blog entry will save you the half hour of trial-and-error I spent on this.

One final word, to the dojo guys. Love your stuff, but please just stop coding and write some docs. Pretty please? Cherry etc?




Posted in Web | Leave a comment

Friday Fun: Infinite Cats

Yeah. So it had to happen. Cats.

And you though this blog was cat free – hah!




Posted in Fun | Leave a comment

Duh! Advertise Your Own Products Dude!

So my blog has this ad banner down the side, which may or may not annoy you, but hey, what's a blog without adwords these days? Anyway, it suddenly struck me that I was failing to advertise my own products on some prize real-estate. Like, Duh!

So, forthwith, there shall ads for my products! Big shiny ones. Really big shiny ones. Right at the top.

I am such a dumbass…




Posted in General | Leave a comment