Anders Tornblad, web developer

I'm all about the web

Monthly archive for March 2013

JavaScript Csv file generator, part 3

The Csv file generator is almost done. All we need to do is to enclose and escape certain special characters. Values containing the separator need to be enclosed in double quotes. Values containing double quotes need to be escaped and, just to be sure, enclosed in double quotes.

Two more requirements and two more unit tests coming up...

Requirements

  1. Values containing the current separator must be enclosed in double quotes.
  2. Values containing double quotes must be escaped (by doubling the double quote characters), and also enclosed in double quotes.
engine.add("Csv.getFileContents should quote fields containing the separator", function(testContext, the) { // Arrange var properties = ["name", "age"]; var csv = new Csv(properties); csv.add({"name" : "Doe, John", "age" : "Unknown"}); csv.add({"name" : "Bunny", "age" : "2"}); // Act var file = csv.getFileContents(); // Assert the(file).shouldBeSameAs("\"Doe, John\",Unknown\r\nBunny,2"); }); engine.add("Csv.getFileContents should quote and escape fields containing double quotes", function(testContext, the) { // Arrange var properties = ["model", "size"]; var csv = new Csv(properties); csv.add({"model" : "Punysonic", "size" : "28\""}); csv.add({"model" : "Philip", "size" : "42\""}); // Act var file = csv.getFileContents(); // Assert the(file).shouldBeSameAs('Punysonic,"28"""\r\nPhilip,"42"""'); }); // Helper function to create one line of text function createTextLine(values, separator) { var result = []; var doubleQuotes = new RegExp("\"", "g"); values.forEach(function(value) { var text = value.toString(); if (text.indexOf(separator) == -1 && text.indexOf("\"") == -1) { result.push(text); } else { result.push("\"" + text.replace(doubleQuotes, "\"\"") + "\""); } }); return result.join(separator); } // New getFileContents implementation Csv.prototype.getFileContents = function(separator, includePropertyNames) { separator = separator || ","; var textLines = []; if (includePropertyNames) { textLines.push(createTextLine(this.propertyOrder, separator)); } this.items.forEach((function(item) { var values = []; this.propertyOrder.forEach(function(propertyName) { values.push(item[propertyName]); }); textLines.push(createTextLine(values, separator)); }).bind(this)); return textLines.join("\r\n"); };

And we're done! EDIT: The finished code can be found on github at /lbrtw/csv.

JavaScript Csv file generator, part 1
JavaScript Csv file generator, part 2
JavaScript Csv file generator, part 3 (this part)

JavaScript Csv file generator, part 2

Last week, I set up a few requirements and unit tests for my CSV file generator. Now it is time to start implementing.

First the constructor and the add method:

function Csv(propertyOrder) { this.propertyOrder = propertyOrder; this.items = []; } Csv.prototype.add = function(item) { this.items.push(item); };

There. The first three unit tests are already passing. Next up is the first version of the getFileContents method.

Csv.prototype.getFileContents = function(separator, includePropertyNames) { var textLines = []; // Add the auto-generated header if (includePropertyNames) { textLines.push(this.propertyOrder.join(separator)); } // We step through every item in the items array this.items.forEach(function(item) { // We create one line of text using the propertyOrder // First create an array of text representations var values = []; this.propertyOrder.forEach(function(name) { values.push(item[name].toString()); }); // Then join the fields together var lineOfText = values.join(separator); textLines.push(lineOfText); }).bind(this); // Return the complete file return textLines.join("\r\n"); };

The only thing left to do now is the saveAs method. Since this requires a functioning window.saveAs implementation to be available, the method is really simple.

Csv.prototype.saveAs = function(filename, separator, includePropertyNames) { var fileContents = this.getFileContents(separator, includePropertyNames); // Create a blob, adding the Unicode BOM to the beginning of the file var fileAsBlob = new Blob(["\ufeff", fileContents], {type:'text/csv'}); window.saveAs(fileAsBlob, filename); };

There! It's done! The only thing left to do is some extra text escaping, but I leave that for next part. EDIT: The finished code can be found on github at /lbrtw/csv.

JavaScript Csv file generator, part 1
JavaScript Csv file generator, part 2 (this part)
JavaScript Csv file generator, part 3

JavaScript Csv file generator, part 1

I came across the need to generate CSV files locally using JavaScript, and set out to create a simple tool for that. It should be small, simple and should just get the job done.

I would like to be able to use the CSV generator something like this:

var propertyOrder = ["name", "age", "height"]; var csv = new Csv(propertyOrder); csv.add({ name : "Anders", age : 38, height : "178cm" }); csv.add({ name : "John Doe", age : 50, height : "184cm" }); csv.saveAs("people.csv");

First things first, so let's start with some unit tests. I use the unit test framework that I have covered in earler posts.

Requirements

  1. The only parameter for the constructor is the order of the properties. This order should be saved.
  2. The add method should add one item to the list of items to export.
  3. For this purpose, the Csv object should contain an items property, containing all added items.
  4. The saveAs method should use the window.saveAs function, requiring a FileSaver shim to be in place.
  5. Mostly for testing purposes, the text content of the file to be generated should be accessible through the getFileContents method.
  6. When calling saveAs or getFileContents, I should be able to specify which field separator to use. The default should be a comma.
  7. When calling saveAs or getFileContents, I should be able to have the property names added automatically as a header row. The default should be not to.
  8. For interoperability purposes, the saved file should contain the correct Byte Order Mark.
engine.add("Csv constructor should save properties order", function(testContext, the) { // Arrange var properties = ["a", "b"]; // Act var csv = new Csv(properties); // Assert the("propertyOrder").propertyOf(csv).shouldBeSameArrayAs(properties); }); engine.add("Csv constructor should initiate the items property", function(testContext, the) { // Arrange var properties = ["a", "b"]; // Act var csv = new Csv(properties); // Assert the("items").propertyOf(csv).shouldBeArray(); the("length").propertyOf(csv.items).shouldBeExactly(0); }); engine.add("Csv.add should add one item", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); // Act csv.add({"a" : 1, "b" : 2}); // Assert the("length").propertyOf(csv.items).shouldBeExactly(1); the("a").propertyOf(csv.items[0]).shouldBeExactly(1); the("b").propertyOf(csv.items[0]).shouldBeExactly(2); }); engine.add("Csv.getFileContents should create file correctly", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); csv.add({"a" : "Abc", "b" : "Def"}); csv.add({"a" : "Ghi", "b" : "Jkl"}); // Act var file = csv.getFileContents(); // Assert the(file).shouldBeSameAs("Abc,Def\r\nGhi,Jkl"); }); engine.add("Csv.getFileContents(separator) should create file correctly", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); csv.add({"a" : "Abc", "b" : "Def"}); csv.add({"a" : "Ghi", "b" : "Jkl"}); // Act var file = csv.getFileContents(";"); // Assert the(file).shouldBeSameAs("Abc;Def\r\nGhi;Jkl"); }); engine.add("Csv.getFileContents(separator, true) should create file correctly", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); csv.add({"a" : "Abc", "b" : "Def"}); csv.add({"a" : "Ghi", "b" : "Jkl"}); // Act var file = csv.getFileContents(";", true); // Assert the(file).shouldBeSameAs("a;b\r\nAbc;Def\r\nGhi;Jkl"); }); engine.add("Csv.saveAs should call window.saveAs correctly", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); csv.add({"a" : "Abc", "b" : "Def"}); csv.add({"a" : "Ghi", "b" : "Jkl"}); var saveAsCalled = false, savedBlob, savedFilename; var mockSaveAs = function(blob, filename) { saveAsCalled = true; savedBlob = blob; savedFilename = filename; }; var oldSaveAs = window.saveAs; window.saveAs = mockSaveAs; // Act csv.saveAs("output.csv", ";", false); // Cleanup window.saveAs = oldSaveAs; // Assert the(saveAsCalled).shouldBeTrue(); the(savedBlob).shouldBeInstanceOf(window.Blob); the(savedFilename).shouldBeSameAs("output.csv"); }); engine.add("Csv.saveAs should add UTF-8 Byte Order Mark to beginning of file", function(testContext, the) { // Arrange var properties = ["a", "b"]; var csv = new Csv(properties); csv.add({"a" : "Abc", "b" : "Def"}); csv.add({"a" : "Ghi", "b" : "Jkl"}); // Mock saveAs function to store the blob that was created var savedBlob; var mockSaveAs = function(blob, filename) { savedBlob = blob; }; var oldSaveAs = window.saveAs; window.saveAs = mockSaveAs; // Act csv.saveAs("output.csv", ";", false); // Cleanup window.saveAs = oldSaveAs; // Assert (reading from a Blob is done using FileReader, which is asynchronous) var bom; testContext.actAndWait(1000, function(testContext, the) { var firstThreeBytes = savedBlob.slice(0, 3); var reader = new window.FileReader(); reader.addEventListener("loadend", function() { bom = reader.result; testContext.actDone(); }); reader.readAsArrayBuffer(firstThreeBytes); }).thenAssert(function(testContext, the) { the(bom).shouldBeSameArrayAs([0xef, 0xbb, 0xbf]); }); });

The next episode will be about implementing Csv. EDIT: The finished code can be found on github at /lbrtw/csv.

JavaScript Csv file generator, part 1 (this part)
JavaScript Csv file generator, part 2
JavaScript Csv file generator, part 3