3 days to create a data visualizer tool from scratch


In this blog I will like to pen down my 3days experience in learning a technology called NodeXL from scratch and creating an app out of that which can be downloaded for usage at http://netme.cloudapp.net/Downloads.aspx.
This could have been done in a shorter span, but considering my laziness, the delay was bound to happen J
Following is the story line best to my knowledge.

March 11, 2013 6:00pm


One of the member of the Microsoft’s Most Valued Professional group,  saw the Data Visualization section in NetMe and asked me about my experience in NodeXL.
NodeXL is a free open source set of libraries which helps you create a graph out of a GraphML or GEXF file format. You can also create your custom graphs using the libraries. This is an easy to learn feature and very effective in Data Visualization.
 The objective was that NodeXL comes with an Microsoft Office Excel dependency, he was looking for an approach how we can remove this dependency and create a windows form app which will fetch the twitter related nodes and its children and show it in a graphical format.
NodeXL was totally new to me and creating an app on that seems to be challenging. I thought to give a try and see how it goes.

March 11, 2013 9:00 pm


I got to know the basic features of what NodeXL do. Downloaded the source code of the same, but diving into 18 projects in the source code repository was not only time consuming but also not feasible for me. So I posted the discussion in NodeXl community forum at http://nodexl.codeplex.com/discussions/436250
To my query, I was really satisfied by the quick response given by the moderator. At around 10 pm IST I received a response from him that it was possible and there was a library set that we can download and start using.
 

March 13, 2013


Till now I was confident enough about the usage in NodeXL. I prepared a small POC wherein I created a graph using a sample graphml file. GraphML file format are used to create directed, undirected and mixed graphs. It is based on XML based file format. A sample GraphML format could be created as likewise
<?xml version="1.0" encoding="UTF-8"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
     http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <graph id="G" edgedefault="undirected">
    <node id="n0"/>
    <node id="n1"/>
    <node id="n2"/>
    <node id="n3"/>
    <node id="n4"/>
    <node id="n5"/>
    <node id="n6"/>
    <node id="n7"/>
    <node id="n8"/>
    <node id="n9"/>
    <node id="n10"/>
    <edge source="n0" target="n2"/>
    <edge source="n0" target="n3"/>
    <edge source="n1" target="n2"/>
    <edge source="n2" target="n3"/>
    <edge source="n3" target="n5"/>
    <edge source="n3" target="n4"/>
    <edge source="n4" target="n6"/>
    <edge source="n6" target="n5"/>
    <edge source="n5" target="n7"/>
    <edge source="n6" target="n8"/>
    <edge source="n8" target="n7"/>
    <edge source="n8" target="n9"/>
    <edge source="n8" target="n10"/>
  </graph>
</graphml>
 

Previous NodeXL community discussions helped me a lot in this. I am not highlighting the code as that can be easily searched in the NodeXL community discussion forum. Rather let me focus on the major problem statement of how to interact NodeXL with Twitter API.
At first I thought to take help of community member on this so posted the second thread on the forum

March 14, 2013


To this I got the whole step by step approach by the moderator on how we can use the NodeXL libraries to fetch twitter data. This was indeed of great help, but at this moment the Azure Mobile script which I wrote for NetMe flashed up. I thought to reuse the code.
The code was to fetch top 18-20 tweets with their username for any particular searched keyword.
Following is the AMS Node.JS code that is fulfilling the requirement
function read(query, user, request) {
    console.log(request.parameters.type);
    readExpert(request);
}
function readExpert(request)
{    console.log(request.parameters.topic);
    var res="<xml>";  
   var req = require('request');     
    req.get({            uri:"http://search.twitter.com/search.json?q="+request.parameters.topic+"&result_type=mixed",            headers: {'content-type': 'application/x-www-form-urlencoded'},      
},

    function(error,response,body){
        if (!error && response.statusCode == 200) { 
               var resp = JSON.parse(body);
               for(var attribute in resp)
               {                  
                   if(attribute == "results")
                   {
                       console.log(resp[attribute]);
                       var len = resp["results"].length;                      
                       for(var loop=0; loop< resp[attribute].length; loop++)
                        {
                                res += "<Tweet>"
                                res += "<UserName>" + resp[attribute][loop].from_user_name +"</UserName>";
                                res += "<UserStatus>" + resp[attribute][loop].text+ "</UserStatus>"
                                res += "</Tweet>";                         
                        }
                   }
               }
               res += "</xml>";
               request.respond(200, res);
                                }   
                                else   
                                {
            console.log(error);
            request.respond(statusCodes.BAD_REQUEST, error);
                                }
    });
}

Now the next step was to use this data and convert it into graph using NodeXL libraries. Here let me make a note for all the libraries I used
1.       Smrf.NodeXL.Adapters
2.       Smrf.NodeXL.Control.WPF
3.       Smrf.NodeXL.Core
4.       Smrf.NodeXL.Layouts
5.       Smrf.NodeXL.Visualization.Wpf
The code for creating a graph is as follows

XmlDocument graphML = new XmlDocument();
 
//Helper class is having the logic of sending the request to Azure Mobile Service which has been mentioned in my previous blog

                HelperClass _helperClass = new HelperClass("https://datavisualizer.azure-mobile.net/tables/datavisualizer", <<Azure Mobile Service Key>>, Guid.NewGuid().ToString());
                string response = _helperClass.ReadExperts(topic);
                string tweet = String.Empty;
                XmlDocument doc = new XmlDocument(); 

//Define NodeXL control
                NodeXLControl nodexl_Graph = new NodeXLControl();
if (response != "")
                {
                    doc.LoadXml(response);
                    IVertex headVertex = nodexl_Graph.Graph.Vertices.Add();
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Circle);
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexLabel, topic);
                    headVertex.SetValue(ReservedMetadataKeys.PerVertexLabelFillColor, System.Drawing.Color.Blue);
                    foreach (XmlNode node in doc.SelectNodes("//xml//Tweet"))
                    {
                        IVertex vertex = nodexl_Graph.Graph.Vertices.Add();
                        foreach (System.Xml.XmlElement item in node)
                        {
                            if (item.Name == "UserName")
                            {
                                vertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Circle);
                                vertex.SetValue(ReservedMetadataKeys.PerVertexLabel, item.InnerText);
                            }
                            if (item.Name == "UserImage")
                            {
                                var webClient = new WebClient();
                                byte[] imageBytes = webClient.DownloadData(item.InnerText);
                                BitmapImage imageSource = new BitmapImage();
                                using (MemoryStream stream = new MemoryStream(imageBytes))
                                {
                                    stream.Seek(0, SeekOrigin.Begin);
                                    imageSource.BeginInit();
                                    imageSource.StreamSource = stream;
                                    imageSource.CacheOption = BitmapCacheOption.OnLoad;
                                    imageSource.EndInit();
                                }
                                vertex.SetValue(ReservedMetadataKeys.PerVertexImage, imageSource);
                                vertex.SetValue(ReservedMetadataKeys.PerVertexLabelFillColor, Colors.Transparent);

                            }
                            if (item.Name == "UserStatus")
                            {
                                IVertex childvertex = nodexl_Graph.Graph.Vertices.Add();
                                childvertex.SetValue(ReservedMetadataKeys.PerVertexShape, VertexShape.Square);
                                childvertex.SetValue(ReservedMetadataKeys.PerVertexLabel, item.InnerText);
                                childvertex.SetValue(ReservedMetadataKeys.PerColor, System.Drawing.Color.Green);
                                IEdge childEdge = nodexl_Graph.Graph.Edges.Add(vertex, childvertex);
                                childEdge.SetValue(ReservedMetadataKeys.PerColor, System.Drawing.Color.Red);
                            }
                            IEdge edge = nodexl_Graph.Graph.Edges.Add(headVertex, vertex);
                        }
                    }
                }
                nodexl_Graph.DrawGraph(true);
 
// NodeXL is a WPF control. If you need to host this control in Windows form you need to create an element host control and assign the control to that.
                elementHost_Graph.Child = nodexl_Graph;

POC

 
This app has been uploaded under the downloads page in http://netme.cloudapp.net/ .  To try this app you need to download the zip folder, extract it and run the msi file. It will get installed in your machine and you will get to see the NetMe shortcut in your desktop. As this is a POC, I have not focused much on the look and feel of the app.
Once the app gets installed you will be able to search for any keyword and see the corresponding graph. You can shift the nodes accordingly.
The major intent of this blog was to highlight how fast and easy it is to code using NodeXL libraries and create your own data visualizer tool within a short time span. Hope you will like it  J

 

Comments

  1. Thanks for your mention, Dan Fay. This is really motivating.

    Dan Fay‏@ddfay18 Mar
    Very Cool RT @nabaruntweet: 3 days to create a data visualizer tool from scratch using NodeXL. Details at http://lnkd.in/wD5Maz

    ReplyDelete

Post a Comment

Popular posts from this blog

Firebase authentication with Ionic creator

Big Data - SWOT Analysis

LINKEDIN api call using NODE.JS OAUTH module