Smurf..An ASA based tool in making- Part -1


This will be a blog series of all the learnings during the course of developing a tool on Azure Stream Analytics (ASA) called Smurf. This introductory blog will start with bits and pieces we want to create this tool.

I will not describe on what is ASA on what it does. It is described well in msdn sites. I will try to focus on all the other different tools which are required to create this project.

What are the components?

 
Azure Stream Analytics can work on 3 kinds of inputs.
1. Data from SQL Azure
2. Data from Azure Storage
3. Data from EventHubs.
 
We will touch each component as we proceed.
 

Starting with Event hub component

I am starting with the Event hub component first. Why I chose it is because of its impressive success story with ASA. As we know event hubs can work on partitioned data and also helps developer to easily scale up. Therefore a story with Event Hub and ASA will indeed be impressive
 
To build Event Hub support these are the following components which we will highlight as we go
 
1. Send data to event hub
2. Process data based on user defined business logic (this seems tricky, let's see how it goes)
3. Receive data from event hub
 

Send data to Event hub

Sending data to event hub is pretty much straight forward. You need to do the following
1. Create the event hub client using event hub name and event hub connection string
2. Create list of eventhubdata objects
3. Start sending these objects asynchronously
4. Verify if all the data has been sent
 

1. Create the event hub client  

/// <summary>
/// Sends the Batch of inputs
/// </summary>
/// <param name="eventDataList">List of EventData</param>
/// <returns>Responds with success </returns>
public async Task<bool> SendBatchAsync(IEnumerable<EventData> eventDataList)
        {
            try
            {
                var eventHubClient = this.GetEventHubClient();
                foreach (var eventData in eventDataList)
                {
                    await eventHubClient.SendAsync(eventData);

                    await Task.Delay(200);
                }
            }
            catch (Exception exception)
            {
                LoggingService.WriteError(exception.Message);
                return false;
            }

            return true;
        }


/// <summary>
/// Gets the Event Hub Client
/// </summary>
/// <returns>EventHubClient Object</returns>
private EventHubClient GetEventHubClient()
        {
            var client = EventHubClient.CreateFromConnectionString(this.EventHubConnectionString, this.EventHubName);
            return client;
        }

 

2. Create list of Eventhubdata objects

3. Start Sending them to Event Hub asynchronously

/// <summary>
/// Sends the Batch of inputs
/// </summary>
/// <param name="eventDataList">List of EventData</param>
/// <returns>Responds with success </returns>
 public async Task<bool> SendBatchAsync(IEnumerable<EventData> eventDataList)
        {
            try
            {
                var eventHubClient = this.GetEventHubClient();
                foreach (var eventData in eventDataList)
                {
                    await eventHubClient.SendAsync(eventData);
                    await Task.Delay(200);
                }
            }
            catch (Exception exception)
            {
                LoggingService.WriteError(exception.Message);
                return false;
            }
            return true;
        }
 

4. Verify using Xunit Test

[Fact]
        public async Task SendEventsToEventHub()
        {
            InitializeEventHub();
            var tasks = new List<Task>();
            var random = new Random();
            var eventDataList = new List<EventData>();
            
            //Create Event Hub Data List
            for (var loop = 0; loop < 5; loop++)
            {
                var info = new MetricData() { DeviceId = random.Next(NumberOfDevice), Temperature = random.Next(100) };
                var serializedString = JsonConvert.SerializeObject(info);
                var data = new EventData(Encoding.UTF8.GetBytes(serializedString))
                {
                    PartitionKey = info.DeviceId.ToString()
                };
                data.Properties.Add("Type", "Telemetry_" + DateTime.Now.ToLongDateString());
                eventDataList.Add(data);
            }
            var sendObj = new Sender(this.EventHubName, this.EventHubConnectionString);
            var sentSuccess = await sendObj.SendBatchAsync(eventDataList);
            sentSuccess.ShouldBeEquivalentTo(true);
        }

Now that the send is working let me capture the learnings on receive in my next blog

Comments

  1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Front end developer Node js Training in Chennai . learn from or Javascript Online Training from India. Nowadays JavaScript has tons of job opportunities on various vertical industry. JavaScript Training in Chennai

    ReplyDelete

Post a Comment

Popular posts from this blog

Firebase authentication with Ionic creator

Big Data - SWOT Analysis

LINKEDIN api call using NODE.JS OAUTH module