Somebody clean my log container, it is growing exponentially

Who has the time to keep a check of log container size. Nowadays most organization rely on cloud  services for storage and maintenance. One of the most common artifact which gets uploaded in these storage account is Logs.

Let me give an example from my experience. I have seen teams using Azure blob storage for storing logs files, dumps, manifest files etc. During investigation these files are downloaded and rendered at user's box. Similarly alert traces go to Azure Table storage.

Now who has the time and effort to keep a check on these storage capacity. I once wrote a simple powershell command which helped me to delete all artifacts older than 60 days as of that day.

Cool! but after 2 months, here comes a mail from one team member saying he is not able to see the containers from CloudXplorer or Azure Explorer.

What happened?? I forgot to rerun my script again and the container grew exponentially again.

OK so now I added a task scheduler which will invoke my script on daily basis.

After another 2-3 months later I got to see a similar mail from another team member. What happened this time?

My machine got formatted because I wanted to deploy a new OS, and everything got deleted. Again I thought I need to write the script and enable the task scheduler.

But Wait!! What if my account stops working? What if again I format my machine??

It will not only be a re-work but also loss of time and effort.

In a midst of confusion one of my team member told me why don't you write a stateless azure fabric service for this and deploy it on one of the azure cluster and forget about it?

Cool! why did not I think about this....

So I started writing this service. Problem statement was delete azure container and azure table which are 60 days older, so Ideally the service will trigger itself once in a day and find out all the container and table older than 60 days and delete it.

Now comes designing of this app. There are 2 actions performed by this service DeleteAzure container and Delete Azure Table. Both depends on time limit of 60 days. The user can also delete a Azure Table or Azure Container by its name.

As per me this resembled a Strategy design pattern. Following is a simple conceptual diagram

Now I hope I will never ever get any issues for cloud storage clean :)


Popular posts from this blog

Firebase authentication with Ionic creator

Big Data - SWOT Analysis

LINKEDIN api call using NODE.JS OAUTH module