Thursday, October 2, 2014

Development with Amazon DynamoDB in C#


In this post I will put on important things you should know before starting development on the Amazon DynamoDB in C#. Following are briefly described concepts:

Table - A table is more or less the same thing you already know and it needs a name and a hash key to be created.

Hash Key - A hash key is like a primary key. You can optionally specify a range key, so by pairing the hash and range keys you must get unique items.

An item is like a record but only the hash key (and the range key if defined) is required, so every item can have a different data structure. 

DynamoDB is a database service in the cloud, so you can’t use Sql statements in order to manipulate data, Query and Scan operations are the way to do it. The Query operation is basically a get having the ID (or hash key) of the item you want to retrieve (that’s why it’s important to choose IDs carefully). By using the Query you can optionally specify the range key. The query has better performance over the Scan. Scan is used in the same way except by the fact that you can specify any attribute as a search condition; hence the table must be full scanned, decreasing the performance. You can use the Query and Scan operations in two ways: one is by retrieving a list of attribute names and values (so you have to read/parse them later) as it’s illustrated in the following sample (note that the simple Query operation uses the hash key only).

List> foo(string hashKeyValue, string tableName)
 {
       QueryRequest queryReq = new QueryRequest().WithHashKeyValue(new        
       AttributeValue().WithN(hashKeyValue));
       queryReq.WithTableName(tableName);

       QueryResponse response = Client.Query(queryReq);
       return response.QueryResult.Items;
 }

The second way to use the Query and Scan operations is by decorating your classes and letting the Amazon API to parse the attributes names and values for you. In this mode you have to add the following attributes to your class:
  • Table Name attribute “[DynamoDBTable("YourTableName")]”: this attribute must be located on top of each type being persisted in DynamoDB and it has the same effect that the tableName variable of the first sample.
  • Hash key and/or hash range attributes “[DynamoDBHashKey]”: this attribute must be located on top of the desired variable being persisted as part of a class.
After properly decorating your class you can use the Query operation as seen in the following sample.

public IEnumerable Query(object key)
{
      return context.Query(key);
}


Querying and Scanning DynamoDB tables are easier by decorating. Here are some issues or restrictions that I have found and probably they can help you in the process of using DynamoDB:
  • DynamoDB does not allow null or empty string attribute values.
  • If you fail to decorate the class you’ll receive a “No DynamoDBTableAttribute on type” exception.
  • If you receive a creepy “The start tag does not match the end tag” exception maybe it’s because you’re behind of a firewall, so you can use any tool like WireShark to make sure the requests can reach the Amazon server.
  • If you’re wondering how to work without SQL don’t worry, double check the Query sample using decorating and you’ll see that an IEnumerable collection is returned, so you can use Linq (and the Sum, Max, Avg of course)

Coding Principles To Follow


The other day I was having discussion with one of my colleague who was frustrated after doing a code review and he was of the opinion that developers do take it lightly to code as per standards; however, in agile model there is lot of pressure on them to deliver in time, leaving the poor code quality. To certain extent, I do agree with developer version however in long term this practice produces lot of serious issues about maintainability, performance and also leaves a technical debt to be borne by the application when it goes to production.

A bad design has following 3 characteristics: 
  • Rigidity - It is hard to change because every change affects too many other parts of the system.
  • Fragility - When you make a change, unexpected parts of the system break. 
  • Immobility - It is hard to reuse in another application because it cannot be disentangled from the current application.
Whenever we code or design we shall keep the following principles in mind which may make our lives simpler:

Open Close Principle
  • Software entities like classes, modules and functions should be open for extension but closed for modifications.
When writing classes, make sure that when you need to extend their behavior you dont have to change the class but to extend it. The same principle can be applied for modules, packages, libraries. If you have a library containing a set of classes there are many reasons for which you'll prefer to extend it without changing the code that was already written (backward compatibility, regression testing). This is why we have to make sure our modules follow Open Closed Principle. When referring to the classes Open Close Principle can be ensured by use of Abstract Classes and concrete classes for implementing their behavior. This will enforce having Concrete Classes extending Abstract Classes instead of changing them. Some particular cases of this are Template Pattern and Strategy Pattern.

  • High-level modules should not depend on low-level modules. Both should depend on abstractions.
  • Abstractions should not depend on details. Details should depend on abstractions.
Dependency Inversion Principle states that we should decouple high level modules from low level modules, introducing an abstraction layer between the high level classes and low level classes. Further more it inverts the dependency: instead of writing our abstractions based on details, the we should write the details based on abstractions.
Dependency Inversion or Inversion of Control are better know terms referring to the way in which the dependencies are realized. In the classical way when a software module(class, framework) need some other module, it initializes and holds a direct reference to it. This will make the 2 modules tight coupled. In order to decouple them the first module will provide a hook(a property, parameter) and an external module controlling the dependencies will inject the reference to the second one.
By applying the Dependency Inversion the modules can be easily changed by other modules just changing the dependency module. Factories and Abstract Factories can be used as dependency frameworks, but there are specialized frameworks for that, known as Inversion of Control Container.

  • Clients should not be forced to depend upon interfaces that they don't use.
This principle teaches us to take care how we write our interfaces. When we write our interfaces we should take care to add only methods that should be there. If we add methods that should not be there the classes implementing the interface will have to implement those methods as well. For example if we create an interface called Worker and add a method lunch break, all the workers will have to implement it. What if the worker is a robot?
As a conclusion Interfaces containing methods that are not specific to it are called polluted or fat interfaces. We should avoid them.

  • A class should have only one reason to change.
In this context a responsibility is considered to be one reason to change. This principle states that if we have 2 reasons to change for a class, we have to split the functionality in two classes. Each class will handle only one responsibility and on future if we need to make one change we are going to make it in the class which handle it. When we need to make a change in a class having more responsibilities the change might affect the other functionality of the classes.

  • Derived types must be completely substitutable for their base types.
This principle is just an extension of the Open Close Principle in terms of behavior meaning that we must make sure that new derived classes are extending the base classes without changing their behavior. The new derived classes should be able to replace the base classes without any change in the code.

Messaging Framework To Send SMS Using TWILIO API


Introduction  
The article is about creating the SMS messaging framework to allow user to send SMS to their application subscribers using Twilio REST Api. It provides a lot boilerplate code for someone to quickly get the task done.

Background 
I got a task to develop a SMS messaging framework using TWILIO api. Most of the things at my end were quite easy to write however things related to TWILIO were missing in most of the forums I searched and there were days of interaction with TWILIO support member to get the resolutions. That is why I thought to write this article in order to save time for others.

Prerequisites 
This article requires knowledge about VS 2010, WCF REST Service and MSMQ. 
Let us start developing the SMS inbound and outbound messaging framework using the TWILIO API. First of all you need to buy an account with TWILIO or use Trial account. They will give you a number which will be used for sending and receiving the messages. Also, you will receive the AccountId and Auth token to be used to access the TWILIO API. 
OK, now so this framework contains the following components:
  1. Outbound Message queue to persist the messages sent from you application. 
  2. Windows service to poll the message queue and send the SMS to the interested parties.
  3. REST API to collect the response received to the SMS send from your application.
  4. Inbound Message queue to collect the response messages. 
  5. Windows service to process the inbound messages. 
Creating Queues  
I am assuming you have the queues installed on your machine otherwise you can install it from Windows Components. Create a queue in the following manner (on Windows 7 OS): 
     Go to Administrative Tools -> Computer Management -> Services and Applications -> Message Queuing -> Private Queues:



   If we enabled the Journaling then we can audit the messages. 




In the same way you can create the “smsinq” for collecting inbound SMS. Create a MessageDTO class with the desired properties and don’t forget to mark it Serializable.
[Serializable]
public class MessageDTO
{
   public MessageDTO(){}
   public Guid MessageGUID{ set; get; }
   public string To { set; get; } //Phone number to whom message need to send
   public string Subject { set; get; }
   public string Body { set; get; }
   public string From { set; get; } //Phone number from whom message need to send
   public DateTime CreatedOn { set; get; }
   public DateTime SentOn { set; get; }
}

Utility.cs 
Now you need the function that will push the messages in the outbound SMS queue (smsoutq). This function which can be put in Utility.cs class to be reusable, is quite easy and can be used for pushing both inbound and outbound messages in their respective queues as follows:
public class Utility
{
    private static string smsOutQueuePath = ".\private$\smsoutq";
    private static string smsInQueuePath = ".\private$\smsinq";
    
    private static Message SetMessageProperties(Message objMsg)
    {
           objMsg.Recoverable = true;
           objMsg.UseDeadLetterQueue = true;
           objMsg.UseJournalQueue = true; 
           objMsg.Formatter = new BinaryMessageFormatter();
           return objMsg;
       }
       
       public static MessageQueue GetMessageQueue(string queuePath)
       {
           MessageQueue msgQ = null;
           try
           {
               if(!MessageQueue.Exists(queuePath))
               {
                    msgQ = MessageQueue.Create(queuePath);
               }                
               msgQ = new MessageQueue(queuePath, QueueAccessMode.SendAndReceive);
               msgQ.Formatter = new BinaryMessageFormatter();
               msgQ.DefaultPropertiesToSend.UseJournalQueue = true;
               msgQ.DefaultPropertiesToSend.Recoverable = true; //Messages remain if server goes down
               msgQ.DefaultPropertiesToSend.UseDeadLetterQueue = true;
               msgQ.MessageReadPropertyFilter.SetAll();
           }
           catch (Exception ex)
           {
                     //your exception logging code
           }
           return msgQ;
       }
 
       public static bool PostToSMSQueue(MessageDTO objMsg, string direction = Constants.MESSAGE_DIRECTION_OUT)
       {
           bool success = false;
           try
           {
               Message objMessage = new Message();
                  objMessage = SetMessageProperties(objMessage);
                  objMessage.Body = objMsg;
                  MessageQueue msgQ = null;
                  
                  if(direction == Constants.MESSAGE_DIRECTION_OUT)
                      msgQ = GetMessageQueue(smsOutQueuePath);
                  else
                      msgQ = GetMessageQueue(smsInQueuePath);
                  msgQ.Send(objMessage);
        success = true;
           }
           catch (Exception ex)
           {
              //your exception logging code
           } 
 
           return success;
       }
       
       public static NameValueCollection ConvertUrlStringToNSCollection(string strUri)
       {
               NameValueCollection nsCol = new NameValueCollection();
               if(!String.IsNullOrEmpty(strUri))
               {
                   try
                   {
                       int endPos = 0;
                       string[] messageArray = null;
                       messageArray = strUri.Split('&');
                       for(int count = 0; count < messageArray.Length; count++)
                       {
                           endPos = messageArray[count].LastIndexOf('=');
                           nsCol.Add(messageArray[count].Substring(0, endPos),
                           messageArray[count].Substring(endPos + 1, messageArray[count].Length - (endPos + 1)));
                       }
                   }
                   catch(Exception ex)
                   {
                       throw ex;
                   }
               }
               return nsCol;
           }
} 
Creating OutboundMsgPoller Service to Send Message to TWILIO
Now you have to write the windows service (OutboundMsgPoller) which will read the messages to the desired parties at some defined time interval and pick the messages and sent them to TWILO using their API to be sent further. Note the code will be same for the ReponseProcessor (Windows service to process the received responses, read inline comments below), so you can reuse it. 
public partial class OutboundMsgPoller : ServiceBase
{
    private static string smsOutQueuePath = ".\private$\smsoutq";
    private static string accountSID = string.Empty;
    private static string authToken = string.Empty;
    private static string callbackURL = string.Empty;
    private Thread m_oPollingThread = null;
    private bool shallRun = false;  
 
    protected override void OnStart(string[] args)
    {
        try
        {
            this.EventLog.WriteEntry("OutboundMessagePoller service started successfully", EventLogEntryType.Information);
            shallRun = true; 
            if (m_oPollingThread == null)
                m_oPollingThread = new Thread(SendMessage);
        m_oPollingThread.IsBackground = false;
        m_oPollingThread.Start();
        }
        catch (Exception ex)
        {
            //your logging code
        }
    }
    
    protected override void OnStop()
    {
               this.EventLog.WriteEntry("OutboundMessagePoller service is OnStop.", EventLogEntryType.Information);
    }
 
    private void SendMessage()
    {
        IList lstMsg = new List();
        int pollInterval = 0; 
        MessageQueue queue = new MessageQueue(smsOutQueuePath, QueueAccessMode.SendAndReceive);
        accountSID = "Your Account ID"; 
        authToken = "Your Auth Token";
        callbackURL = string.Empty;//in case you want to collect the message status then you can specify a url for your application where TWILIO can POST the status. 
        
        do
        {
            try
            {
                using (MessageEnumerator messageEnumerator = queue.GetMessageEnumerator2())
                {
                    while (messageEnumerator.MoveNext(new TimeSpan(0, 0, 1)))
                    {
                        Message msg = queue.ReceiveById(messageEnumerator.Current.Id, new TimeSpan(0, 0, 1));
                        lstMsg.Add(msg);
                    }
                }
                queue.Close();
                
                if (lstMsg != null && lstMsg.Count > 0)
               {
                //for OutboundMsgPoller service
                this.SendMessageToTwilio(lstMsg, accountSID, authToken, callbackURL);
                //for ReponseProcessor service, you have to write your own method
                YourMethod(lstMsg);
            }
            lstMsg.Clear();
            //Wait... 
            {
                pollInterval = 30000; //time in milliseconds, can be made configurable
            }
            Thread.Sleep(pollInterval);
        }
        catch (Exception ex)
        {
            shallRun = false;
            //your logging code
        }
        } while (shallRun);
    } 
 
    private void SendMessageToTwilio(IList objMessage, string accountSID, string authToken, string callbackURL)
    {
        try
        {
            if (objMessage != null && objMessage.Count > 0)
            {
                // instantiate a new Twilio Rest Client
                var client = new TwilioRestClient(accountSID, authToken); 
                // iterate over message
                foreach (Message ms in objMessage)
                {
                    ms.Formatter = new BinaryMessageFormatter();//this is important
                    if (ms.Body != null)
                    {
                        var objTw = client.SendSmsMessage(((MessageDTO)ms.Body).From, ((MessageDTO)ms.Body).To, ((MessageDTO)ms.Body).Body, callbackURL);
                            if (objTw.RestException != null)// this is important
                            {
                                throw new Exception("SendMessageToTwilio - Twilio Error Code - " + objTw.RestException.Code + "|| Error Message - " + objTw.RestException.Message);
                            }
                            else
                            {
                                //Update the SMSID received from TWILIO in your DB or other appropriate repository.
                            }
                        }
                    }
                }
        }
        catch (Exception ex)
        {
             //your logging code
        }
    }
}  
Creating REST Service to Collect TWILIO Responses
In case you don’t have please install the install the WCF REST template from given link: http://visualstudiogallery.msdn.microsoft.com/fbc7e5c1-a0d2-41bd-9d7b-e54c845394cd
Once you install it, create a new project using this template as follows:

TWILIO Response Class
You will be needing to create a Twilio Response class as follows: 
[Serializable]// this is important
public class TwilioRespDTO 
{   
    public TwilioRespDTO(){}
    public string SmsSid { get; set; }
    public string AccountSid { get; set; }
    public string From { get; set; }
    public string To { get; set; }
    public string Body { get; set; }
} 
Response Collection REST API
[ServiceContract]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
// NOTE: If the service is renamed, remember to update the global.asax.cs file
public class Responder
{    
    [WebInvoke(UriTemplate = "response/sms", Method = "POST")] 
    //IMPORTANT: response/sms is the part of the uri you will register with Twilio account for getting responses.
    public void GetTwilioSMSResponse(Stream sr)
    {
        if (sr != null)
        {
            try
            {
                string postData = string.Empty;
                NameValueCollection nsCol = new NameValueCollection();  
 
                using (StreamReader srt = new StreamReader(sr))
                {
                    postData = srt.ReadToEnd();
                }
 
                nsCol = Utility.ConvertUrlStringToNSCollection(postData); 
 
                if (nsCol != null && nsCol.Count > 0)
                {
                    TwilioResponseDTO objTw = new TwilioResponseDTO();
                    objTw = CreateTwilioResponseObject(nsCol);
                    Utility.PostToSMSQueue(objResp, Constants.MESSAGE_DIRECTION_IN);
                }
            }
            catch (Exception ex)
            {
               //your error handling code here
            }
        }
    } 
 
    private TwilioResponseDTO CreateTwilioResponseObject(NameValueCollection nsCol)
    {
        MethodInfo objMethod = (MethodInfo)MethodBase.GetCurrentMethod();
        TwilioResponseDTO objTw = new TwilioResponseDTO();
        objTw.SmsSid = nsCol.Get("SmsId");
        objTw.AccountSid = nsCol.Get("AccountId");
        objTw.From = HttpUtility.UrlDecode(nsCol.Get("From"));
        objTw.To = HttpUtility.UrlDecode(nsCol.Get("To"));
        objTw.Body = nsCol.Get("Body");
        return objTw;
    }   
}            
Conclusion
I hope you enjoyed reading about this article as much as I enjoyed writing it. I also hope that it will relieve you from writing boilerplate code and allow you to concentrate on the task at hand instead. Feel free to borrow the code and use it for your needs.