How to work with Azure Service Bus efficiently, part 1

Introduction

This is a small series of articles about how to work with Azure Service Bus. The code is written in .net core and .net standard. No fancy Python in place here!

• Part 1: (this one) Talk about the basics
• Part 2: Show the code
• Part 3: Has everything been recognized?

Straighten out requirements

I did work with Azure Service Bus quite a lot in the past. When talking about messaging, there is a lot of different requirements in sending and retrieving messages. Let’s see what that is:

• Performance: How many messages can I send from a single (micro)-service?
• Performance: How many messages can I retrieve and work on synchronously and asynchronously?
• Architecture: How can I remove Service Bus implementations with minimum effort against anything else?
• Reliability: How can I ensure reliable work loads?
• Testability: How do I test all that?
• Deployment: Let’s keep out the deployment of the tasks for these articles. It is surely necessary and important, but I don’t picture this in these articles.
• Code: How to I structure my code that initialisation and working on service bus messaging is reliable, transparent and maintainable?
• Code: How do I avoid having to write and repeat a lot of boilerplate code?
• Messaging: Does the messaging is used to notify or to process?
• Exception Handling: How do I determine that a message cannot be worked on and delete it?
• Exception Handling: Where to put messages that always fail?
• Exception Handling: How do I know that a failed message was due to e.g. connectivity or the fact that the system can anyway not handle it?
• Exception Handling: How can I repeat messages in case of failure?
• Services: How do my choice of infrastructure influence the questions above?

Choosing Azure Services

Let’s first answer the last question from paragraph above. This is necessary as some services free the developers from the burden of most of these infrastructural thoughts.

What kind of Azure services are out there for let code run that is likely to retrieve or send Azure Service Bus messages?

Let’s talk about Azure Functions

First of all, Azure Functions makes it very easy to work with Azure Service Bus messages. As it is able to remove the necessity of initialisation and maintenance of infrastructural necessities (Table Storage, Service Bus, Cosmos Db, …) connection strings can be defined declaratively. The code then just handles the rest. Is this best choice? For sure not all the time. But in fact, it solves issues, let’s have a look onto that:

[FunctionName("ServiceBusQueueTriggerCSharp")]                    
public static void Run(
    [ServiceBusTrigger("myqueue", Connection = "ServiceBusConnection")] 
    string myQueueItem,
    Int32 deliveryCount,
    DateTime enqueuedTimeUtc,
    string messageId,
    ILogger log)
{
    log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
    log.LogInformation($"EnqueuedTimeUtc={enqueuedTimeUtc}");
    log.LogInformation($"DeliveryCount={deliveryCount}");
    log.LogInformation($"MessageId={messageId}");
}

It is quite easy to send and retrieve messages from queues or topics. Just define the connection string directly in code (not recommended) or define in a settings.json file. Only care about the logic to be applied when sending or retrieving. Scaling will be applied automatically. There are some possibilities to influence the scaling. Certainly also pricing comes into the play. For more information about cold start, limits, scaling options and pricing have a look at Azure Functions event driven scaling.

But wait, what if that doesn’t fit your requirements?

• There is the need for more options in terms of scaling
• Azure Functions is not cost efficient on the task at hand
• Messages that need to be processed taking longer than 5 minutes
• Language of choice is not supported by Azure Functions
• Azure Functions does not fit the overall architectural procedure (read as: containers, Service Fabric, …)

But actually, Azure Functions hit one thing perfectly: Infrastructure is something the developer “just” uses. Anyway messages need to be cut in a certain way, queues need to be set up properly, etc. But I consider Azure Functions to have a smart solution on handling infrastructural needs.

Infrastructure should be as hurdle free as possible to the developer

All the other services

For all the other services, Service Fabric, Azure Kubernetes Service (AKS), App Service, Container Instances and Batch infrastructure needs to be treated by oneself. That means if you jump between these services or just implement on project after the other, infrastructure handling needs to be considered every time, and probably implemented every time in a slightly different way.

How I would like to work with messaging brokers

Let’s put some requirements to code and infrastructure.

• Abstraction: I would like to write my message handlers, the piece of code that actually implements the logic for processing a certain message, independently from the actual broker. It should not matter if I decide to work with Azure Service Bus , Kafka or RabbitMQ. This should not change the implementation of the actual processing.
• Simplicity: I need to have configuration as simple and transparent as possible.
• Simplicity: I need to have initialisation as simple and transparent as possible
• Reliability: Processing needs to be reliable
• Exception Handling: Needs to be straight, transparent and understandable
• Code Quality: I do not want to repeat my self as far that it is possible
• Messaging: I prefer to get messages by push. Polling does make sense in certain cases, but can lead to more costs and is an additional burden.
• Messaging: Messages, that exceed the limits in terms of size, need to be handled gracefully.
• Messaging: Adding senders or receivers of messages must work seamlessly

How can that be reached?

Creating a sample

Let’s create a small sample to outline expectations.

Objectives and assumptions

• Messages need to be processed in a reliable, scalable way
• Messages can be sent from multiple senders
• Messages can exceed the size limit of 1MB
• Blob Storage and Service Bus instance can replaced by other services serving the same functionality with changing as less code as possible – in best case just configuration
• Write as less code as possible, focus on the implementation of the actual business logic, don’t fight against infrastructure all the time

In next article, of the series, I’ll enhance the sample with actual code. Follow me on that!