IoT architectures – automation stunts with .net framework, part 4

Part 4: Going on with alternative deployment options

This is the fourth of at six articles about how an IoT project with certain constraints and prerequisites was implemented. In recent article I described why it was not possible to create the docker container for OSISoft AKSDK lib and that is was not possible to leverage old-school Azure Classic Cloud Services. This article goes on with the alternative deployment options:Azure Service Fabric, Azure Functions and Windows Services.

Again move the code.

As the code against needs to adopt to a different cloud service, let’s first have a look onto how the code has been structured for docker containers and Azure Classic Cloud Services. In previous article I already stated, the main areas for adoption are Configuration, Logging and Bootstrap. Let’s have a look onto the Bootstrap part.

Bootstrap for Docker

Actually there are always three parts. A bootstrapper and the actual start up class. Additionally one-to-n installer classes. Using dependency injection, all of the services are captured in modules / installers. An installer looks like this, implementing an interface:

using Iot.Contracts.Ioc;
using Osi2Hdf.Data.Configuration;
using Osi2Hdf.Data.Messages;
using Osi2Hdf.Reader.Services.MessageHandlers;
using Microsoft.Extensions.DependencyInjection;

namespace Osi2Hdf.Reader.Services.Installers
{
    public class Osi2HdfReaderInstaller : IInstaller
    {
        public void Install(IServiceCollection services)
        {
            services.AddSingleton<OsiServerConfiguration>();
            services.AddSingleton<PathDelimiterReplacer>();
            services.AddSingleton<IMessageHandler<ReadOsiRecordsMessage>, ReadOsiMessageHandler>();
            services.AddSingleton<ObjectSizeCalculator>();
            services.AddSingleton<IBlobFileManagerConfiguration, BlobFileManagerConfiguration>();
            services.AddSingleton<BlobFileNameCreator>();
            services.AddSingleton<MessageCreator>();
            services.AddSingleton<GroupedSensorRecordCreator>();
            services.AddSingleton<ApplicationUnhandledExceptionHandler>();
        }
    }
}

The installers are topic related. As the overall size of the program is rather small, there are only two installers. One for configuration, the other for the whole program. With that installer, the Bootstrapper class is implemented. It collects all necessary installer and registers the contained services to the dependency injection framework.

using System;
using Iot.Cloud.Common.Blob.Services.Installers;
using Iot.Cloud.Common.CosmosDb.Services.Installers;
using Iot.Cloud.Common.ServiceBus.Services.Installers;
using Iot.Common.Services.Installers;
using Henkel.Iot.Contracts.Ioc;
using Osi2Hdf.Core.Services.Installers;
using Osi2Hdf.Reader.Services.Installers;
using Microsoft.Extensions.DependencyInjection;

namespace Osi2Hdf.Reader.Bootstrap
{
    public class Bootstrapper
    {
        public IServiceProvider Initialize()
        {
            var services = new ServiceCollection();
            var installers = new IInstaller[]
            {
                new Osi2HdfReaderInstaller(),
                new Osi2HdfReaderConfigurationInstaller(),
                new Osi2HdfCoreInstaller(),
                new CloudCommonServiceBusCoreInstaller(),
                new CloudCommonServiceBusQueuesInstaller(),
                new IotCommonInstaller(),
                new IotCommonLoggingInstaller(), 
                new CloudCommonBlobInstaller(),
                new CloudCommonCosmosDbInstaller()
            };

            foreach (var installer in installers)
            {
                installer.Install(services);
            }
            return services.BuildServiceProvider();
        }
    }
}

As you may have seen, there are more installers used than provided by the Reader implementation. Obviously there are parts in other assemblies for separation of concerns or due to reusability. A common area provides logging/ configuration functionalities. The cloud common assemblies provide abstractions for Blobs, Service Bus and CosmosDb. These both parts will not change when moving from one Service to another. They will just be used differently. Coming to the Docker implementation of the .net framework executable:

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Iot.Cloud.Common.Blob.Contracts;
using Iot.Cloud.Common.ServiceBus.Contracts;
using Iot.Cloud.Common.ServiceBus.Services;
using Iot.Contracts;
using Iot.Contracts.Ioc;
using Osi2Hdf.Data.Configuration;
using Osi2Hdf.Reader.Services;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

namespace Osi2Hdf.Reader.Bootstrap
{
    class Program
    {
        private static ILogger<Program> _logger;
        private static IServiceBusQueueClientListener<ServiceBusQueueClientListenerConfiguration> _serviceBusQueueListener;
        private static readonly ManualResetEvent CompletedEvent = new ManualResetEvent(false);
        private static IServiceBusQueueClientSender<ServiceBusQueueClientSenderConfiguration> _serviceBusQueueSender;
        private static IUpdater _osiMetaDataUpdater;
        private static ApplicationUnhandledExceptionHandler _applicationUnhandledExceptionHandler;
        private static IBlobFileManager<IBlobFileManagerConfiguration> _blobFileManager;
        private static ServiceBusClientCollection _serviceBusClientCollection;
        private static IServiceBusQueueClientSender<ServiceBusQueueClientSenderBlobConfiguration> _serviceBusQueueBlobSender;

        static void Main(string[] args)
        {
            try
            {
                var applicationInitializer = new Bootstrapper();
                var serviceProvider = applicationInitializer.Initialize();
                var objectResolver = (IObjectResolver)serviceProvider.GetService(typeof(IObjectResolver));
                _logger = objectResolver.Resolve<ILogger<Program>>();
                _serviceBusClientCollection = objectResolver.Resolve<ServiceBusClientCollection>();
                _serviceBusQueueListener = _serviceBusClientCollection.GetListener<ServiceBusQueueClientListenerConfiguration>();
                _serviceBusQueueSender = _serviceBusClientCollection.GetSender<ServiceBusQueueClientSenderConfiguration>();
                _serviceBusQueueBlobSender = _serviceBusClientCollection.GetSender<ServiceBusQueueClientSenderBlobConfiguration>();
                _blobFileManager = objectResolver.Resolve<IBlobFileManager<IBlobFileManagerConfiguration>>();
                _osiMetaDataUpdater = objectResolver.Resolve<IUpdater>();

                _applicationUnhandledExceptionHandler = objectResolver.Resolve<ApplicationUnhandledExceptionHandler>();
                _applicationUnhandledExceptionHandler.Bind();

                var taskList = new List<Task>();
                taskList.Add(Task.Factory.StartNew(() => _osiMetaDataUpdater.Update()));
                Task.WaitAll(taskList.ToArray());

                taskList.Add(Task.Factory.StartNew(() => _blobFileManager.Initialize()));
                taskList.Add(Task.Factory.StartNew(() => _serviceBusQueueSender.Initialize()));
                taskList.Add(Task.Factory.StartNew(() => _serviceBusQueueBlobSender.Initialize()));
                taskList.Add(Task.Factory.StartNew(() => _serviceBusQueueListener.Start(new CancellationToken())));
                Task.WaitAll(taskList.ToArray());
                CompletedEvent.WaitOne();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
                throw;
            }
        }
    }
}

The program file does not need to do too much. It uses the bootstrapper class to register all necessary services then it does it actual job: Initialize the services that are needed for startup.

Bootstrap for Azure Classic Cloud Services

Following the structure above, let’s focus on Bootstrapper and start up class. As stated before, the Bootstrapper class doesn’t change at all. It is only the start up class that needs to be changed. This is for Azure Classic Cloud Services a so-called „WorkerRole“ or „WebRole“. This is an unattended service without any user interface or public contract. Therefore it is a WorkerRole. Let’s have a look onto the class.

using System;
using System.Diagnostics;
using System.Threading;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.Extensions.Logging;
using System.Threading.Tasks;
using System.Collections.Generic;
using Henkel.Iot.Cloud.Common.Contracts.Messaging;
using Henkel.Iot.Contracts;
using Henkel.Iot.Contracts.Ioc;
using Henkel.Osi2Hdf.Data.Configuration;
using Henkel.Osi2Hdf.Reader.Bootstrap;

namespace Henkel.Osi2Hdf.Reader
{
    public class WorkerRole : RoleEntryPoint
    {
        private ILogger<WorkerRole> _logger;
        private IServiceBusQueueClientListener<ServiceBusQueueClientListenerConfiguration> _serviceBusQueueListener;
        private readonly ManualResetEvent CompletedEvent = new ManualResetEvent(false);
        private IServiceBusQueueClientSender<ServiceBusQueueClientSenderConfiguration> _serviceBusQueueSender;
        private IUpdater _osiMetaDataUpdater;

        public override void Run()
        {
            Trace.WriteLine("Starting processing of messages");

            try
            {
                var taskList = new List<Task>();
                
                taskList.Add(Task.Factory.StartNew(
                    () => _osiMetaDataUpdater.Update()));
                Task.WaitAll(taskList.ToArray());
            }
            catch (Exception ex)
            {
                _logger.LogError("Failed to execute osiMetaDataUpdater", ex);
            }

            try
            {
                var taskList = new List<Task>();
                taskList.Add(
                    Task.Factory.StartNew(
                        () => _serviceBusQueueListener.Start(new CancellationToken())));
                taskList.Add(
                    Task.Factory.StartNew(
                        () => _serviceBusQueueSender.Initialize()));
                Task.WaitAll(taskList.ToArray());
            }
            catch (Exception ex)
            {
                _logger.LogError("Failed to initialize ServiceBusListener or Sender.", ex);
            }
          
            CompletedEvent.WaitOne();
        }

        public override bool OnStart()
        {

            try
            {
                var applicationInitializer = new Bootstrapper();
                var serviceProvider = applicationInitializer.Initialize();
                var objectResolver = (IObjectResolver)serviceProvider.GetService(typeof(IObjectResolver));
                _logger = objectResolver.Resolve<ILogger<WorkerRole>>();
                _serviceBusQueueListener = objectResolver.Resolve<IServiceBusQueueClientListener<ServiceBusQueueClientListenerConfiguration>>();
                _serviceBusQueueSender = objectResolver.Resolve<IServiceBusQueueClientSender<ServiceBusQueueClientSenderConfiguration>>();
                _osiMetaDataUpdater = objectResolver.Resolve<IUpdater>();
            }
            catch (Exception ex)
            {
                _logger.LogError("Failed to start WorkerRole", ex);
                throw;
            }

            return true;
        }

        public override void OnStop()
        {
            CompletedEvent.Set();
            base.OnStop();
        }
    }
}

It is in fact longer than the program.cs in docker container implementations. There is a certain interface/ inheritation that needs to be followed. At least OnStart and Run methods need to be implemented. If you wonder why exception are caught, this is due to the nature of this implementation. If OnStart or Run method fail, the service is directly restarted, even when deploying from Visual Studio. Deployment will not end. So I tend to have a look onto the log and fix things, if necessary.

Having the code prepared for change makes live pretty easy

Okay, we all know, that’s not the full story. Logging and Configuration also need to be taken in consideration. But it works pretty much the same. Some sensible interfaces in common area of the assemblies allow for adapter-like usage. So the effort is pretty low.

Bootstrap for Azure Service Fabric

Going on, let’s have a look onto the necessities of Azure Service Fabric. Again a different approach, different configuration items, logging and startup.

using System;
using System.Collections.Generic;
using System.Fabric;
using System.Threading;
using System.Threading.Tasks;
using Iot.Cloud.Common.Contracts.Messaging;
using Iot.Contracts;
using Iot.Contracts.Ioc;
using Osi2Hdf.Data.Configuration;
using Osi2Hdf.Reader.Bootstrap;
using Microsoft.Extensions.Logging;
using Microsoft.ServiceFabric.Services.Communication.Runtime;
using Microsoft.ServiceFabric.Services.Runtime;

namespace Osi2Hdf.ServiceFabric.Reader.Bootstrap
{
    /// <summary>
    /// An instance of this class is created for each service instance by the Service Fabric runtime.
    /// </summary>
    internal sealed class Reader : StatelessService
    {
        private static ILogger<Reader> _logger;
        private static IServiceBusQueueClientListener<ServiceBusQueueClientListenerConfiguration> _serviceBusQueueListener;
        private static IServiceBusQueueClientSender<ServiceBusQueueClientSenderConfiguration> _serviceBusQueueSender;
        private static IUpdater _osiMetaDataUpdater;

        public Reader(StatelessServiceContext context)
            : base(context)
        { }

        /// <summary>
        /// Optional override to create listeners (e.g., TCP, HTTP) for this service replica to handle client or user requests.
        /// </summary>
        /// <returns>A collection of listeners.</returns>
        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[0];
        }

        /// <summary>
        /// This is the main entry point for your service instance.
        /// </summary>
        /// <param name="cancellationToken">Canceled when Service Fabric needs to shut down this service instance.</param>
        protected override async Task RunAsync(CancellationToken cancellationToken)
        {
            try
            {
                var applicationInitializer = new Bootstrapper();
                var serviceProvider = applicationInitializer.Initialize();
                var objectResolver = (IObjectResolver)serviceProvider.GetService(typeof(IObjectResolver));
                _logger = objectResolver.Resolve<ILogger<Reader>>();
                _serviceBusQueueListener = objectResolver.Resolve<IServiceBusQueueClientListener<ServiceBusQueueClientListenerConfiguration>>();
                _serviceBusQueueSender = objectResolver.Resolve<IServiceBusQueueClientSender<ServiceBusQueueClientSenderConfiguration>>();
                _osiMetaDataUpdater = objectResolver.Resolve<IUpdater>();

                var taskList = new List<Task>();
                taskList.Add(Task.Factory.StartNew(
                    () => _osiMetaDataUpdater.Update()));
                Task.WaitAll(taskList.ToArray());

                taskList.Add(
                    Task.Factory.StartNew(
                        () => _serviceBusQueueListener.Start(new CancellationToken())));
                taskList.Add(
                    Task.Factory.StartNew(
                        () => _serviceBusQueueSender.Initialize()));
                Task.WaitAll(taskList.ToArray());

                long iterations = 0;

                while (true)
                {
                    cancellationToken.ThrowIfCancellationRequested();

                    ServiceEventSource.Current.ServiceMessage(this.Context, "Working-{0}", ++iterations);

                    await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
                throw;
            }
        }
    }
}

So far so good. Code is under control. But the infrastructure? Let’s see.

Going for Azure Service Fabric

I did skip Azure Functions. I didn’t want to go with the very old 1.x version of Azure Functions which would be the only choice for .net framework.

Service Fabric can be compared to AKS from a functional point of view. It is a self balancing, multi node system. It comes with a out-of-the-box user interface for administration of all nodes. Containers can be deployed but it feels like they are not first-class citizens. Keep in mind, it’s all more windows focussed. There are a couple of comparisons out there, you may want to have a look onto this stackoverflow question.

Due to Service Fabric has a lot of capabilities, configuration and maintenance is a pretty big task. Anyway it comes with pretty much the same functionality like Azure Classic Cloud Services. It is possible to configure a start up batch file that will install the necessary setup.

To make a long story short: I didn’t make it.

Exceptions from a batch file when running a setup are very hard to dig up and understand. From cost point of view, Service Fabric is way to huge for the size of implementation that I planned to do. So I stopped pretty soon with trying. All of these tasks also cost money, due to I spend my time. So I need to speed up this.

Any other options?

What now? I had a lot of options and I don’t get it running. This was one of the most frustrated moments in my developer life. I never needed to go for so many options and still didn’t have a solution at hand. Additionally, time’s flying and I need a solution for making this system run. What do I need?

  • Have a system where I can install this setup seamlessly
  • There is need to scale, in best case horizontally. If that doesn’t work, vertically would be fine, but Availability still needs to be considered.
  • Deployment of sources shall be done automatically
  • Deployment of services shall be done automatically

Okay, last exit for the lost: I need a VM. I can scale with single processes vertically. Sources can be deployed on this vm. VM could be hosted in Azure as well. Certainly there are some constraints. For reaching out to OSIsoft, the VM needs a two-cross vnet and read data via VPN inside of the companies‘ network. I do not like this solution. But it is a solution. And one that I can do immediately.

Going for a Windows Service

This has been quite some time that I created a Windows Service. I needed to dig it up. Startup is a easy as for all the other variants.

using System;
using System.ServiceProcess;
using System.Threading;
using System.Threading.Tasks;
using Iot.Contracts.Configuration;
using Iot.Contracts.Ioc;
using Microsoft.Extensions.Logging;

namespace Osi2Hdf.Reader.WindowsService.Services
{
    public partial class ReaderManagerService : ServiceBase
    {
        private ILogger<ReaderManagerService> _logger;
        private readonly ReaderProcessManager _readerProcessManager;
        private readonly IConfigurationReader _configurationReader;
        private readonly CancellationTokenSource _cancellationTokenSource;

        public ReaderManagerService()
        {
            InitializeComponent();

            var bootstrapper = new Bootstrapper.Bootstrapper();
            var serviceProvider = bootstrapper.Initialize();
            var objectResolver = (IObjectResolver)serviceProvider.GetService(typeof(IObjectResolver));
            _logger = objectResolver.Resolve<ILogger<ReaderManagerService>>();
            _logger.LogDebug("ctor");
            _readerProcessManager = objectResolver.Resolve<ReaderProcessManager>();
            _configurationReader = objectResolver.Resolve<IConfigurationReader>();
            _cancellationTokenSource = new CancellationTokenSource();
        }

        protected override void OnStart(string[] args)
        {
            _logger.LogDebug(nameof(OnStart));
            Task.Run(async () => await StartInternally());
        }

        internal async Task StartInternally()
        {
            _logger.LogDebug(nameof(StartInternally));
            try
            {
                AppDomain.CurrentDomain.UnhandledException += (sender, eventArgs) =>
                {
                    _logger.LogError(eventArgs.ExceptionObject as Exception, "Unexpected Error");
                };
                int processCount = _configurationReader.Read<int>("ProcessCount");
                var readerPath = _configurationReader.Read<string>("ReaderPath");
                _readerProcessManager.Initialize(processCount, readerPath);
                _readerProcessManager.Start();

                while (!_cancellationTokenSource.IsCancellationRequested)
                {
                    _readerProcessManager.KillNotRespondingProcesses();
                    _readerProcessManager.KeepProcessesAlive();
                    await Task.Delay(1000);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Service OnStart failed.");
                throw;
            }
        }
        protected override void OnStop()
        {
            _logger.LogDebug(nameof(OnStop));
            try
            {
                _readerProcessManager.Stop();
                _cancellationTokenSource.Cancel();
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to execute OnStop");
            }
            finally
            {
                base.OnStop();
            }
        }
    }
}

There is a certain procedure that is necessary to be followed to actually install or uninstall a windows service. On the executing machine nobody should use the SYSTEM user for a service due to security considerations. But the code itself is still pretty easy.

Why do I think that infrastructure is going to be more important than coding?

Considering the time that I needed for implementation, latest in that point in time, trying to find the right infrastructure was more time consuming. And I didn’t even start to automate the infrastructure with Terraform and Ansible.

The windows service will just be a keeper service. It will not read on its own. Rather than this it is responsible for starting executables for reading data from OsiSoft. Each executable can read 1-n messages in parallel. In this way I have two different ways of adjusting the vertical scaling.

Again coding was easy. But deployment?

Let me put it like this: At my company it is not possible to automatically provision an VM and deploy code on it from Azure DevOps. VMs are only allowed as Build machines. This is not what I want.

Anyway, the code and the approach works. Let me go even deeper in the next article.