Part 2: Let’s go with containers

This is the second of six articles about how an IoT project with certain constraints and prerequisites was implemented. This time it is about trying to create windows containers with a massive setup of a proprietary piece of software.

• Part 1: Analysis & Design
• Part 2: (this one) Let’s go with containers
• Part 3: What about alternative deployment options?
• Part 4: Going on with alternative deployment options
• Part 5: Final architecture & learnings

How to host that?

Sure containers as hyped as useful. Putting everything in containers seems to be quite natural. Why sticking to anything else? Let’s have a look onto the advantages though:

• Portability between different platforms and clouds—it’s truly write once, run anywhere.
• Efficiency through using far fewer resources than VMs and delivering higher utilization of compute resources
• Agility that allows developers to integrate with their existing DevOps environment.
• Higher speed in the delivery of enhancements. Containerizing monolithic applications using microservices helps development teams create functionality with its own life cycle and scaling policies.
• Improved security by isolating applications from the host system and from each other.
• Faster app start-up and easier scaling.
• Flexibility to work on virtualized infrastructures or on bare metal servers
• Easier management since install, upgrade, and rollback processes are built into the Kubernetes platform.

That all sounds great. The first point is actually partly untrue at least for windows containers, there are prerequisites. Anyway, let’s have a look onto Azure services and pick the right one for our task. What is available for hosting, deployment, maintenance of containers and which one is the best choice?

Hosting choices on Azure for containers

AKS: Does it make sense to use AKS for at least three types of containers? Even if it is necessary to have a lot of instances, it seems to be some kind of overengineering. As I cannot estimate how many containers instances I need, I am unsure if the complexity of an AKS is going to pay out.

App Service: The only container instance that probably could have an http endpoint is the scheduler. Reader and Writer are unattended and work via messages, so no need. Actually it is possible to host containers, but if you don’t have an http endpoint, health probes will fail. That means, the container is going to be restarted as App Service “thinks” the container is unresponsive. This is something I learned later.

Container Instances: Had a short look. This sounds promising esp. due to the early stage of implementation. With a container registry, automatic deployment is possible, I do not need to care about network, ingest, security within an AKS. So let’s go with this service.

Batch: Actually I didn’t see this service when I started the implementation. Would have been a good idea at least to replace the scheduler. Not sure about the compute capabilities. I’ll have a look onto this service in a later project.

Service Fabric: I used Service Fabric some years before. Still 30% of all Azure services run on Service Fabric, so don’t underestimate the power and flexibility of this framework. You know, great power comes with great responsibility. Service Fabric is as AKS too complex for this simple solution. For production system, at least 5 nodes are necessary, definition and running services is specific and need a lot of configuration.

Container Registry: Sure, I need it :-).

Open Shift: Always wanted to try OpenShift, but for this project it doesn’t seem to be the right choice to go into “testing mode”. Actually customer pays and this kind of eval is not included, unluckily.

Windows containers, pure fun

For creation of a windows container, I need a windows environment. I do work on a mac, so I got a VM in our data center for doing. This is Windows Server 2016. Is that a good idea? I am not sure.

Let’s start with the more simple Docker file definition. asp.net core can run pretty much everywhere and it isn’t as large as the actual windows server core image. Looks like this.

# Indicates that the windowsservercore image will be used as the base image.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1

# Metadata indicating an image maintainer.
LABEL maintainer="holger.leichsenring@henkel.com"

EXPOSE 80
# copy binaries
COPY ".\bin\Debug\netcoreapp3.1\publish" "c:\app"

WORKDIR "C:\app"

CMD [ "Henkel.Osi2Hdf.Writer.exe" ]

Okay, happy to start it up. What’s this error message? Ah, come on.

C:\Windows\system32>docker pull mcr.microsoft.com/windows/servercore:1607 1607: 
Pulling from windows/servercore 3889bb8d808b: Extracting [==================================================>]   4.07GB/4.07GB 
846a2223e9e7: Download complete failed to register layer: rename
 C:\ProgramData\docker\image\windowsfilter\layerdb\tmp\write-set-148328921
 C:\ProgramData\docker\image\windowsfilter\layerdb\sha256\f358be10862ccbc329638b9e10b3d497dd7cd28b0e8c7931b4a545c88d7f7cd6: Access is denied.

The first time I started with Docker on Windows is long ago. I remembered having no fun at all. Have a look on Docker containers on Windows Server 2016 with Containers: HNS failed with error: Unspecified error orDocker containers on Windows 10: HNS failed with error : Unspecified error. Experiencing these issue had been on a physical machine, but that doesn’t make too much difference. I tried the same solutions, firstly, switch off the virus scan. Working in a large company means, this can take days for a small time window where I can check if it is the cause.

Switching off the virus scan solved the problem with downloading, but still it was not possible to create docker images. It always failed somewhere in the middle with Access Denied error messages as well. What does it mean?

Windows Server 2016 is unreliable for creating Docker images

Lots of things are processualized within large companies. Ugrading the image took days. But here we go, I finally was able to create the image for the Writer component. All fine.

.net Framework, Containers and a massive setup

Pretty optimistic me, I searched the internet and thought:

Come on, cannot be that hard to create a docker image for a .net framework lib!

I did what all guys do: I used my Google Fu to find the right thing. I thought I was successful, when I saw a docker image definition for the AFSDK lib of OSISoft. This is how the Dockerfile looks like.

# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference ='SilentlyContinue';"]
RUN New-Item "C:/PI" -ItemType Directory
COPY ./AFClient.zip C:/PI/
RUN Expand-Archive C:/PI/AFClient.zip C:/PI/AFClient/
COPY ./kst.ini C:/PI/AFClient/
CMD powershell

Uh, pretty scary. Powershell. Setups. Ah, come on. Will work. In his explanation the guy stated, this is what is going to be needed:

Make(run) a Container and execute powershell on it.Then,
 

PS>cd C:\PI\AFClient
PS>Start-Process "Setup.exe" "-f silent.ini" -PassThru | Wait-Process

Wait a few minutes. After Installation finished, Test the connection to PIDA.
 
PS>[System.Reflection.Assembly]::LoadWithPartialName("OSISoft.AFSDK")
#AFSDK information will be displayed
PS>$PI=New-Object OSIsoft.AF.PI.PIServers
PS>$PI.Connect()
PS>$PI.DefaultPIServer

To be honest, that sounds odd. Weird. I’ll find more adjectives.

Anyway, I need to have it in place, so I followed instructions. Having a windows 2019 server at least creation of docker images worked. And then I spent a lot of time on running this setup. It is not my perferred way of creation of Docker containers to have

• a zip file that needs to be copied
• then unzipped
• installed via weird setup.ini file that is as long as the Bible
• not seeing anything as a result in terminal. Setup just finishes with “success” or “failed”.
• It is necessary to open the log to get the details.

Executing windows setups is pure love inside of docker containers (warning: Ironic mode)

Isn’t it funny when a setup tells you, it wants to restart the “machine” when being installed in a docker container?

Have a look onto the setup.ini

The setup needs to run silently. In Docker containers I wont be able to accept dialogs so double check the ini file. Let’s have a look. It’s massive.

;---------------------------------------------------------------------------------------------
; SILENT INSTALLATION INSTRUCTIONS
;
; To run a silent installation, use the -f option with Setup.exe:
;
;     setup.exe -f silent.ini
;
; Prior to running the silent install, update the COMMANDLINE section listed below as needed.
; Refer to the CUSTOMIZATION OPTIONS section for a description of the properties that can be set.
;
; One of the customization options allows you to configure a separate KST_INI_FILE (refer to section 'CUSTOMIZATION OPTIONS' below for additional information). 
; The structure of KST_INI_FILE is shown below (only NUM_SERVERS and PI_SERVER are required):
; [NUM_SERVERS]
; NUM = ##
; [PI_SERVER]
; # = ServerName
; [PI_PATH]
; # = ServerPath.osi.com - defaults to ServerName
; [PI_PORT]
; # = 5450 - defaults to 5450 for PI3, 545 for PI2
; [PI_USER]
; # = piadmin - defaults to pidemo
; [PI_DEFAULT_SERVER]
; # = TRUE - should only be one server which is default
; [PI_SERVER_ID]
; # = xxxxxxxxxx - not needed
; [PI_TYPE]
; # = 3 - defaults to PI3 (2, 3, PI2, or PI3 allowed)
; [PI_ALIAS]
; # = AliasName - creates alias if specified
;---------------------------------------------------------------------------------------------
[SETUPKIT]
NAME = SetupAFClient2_x
DISPLAYNAME = PI AF Client 2018 SP3 Patch 1
SHOWSKIPPEDMODULES = FALSE
FEATUREUPGRADE = FALSE
; These suppress the setup wrapper dialogs, not those for the individual setup modules
SUPPRESSCOMPLETIONMESSAGE = FALSE
SUPPRESSPROGRESSMESSAGE = FALSE
SUPPRESSHEADERMESSAGE = TRUE
SUPPRESSDIALOGS = TRUE
[NUMSETUPMODULES]
NUM = 13
[SETUPMODULES]
1 = dotnet48Setup
;   dotnet48Setup required by AFClient
2 = PreInstallTasks
;   PreInstallTasks required by pibufss.msi and pibufss_x64.msi
3 = MSVC9SP1x86Redistrib
4 = MSVC9SP1x64Redistrib
;   MSRuntimes VS2008 SP1 required by: PIBufss (via the piapi). for x64 installations, only need the x64 runtimes.
5 = MSRuntimesVS2019_x86
6 = MSRuntimesVS2019_x64
;   MSRuntimes VS2019 (32-bit and 64-bit) required by: MDA components, PIBufss (32 bit and 64 bit), and PINS utilities and dll's: pictrdll.dll, pigetmsg.exe, piconfig.exe, pidiag.exe
7 = PINS
; PINS or PISDK install is required by: PIBuffss, which is an optional component for AF Client.  PINS is preferred over PI SDK unless you have a specific need for PI SDK.
; If PI SDK is needed for your application, then include both the PI SDK 32-bit and 64-bit installation kits here, and include the VS2015 Update1 (32 bit and 64 bit) redist kits.
; Re-number the sections below this line, and update [NUMSETUPMODULES], [UPGRADECOMMANDLINE], and [UNSUPPORTED_OPERATING_SYSTEMS] sections.
8 = PIBufss.msi
9 = PIBufss_x64.msi
;   PIBufss installs are optional for AFClient.  Install if you require buffering.  Buffering is recommended when writing to HA Data Archives.
10 = AFClient_Net35
; AFClient_Net35_Check to check if the .net 3.5 sdk feature was installed
11 = AFClient_x86.msi
12 = AFClient_x64.msi
;  the AFClient installation
13 = VSTOWrapper
;   VSTOWrapper Required by PIBuilder in AFClient.
[COMMANDLINE]
; dotnet48Setup - typical installation of .net 4.8.0 will require a reboot. if you prefer the silent install to not reboot then add the /norestart option.
1 = /q
; PreInstallTasks  (Must come after DotNet 4.8 installation)
2 = 
; MSRuntimes VS2008 SP1 (32-bit and 64-bit) 
3 = /q /norestart
4 = /q /norestart
; MSRuntimes VS2019 (32-bit and 64-bit)
5 = /q /norestart
6 = /q /norestart
; PINS (32-bit / 64 bit support)
7 = ALLUSERS=1 REBOOT=Suppress /qn
; If PI SDK is needed for your application, use the following cmdline options for both 32 bit and 64 bit kits:  
; = ALLUSERS=1 REBOOT=Suppress NOPISDKBUFFERING=1 /qn

; PI Buffer Subsystem (32-bit and 64 bit)
8 = ALLUSERS=1 REBOOT=Suppress CLEANUPKST=0 /qn 
9 = ALLUSERS=1 REBOOT=Suppress CLEANUPKST=0 /qn
10 = SKIPFULLAFSDKCHECK=1 ALLUSERS=1 REBOOT=Suppress /qn
; AFClient (32-bit or 64-bit) CUSTOMIZATION OPTIONS  
;  AF_SERVER=machineName   Defines default Asset Server (new install only)  
;
;  Configuration options used to add a single PI Data Servers to the Known Servers Table:
;  PI_SERVER=machineName   Defines default Data Server 
;  PI_USER=pidemo  
;  PI_ALIAS=ServerNameAlias
;
;  Configuration options used to add multiple PI Data Servers to the Known Servers Table:
;  IMPORT_KST=1                 Set to 1 to configure multiple PI Data Servers.
;  KST_INI_FILE="c:\kst.ini"    Specify path to an ini file that defines the list of PI Data Servers to be added to the KST. 
;
; Examples:  (single server)    -->  13 = /qn REBOOT=Suppress ALLUSERS=1 INSTALLDIR="C:\Program Files\PIPC" PI_USER=pidemo PI_SERVER=ServerName PI_ALIAS=ServerNameAlias
;            (kst.ini import)   -->  13 = /qn REBOOT=Suppress ALLUSERS=1 INSTALLDIR="C:\Program Files\PIPC" IMPORT_KST=1 KST_INI_FILE="c:\kst.ini"
;
;  REMOVE_KST=1  This option will permanently delete the Known Servers Table (KST), and should only be set if other PI Products that use the KST are not installed.
;
;  NOPISDKBUFFERING=1 Specify this option to update the PISDK buffering setting. If the value is not specified then no changes are made to users configuration.
;
;  ONLYSHOWSERVER=1  Specifies the install to suppress all dialogs except DLG_DefaultServer_Info. (default 0 does nothing)
;
;  AFSDKONLY=1 Specifies the install to ONLY install FD_AFSDK on a clean install, or upgrade existing components if AFClient already installed.
;
;  SENDTELEMETRY=1   Specifies participation in the PI System Customer Feedback Program if set to 1. Default value during silent installation is 0.
;
;  SHUTDOWN_OPTIONS=2   Specifies for silent install to not shutdown services and applications that have the AFSDK.dll file locked and in use at the time of the upgrade. 
;                       The default behavior is to shutdown and restart applications that have the AFSDK.dll file in use to prevent user from having to reboot box at end of installation.
;	                     If SHUTDOWN_OPTIONS=2 and this was an upgrade and the AFSDK.dll file was in use, then an automatic reboot will occur unless the REBOOT=Suppress option is set.
;
;  SKIPLAUNCHCONDITIONS=1 Set this option to bypass verification of pre-requistes and allow the msi to install even when a condition is not met. Setting this value is NOT a recommended installation.
;  
;  ADDLOCAL  Following list of features are possible, property is case sensitive and does not allow space:
;		FD_AFSDK				Installs the AFSDK and AFRegPlugIn; it is required
;		FD_AFExplorer			Installs PI System Explorer, AFImport, and AFExport
;		FD_AFBuilder			Installs the PI Builder add-in to Excel
;		FD_AFDocs				Installs the help files 
;		FD_AFAnalysisMgmt		Installs the Management UI components. The default behavior is to not install this feature.
;		FD_AFCollectiveManager	Installs the AF Collective manager. The default behavior is to not install this feature. 
;		ALL						Installs all features
;  
11 = /qn REBOOT=Suppress ALLUSERS=1 ADDLOCAL=FD_AFSDK,FD_AFExplorer,FD_AFBuilder,FD_AFDocs CLEANUPKST=0
12 = /qn REBOOT=Suppress ALLUSERS=1 ADDLOCAL=FD_AFSDK,FD_AFExplorer,FD_AFBuilder,FD_AFDocs CLEANUPKST=0
; VSTOWrapper
13 = /q /norestart
[UPGRADECOMMANDLINE]
; If PI SDK is needed for your application, use the following upgrade command line options for both 32 bit and 64 bit kits: 
; = REINSTALL=ALL REINSTALLMODE=vemus
[DISPLAYNAME]
3 = Microsoft Visual C++ 2008 Redistributable x86
4 = Microsoft Visual C++ 2008 Redistributable x64
[VERSION]
[CHECKFORINSTALLEDONLY]
[UNSUPPORTED_OPERATING_SYSTEMS]
;-----------------------------------------------------------------------------------------------------------
; Defined Operating Systems
; (Win95, Win98, WinME, WinNT, Win2K,  WinXP, WinNET, VistaNoSp, VistaSp1, Win2008NoSp, Win2008Sp1, Win2008R2NoSp)
;
; If Operating System listed with NO SP, then not supported on that OS regardless of SP version installed.
; If Operating System listed with SP, then that particular OS has a service pack requirement. 
;
; A 0 entry denotes that the setup can not run on the given operating systems.
; Entries that corresponds to an entry in the [SETUPMODULES] section denotes
; that the particular setup module will be skipped on the given operating system
;-----------------------------------------------------------------------------------------------------------
0 = Win95, Win98, WinME, WinNT, Win2K, WinXP, WinNET, Vista, Win2008NoSp, Win2008Sp1, Win2008Sp2, Win2008SP2_X64_CORE, Win7NoSp, Win7Sp1, Win2008R2NoSp, Win2008R2Sp1, Win2008R2SP1_X64_CORE, Win8NoSp, Win8Sp1, Win8Sp2
2 = CORE
3 = X64
4 = X86
6 = X86
8 = X64
9 = X86
11 = X64
12 = X86
;MSI's or Setup wrapper Dll's can write to this section to dynamically change 
;the command line for an MSI at setup runtime.
[RUNTIME_PARAMETERS]
;This section is used by the pre-install tasks and post-install tasks.
[PIBUFSS]

Are these guys kidding? Millions of options. Actually I could have used that face from Jim Carrey all over the place. Sometimes it is good that webcam doesn’t capture the face when doing this weird stuff.

At least some comments in that file! You can imagine, I checked a lot of variations:

• which ones are necessary for the program to work correctly?
• how can I prevent the setup’s demand for restarts?
• where are restarts probably necessary? This is maybe necessary for installing .net 4.8. So get the right image and skip this installation… and so on.

Finally I arrived with this definition of the setup.ini:

[SETUPKIT]
NAME = SetupAFClient2_x
DISPLAYNAME = PI AF Client 
SHOWSKIPPEDMODULES = FALSE
FEATUREUPGRADE = FALSE
SUPPRESSCOMPLETIONMESSAGE = TRUE
SUPPRESSPROGRESSMESSAGE = TRUE
SUPPRESSHEADERMESSAGE = TRUE
SUPPRESSDIALOGS = TRUE

[NUMSETUPMODULES]
NUM = 7

[SETUPMODULES]
# 1 = PreInstallTasks
# 2 = MSVC9SP1x64Redistrib
# 3 = MSRuntimesVS2015u2_x64
# 3 = PINS
# 4 = AFClient_x64.msi
# 5 = PIBufss_x64.msi
# 6 = VSTOWrapper
# 1 = dotnet48Setup

1 = PreInstallTasks
2 = MSVC9SP1x64Redistrib
3 = MSRuntimesVS2019_x64
4 = PINS
5 = AFClient_Net35
6 = AFClient_x64.msi
7 = VSTOWrapper

[COMMANDLINE]
# 1 = 
# 2 = /q /norestart
# 3 = /q /norestart
# 3 = ALLUSERS=1 REBOOT=Suppress /qn
# 4 = /qn REBOOT=Suppress ALLUSERS=1 ADDLOCAL=FD_AFSDK,FD_AFExplorer,FD_AFBuilder,FD_AFDocs IMPORT_KST=1 KST_INI_FILE="C:\PI\AFclient\kst.ini"
# 5 = ALLUSERS=1 REBOOT=Suppress /qn
# 6 = /q /norestart

# 1 = /q
1 = 
2 = /q /norestart
3 = /q /norestart
4 = ALLUSERS=1 REBOOT=Suppress /qn
5 = SKIPFULLAFSDKCHECK=1 ALLUSERS=1 REBOOT=Suppress /qn
6 = /qn REBOOT=Suppress ALLUSERS=1 ADDLOCAL=FD_AFSDK,FD_AFExplorer,FD_AFBuilder,FD_AFDocs CLEANUPKST=0
7 = /q /norestart

[UPGRADECOMMANDLINE]
[DISPLAYNAME]
[VERSION]
[CHECKFORINSTALLEDONLY]
[UNSUPPORTED_OPERATING_SYSTEMS]
[RUNTIME_PARAMETERS]
[PIBUFSS]

Giving up

I really spent a lot of time.

Implementing the functionality was done in 1.5 weeks. Doing the infrastructure automation took already two days and was completely unsatisfying.

I didn’t make it. There was always a weird error message in setup logs, that was not understandable. I had been pretty creative. Lots of variations. When it takes to much time…

When consuming too much time for a task it looks like you need help

So I contacted the support of OSISoft asking for a docker image from the provider. This was pretty interesting and more a political than a technical issue. OSISoft already has a REST-based Http interface in place that can be used for the same reasons. And I would have loved to use that. But it generates a lot of costs, so it was not feasible to switch to http. God, this would have been eased everything, also the implementation. Anyway.

I always tough times, when I need to confess that something doesn’t work. I tend to spend even more time. Try to kill the issue with more time and effort. Sometimes that does work, but mostly it doesn’t. And then I even spend more time.

It is like to spend money on a project that it is known it is doomed to fail but because there had been an investment it is hard to stop

Okay, change direction. Docker will not work with .net Framework and this massive old-school setup. Let’s have another look onto different options in the next article.

How can .net framework be hosted?

• Cloud Services
• Service Fabric
• Windows Service on a VM

All not the options I wanted, but life’s hard. Follow me to the next one. Even darker.