Indicium deployment

Indicium is a .NET Core application which means that it is capable of hosting itself with a built-in web server. However, for security and management reasons, we do not recommend exposing the built-in web server in production environments. Instead, we recommend hosting Indicium behind an IIS proxy, since IIS is rich in features and offers great management tools. This chapter will explain how Indicium can be integrated in an IIS proxy.

note

When Automated Machine Learning is used in the environment, an additional component needs to be installed and configured. More can be found here.

About Indicium

Two types of the Thinkwise Indicium Application Tier are available:

• Indicium Basic: for use with the Windows GUI and Mobile GUI. This basic version does not support, for example, system flows and OpenID.
• Indicium: for use with the Universal GUI and via API's. This version uses the full range of Indicium functionality.

Quick reference guide Indicium deployment

Instead of using the Deployment Center, this document describes how to deploy Indicium manually (on premise).

To deploy Indicium in the cloud:

• Microsoft Azure
• Amazon AWS.

Follow these steps to manually deploy Indicium on premise (using IIS):
1. Check - Before you start, check if your systems meet the prerequisites.
2. Download - Download Indicium and save the files.
3. Application pool - Create an application pool in IIS.
4. Web application - Create the web application.
If necessary: Troubleshooting or Monitoring Indicium live.

For a video explanation on how to manually deploy Indicium (including troubleshooting), see: Masterclass Indicium. It is an extensive video that also covers:

• the deployment of Universal GUI (1:13:00)
• the deployment of Windows GUI (1:19:00)
• API's in Indicium (1:28:00).

Prerequisites

A number of prerequisites needs to be met before Indicium can be installed on IIS. This chapter describes all of the necessary steps.

Indicium

Required	More information
IIS	Check if IIS has been installed. See: Configuring IIS.
.NET 6	Indicium is built using ASP.NET Core 6. As such the ASP.NET Core 6 runtime must be installed on the web server. If you also need to run Indicium on IIS, you must install the ASP.NET Core Runtime 6 Hosting Bundle: https://dotnet.microsoft.com/en-us/download/dotnet/6.0. This module enables IIS to take control of starting and stopping Indicium, amongst other things. To check if the module has already been installed, open Modules on a website in the IIS Manager.
URL Rewrite Module	Finally, it is necessary to install Microsoft's URL Rewrite Module for IIS. URL rewriting is a feature that is not available in IIS by default, but it is necessary to facilitate the integration of Indicium. The availability of this module can also be checked in the IIS Manager, much like the previous module. If the URL Rewrite module has not been installed yet, then use the link below to install it: https://www.iis.net/downloads/microsoft/url-rewrite.

Optional	More information
.NET Framework	Some features of Indicium currently cannot be ported to run on .NET 6. Therefore, these features are run as plugins in separate .NET Framework processes. The minimum .NET Framework version to run the plugin processes is 4.7.2. To check which version has been installed: https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/how-to-determine-which-versions-are-installed. If the web server has no .NET Framework or a lower version, then use the following link to install the latest version: https://www.microsoft.com/net/download/thank-you/net472. This affects the following features: Crystal Reports SSRS Reporting DevExpress Reporting
DB2 Core driver	See: IBM i DB2 drivers for the Universal GUI with Indicium Note: Using DB2 with any of the reporting types mentioned in the .NET Framework section above, still require the old iSeries DB2 driver.

Indicium Basic and builds older than 2021.1.17

Required	More information
IIS	Check if IIS has been installed. See: Configuring IIS.
.NET Framework	Indicium is built with version 4.7.2 of the .NET Framework. As such, this is the minimum version of the .NET Framework that needs to be installed on the web server. To check which version has been installed: https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/how-to-determine-which-versions-are-installed. If the web server has no .NET Framework or a lower version, then use the following link to install the latest version: https://www.microsoft.com/net/download/thank-you/net472.
.NET Core Windows Server Hosting Module	To facilitate the integration of Indicium in IIS, it is necessary to install Microsoft's .NET Core Windows Server Hosting Module on the web server. This module enables IIS to take control of starting and stopping Indicium, amongst other things. To check if the module has already been installed, open Modules on a website in the IIS Manager. If the module has not been installed yet, then use this link to install it: https://dotnet.microsoft.com/download/dotnet/thank-you/runtime-aspnetcore-5.0.4-windows-hosting-bundle-installer.
URL Rewrite Module	Finally, it is necessary to install Microsoft's URL Rewrite Module for IIS. URL rewriting is a feature that is not available in IIS by default, but it is necessary to facilitate the integration of Indicium. The availability of this module can also be checked in the IIS Manager, much like the previous module. If the URL Rewrite module has not been installed yet, then use the link below to install it: https://www.iis.net/downloads/microsoft/url-rewrite.

Download and save Indicium

1. When all prerequisites mentioned above are met, you can download the Indicium software in TCP - Thinkwise products and save it.

2. If necessary, remove the security block in the zip file's Properties. (This won't be necessary if you use 7-ZIP.)

3. Then extract the files.

4. In IIS, right-click on the site where you wish to install Indicium and select Explore. Typically, this location will be something like C:\inetpub\wwwroot\indicium.

5. Add a folder and give it a name, eg 'indicium_universal'.

6. Then copy all the extracted files to this folder.

Create an Application Pool

Create in IIS a separate application pool for each Indicium: right click Application Pools and choose Add Application Pool.

warning

Always create a separate pool for each Indicium. Several Indiciums in one pool won't work. Moreover, it ensures that each Indicium can operate separately from the others and can be independently stopped and restarted.

• Name: name the Application Pool the same as the folder you created (eg, as in the example above: 'indicium_universal').

• .NET CLR version: Indicium is capable of hosting itself and IIS is merely used as a proxy to expose Indicium to the network and/or the internet. As such, all the heavy lifting is performed by Indicium's own web server and the Application Pool in IIS does not even need to initialize a .NET runtime. Because of this, select No Managed Code when creating an Application Pool for Indicium, as it will positively affect the start up time.

Configuring pool users

warning

When configuring the Database Pool User in appsettings.json, you can only use database accounts and no domain accounts. Domain accounts in appsettings.json are not supported because they can cause severe performance degradation and are considered a potentially dangerous security breach.

1. After creating the application pool in IIS, open the Advanced Settings (under the Actions menu) of the application pool.

2. Now configure an Identity (click the ...). This is the Application Pool Identity. Use for example a dedicated service account. The identity requires the following permissions:

   • Full access (read, write, delete, list, execute) to the Indicium folder so, for example, logs and caches can be written.
   • Read/write/modify control for all file system storage locations so files can be written, read, and deleted.

What these disk locations are depends on where the application stores its images and other files, and to which locations the application can upload files.

Indicium handles authentication and authorization internally and will perform all database traffic with a single user, the Database Pool User. By default, the Application Pool Identity (as set in IIS) is used as Database Pool User. To deviate from this, you can choose to override it by adding the PoolUserName and PoolPassword properties in the appsettings.json file.

{
  "MetaSourceConnection": {
    "PoolUserName": "[username]",
    "PoolPassword": "[password]"
  }
}

Make sure the Database Pool User only has the minimal required permissions:

• It should only facilitate the communication with the database.
• It needs full access to all the databases present in IAM, including the IAM database itself. Therefore, it should have the role 'db_owner' for the IAM database and all end application databases.
• There is no need to add the Database Pool User as a user to IAM, since it does not need to log in to the end application.

None of the end users accessing the applications requires any physical permissions on any database, as this is regulated by the pool user.

Idle time-out

In IIS, ensure in the application pool's Advanced Settings that the Idle Time-out (minutes) is set to 0 (disabled) instead of 20, so scheduled process flows will continue running even if there is no user activity.

Start mode

Furthermore, check in IIS in the application pool's Advanced Settings that the StartMode is set to AlwaysRunning instead of OnDemand. This will ensure Indicium will be active even if no web requests have been made yet.

warning

If you change the StartMode setting to AlwaysRunning, you must enable the Application Initialization feature.

Create the Web Application

1. Next, open the appsettings.json file (as Administrator) with a text editor to configure the connection to the IAM database.

   For more information about PoolUserName and PoolPassword and whether you should use it or not, see above.

   note

   Double quotes (") and backslashes (\) in the appsettings.json file, for instance in the server address, need to be escaped by an extra backslash. For example: server\instance should be server\\instance.

   {
     "Logging": {
       "IncludeScopes": false,
       "LogLevel": {
         "Default": "Information",
         "System": "Information",
         "Microsoft": "Warning",
         "Indicium": "Debug"
       }
     },
   
     "MetaSourceConnection": {
       "Server": "[server]",
       "Database": "[iam_database]",
       "PoolUserName": "[username]",
       "PoolPassword": "[password]"
     }
   }

2. Now create a new Web Application: right click on the folder (eg indicium_universal) and select Convert to application.

3. Choose an Alias, the created Application Pool and the Physical path to Indicium.

   

4. The Web Application will attempt to listen on port 80 by default. If this port is already in use, it is necessary to choose another port. This can be done in IIS by right-clicking Default Web Site and choosing Edit bindings.

5. In addition to the AlwaysRunning setting on the application pool, ensure Indicium is activated properly after the application pool has been (re)started. This can be done in the Advanced settings of the Indicium web application. Enable the Preload option.

   Note that the Preload feature of IIS will not work correctly if the website only accepts traffic via https, as the preload request is fired on http. A resolution for this problem can be found here.

   

6. When done, the Web Application can be started to verify that Indicium is running and configured correctly. This is possible in two ways:

   • Open a browser and navigate to https://server/indicium/.
   • Or select the application in IIS and choose Browse :80 (http) (depending on the set port).

   

note

If you do not get a result like the image above, then please refer to Troubleshooting issues below for steps to resolve your problem.

Troubleshooting Indicium deployment

Startup errors

If the root URL of Indicium (eg https://server/indicium/) gives an error, then it will most likely state "An error occurred while starting the application" and/or show error code 502.21. With version 2.0.6+ of the AspNetCoreModuleV2 for IIS, the error page should include the cause of the error, which is typically informative enough to point you in the right direction. If not, then follow the steps below in their stated order.

note

When Indicium encounters a database connection failure during start-up, a retry occurs every two minutes for up to 10 minutes. The cause of the connection failure can be a temporary issue with the database server or network that resolves itself.

Database connection and version mismatch

Check Indicium's Logs folder for more information about what could be wrong:

• Incorrect version of Indicium or IAM.
• Incorrect database connection settings. Ensure that the information in the appsettings.json is correct. Double-check the server address and the database. See: Create the web application.
• Database Pool User lacks permission. Ensure that the Database Pool User has access to the configured IAM database and product databases. See: Configuring pool users.

Configuration and system requirements issues

Check Indicium's Logs folder for more information about what could be wrong.

If it's empty: start Indicium.exe (from the root folder) as the Database Pool User (make sure that it's not overruled by the Application Pool User in the appsettings.json). See if it runs successfully or displays an error. If so, make a screen shot because it disappears quickly.

• Syntax error in appsettings.json. Ensure that the JSON in the appsettings.json is valid. The most common mistakes are forgetting to escape backslashes , either in the server address or the pool username, or forgetting a comma after a property which is not the last one.

  

  If you want to be certain that the JSON is valid, you could use this website to validate it (don't forget to clear any sensitive information first): https://jsonlint.com/.

• Not having the necessary version of the .NET Framework or .NET Core runtime installed. Double-check if your system meets the required versions as mentioned in Prerequisites.

• Indicium's files are blocked by file security properties.

IIS hosting issues

• Not having the necessary version of the .NET Core Windows Hosting bundle installed. Double-check if your system meets the required versions as mentioned in Prerequisites.

• Not having the IIS Rewrite module installed. Double-check if your system meets the required versions as mentioned in Prerequisites.

• The Application Pool User has no access to Indicium's folder. See: Configuring pool users.

• The Application Pool user has no access to the certificate store. The following error indicates insufficient access rights to the windows certificate store for the application pool identity:

  System.Security.Cryptography.CryptographicException: The system cannot find the file specified.

  Right click on your Application Pool, open the Advanced Settings and make sure that the correct Identity is set. Note that Load User Profile should also be set to True.

• Multiple applications on one Application Pool (Indicium). Make sure you use a separate Application Pool for each Indicium.

Log files

If none of the above-mentioned solutions solves Indicium's startup problems, gather the log files in the Logs folder in the root of Indicium and send them to your contact at Thinkwise to receive further assistance.

Login failed

Full message: "Login failed. Please contact your system administrator".

The credentials are correct but the corresponding user in IAM is probably configured wrongly:

• Check if IAM contains a user entry corresponding to the user's email (menu Authorization > Users).
• Check in IAM if the corresponding user is configured with the External authentication type (menu Authorization > Users > tab Form > tab User > field Authentication type).

401 - Unauthorized

If you are getting "401 - Unauthorized" results for a user (i.e. if the browser keeps prompting you to log in) and you are certain that the specified credentials are correct, then check if the authentication method of the user is set correctly.

404 - Application not found

If an application URL (e.g. https://server/indicium/iam/myAppl/) gives the result "404 - Application not found", and you are certain that the URL is correct, ensure that the user that you are using to log in as has access to the application in IAM.

When accessing an application present in the Software Factory (e.g. https://server/indicium/sf/myAppl/), ensure the user logged on to Indicium has this application activated.

Monitoring Indicium live

Indicium is capable of logging automatically to Azure Application Insights. This service can be used for monitoring Indicium live. You can, for example, inspect failed requests, monitor response times, or investigate exceptions.

Read the example below to learn how Azure Application Insights can help you in the development process. Especially when you don't have direct access to Indicium's error logs, having such logging capabilities is extremely useful. But even when you have direct access to the log files, the extra features offered by Azure Application Insights make it worth enabling it.

Setting up Azure Application Insights

This feature is enabled by default, but to use it, you need a paid Azure account with Application Insights enabled as a resource.

Once this has been set up in Azure, the next step is to find your Application Insights instrumentation key and add it to Indicium's appsettings.json.

Azure Application Insights instrumentation key

{
  "ApplicationInsights": {
    "InstrumentationKey": "actualinstrumentationkey"
  }
}

Similar to the standard logging, you can control the logging levels by adding the following to the appsettings.json file:

{
  "Logging": {
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information",
        "System": "Information",
        "Microsoft": "Warning",
        "Indicium": "Debug"
      }
    }
  }
}

More information about the different logging levels can be found here.

Using Azure Application Insights

This paragraph gives an example of how to use Azure Application Insights to inspect failed requests.

1. Navigate to your Azure Application Insights resource. The first screen you will see is the overview. This overview features several graphs about your application's performance, as you can see in the next image.

Azure Application Insights overview

As you can see, there were some failed requests. Application Insights can help to investigate these failures.

2. Click on the Failed Requests graph to open the following screen:

Azure Application Insights failed requests

This screen gives an overview of all failed requests within a specific time interval, the last 30 minutes in this example. The request with the highest number of failures seems to be a GET request for a badge on generate_sales_invoice.

3. Click on the request to get more detailed information.

Azure Application Insights badge request

In this case, Indicium is returning a 401 Unauthorized response code for this resource.

Something else that might be worth investigating is the logged SqlException.

4. Navigate to the Exception tab and click on the exception message on the right of your screen. The following screen gives highly detailed information about this exception.

Azure Application Insights exception

note

This screen shows one of the advantages of debugging with Application Insights. Not only the SqlException is shown with a full stack trace, but it also gives an overview over time about what has led to the actual exception.

In this example, the exception is preceded by a GET request on the employee_hierarchy resource. Apparently, while processing the request, some piece of SQL code is giving a maximum recursion error.

With this information, you now know that there must be a problem in a SQL template, and as it turns out, the problem is caused by the employee_hierarchy view. This view is using a recursive cte that seems to end up in an infinite loop.

Measure Server-Timing responses

Indicium

You can view several performance metrics for requests using Server-Timing response headers. Browser developer tools can interpret these headers and visualize the performance. Performance metrics can provide a detailed insight and help solve potential performance issues, often caused by specific queries, layout procedures, or file storage operations. The available performance metrics are:

CRUD operations

• All SELECT queries for requested subjects of an application (tables and views).
• When the $count operation requests the total number of records, this will be measured and returned separately.
• When the $expand operation joins a lookup or a detail subject, these will each be measured and returned separately.
• Insert statements.
• Update statements.
• Delete statements.
• Expression queries while adding or editing a record.
• Default value queries when entering edit mode to add a new record or start a task or report.
• Lookup validations when changing a lookup value or a lookup filter value.

Application logic

• Tasks.
• Subroutines.
• Default procedures.
• Layout procedures.
• Context procedures.
• Badge procedures.
• Change detection procedures (the new logic concept).

File storage

• Fetching the database record for a file to be downloaded.
• Calculating the hash of a file.
• File access (this can be considered as obtaining a file connection, but does not include the entire download process).
• Writing files (this does include the entire upload process).

Miscellaneous

• Fetching records by primary key. This is, for instance, done when starting a task in the context of a record. This record will first need to be fetched.
• Loading the application model. After restarting Indicium or changing the application model, the model needs to be loaded, which may take multiple seconds on top of the request performed.

Performance metrics are listed under Server timings in your browser's developer tools.

Some examples (partially in Dutch) are shown below:

Examples of Servertiming metrics

Running Indicium in Development mode

warning

As convenient as it may seem to always run all instances of Indicium in Development mode, please do not do this. Only run Indicium in Development mode when you are actually developing and debugging problems or if Indicium is only accessible by trustworthy users.

By default, Indicium runs in Production mode. In this mode, Indicium limits the amount of information it sends to clients when database errors occur for security reasons. Indicium in Production mode will send the translatable database errors, such as primary key violations, to the client.

Production mode does not influence the amount of information that is logged to disk when errors occur. Indicium will always log all errors, whether related to the database or not, to the Logs folder in the root of Indicium.

If you want to run Indicium in Development mode, then you need to set the Development environment variable in the Web.config file in the root of Indicium:

<configuration>
    <system.webServer>
        <handlers>
            <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified"/>
        </handlers>

        <!-- Set Development environment variable -->
        <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" forwardWindowsAuthToken="false">
            <environmentVariables>
                <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
            </environmentVariables>
        </aspNetCore>

        ...

    </system.webServer>
</configuration>