Indicium troubleshooting

Introduction

This page provides information on how to troubleshoot Indicium.

For more information about possible startup errors, see Startup errors in the Indicium setup guide.

Possible issues

Login failed

Full message: "Login failed. Please contact your system administrator".

The credentials are correct but the corresponding user in IAM is probably configured wrongly:

• Check if IAM contains a user entry corresponding to the user's email (menu Authorization > Users).
• Check in IAM if the corresponding user is configured with the External authentication type (menu Authorization > Users > tab Form > tab User > field Authentication type).

401 - Unauthorized

When a user gets "401 - Unauthorized" results, the browser keeps prompting to log in. If you are certain that the specified credentials are correct, check if the user's authentication method is set correctly.

404 - Application not found

If an application URL (e.g. https://server/indicium/iam/myAppl/) gives the result "404 - Application not found", and you are certain that the URL is correct, ensure that the user that you are using to log in has access to the application in IAM.

When accessing an application present in the Software Factory (e.g. https://server/indicium/sf/myAppl/), ensure the user logged on to Indicium has this application activated.

Stuck on Indicium logout page

When Indicium is running on a domain other than the Universal GUI (not recommended for production environments), users might get stuck on the Indicium logout page after logging out.

To work around this, add the Universal GUI URL to the whitelist of allowed URLs that Indicium can use to redirect users when needed:

1. Log in to IAM.
2. Navigate to menu Settings > Global settings > tab Client redirect.
3. Enter the URL of the Universal GUI.

Indicium log files

You can access Indicium log files in two ways:

• Multiple log files (web server): Gather multiple log files from the web server Indicium is running on.
• Latest log file (landing page): Download the latest log file from Indicium's landing page.

note

• Sometimes, errors are caused by incorrect model settings. You can check this in the Software Factory: menu Quality > Validations.
• If the log files do not provide enough information, you can also check:
  • The Windows Event Viewer (Windows Logs > Application).
  • The Live Error Log (see below).

Multiple log files (web server)

To gather multiple log files, you can go to the web server Indicium is running on. You must have access to this web server and to the Indicium deployment directory.

To gather multiple log files:

1. Go to the web server where Indicium is running.
2. Go to the the root of the Indicium deployment directory.
3. Go to the Logs folder.
4. Gather the relevant log files.

Latest log file (landing page)

Main Administrator

If you only need the latest log file, or do not have access to the web server Indicium is running on, you can also go to Indicium's landing page.

To download the latest log file:

1. Go to the Indicium landing page.
2. Click on the link to log in. Use the credentials as specified in the relevant IAM.
3. Return to the Indicium landing page if this was not done automatically.
4. Click the error log link to download the latest log file.

Live error log

If you have access to downloading Indicium's error log, you can also access the Live Error Log page. You can find it at the same location as the regular error log - on Indicium's landing page. See Indicium log files.

The Live Error Log has some advantages over the regular error log:

• Text-based filtering.
• Minimum error level filtering (for example, errors only)
• The request URL (RequestPath) that caused the error

When you open the Live error log, it shows the last 100 (configurable) log entries that you would also see in the error log file. You can expand each entry to see more details.

• If web sockets are enabled on the web server, the Live Error Log maintains an open connection with Indicium. New log entries are added automatically as they occur.
• If web sockets are not enabled, you must refresh the page to see the latest log entries.

note

If Indicium is horizontally scaled, and Redis is configured for state management, the Live Error Log shows the log entries of all Indiciums simultaneously rather than just the Indicium to which you are connected.

An expanded error in the Live Error Log

Change the number of log entries

You can change the number of log entries that Indicium keeps track of for the Live Error Log by adding the following configuration setting to the appsettings.json file:

{
  "LiveErrorLog": {
    "MaxTailLength": 100
  }
}

Monitoring Indicium live

Indicium is capable of logging automatically to Azure Application Insights. This service can be used for monitoring Indicium live. You can, for example, inspect failed requests, monitor response times, or investigate exceptions.

Read the examples below to learn how Azure Application Insights can help you in the development process. Such logging capabilities are extremely useful, especially when you do not have direct access to Indicium's error logs. But even when you have direct access to the log files, the extra features offered by Azure Application Insights make it worth enabling.

Set up Azure Application Insights

This feature is enabled by default, but to use it, you need a paid Azure account with Application Insights enabled as a resource.

The next step is to find your Application Insights Instrumentation key and add it to Indicium's appsettings.json file.

Azure Application Insights instrumentation key

{
  "ApplicationInsights": {
    "InstrumentationKey": "actualinstrumentationkey"
  }
}

Controlling the logging levels

Similar to the standard logging, you can control the logging levels by adding the following to the appsettings.json file:

{
  "Logging": {
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information",
        "System": "Information",
        "Microsoft": "Warning",
        "Indicium": "Debug"
      }
    }
  }
}

More information about the different logging levels can be found here.

Viewing SQL commands in Application Insights

It is possible to view SQL commands in Application Insights.

To enable the logging of SQL commands, add the following to the appsettings.json file:

{
  "Telemetry": {
    "EnableSqlCommandTextLogging": true
  }
}

After configuring Indicium, SQL commands will be visible in Application Insights.

Analyze login events

To analyze failed and successful login events in Azure Application Insights:

1. Go to the menu Monitoring > Logs.
2. Run the following query:

traces | where customDimensions ["ProviderName"] == "Thinkwise-Login"

3. Inspect the events:

   • Analyze the individual data in the tab Results:

   The query and its results

   • Analyze the accumulated data in the tab Chart:

   A chart with the accumulated login results

Analyze failed requests

This topic shows an example of how to inspect failed requests in Azure Application Insights:

1. Go to your Azure Application Insights resource. The first screen is the overview. It features several graphs about your application's performance.

Azure Application Insights overview

To investigate failed requests:

1. Click on the Failed Requests graph.

An overview opens of all failed requests within a specific time interval, the last 30 minutes in this example. The request with the highest number of failures seems to be a GET request for a badge on generate_sales_invoice.

Azure Application Insights failed requests

1. Click on the request to get more detailed information. In this case, Indicium is returning a 401 Unauthorized response code for this resource.

Azure Application Insights badge request

Something else that might be worth investigating is the logged SqlException.

1. Go to the Exception tab and click on the exception message on the right of your screen. The following screen gives highly detailed information about this exception.

Azure Application Insights exception

note

This screen shows one of the advantages of debugging with Application Insights. Not only is the SQLException shown with a full stack trace, but it also gives an overview of what led to the actual exception over time.

In this example, the exception is preceded by a GET request on the employee_hierarchy resource. Apparently, while processing the request, some piece of SQL code is giving a maximum recursion error.

With this information, you now know that there must be a problem in a SQL template, and that the problem is caused by the employee_hierarchy view. This view uses a recursive common table expression (CTE) that seems to end in an infinite loop.

Measure Server-Timing responses

Indicium

You can view several performance metrics for requests using Server-Timing response headers. Browser developer tools can interpret these headers and visualize the performance. Performance metrics can provide a detailed insight and help solve potential performance issues, often caused by specific queries, layout procedures, or file storage operations.

Each measurement gets its own Server-Timing header. That way, you can easily distinguish between different reasons for performance issues. For example, a request might take a long time because one operation that performs poorly slows it down. It might also take longer because one of its operations is performed multiple times.

The available performance metrics are:

CRUD operations

• All SELECT queries for requested subjects of an application (tables and views).
• When the $count operation requests the total number of records, this will be measured and returned separately.
• When the $expand operation joins a lookup or a detail subject, these will each be measured and returned separately.
• Insert statements.
• Update statements.
• Delete statements.
• Expression queries while adding or editing a record.
• Default value queries when entering edit mode to add a new record or start a task or report.
• Lookup validations when changing a lookup value or a lookup filter value.

Application logic

• Tasks.
• Subroutines.
• Default procedures.
• Layout procedures.
• Context procedures.
• Badge procedures.
• Change detection procedures (the new logic concept).

File storage

• Fetching the database record for a file to be downloaded.
• Calculating the hash of a file.
• File access (this can be considered as obtaining a file connection, but does not include the entire download process).
• Writing files (this does include the entire upload process).

Miscellaneous

• Fetching records by primary key. This is, for instance, done when starting a task in the context of a record. This record will first need to be fetched.
• Loading the application model. After restarting Indicium or changing the application model, the model needs to be loaded, which may take multiple seconds on top of the request performed.

Performance metrics are listed under Server timings in your browser's developer tools. Some examples:

Examples of Server-Timing metrics

Running Indicium in Development mode

warning

As convenient as it may seem to always run all instances of Indicium in Development mode, do not do this. Only run Indicium in Development mode when you are actually developing and debugging problems or if Indicium is only accessible by trustworthy users.

By default, Indicium runs in Production mode. In this mode, Indicium limits the amount of information it sends to clients when database errors occur for security reasons. Indicium in Production mode will send the translatable database errors, such as primary key violations, to the client.

Production mode does not influence the amount of information that is logged to disk when errors occur. Indicium will always log all errors, whether related to the database or not, to the Logs folder in the root of Indicium.

If you want to run Indicium in Development mode, you need to set the Development environment variable in the Web.config file in the root of Indicium:

<configuration>
    <system.webServer>
        <handlers>
            <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified"/>
        </handlers>

        <!-- Set Development environment variable -->
        <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" forwardWindowsAuthToken="false">
            <environmentVariables>
                <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
            </environmentVariables>
        </aspNetCore>

        ...

    </system.webServer>
</configuration>