Upcycler

Introduction to the Thinkwise Upcycler

The goal of the Thinkwise Upcycler is to jump-start the modernization of your existing legacy application. Depending on your legacy technology, 20% to 50% of the development time will be saved by using the Upcycler.

The Upcycler completely replaces the legacy application and migrates its data automatically to the new RDBMS of your choice. The modernized application has a fully responsive web app user interface and an integrated service tier. After having upcycled your legacy application, you can further develop your new application.

warning

The Upcycler is for advanced users only. For a successful replacement of your legacy application, you will need a team with the expertise of your application, your legacy technology, and the Thinkwise Platform. Please contact Thinkwise or a Certified Partner for more information.

The Thinkwise Upcycler comes with Upcyclers for a large number of relational databases. For an overview, see Available Technology Upcyclers.

• All Upcyclers can be tailored to suit your specific needs.
• It is possible to develop and use your own Upcyclers for other technologies.

Thinkwise Upcycler installation

The Thinkwise Upcycler is a separate application within the Thinkwise Platform and should be deployed as such. It should be downloaded as part of the platform installation package and deployed through the Thinkwise Deployment Center.

For more information, see the Thinkwise Deployment Center manual.

An overview of the upcycling process

1. Metadata - The first step is to derive as much metadata as possible from your legacy application. Doing this right at the start of your upcycle project saves an enormous amount of time later. Thinkwise provides a separate utility program for this, the Thinkwise Extractor. It extracts the metadata from your legacy application and puts it into a file that can be read by the Thinkwise Upcycler. For some legacy technologies, you can use the export facilities of that technology instead.

2. Conversion - The next step is to convert your source's metadata into metadata for the Thinkwise Software Factory with the help of an Upcycler. An Upcycler contains a series of steps at three levels, implemented in SQL. The levels inherit their steps from each other.

   • Generic level - This level consists of generic upcycle steps and generic data import steps that have been defined by Thinkwise and cannot be modified. It gives you an easy start in creating your own Upcycler if one is necessary.

   See Generic upcycler.

   • Technology level - The Thinkwise Upcycler comes with several ready-made technology-specific Upcyclers that cannot be modified. If no upcycler is available for your legacy application's technology, you can create your own Technology Upcycler. A Technology Upcycler consists of a technology definition, upcycle steps, and data import steps. The Technology level inherits its steps from the Generic level.

See Available Technology Upcyclers and Create your own Technology Upcycler.

• Application level - At this level, you can tailor your Technology Upcycler for upcycling a specific legacy application. It consists of an application definition, upcycle steps, and data import steps. The Application level inherits its steps from the Technology level.

See Fine-tune an Upcycler for your application.

3. Data import - The next step is the Data Importer. This process is structured in the same way as the Upcycler process.

4. Enrichment - The final step is Enrichment. This process is structured similarly to the Upcycler process, but each step performs an analysis that delivers proposed actions you can select and apply.

Extract the metadata from your legacy application

To upcycle an application, you first need to extract the model and the data from your legacy application. Usually, the extraction is performed by either the Thinkwise Extractor or the export functionality of your legacy platform. For more information on the export functionality, see the Technology Upcyclers.

The Thinkwise Extractor is a command-line utility for extracting the model and data from legacy applications.

It consists of one executable and a folder containing some scripts. This makes it easy to use, even when no Thinkwise installation is available yet. Scripts for all supported technologies are pre-delivered.

note

Use the 32-bit version of the Thinkwise Extractor if you have a 32-bit data source and the 64-bit version if you have a 64-bit data source.

To use the Thinkwise Extractor:

1. Copy the executable and the scripts to a computer with access to the legacy database.
2. Run twupcycle.exe from either the win-x86 or the win-x64 folder, using the appropriate command line parameters.

twupcycle.exe <COMMAND> <ACCESS METHOD> <OPTIONS>

COMMAND
schema	Extract the schema of a datasource.
data	Extract the data of a datasource.

ACCESS METHOD
odbc	For now: always odbc.

OPTIONS
-h, --h	Prints help information.
--dsn	The DSN (Data Source Name) of the connection. Required, unless -c, --connection-string has been specified.
-c, --connection-string	The string for connecting to the data source. Required, unless --dsn has been specified.
--uid	Option to explicitly set the user name for the connection string.
--pwd	Option to explicitly set the password on the connection string. If no --uid has been specified, this option is ignored.
-e, --extractor	This can be either a zip file or a directory containing script entries under a schema/odbc path relative to the root. - If you enter a relative path, the Extractor will try to find a match in the current directory and the location of the program. - The default extension of an archive is .extractor. - A path with an extension will always try to look for a zip file with the specified extension instead of .extractor. For example, --extractor mssql will try to find an extractor archive in the following order: -- <current working directory>/mssql.extractor -- <current working directory>/mssql -- <program location>/mssql.extractor -- <program location>/mssql If this option has not been specified and the input has not been redirected, the Extractor will show a list of .extractor files to choose from, if it detects any.
-o, --output-file	The file path where the output file(s) should be written. - For the schema command, you will be prompted for a path if this option has not been specified, unless the output has been redirected (e.g., due to a pipe). - For the data command, by default, the files will be written to the current working directory. - Relative paths will be considered relative to the current working directory. - Output is always written using UTF-8 encoding with a byte order mark (BOM).

Examples

Here are some examples for extracting schema and data. You can adapt them to your own needs or write your own lines.

This example extracts the schema from an MS SQL Server database. Here, it contains --connection-string. This can be replaced by --dsn.

twupcycle.exe schema odbc --connection-string "Driver={ODBC Driver 17 for SQL Server};Server=MyServer\MyInstance;Database=AdventureWorks2019;Trusted_Connection=yes;" --extractor mssql --output-file schema.json

This example extracts the schema from a MySQL database.

.\Extractor\twupcycle.exe schema odbc --dsn MySQL_local --extractor .\Extractor\Scripts\MySQL --output-file schema.json

This example extracts the schema from an Oracle database.

..\Extractor\twupcycle64.exe schema odbc --dsn OracleDSN --uid ******** --pwd ******** --extractor oracle --output-file .\schema\schema.json

This example extracts the schema from an DB2 database. Note that the default folder for the extractor has been changed here. You can add a user name and password with --uid myuser and --pwd mypassword. If you only add a user name (--uid my user), the user will be prompted for the password.

twupcycle.exe schema odbc --dsn DB2Thinkwise --extractor .\Scripts\DB2 --output-file schemaDB2.json

This example extracts the data from a ProgressOpenEdge database:

twupcycle.exe data odbc --dsn ADTEST99ZZ --uid myuser --pwd mypassword --extractor .\Extractor\Scripts\PROGRESS --output-directory .\Data

This example extracts the data from a PostgreSQL database:

twupcycle64.exe data odbc --connection-string "Driver={PostgreSQL ODBC Driver(UNICODE)};Server=localhost;Port=5432;Database=dvdrental;" --uid myuser --pwd mypassword --extractor postgresql --output-directory .\data

This example extracts the data from a MySQL database:

twupcycle.exe data odbc --dsn MySQL_local --extractor .\MySQL --output-directory .\Data

This example extracts the data from an Oracle database:

..\Extractor\twupcycle64.exe data odbc --dsn OracleDSN --uid ******** --pwd ******** --extractor oracle --output-directory K:\temp\oratestdata\

Upcycle your application

1. Create an application in the Thinkwise Upcycler

After extracting the model and data from your legacy software, you need to add an application to the Thinkwise Upcycler:

menu Upcycle > Applications > tab List

1. Click Add .
2. In tab Form, enter the required information about the application, the source model, the source data, the target model, and the target application database:

Application
Field	Description
Name	The application's name in the Thinkwise Upcycler.
Technology	The legacy technology in which the application was build.
Active	Inactive applications will not be shown by the default prefilter.

Source model
Field	Description
Source model	Preview field for a part of the source model, once it has been imported.
Model file type	Type of data in the model file, gets a default value from the Technology level.
Model file extension	File extension of the model file, gets a default value from the Technology level.
Model file encoding	Encoding of the model file, gets a default value from the Technology level.

Source data
Field	Description
Data folder	Folder where the extracted data files will be saved. This has to be a folder on the webserver where Indicium is installed.
Data file type	Type of data in the data files, gets a default value from the Technology level.
Data file extension	File extension of the data files, gets a default value from the Technology level.
Data file encoding	Encoding of the data files, gets a default value from the Technology level.

Target model
Field	Description
Project	Project ID that will be created for the upcycled application.
Project version	Version ID that will be created for the upcycled application.
Primary application language	The language for which translations will be generated. For multi-language applications, this will be used as a fallback.
Project folder	Project folder that will be used for your project.
SF database server	The database server on which your Software Factory is located. By default, this is the database server of the Thinkwise Upcycler.
SF database name	The name of the Software Factory database.

Target application database
Field	Description
Application database server	The database server on which your application database must be created.
Application database name	The name of the database that will be created. It will be used for importing your application data.

If necessary: add languages

Some technologies support multi-language applications. If the matching Technology Upcycler supports it, you need to add the application languages. This will translate the language codes of your legacy application to the language codes of the Thinkwise Platform.

1. Add the languages in the form on the right.

Form for adding a new application with multiple languages to the Upcycler

2. Import the source model

1. Execute the Import source model task to load the source model into the application.

After importing the source model, a preview of the model becomes visible in the Source model field.

A preview of the model in the Source model field

Now all information that your Upcycler needs has been entered into the Thinkwise Upcycler.

3. Execute the upcycle run

menu Upcycle > Upcycle runs > tab Application > tab Current

1. Select your application.
2. Execute the Execute upcycle task.

This will execute the upcycle steps as defined in your Technology Upcycler. Where needed, it is overruled by steps at the Application level. In the Steps tab, you can track the progress.

Progress of the upcycle run

4. Inspect the upcycle run

menu Upcycle > Upcycle runs

After the upcycle run has finished, carefully inspect all messages to see if it is necessary to adjust the upcycle steps.

When at least the PROJECT: Create project step was executed, a project has been added to your target Software Factory. If not all steps were successful, the project is incomplete. In that case, remove the project, make the necessary changes to the Upcycler, and execute the Upcycle run again.

• A list of all messages is available in tab Application > tab Current > tab Report:

  Messages in the Report tab

• To inspect the results in more detail, consult tab Application > tab Current > tabs Steps/Details:

  • Information message.
  • Error message.

  Inspect the results in more detail

• To inspect what data was transferred to the Software Factory, look at tab Application > tab Current > tab Data:

  Transferred data in the executed upcycle steps

5. Inspect your application in the Software Factory

After a successful upcycle run, your new project will be available in the Software Factory. Take your time to inspect the result.

warning

Do not make any changes yet, as you may want to fine-tune your Upcycler and run it again. Moreover, if you change the data model, you will not be able to import the data.

If you think the result is satisfactory, you can move on to the next step: Create your application.

6. Create your application in the Software Factory

In the Software Factory, create your application as usual, by running the Creation process.

1. Run the Generate definition and Validate definition steps separately, so you can easily inspect the result.
2. Check the messages carefully to see whether you can or need to fine-tune your Upcycler.
3. If you think the result is satisfactory, you can run the Generate source code and Execute source code steps.
4. If Execute source code is successful, you can start your application. At this stage, there is no data in your application yet.

7. Execute a data import run

menu Upcycler > Data import runs

1. Select your application.
2. Execute the Execute data import task.

This will execute the data import steps as defined in your Technology Upcycler, where necessary overruled by steps at the Application level.

8. Inspect the data import

menu Upcycler > Data import runs > tab Application > tab Current > tab Steps

1. After the data import run has finished, inspect the information and error messages carefully to see if it is necessary to adjust the data import steps.

   • An overview of all problems is available in tab Steps.

   If necessary, analyze the errors in the Steps tab

   • The full error message for a problem is available in tab Details > Message.

   The full error message is available in tab Details > Message

   • The generated SQL for analysis of a failed data import is available in tab Transfer code. If the import was successful, the generated SQL was removed to avoid storage of a very large data volume in the Thinkwise Upcycler.

   The generated SQL for a failed data import

9. Inspect your application

1. Start your application. It will now have data.
2. Inspect your application to see if you still want to improve anything in the upcycle process.

10. Enrich your application

Now that your application is running with data, you can enhance it with bulk updates if necessary.

To support you in this step, Enrichment runs are available. Enrichments are a way to make mass changes to an application that would be unfeasible to do by hand.

• For general information on the enrichment run, see chapter Introduction to enrichment runs.
• The Upcycler contains several predefined enrichments. For an overview, see chapter Available enrichments.
• To fine-tune the available enrichment steps, see chapter Fine-tune the enrichment steps.
• To create custom steps for your own Upcycler, see chapter Create custom enrichment steps.
• If necessary, program your own enrichment steps. See chapter Programming an enrichment.

First, execute an enrichment analysis. This will execute the enrichment analysis steps as defined in your Technology Upcycler, where necessary overruled by steps at the Application level. The run will produce enrichment action steps.

menu Upcycle > Enrichment runs > tab Application

1. Select your application.
2. Navigate to tab Analysis steps.
3. Execute the Execute enrichment analysis task.
4. Enter the project version.
5. Press Execute.

Execute the enrichment analysis steps

Then, execute the enrichment actions. This will execute the enrichment action steps that were produced by the analysis run.

6. Navigate to tab Action steps.
7. Select or deselect the steps you wish to execute.
8. Execute the Execute enrichment actions task.
9. Answer 'Yes' to the question: "Do you want to execute task Execute enrichment actions?'.

Execute the enrichment action steps

11. Inspect the enrichments

Check the database, Software Factory, and your end-application whether the enrichment actions in the model and data have been executed as expected.

12. Plan towards a full-featured application

If the upcycle process has finished, make a plan for turning your upcycled application into a full-featured Thinkwise application.

Fine-tune an Upcycler for your application

If your upcycle run needs improvement, you can fine-tune the Technology Upcycler you are using for your legacy application.

tip

Even if you think that an improvement is valuable to all applications in this technology, it is best to try it on your specific application first.

menu Upcycle > Upcycle runs

1. Select your application.
2. Open tab Application upcycle steps.
3. Add upcycle steps specific to your application, modify generic steps, or deactivate steps.

In tab List, you can check the status:

Status	Description
Base status	The status of the step at the Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step is specific for this technology and has no underlying generic step.
Current status	The status of the step at the Application level. - Inherited - the step has not been modified for this application. - Modified - the step has been modified for this application. - Specific - the step has been added specifically for this application.

In tab Form, the following settings are available:

Setting	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Order no	The order in which the steps will be executed.
Reason	The reason why you are creating a variation on a step. This field becomes available when you add or modify a step.
Description	The purpose of the step's functionality.
Step code	The SQL code that is executed for this step.

tip

If you have made a change and wish to revert it to the original technology upcycle step, use the Revert to Technology upcycle step task.

Fine-tune the Application upcycle steps

Re-run the upcycle

After finetuning the steps, you can rerun the Upcycler.

1. In the Software Factory, delete the project that the Thinkwise Upcycler has created earlier for this application.
2. Open the Thinkwise Upcycler (menu Upcycle > Upcycle runs).
3. Select your application.
4. Execute the Execute upcycle task.

Fine-tune the data import

menu Upcycle > Data import runs

1. Select your application.
2. Open tab Application data import steps.
3. Add data import steps specific to your application, modify generic steps, or deactivate steps.

In tab List, you can check the status:

Status	Description
Base status	The status of the step at the Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step is specific for this technology and has no underlying generic step.
Current status	The status of the step at the Application level. - Inherited - the step has not been modified for this application. - Modified - the step has been modified for this application. - Specific - the step has been added specifically for this application.

In tab Form, the following settings are available:

Setting	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Order no	The order in which the steps will be executed.
Reason	The reason why you are creating a variation on a step. This field becomes available when you add or modify a step.
Description	The purpose of the step's functionality.
Step code	The SQL code that is executed for this step.

tip

If you have made a change and wish to revert it to the original technology data import step, execute the Revert to Technology data import step task.

Fine-tune the data import steps

Re-run the data import

1. For re-running the data import, you need a generated database with empty tables. This can be achieved in two ways:

   • With your database management tool, delete all records from all tables in your application database. This can be complicated because of the referential integrity constraints between tables.
   • With your database management tool, drop the application database.

2. Create a fresh database in the Software Factory.

3. Open the Thinkwise Upcycler (menu Upcycle > Data import runs).

4. Select your application.

5. Execute the Execute data import task.

Fine-tune the enrichment steps

menu Upcycle > Enrichment runs

1. Select your application.
2. Navigate to the tab Application enrichment analysis steps.
3. Add enrichment analysis steps specific to your application, modify generic steps or deactivate steps.

In tab List, you can check the status:

Status	Description
Base status	The status of the step at the Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step is specific for this technology and has no underlying generic step.
Current status	The status of the step at the Application level. - Inherited - the step has not been modified for this application. - Modified - the step has been modified for this application. - Specific - the step has been added specifically for this application.

In tab Form, the following settings are available:

Setting	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Order no	The order in which the steps will be executed.
Reason	The reason why you are creating a variation on a step. This field becomes available when you add or modify a step.
Description	The purpose of the step's functionality.
Model query	The SQL code executed against the model for this step.
Data query	The SQL code executed against the data for this step.

tip

If you have made a change and wish to revert it to the original technology data import step, execute the Revert to technology enrichment analysis step task.

Fine-tune enrichment steps

Re-run the enrichment analysis

You can re-run the enrichment analysis without consequences until it produces the desired results.

1. Create a fresh database in the Software Factory.
2. Open the Thinkwise Upcycler (menu Upcycle > Enrichment runs).
3. Select your application.
4. In tab Analysis steps, execute the Execute enrichment analysis task.

warning

Be extremely careful when running the resulting enrichment actions. They can change both the model and the data. Therefore, create a new version of your project and a backup of the data before running the actions.

5. In tab Action steps, execute the Execute enrichment actions task.

Create your own Upcycler

The Thinkwise Upcycler comes with Upcyclers for a number of technologies. If no Upcycler is available for the technology you need, you can add your own.

warning

Only create your own Upcycler if you have sufficient knowledge of the technology, the legacy application, and the Thinkwise platform. Contact Thinkwise or one of our Certified Partners for advice on creating a new Upcycler.

1. Extract the model and data

When you are building your own Technology Upcycler, it might be necessary to create scripts for the Thinkwise Extractor as well. However, not all Technology Upcyclers use the files the Thinkwise Extractor produces, and for Platform Upcyclers, it can be more convenient to use the platform's export facilities.

Therefore, you first need to decide how to extract the model and the data from your legacy application.

• The legacy technology that needs to be upcycled is a database - The easiest way is to use the Thinkwise Extractor. If the database system supports the Information Schema, it is the recommended way since it will result in maximum reuse of generic steps in the Upcycler. If not, use the proprietary catalog tables of the database system.

  • There needs to be one script per table in the catalog or Information Schema. You can use the scripts delivered with the Thinkwise Extractor as examples.

  • Each technology needs a script for extracting the data. You can use the pre-delivered scripts for the supported technologies as examples.

    The default structure of the folders is:

  The Extractor's folder structure

• The legacy technology that needs to be upcycled is a platform - Investigate whether your platform has an export functionality for the model information and if you comprehend its resulting file(s). The alternative is to use the Thinkwise Extractor to extract the tables in which the application is defined.

2. Create a new technology in the Upcycler

If the technology you need is not available by default in the Thinkwise Upcycler, you can add one.

menu Definition > Technology upcyclers

1. Add a new Upcycler.

Field	Description
Name	Name of the technology.
Description	A short description of the technology, for example, for which versions of the technology you are creating this Upcycler.
Logo	The technology's logo.
Active	Only active technologies will be shown by the default prefilter.
Default model file type	Type of data in the model file, provides a default for the application.
Default model file extension	Extension of the model file, provides a default for the application.
Default model file encoding	Encoding of the model file, provides a default for the application.
Default data file type	Type of data in the model file, provides a default for the application.
Default data file extension	Extension of the data files, provides a default for the application.
Default data file encoding	Encoding of the data files, provides a default for the application.

Add a new Technology Upcycler

3. Add the Upcycle steps

menu Definition > Technology upcyclers > tab Technology upcycle steps

On opening, tab List shows the steps that have been inherited from the Generic level.

1. In tab Form, Add new steps specific to your technology or modify generic steps to make them work for your technology. You can also disable steps. Use the Upcyclers delivered by Thinkwise as examples. The modified fields turn yellow.

Field	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Status	The status of the step at this Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step this specific for the technology and has no underlying generic step.
Technology	The technology to which this step belongs.
Order no	The order in which the steps will be executed.
Reason	The reason why you are creating a variation on a step. This field becomes available when you add or modify a step.
Description	The purpose of step's functionality.
Step code	The SQL code that is executed for this step.

tip

If you have made a change and wish to revert it to the generic upcycle step, execute the Revert to generic upcycle step task.

Add upcycle steps to your Upcycler

4. Test your Upcycler

Before continuing to data migration, test your Upcycler thoroughly.

tip

Before testing it with your full legacy application, find or create a small demo application.

5. Create custom data import steps

menu Definition > Technology upcyclers > tab Technology data import steps

On opening, tab List shows the steps that have been inherited from the Generic level.

1. In tab Form, Add new steps that are specific to your technology or modify generic steps to make them work for your technology. You can also disable steps. Use the Upcyclers delivered by Thinkwise as examples.

   • For a Technology Upcycler based on the Information Schema, you only need minimal changes on top of the generic steps. Ideally, only the translation of data types needs to be customized.
   • For a Technology Upcycler for a platform, it may be necessary to put more effort into the data import steps, although it will be less than for the upcycle steps.

The modified fields turn yellow.

Field	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Status	The status of the step at the Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step is specific for this technology and has no underlying generic step.
Technology	The technology to which this step belongs.
Order no	The order in which the steps will be executed.
Reason	The reason why you are creating a variation on a step. This field becomes available when you add or modify a step.
Description	The purpose of step's functionality.
Step code	The SQL code that is executed for this step.

tip

If you have made a change and wish to revert it to the generic upcycle step, execute the Revert to generic upcycle step task.

Create data import steps

6. Test the custom data import steps

Test your customized data import steps thoroughly.

tip

Before processing a full production database, test a smaller set first.

7. Create custom enrichment steps

menu Definition > Technology upcyclers > tab Technology enrichment analysis steps

On opening, tab List shows the steps that have been inherited from the Generic level.

1. In tab Form, Add new steps specific to your technology or modify generic steps to make them work for your technology. You can also disable steps. Use the Upcyclers delivered by Thinkwise as examples. The modified fields turn yellow.

Field	Description
Name	The name of the step.
Active	Only active steps will be executed and shown by the default prefilters.
Status	The status of the step at the Technology level. - Generic - the step has not been modified for this technology. - Modified - the generic step has been modified for this technology. - Specific - the step is specific for this technology and has no underlying generic step.
Technology	The technology to which this step belongs.
Order no	The order in which the steps will be executed.
Description	The purpose of the step's functionality.
Model query	The SQL code that is executed against the model for this step.
Data query	The SQL code that is executed against the data for this step.

tip

If you have made a change and wish to revert it to the generic upcycle step, execute the Revert to generic enrichment analysis step task.

Create custom enrichment steps

8. Test the custom enrichment steps

Test your customized enrichment steps thoroughly.

You can re-run the enrichment analysis without consequences until it produces the desired result. Before processing a full production database, test a smaller set first.

warning

Be extremely careful when running the resulting enrichment actions. They can change both the model and the data. Therefore, create a new version of your project and a backup of the data before running the actions.