Add script calculations

You can enhance a dataset by adding scripts for your calculations. The R and Python script languages are supported in Data Preparation.

Prerequisites

• You have a respective AI connection to process your scripts. If applicable, you can use a default built-in Python connection. For details, see Add AI connections.
• You are creating a dataset.

Procedure

1. Navigate to the calculations editor

   While creating or editing a dataset:

   • If you are on Step 2 – Refine, select a data source, and then click Calculations.

     

   • If you are on Step 3 – Join and preview, next to the data source name, click  More options, and then Calculations.

     

   The Calculations dialog appears.

   

2. Define script calculation

   1. In the Type calculation field, type script or scalar, and then click the Edit script button that appears.

      Alternatively, you can drag the SCRIPT or SCALAR function from the list. You can switch between the types at any time before saving the calculation.

      

      Vector calculation

      If using the Vector calculation type, multiple rows are sent as a table to a server for processing in one request. A result will be an array of values–a set of rows for a column. The following is an example of the outlier detection script sending data to a table in Python.

      import numpy as np
      from sklearn.cluster import DBSCAN
      from sklearn.preprocessing import StandardScaler
      X = np.column_stack([_arg1, _arg2])
      X = StandardScaler().fit_transform(X)
      db = DBSCAN(eps=1, min_samples=3).fit(X)
      result = db.labels_.tolist()
      return result
      

      Scalar calculation

      With the Scalar calculation type, each row is sent separately and calculated in the server one by one.

      Example of a scalar function that converts values in each row to uppercase:

      return _arg1.upper()

   2. In the script editor that appears, do the following:

      1. Select the following settings:

         • Connector – Python (built-in) or any other defined AI connector.
         • Calculation – Specify how to calculate the data in the script by selecting a respective calculation type:

           • Vector – Multiple rows are sent as a table in one request to the server for processing.

           • Scalar – Each row is sent and calculated in the server separately, one by one.

         • Result – The format of the returned value: double, string, or integer.

         

      2. Click in the Data frame columns field, and select a column that you want to use as the first argument for the script. Repeat this step for each argument of the script. You can reorder the columns by dragging a column to another place.

         If you need to change the aggregation for a column, click a column name, and select a new aggregation type.

         For scalar calculations, set the aggregation to None.

         

         Preview input data

         To view the raw data values for input, click Preview input data frame.

         

         Create data frame stub

         To create a data frame stub template, click Generate data frame stub. Click Copy to reuse it in a script.

         

      3. In the Script field, enter the code.

         Note: You can use only scripts that return data in one column.

      4. (Only for scalar) Click Preview result to preview the script result in the Calculation column.

         The tabular preview is generated.

         

         Click Close.

      5. Click Apply.

         The script is added to the calculation pane.

   3. In the Calculation name field, type a name for the calculation.

   4. In the dropdown list next to the calculation name, select the calculation’s data role (such as dimension, measure, and date).

   5. Click Save.

   Now, the new calculated column is added to the dataset.