Skip to content

Pipelines

Pipelines are used to transform a resource into one or several entries. A pipeline is an entry of GraphType Pipeline that defines several properties that are used in the transformation process. A pipeline is executed by submitting its entry URI to the context's execution resource.

Currently only synchronous execution is supported.

Pipeline properties

The following properties (in the EntryStore namespace) are supported in the pipeline's resource graph (if the resource graph is empty, the metadata is used instead):

  • transform: points to the root-node of a transform
    • transformPriority: every pipeline can chain together several transforms, the order is determined by the transforms' priorities
    • transformType: determines which type of transform should be carried out
    • transformArgument: points to one or more transformArgumentKey-transformArgumentValue pairs to provide transform-specific parameters
  • transformDestination: the destination entry of a transform that results in a graph (optional)
  • transformDetectDestination: if true, the entries are detected from the resulting graph instead of using transformDestination (optional)

Execution

Request

POST https://domain/store/context-id/execute

HTTP header: Content-type: application/json

Request body

{
  "pipeline": "https://domain/store/context-id/entry/20",
  "source": "https://domain/store/context-id/entry/21"
}

The source parameter is optional. Both the pipeline and the source must belong to the context of the request. The source must be a local entry with an uploaded file. Executing a pipeline requires write access to the context's resource and read access to the pipeline and source entries.

Response

HTTP status if successful: 201 CREATED, or 200 OK if no entries were created or updated.

Response body

The response body of a successful request returns a list of entries that were created or updated during the pipelining process.

{
  "result": [
    "https://domain/store/context-id/entry/22",
    "https://domain/store/context-id/entry/23",
    "..."
  ]
}

Errors

Status Reason
400 Malformed JSON, missing pipeline, pipeline or source outside the context, or the pipeline cannot be executed with the given input
401 / 403 Missing write access to the context or read access to the pipeline or source
404 The context, pipeline or source entry does not exist
409 The pipeline entry is not of GraphType Pipeline, or the source is not a local entry with an uploaded file
415 The request is not application/json
502 / 504 A service the pipeline calls (e.g. RowStore) failed or did not answer in time (6.x)

EntryStore 5.x

In 5.x some of these cases are answered differently, e.g. a missing pipeline or source entry with 500 and failures of RowStore with 400.

CSV2RowStore Pipeline

The CSV2RowStore pipeline ("transformType": "rowstore") supports the following arguments:

  • action: determines what action to perform; one of the following values:
    • create (default if action parameter is not provided): creates a new dataset in RowStore
    • replace: replaces the data of an already existing dataset, requires the parameter datasetURL
    • append: appends data to an already existing dataset, requires the parameter datasetURL
    • setalias: sets or removes an alias of an existing dataset. Requires the parameters datasetURL and alias. The parameter alias is used to provide the alias to be set (should consist of alpha-numeric unicode characters only), must be omitted or left empty to remove an active alias.

All actions except setalias require a source entry with the CSV file. All actions return a reference to exactly one result entry in the response body. In the case of create this will be a newly created entry (a Reference entry of GraphType PipelineResult pointing to the dataset), and for replace, append and setalias the previously existing result entry is returned. An unknown action is answered with 400.

The transform requires entrystore.rowstore.url to be set. In 6.x the actions replace, append and setalias only accept a datasetURL with the same origin (scheme, host and port) as entrystore.rowstore.url, so RowStore's own base URL must use that origin. Other dataset URLs are answered with 400 and no request is sent to them.

If RowStore rejects a request (4xx, e.g. an invalid or taken alias), the execution is answered with 400. If RowStore cannot be reached, fails or answers unexpectedly, it is answered with 502, and with 504 if RowStore does not answer in time (6.x).