Pipelines¶
Pipelines are used to transform a resource into one or several entries. A pipeline is an entry of GraphType Pipeline that defines several properties that are used in the transformation process. A pipeline is executed by submitting its entry URI to the context's execution resource.
Currently only synchronous execution is supported.
Pipeline properties¶
The following properties (in the EntryStore namespace) are supported in the pipeline's resource graph (if the resource graph is empty, the metadata is used instead):
transform: points to the root-node of a transformtransformPriority: every pipeline can chain together several transforms, the order is determined by the transforms' prioritiestransformType: determines which type of transform should be carried outtransformArgument: points to one or moretransformArgumentKey-transformArgumentValuepairs to provide transform-specific parameters
transformDestination: the destination entry of a transform that results in a graph (optional)transformDetectDestination: iftrue, the entries are detected from the resulting graph instead of usingtransformDestination(optional)
Execution¶
Request¶
POST https://domain/store/context-id/execute
HTTP header: Content-type: application/json
Request body¶
{
"pipeline": "https://domain/store/context-id/entry/20",
"source": "https://domain/store/context-id/entry/21"
}
The source parameter is optional. Both the pipeline and the source must belong to the context of the request. The source must be a local entry with an uploaded file. Executing a pipeline requires write access to the context's resource and read access to the pipeline and source entries.
Response¶
HTTP status if successful: 201 CREATED, or 200 OK if no entries were created or updated.
Response body¶
The response body of a successful request returns a list of entries that were created or updated during the pipelining process.
{
"result": [
"https://domain/store/context-id/entry/22",
"https://domain/store/context-id/entry/23",
"..."
]
}
Errors¶
| Status | Reason |
|---|---|
| 400 | Malformed JSON, missing pipeline, pipeline or source outside the context, or the pipeline cannot be executed with the given input |
| 401 / 403 | Missing write access to the context or read access to the pipeline or source |
| 404 | The context, pipeline or source entry does not exist |
| 409 | The pipeline entry is not of GraphType Pipeline, or the source is not a local entry with an uploaded file |
| 415 | The request is not application/json |
| 502 / 504 | A service the pipeline calls (e.g. RowStore) failed or did not answer in time (6.x) |
EntryStore 5.x
In 5.x some of these cases are answered differently, e.g. a missing pipeline or source entry with 500 and failures of RowStore with 400.
CSV2RowStore Pipeline¶
The CSV2RowStore pipeline ("transformType": "rowstore") supports the following arguments:
action: determines what action to perform; one of the following values:create(default if action parameter is not provided): creates a new dataset in RowStorereplace: replaces the data of an already existing dataset, requires the parameterdatasetURLappend: appends data to an already existing dataset, requires the parameterdatasetURLsetalias: sets or removes an alias of an existing dataset. Requires the parametersdatasetURLandalias. The parameteraliasis used to provide the alias to be set (should consist of alpha-numeric unicode characters only), must be omitted or left empty to remove an active alias.
All actions except setalias require a source entry with the CSV file. All actions return a reference to exactly one result entry in the response body. In the case of create this will be a newly created entry (a Reference entry of GraphType PipelineResult pointing to the dataset), and for replace, append and setalias the previously existing result entry is returned. An unknown action is answered with 400.
The transform requires entrystore.rowstore.url to be set. In 6.x the actions replace, append and setalias only accept a datasetURL with the same origin (scheme, host and port) as entrystore.rowstore.url, so RowStore's own base URL must use that origin. Other dataset URLs are answered with 400 and no request is sent to them.
If RowStore rejects a request (4xx, e.g. an invalid or taken alias), the execution is answered with 400. If RowStore cannot be reached, fails or answers unexpectedly, it is answered with 502, and with 504 if RowStore does not answer in time (6.x).