Qatium Ingest API

Introduction

This document details the behaviour of the Qatium Ingest API, whose purpose is to allow the loading of external data to a model for its subsequent visualisation in the application. In addition, it allows users to know the status of their ingested data.

Within this documentation, to the right of the screen there is an interactive Swagger to facilitate the integration with the API, and allows to test all the API requests. This requires the use of a token, which can be a test token (provided) or a real token in the 'Authorize' section.

IMPORTANT: All requests made with a real token will generate an ingest in the Qatium model related to it.

Headers

Authorization

In order to be able to make requests to the API, we must have a valid token provided by Qatium.

Once we have this token, we must send it in the Authorization HTTP header as a Bearer Token in all requests.

Content-Type

Due to the multiple formats supported by the API, it is essential to indicate the format of the request's content

Content-Type: application/json

Content-Type: multipart/form-data

Ingest Request

POST /ingest

This endpoint allows the upload of readings to the model and will respond with the number of rows uploaded.

The API is designed for realtime and near-real time updates, for historical data uploads please contact Qatium.

Qatium uses a mapping between variables and assets that must be taken into consideration when sending data:

Any information received that does not correspond to that specified in the table will be discarded.

Prepare the data

In order to be able to interpret the readings correctly, each reading shall contain the following properties:

In addition, validation is applied for the following metrics:

Supported Formats

JSON Body

Only up to 50 MB are accepted in this format.

JSON File

It is not recommended to use files larger than 1.5GB.

CSV File

It is not recommended to use files larger than 1.5GB.

Headers are not mandatory, however if they are not sent, the data must be sorted in the following order:

"time","asset","metric","unit","value"

Endpoint response

When a data upload is successful, the response will include the status of the data in percentages:

...
"status": {
  "successful": 80,
  "quarantined": 20,
  "pending": 0
}
...

The possible response status codes are the following:

Rate Limit

This endpoint has a rate limit of 100 requests per minute. If this limit is exceeded, the API will return a 429 status code.

Request Job Statuses

GET /jobs/{jobId}

This endpoint is used to visualise the status of a data ingestion.

To do this, we must use the 'job_id' obtained as a response in the ingest request.

Depending on the result of the data load, one of the following statuses will be returned:

Endpoint response

Request Job Details

GET /jobs/{jobId}/details

This endpoint allows users to know the exact details of why the data of an ingestion has been quarantined. To do so, we must use the ‘job_id’ obtained in the response of the ingestion.

Endpoint response

When sending data to the Ingest API, you will get a response.

Successful response example:

{
  job_id: 'job id',
  job_status: 'ok',
  message: `Your request has been processed without either errors or warnings, 10 rows have been appended`,
  status: {
    successful: 10,
    quarantined: 0,
    pending: 0,
  }
}

Unauthorized response example:

{
  job_id: 'job Id',
  message: 'Job id not found, or access to specified job unauthorized. Check your job id and token are correct.',
}

Ingested with warnings response example:

{
  job_id: 'job Id',
  job_status: 'warning',
  message: `Your request has been processed with warnings so 10 rows have been appended and 20 rows could not be appended.`,
  status: {
    successful: 10,
    quarantined: 20,
    pending: 0,
  },
}

Invalid request response example:

{
  job_id: 'job Id',
  job_status: 'invalid',
  message: `Your request has been processed with errors so no rows have been appended. Warning data is included in the request.`,
  warnings: [<INVALID READINGS>]
}

Pending response example:

{
  job_id: 'job Id',
  job_status: 'pending'
  message: 'Your request is waiting to be processed, please try again in a few minutes',
}

FAQs

When sending data to the Ingest API, you will get a response.

1. Can I retrieve, delete or update data through the API?

   The Ingest API is an append only API, if you wish to delete part or all of the data you submitted through the Ingest API, please contact us at q@qatium.com

2. What happens if two values are submitted for the same asset with identical timestamps?

   Qatium will use the last value received. The previous value(s) will be ignored and not visible in the application.

3. Do the API accept epoch time format?

   Yes, it also accepts epoch time format if that's your preferred flavor. Epoch times will always be considered UTC.

4. Do users need to send AMI data to another endpoint URL?

   No, Qatium will process the data as AMI if a flow feed is associated with a junction and as SCADA if the flow feed is associated with a pipe. Please contact us at q@qatium.com if you would rather send the raw meter readings instead of flow data.