Endpoints

New in DataGrab 5.3 are Endpoints. DataGrab Imports are designed to seek out and pull data from an external file or remote URL. Endpoints are a way for DataGrab to receive incoming data from an external application. Usually this is referred to as a webhook. An event happens in an application and it triggers a webhook, which POSTs data to a URL you've configured. Now with DataGrab you can accept incoming data to create or update entries in ExpressionEngine.

Endpoints are always paired with a previously configured Import. Imports usually handle many entries at a time from a remote file. Endpoints can also handle many entries, but generally the payload will be much smaller, such as 1 entry at a time. Regardless of the number of entries, the format of the payload sent by the external application must match the format of an already configured import.

To use endpoints:

1. Create an Import: Begin by creating an import using a static or remote JSON or XML file that matches the sender’s data format. Run the import through the DataGrab interface and verify it works as expected.

2. Create the Endpoint: Assign the import to a new endpoint. The sender will POST data to the Endpoint URL, including proper authentication.

Endpoints require:

1. A configured JSON or XML-based import.

2. POST requests must include body content that matches the import’s format.

3. A crontab process running with the --consumer flag to process data received by the endpoint. If you have multiple endpoints with different imports, each import requires its own dedicated consumer process.

That last step is very important. You must configure a crontab to consume the data from an endpoint. Otherwise it will sit in the queue and never become an ExpressionEngine entry.

Creating an Endpoint

Creating an Endpoint is very easy. Give it a name and associate it to an existing Import. You must also provide some method of authentication. Endpoints can accept authentication via GET variables, or HTTP Headers. You can add more than 1 parameter, though 1 or 2 is all that is usually required. If the parameters do not match exactly what is sent the request will fail.

To test an endpoint you can use an application such as Bruno, Insomnia, or any other rest client. In these screenshots we're using Bruno. First start with defining the query parameters to the Endpoint URL, which is generated after you create an Endpoint.

Add the POST body content. This should match the value of a full import, except it will contain data for 1 entry.

Most importantly, add your authentication values. You don't want to openly accept any data sent to your endpoint.

After submiting the test request from your preferred rest client you should see a row in your exp_datagrab_jobs table (if you are using the database queue driver, which is the default). If you do not see anything in this table, and you have the DataGrab module page open in a browser tab it is possible that DataGrab already picked up the job from the queue. In this case you should see the new entry in ExpressionEngine. If it still does not appear to be working, troublehoot it with the log file.