Troubleshooting

If you see an error similar to the one below, there is a 99.9% chance that this is not a DataGrab issue. Please do not submit a ticket until you have fully read this guide and exhausted all other posibilities.

File validation

Make sure your import files are valid. Use an online validation checker to ensure there are no errors with the file formatting. DataGrab can not read invalid XML, JSON, or CSV files.

There have been multiple bug reports of CSV file imports failing that have boiled down to a character encoding issue. Either a special or hidden character in the file that breaks the import. DataGrab 6 will attempt to convert the encoding to UTF-8. If DataGrab still can't read the file because of a character encoding issue then you will need to find and resolve the issue yourself. When this happens it will not show up in the DataGrab log file as a clearly identified character issue. It'll likely just stop importing after a certain point in the file (e.g. it stops after importing 27 entries), and that is where you will likely find the issue, on entry #28.

Import Configuration

Most of the issues encountered with DataGrab occur when configuring an import. Usually it's because the import file is invalid or inaccessible.

Check the following:

• The path in the config is correct

• Try running curl https://site.com/path/to/feed.xmlin your command line and see what the result is. Often times this might return something like a CloudFlare error page, or some other HTML based page that is not the expected JSON or XML response, which means there is an issue with your server that needs to be resolved before DataGrab can read the file. Sometimes loading the URL in a browser window appears fine making it appear as if it is a DataGrab issue, but using curlwill reveal the real issue.

• Make sure your environment has cURL enabled, and make sure you're not running into a self-referential request failure (e.g. making requests to URLs on the same server)
• Make sure $config['datagrab_verify_peer'] = 'y'; is not in your config.php file, or it is set to 'n'

• Open the import feed in a separate browser window, if it does not load and you do not see the XML or JSON data, neither can DataGrab.

• If you are importing from a local file or URL and using basic auth to block access to your site while in development, then be sure to include the credentials in your import URL as well. The import process does not use the same authenticated session as your browser. For example: http://user:[e-mail]/import-file.json

• If you are importing from a remote file or URL that is behind a basic auth, make sure it is correct, and make sure the username and password don't contain an @. If you open the feed with the username and password in a separate browser window and you receive a login prompt, then DataGrab sees the same thing and can not read the file. This likely indicates your username or password is wrong, or something else is misconfigured on the server.

• Make sure your site and feed protocols match. Don't load a feed from https if your site is running from http.

Check the logs

If your import file is valid and everything is configured and you are imports checking the log file is the best place to go to monitor the import activity. By default DataGrab will log to the system/user/cache/DataGrab-import.log file. Each time an import starts, it will purge old logged items from the file to avoid compounding the file size. You can change the logging type by adding the following option to your config.php file:

$config['datagrab_log_type'] = 'file|developer|php|off';

'file' is the default value. You do not need to add this to your config file unless you want to use one of the other options.

If you set the log type to 'developer' it will log to ExpressionEngine's Developer Log that is accessible in the control panel. Go to Tools > Logs >Developer Log and review the log for any error messages. It may be helpful to fully clear the entire log, then run an import again to ensure you have up-to-date log messages.

If you set the log type to 'php' it will write to your server's PHP log file. Depending on your system and how you have PHP configured, this file path may not be in a standard location such as /var/log/. If you do not want to log to your database (e.g. ExpressionEngine's Developer Log) or the user/cache directory, then you should use the php log option. If you choose 'php' as your logging option you can also change the path if you use the following config option:

$config['datagrab_log_file'] = '/var/www/log/DataGrab-import.log';

Note that if you change the log file make sure the directory is writable. We do not provide support for system configuration settings if the log is not outputting to the value you've defined. This option will not set your entire php error logging to this file. It will only log DataGrab events and errors to this file. Other ExpressionEngine or similar PHP errors will continue to be logged to your system defined log file.

If you do not set this option in your config file DataGrab will continue logging to the DataGrab-import.log file in your user/cache directory as mentioned above. You can turn off all logging by setting the log type to 'off';

If you disable logging and create a support ticket, you will be asked to re-enable and provide a log output of the import issue you're experiencing. The php or default DataGrab-import.log file is preferred for support requests since I can not read your Developer log without control panel access.

Try the Import URL

If you attempt to run an import and the Status column does not render a progress bar, but instead renders an error message, or something that looks like the screenshot below, then use the Import URL option and attempt to run the import in a new browser tab. From there, the full error message will be revealed. It is also highly possible the error will be in your import log. Please include the full error message if you're creating a support ticket.