BoldMinded Docs
  • Overview
  • Advanced Categories
    • Docs
      • Installation
      • Requirements
      • Setup & Configuration
        • Hidden Config Options
        • Multi-Site Manager
      • Template Tags
      • FAQs
  • Ansel
    • Docs
      • Installation
      • Upgrading
      • Requirements
      • Settings
      • Usage
        • Channel:Form
      • Troubleshooting
      • Template Tags
    • FAQs
  • Bloqs
    • Docs
      • Why Bloqs?
      • Features
        • Add Bloq Menu
        • Nesting
        • Cloning
        • Drafts
        • Bloq Usage
        • Deprecated Bloqs
        • Caching
        • Profiling
        • Bloq Components
        • Bloq Groups
      • Installation
      • Requirements & Compatibility
      • Setup & Configuration
        • Hidden Config Options
      • Creating Bloqs
      • Template Tags
        • Twig
        • Examples
      • Context Variables
      • Add-on Developers
    • FAQs
  • Carson
    • Docs
      • Installation
      • Requirements
      • Fields
        • Omni
          • Hidden Config
        • Assistant
        • SEO
      • Troubleshooting
  • Custom System Messages
    • Docs
      • Installation
      • Requirements
      • Variables
      • Template Tags
  • DataGrab
    • Docs
      • Installation & Upgrading
      • Requirements
      • Troubleshooting
      • Configuring Imports
      • Running Imports
      • Automatic Imports
        • Importing with cron
        • Importing with cron (Deprecated)
      • Endpoints
      • Configuration Options
        • Config File Options
        • Increasing PHP memory limit
      • Import Types
        • CSV
          • Importing into Grid or Matrix field
        • JSON
          • Example file
        • WordPress
        • XML
          • Example file
        • Creating your own import type
      • Assigning Authors
      • Field Types
        • Assets
        • Ansel
        • Bloqs
        • Calendar
        • Channel Images
        • Date
        • File
        • File Grid
        • Fluid
        • Grid
        • Low Events
        • Relationships
        • Simple Grids & Tables
        • Tag & Tagger
        • Matrix (Deprecated)
        • Creating your own fieldtype
      • Publisher
      • Version 5.0
      • Version 6.0
    • FAQs
  • Feature Flags
    • Docs
      • Installation & Upgrading
      • Requirements
      • Configuration
      • Template Tags
      • A/B Testing
    • FAQs
  • Fluidity
    • Docs
      • Installation & Upgrading
      • Requirements
      • Configuration
      • Demos
    • FAQs
  • Logit
    • Docs
      • Installation
      • Requirements
      • Configuration
    • FAQs
  • Publisher
    • Docs
      • Installation
      • Requirements
      • Issues & Tips
      • Languages
      • Template Tags
        • Forms
        • Email Notification Templates
        • Channel:Form
        • Twig
      • URL Translations
      • Auto Translations
      • Diffs
      • Drafts
      • Categories
      • Phrases
      • Persistence
      • Performance
      • Add-ons
        • First Party
        • Third Party
      • Hidden Config
      • Extending Publisher
    • FAQs
  • Reading Time PRo
    • Docs
      • Installation
      • Requirements
      • Configuration
      • Template Tags
  • Reel
    • Docs
      • Installation
      • Requirements
      • Settings
      • Field Tags
    • FAQs
  • Simple Grids & Tables
    • Docs
      • Installation
      • Requirements
      • Template Tags
      • CSV File Imports
      • Field Settings
      • GraphQL
      • Advanced Configuration (deprecated)
    • FAQs
  • Sitemap
    • Docs
      • Installation
      • Requirements
      • Configuration & Usage
    • FAQs
  • Snaptcha
    • Docs
      • Installation
      • Requirements
      • Configuration
      • Template Tags
      • Developers
    • FAQs
  • Speedy
    • Docs
      • Installation & Updating
      • Requirements
      • Configuration
      • Template Tags
      • Static Caching
        • Real World Example
      • Frontedit Support
      • Control Panel
      • Migrating from CE Cache
      • Diagnostics
      • CLI Commands
      • Reverse Proxy Purging
    • FAQs
  • Trek (unreleased)
    • Docs
      • Configuration
    • FAQs
  • Queue
    • Docs
Powered by GitBook
On this page
  • Toolbar Actions
  • Memory Management and timeout issues

Was this helpful?

  1. DataGrab
  2. Docs

Running Imports

Last updated 4 months ago

Was this helpful?

Imports can be executed directly in the ExpressionEngine control panel, or by requesting the Import URL. Note that the Import URL does not require the user to be logged in to execute. Therefore, if a random user discovers your Import URL, they can run a import for you without you knowing! For this reason, it is recommended to use a passkey, which you can set when configuring the import.

If you encounter any issues running an import, the first step you should take is look at your import log, as described in the . DataGrab has pretty extensive logging, so there is a good chance you will find what you are looking for in the logs.

Since DataGrab operates using a queue, there may be times where you view the DataGrab imports list and a status changes, such as displaying a loading bar, on one of the imports without you taking action. This may happen if an item from that import is in the queue and DataGrab takes action on it. When this page is open in the browser the it is always checking for status updates of imports.

Toolbar Actions

Import URL simply gives you the URL to the import so you can run it in a cron, or outside of the control panel.

Memory Management and timeout issues

starts, or continues an import.

restarts an import from the beginning. This button is only visible if an import is WAITING.

lets you edit the import settings, such as it's name, description, and passkey.

lets you configure the import itself, e.g. mapping import file properties to fields in ExpressionEngine.

If an import was halted for any reason, such as reloading the control panel page while it was in progress, it will switch to the WAITING status. You can resume the import at any time by clicking on the icon.

DataGrab uses import queues. When an import initiated it reads your import file and creates records in the queue database table for all the items it needs to import. This could be 5 or 5000 entries. It then uses consumer processes to import the entries in batches, defined by your batch import size setting (defaults to 50). Consumers will stop importing when it hits that limit, or if it reaches a PHP script max execution time. DataGrab will start new consumers when the import is executed through the control panel (see the screenshot above), or if using the ACT url to import. However, if you are importing with the CLI, you will need to manually start new consumers. Fortunately there are things like cron and supervisord that will automatically start new consumers on whatever interval you define. Read more about this in the section of the docs.

Automatic Imports
troubleshooting section