# Overview

If you can't find what you are looking for in our documentation you can submit a support ticket at [boldminded.com/support](https://boldminded.com/support).

{% hint style="warning" %}
All add-ons currently require PHP 8.2 or greater.

**On July 4th 2024 PHP 8.2 became the new minimum requirement for all add-ons**. Expect any add-on released after that date to require 8.2 or greater. Some releases may not immediately take advantage of 8.x specific features in PHP, which means you *might* be able to continue using new releases in PHP 7.4, however, if you experience an error the first thing you should do is update to PHP 8.2 *then* create a support ticket if the error persists.
{% endhint %}

{% content-ref url="/pages/puLaEUuQXlYKiuOVid2c" %}
[Advanced Categories](/advanced-categories/docs)
{% endcontent-ref %}

{% content-ref url="/pages/8Islf32r3CN9A2F5Ni13" %}
[Ansel](/ansel/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-LyL6z1cJM9KfjYeUBuV" %}
[Bloqs](/bloqs/docs)
{% endcontent-ref %}

{% content-ref url="/pages/HU3UkPny8fqRzeqLIMCb" %}
[Carson](/carson/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-LyovuWaL\_p06BX3tf-A" %}
[Custom System Messages](/custom-system-messages/docs)
{% endcontent-ref %}

{% content-ref url="/pages/VLcBbSOCLHXLUHehfD6p" %}
[DataGrab](/datagrab/docs)
{% endcontent-ref %}

{% content-ref url="/pages/0R5yDj5TqhQP04WZ7WhY" %}
[Dexter](/dexter/docs)
{% endcontent-ref %}

{% content-ref url="/pages/05D2qBsmEXGrzd1lAnqu" %}
[Fluidity](/fluidity/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-MUPHzpS9okMnUbOkBKt" %}
[Logit](/logit/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-Ly1iZcylBgjZTVd5ijR" %}
[Publisher](/publisher/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-LyovpC5XYnt7LfTloTw" %}
[Reading Time PRo](/reading-time-pro/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-LyoviZGQdLEzAoZ5qDW" %}
[Reel](/reel/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-LyovnPhMV9ko8hSpIXa" %}
[Simple Grids & Tables](/simple-grids-and-tables/docs)
{% endcontent-ref %}

{% content-ref url="/pages/CYSkfanBGptSrsONpeSq" %}
[Sitemap](/sitemap/docs)
{% endcontent-ref %}

{% content-ref url="/pages/0zJ6HIL5FOwUPXJ2TmbV" %}
[Snaptcha](/snaptcha/docs)
{% endcontent-ref %}

{% content-ref url="/pages/-Ly1iUSY-9XhRmMXotEx" %}
[Speedy](/speedy/docs)
{% endcontent-ref %}


# Docs

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FkMJ6RxsleBhDAn8MrKfk%2Ficon-advanced-categories.png?alt=media\&token=f91473d6-f26d-44bb-90ec-def3d3d17ca9)


# Installation

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

To install Advanced Categories, unzip the zip file and place the `advanced_catgories` folder into your `/system/user/addons` folder. Then login to the Control Panel and go to ***Developer >*** ***Add-ons*** from the main menu and click ***Install*** Advanced Categories.


# Requirements

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

## Required

* ExpressionEngine 6.4.0 or greater
* PHP 7.4 or greater

### Recommended

[Good use of ExpressionEngine's Publish Layouts](/advanced-categories/docs/setup-and-configuration)


# Setup & Configuration

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

Advanced Categories is a custom fieldtype that acts as a facade to manage assignment of categories to entries in ExpressionEngine. Before an entry is saved, the values from the Advanced Categories field are assigned to the ChannelEntry object the same way a normal category field assigns its values. As far as ExpressionEngine is concerned, an Advanced Category field is just a category group assigned to an entry.

{% hint style="warning" %}
**There are 2 important things to note when using Advanced Categories**:

1. Category groups must still be assigned to a channel. You can not add an Advanced Categories field to a channel, assign categories to the field, and expect it to work. Advanced Categories will only display categories that are also natively assigned to the channel through the channel management settings.
2. When using Advanced Categories it is highly recommended to use Publish Layouts to change the display of the default category groups in the Categories tab. Hide category groups that are made irrelevant based on how your Advanced Categories field is configured.
   {% endhint %}

On the left, in the image below, is a default view of an entry publish page when 3 category groups are assigned to the channel, and on the right are 2 Advanced Category fields assigned to the channel. They look very similar, and that is by design, however, they behave differently. The "Call Signs" field on the right has been configured to require at least 1 category selection, but no more than 3. It also happens to be listing the same categories from the Call Signs category group on the left. The rules, and each of the category options were selected to display in the "Call Signs" field settings page. If the default category group remains visible on the publish page users can still interact with it, but if *any* category in the default category group is assigned as an option in an Advanced Categories field, then the categories selected in the Advanced Categories field will superceed anything selected in the default category group.

If an Advanced Categories field lists category options from more than 1 category group, and only a single category is selected, such as the Primary Category field on the right, then all other category groups that have a category as an option in the Advanced Categories field will be deselected. For example, if the default category group "Fruits" was not removed from the Categories tab using a Publish Layout, and the user selected "Apple", but then selected "First Order" from the "Primary Category" field, "Apple" would be deselected and not saved to the entry. This would happen even if the "Primary Category" field allowed more than 1 category to be selected. If no categories from the "Fruits" group were selected in an Advanced Categories field, then no categories from that group are saved to the entry.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F4VXK2Tcip5lvrlvQxWo5%2Fadvanced-categories-1.png?alt=media\&token=a27f4298-be6a-4dfb-9cb9-a8231d18b474)

## Using Publish Layouts

Now that you know how Advanced Category fields work, you can take advantage of Publish Layouts. Below we see the Advanced Categories field called "Primary Category" along side the default "Call Signs" category group in the Categories tab. By taking advantage of Publish Layouts, you can use Advanced Categories to offer more control over entry category assignment. If an Advanced Categories field is meant to replace an existing category group (or multiple groups), then you may want to consider hiding those groups.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FcA2tZ7hQ2XwyUWT5OOfP%2Fadvanced-categories-5.png?alt=media\&token=f9776c78-db39-4317-b0a4-5f66e23b6de1)

## Rules

One of the main reasons Advanced Categories exists is to be able to assign rules to entry category assignments. The default category group fields do not offer the ability to require a number of categories to be selected, nor can they prevent more than 1 category from the group from being selected. Sometimes you need to be able to have such control over category assignments.

In this example, the field has a rule to require at least 1, but no more than 3 categories to be selected. Once 3 selections have been made the other options are disabled preventing additional selections. You can also add custom instruction to the field, which is something you can not do with a category group.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F4d9nii8Yu5z2y2JxBwxB%2Fadvanced-categories-3.png?alt=media\&token=94aecba0-0009-4b03-b8be-8ebcf882b94c)

If a field does not pass validation, you will be informed as such, just like you would any other custom field on the entry.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FMLcM00QvNiKCvzLxutpx%2Fadvanced-categories-4.png?alt=media\&token=0bb8ebfe-85ac-4bc9-98a4-9462e0615f9f)


# Hidden Config Options

`$config['advanced_categories_nuclear'] = 'yes';`

Enabling this option will only save categories from Advanced Categories fields. This means if there is 1 Advanced Categories field, and 5 category groups assigned to an entry then any selected category in the native category field groups will be unselected unless it's selected in an Advanced Categories field on the entry.


# Multi-Site Manager

Advanced Categories works in an MSM install, however, there are some things to be aware of.

* ExpressionEngine lets you share custom fields across multiple sites.
* ExpressionEngine does not let you share category groups across sites.

For those reasons, and based on what Advanced Categories is intended to do, you can not share an Advanced Category field across multiple sites. You will need to make an Advanced Category field unique to each MSM site you are running.


# Template Tags

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

Advanced Categories is a fieldtype, and as such it works just like any other fieldtype in ExpressionEngine. The following examples assume you created a custom field with the label  of "My Category Field", and the field template short name is `{my_category_field}`

Every Advanced Categories field is a tag pair, and even though it is a field, the tag pair functions almost exactly the same as the `{categories}{/categories}` tag pair from the `{exp:channel:entries}` tag ([see ExpressionEngine docs](https://docs.expressionengine.com/latest/channels/entries.html#categories)).&#x20;

```
{exp:channel:entries}
    {title}
    {my_category_field}
        {category_id}
        {category_name}
        {category_image}
        {category_description}
        {category_group}
        {my_custom_category_field_1}
        {my_custom_category_field_2}
        {path="path/to/category/page"}
        {category_total_results} or {total_results}
        {category_count} or {count}
    {/my_category_field}
{/exp:channel:entries}
```

{% hint style="warning" %}
This tag pair does not have support for the `show` or `show_group` parameters
{% endhint %}

Since it is also a fieldtype you have access to several tag modifiers:

`{my_category_field:ids}`\
This will list all the IDs of the categories chosen in the field separated by pipe. For example: `1|2|3`. If only a single category is selected, then its output would be that category ID.

`{my_category_field:url_titles}`\
This will list all the IDs of the categories chosen in the field separated by pipe. For example: `alpha|beta|charlie`. If only a single category is selected, then its output would be that category ID.

`{my_category_field:category_names}`\
This will list all the IDs of the categories chosen in the field separated by pipe. For example: Alpha`|Beta|Charlie`. If only a single category is selected, then its output would be that category ID.

All tag modifiers above support the `separator=","` parameter where you can change the pipe separator to another character of your choice. For example`{my_category_field:ids separator=","}` will output `1,2,3`

If you wanted to see if an entry is assigned to a category you can use a simple conditional:

```
{exp:channel:entries}
    {title}
    {if my_category_field:ids ~ "/(1|2|3)/"}
        This entry is assigned to either category 1, 2 or 3
    {/if}
{/exp:channel:entries}
```


# FAQs

{% hint style="danger" %}
Advanced Categories has been deprecated and is no longer for sale or supported. ExpressionEngine 7.4 introduces many of Advanced Categories features as natively to ExpressionEngine.
{% endhint %}

### Why would I need Advanced Categories?

Just like a lot of add-ons Advanced Categories is not required for every site, but some sites need more control over how categories are assigned to entries, and ExpressionEngine's native category fields do not have any rules around category selection. Also, as your site ages, its possible that some categories become deprecated, or category usage changes.&#x20;

For example, maybe you don't want any new entries to be assigned to certain categories or category groups, but have hundreds or thousands of old entries where those category assignments are still relevant.

Another example, maybe your category groups make sense and are still used to display content on the front-end, but you only want the user to be able to assign a single category from one of two or more category groups. Advanced Categories lets you create a custom field with a single list of categories from multiple groups, and limit it to a single category selection.

With Advanced Categories you can customize which categories, from any group, are presented to an user when editing an entry. There are a lot of different reasons someone may want to drastically alter how categories are managed on an entry. Advanced Categories is designed to fit those needs.

### Does Advanced Categories replace the category groups on the entry publish page?

No, it does not replace the the Categories tab or the category group fields that appear on you rentry publish pages. Advanced Categories is a separate, standalone fieldtype that acts as a facade for assigning categories to an entry.

### What happens if I uninstall Advanced Categories?

Since Advanced Categories is just a custom fieldtype that acts as a facade for saving categories to an entry, uninstalling the add-on will not remove any assigned categories to any entries. Uninstalling it will simply revert the entry publish form to the state it was in before you installed it. If you're using the Advanced Categories custom fields in your front-end templates, such as `{primary_category:ids}` then obviously you'll need to remove the field tags from your templates.&#x20;

### Wouldn't adding separate category fields to an entry publish page confuse editors?

Well, this is always a possiblity, but this is why it is **strongly** suggested to use the Publish Layouts feature in ExpressionEngine to either hide the default Categories tab entirely, or certain category groups. You can do this globally, or by user group, depending on what level of experience or knowledge your users have with the system. For Super Admin users, or highly experienced users maybe you want to keep the default Categories tab and category groups visible. For less experienced, or trusted users, hiding the default Categories tab and or category groups may be a better apporach.

### Is there a template tag to list categories?

If this is a question asking about a module template tag to replace the `{exp:channel:categories}` tag, then no. This is fieldtype only to assign categories to an entry. It is not, and will not, include any functionality to list categories outside of the context of a custom field.

### How do I display nested categories?

You can't, and there are no plans to support it. Since Advanced Categories lets you customize which categories are displayed, which means you can create more than 1 Advanced Categories field to display child relationships. Advanced Categories is intended to help *simplify* category display and selection. It is of our opinion that nested categories only complicate matters. Simple, flat category listing or bust :)&#x20;


# Docs


# Installation

To install Ansel move the `/addons/ansel/` directory to your `/system/user/addons/` folder. Also copy the `/themes/user/ansel/` folder to the `/themes/user/` directory in your ExpressionEngine install. Login to your control panel and visit the ***Developer > Add-ons*** page, and click the Install link for Ansel.


# Upgrading

### If upgrading from Ansel 2.x to Ansel 3.x, the process should be seemless, but there are a few things to note.

{% hint style="warning" %}
The most noticeable functional change is that Ansel now only supports the native File Manager. The Assets and Treasury add-ons not longer work with Ansel. You can [read more about this decision in the FAQs.](/ansel/faqs)
{% endhint %}

In an effort to be more aligned with the native ExpressionEngine File Manager, the Caption field has been renamed to Description. When upgrading all field settings and database columns will be updated as such. If your template is using the `{img:caption}` variable it will still work. There is now an `{img:description}` tag.

Since Ansel and the native File Manager both have a Title and Description field, there is a new hidden config variable that when enabled will keep the two in-sync. This config value can be added or removed at any time (it defaults to 'n'), but if you add this to your config.php file prior to upgrading it will copy all the Title and Description field values from your Ansel files to the native File Manager. E.g. `$config['ansel_sync_meta_fields'] = 'y';`&#x20;

{% hint style="info" %}
Not using the `ansel_sync_meta_fields` variable, or setting it to `n`, will make Ansel operate as it did in previous versions. Adding it will turn on the new metadata sync behavior.
{% endhint %}

Once an image is saved, the edit meta button will become available (screenshot below). Clicking this will let you edit the meta data of the cropped image, not the original source image. This lets you add a Title and Description that is unique to each usage. This is where you might want to decide if you want to keep the meta fields in-sync with the native FIle Manager. This does not change the front-end behavior when displaying images within an Ansel field. The syncing is mostly available for convenience and consistency. For example, if a content editor who isn't aware of how Ansel or the File Manager worked viewed a cropped image from within the ExpressionEngine File Manager, and chose to update the Title or Description fields it would not update the values of the image in the entry, and thus not reflect the changes on the front-end. However, if you enable syncing, the cropped image will be updated as well and the changes will be reflected on the front-end, and the next time the entry is viewed and edited.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FMQipD38utae7S8CTFULE%2Fansel-edit-meta-link.png?alt=media&amp;token=4c7a5fbf-77f3-48a5-b405-b9b8e7a576d9" alt="" width="262"><figcaption></figcaption></figure>

If `ansel_sync_meta_fields` is enabled, then changes made to the Title and Description field in the image slide out will update the the Title and Description in Ansel. Conversely, if the Title or Description is updated in the Ansel field, and the entry is saved, then the Title and Description field in the slide out will also be updated. These are separate fields stored in separate database tables, only when `ansel_sync_meta_fields` is enabled will they be kept in-sync.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FYjyK16eGfWRykUhKU7Nh%2Fansel-meta-modal.png?alt=media&amp;token=7b7b3aa9-c066-488f-a063-1b48109e2d7a" alt=""><figcaption></figcaption></figure>


# Migrating from Assets

If you are using the Assets module to manage files, and want to switch to ExpressionEngine's native File Manager by changing the Upload, Save, and Preview folder settings in an Ansel field you will need to run a migration command, which will update the file IDs in the `exp_ansel_images` table.

**Please backup your database prior to running this command.**

In this example change `4` to whatever upload directory ID that Assets and Ansel are using. Run the command for each directory you need to update.

```bash
php system/ee/eecli.php ansel:migrate-assets --dir=4
```


# Requirements

* ExpressionEngine 7.4 or greater
  * Using native File Manager (compatibility mode on or off)
* PHP 8.2 or greater
  * GD (or GD 2) library. ImageMagick is also supported if installed.
  * At least 256 megabytes of memory allocated and available to PHP
    * 512 megabytes is recommended (and more if you can spare it). Manipulating images can take a lot of memory. And particularly, the larger image, the more memory it will take.
  * `jpegoptim` for jpeg optimization
  * `gifsicle` for gif optimization
  * `optipng` for PNG optimization


# Settings

### Field Type Contexts

Ansel is compatible with the following contexts, and regardless of the context uses the same settings presented as uniformly as possible.

* [Channel Entries](https://docs.expressionengine.com/latest/control-panel/create.html)
* [Fluid Field](https://docs.expressionengine.com/latest/fieldtypes/fluid.html)
* [Grid](https://docs.expressionengine.com/latest/fieldtypes/grid.html)
* [Bloqs](https://boldminded.com/add-ons/bloqs)
* [Pro Variables](https://docs.expressionengine.com/latest/add-ons/pro-variables/overview.html)

### Storage Locations

Ansel is compatible with ExpressionEngine's new File Manager added in version 7. It supports sub-folders or compatibility mode (no sub-folders). *Ansel 3.0 no longer supports Assets or the Treasury modules.* ExpressionEngine's new File Manager is extremely robust and is the recommended path forward for the best experience. [You can read more about this change in the FAQs.](/ansel/faqs)

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F12VQ2ZrLaR4fxN3p3r4C%2Fansel-field-settings.png?alt=media&amp;token=008881b8-630d-4fc3-b514-472aae8f0188" alt="" width="563"><figcaption><p>Ansel's Field Settings</p></figcaption></figure>

***

### Upload, Save, and Live Preview Locations

The first two settings for any Ansel field are very related and require a tiny bit of explanation.

The Upload directory is where source images are drawn from or uploaded to. It is meant to be a user facing directory — the one your users will see when they add an image to an Ansel field.

The Save directory is where Ansel saves images the user crops or otherwise adds to the field. The end user should not ever need to see this directory. Ansel names the files with image IDs and timestamps for cache breaking purposes and so on when the user updates the images in any field.

The Preview directory is where Ansel saves images when viewing an entry in Live Preview mode. The contents of this directory are temporary, and when an entry is saved the contents of the directory are deleted. You can use the same directory for all Ansel fields, or create a unique directory for each field. If a user previews an entry and does not save it, then the images will not be deleted, and this is fine. The next time any entry using an Ansel field is saved the previous preview images will be removed even if they are not used in the entry.

{% hint style="info" %}
There are a few different approaches and ways of thinking about this. You could create a source and save directory for each field and keep everything separate, or you could create one general upload directory for all source images, and a save directory for each field. Either approach has its merits. Either way, it is highly recommended that you create a save directory for each field you create. This allows you to keep your images well segregated and create unique image manipulations per directory and field as well.
{% endhint %}

#### Tile View

Tile View is the new default layout introduced in Ansel 3. It is a more compact view that is more mobile friendly and more subitble to fields that include a lot of images. Disabling it will revert to the legacy vertical table layout.

#### Min and Max Quantity

Each Ansel field can hold any number of images, and these settings allow you to define how few and how many images a field can hold.&#x20;

If tile view is enabled, max is set to 1, and prevent uploads when max quantity is enabled, Ansel will be put into single image view mode, which automatically displays the meta fields if enabled and removes the ability to toggle their display. The Cover metafield will also be disabled. If you're displaying only 1 image then there is no group of images it needs to be differentiated from as a cover.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FhgCmiAQyQIn45MoFqqhB%2Fansel-single-image-display.png?alt=media&amp;token=8c4d055a-8672-4253-8ed0-563175e69d93" alt="" width="375"><figcaption></figcaption></figure>

#### Image Quality

Set the JPEG quality from 1 to 100.

{% hint style="info" %}
It is not normally a good idea to set the quality to 100. Anywhere from 70 to 90 is usually a good area to be in terms of quality vs. file size.
{% endhint %}

#### Force JPEG

Normally, image files are manipulated and saved in their original format. However, if you want all images to be saved as JPEG, you can specify so with this setting.

#### Retina Mode

When set to yes, the dimensions you enter for min/max width/height below will all be doubled. This allows you to set dimensions based on how you would think or normally specify dimensions in CSS, but doubles those dimensions so that your images will be retina ready.

#### Min/Max Width/Height

Use these fields to define image dimension requirements. Ansel will ensure that the image the user selects is at least as large as the minimum dimensions specified and that they cannot select a crop smaller than that. And Ansel will resize the image to meet the maximum specified size if the user selects a crop are that is larger than specified.

#### Crop Ratio

Enter a crop ratio to make sure the image is constrained to a certain aspect. The ratio is entered as 16:9, where the first number on the left side of the colon is the width of the ratio, and the number on the right side of the colon is the height of the ratio.

#### Named Crop Variants

Version 3.2 adds named crop variants with per-field crop definitions that let editors set independent crop regions for different viewports or display contexts. Each variant produces its own image file and is exposed as template variables and a ready-made `<picture>` tag.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FppagGZZXDZfqzocR42eG%2Fansel-named-crops-settings.png?alt=media&amp;token=216537e3-1318-4afd-9f95-4b5ffb30d620" alt=""><figcaption></figcaption></figure>

#### Display Fields

There are three optional fields that can be set to show or hide, required, and the label of the fields can be customized. The title and caption fields are text input fields, and the cover field is a checkbox. Only one can be selected per field.

### Config File Settings

Ansel has some settings that are only controlled via the `config.php` file.&#x20;

Disabled by default, but if you want the Title and Description meta fields in Ansel to stay in sync with the Title and Description field in the File Manager enable this setting.

```php
$config['ansel_sync_meta_fields'] = 'y';
```

Ansel defaults to ImageMagick, then uses GD if ImageMagick is not available. If you have both installed on your server and want to use GD instead, then you'll need add this to your `config.php` file

```php
$config['ansel']['forceGD'] = 'y';
```

If you're using image manipulations and optimizations you can optionally display any errors that might occur when performing these actions.

```php
$config['ansel']['optimizerShowErrors'] = 'y';
```


# Usage

### Getting To Know Ansel

Ansel will feel very familiar to use in the publishing workflow because it builds on the basic ExpressionEngine control panel styles.

There are two options for getting an image into your field:

* Drag and drop an image to the drop area or click it to select an image to upload
* Choose an existing image in the file manager (or use the file manager to upload an image)

One of the advantages to using the drag and drop interface is the image will never be uploaded if it does not meet requirements — whereas if you go through the file manger interface and upload from there, the image will be uploaded by EE then evaluated by Ansel.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FoesnEclWIgEpnaKpCByj%2Fansel-meta-fields-visible.png?alt=media&amp;token=0cf3c6e2-5be4-4aa7-9801-be03198ba284" alt=""><figcaption><p>Tile view with meta fields toggled</p></figcaption></figure>

### Adding Images

When you drag and drop an image over the upload area in a field (or choose one from the existing images in the file manager), Ansel will automatically select the largest portion of the image possible given the constraints set up for the field and add an image row.

You can use the re-order handles to change the order of the images in the field.

To change the selected area and/or crop of the image, click the image thumbnail which will bring up the cropping interface with a lasso to select the relevant portion of the image. When you have selected the area of the image you want, click the green check mark or press the enter key.

If you wish to cancel the crop operation, click the red X, or press escape.

### Using metadata fields

Title, Description, and Cover fields are optional metadata that can be enabled in the field's settings. Ansel saves these values into it's own table along with the cropped images, but it also sync's the Tile and Description fields to ExpressionEngine's native Files storage. The Cover field has no equivalent in ExpressionEngine, so it is not synced. If you use the `ansel_sync_meta_fields` config value this will be a two way sync. Read more about this in the [Upgrading guide](/ansel/docs/upgrading).


# Channel:Form

Ansel works in channel:form, however you will notice that it will appear largely unstyled. This is because it relies on many styles and icons normally found within ExpressionEngine's control panel, but are not easily available on the front-end.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FoUKvaRwTBlF1OUkixTMD%2Fansel-cf-unstyled.png?alt=media&amp;token=038ca724-9996-47f2-8fcf-73d4be109ad9" alt=""><figcaption></figcaption></figure>

Including the following css file, per the channel:form docs, is not sufficient.

```html
<link href="{path='css/_ee_channel_form_css'}" type="text/css" rel="stylesheet" media="screen">
```

You can include the entire `common.min.css`file, which is what is used by ExpressionEngine in the control panel, but it may apply more styles than necessary. If you include this you will notice that the Ansel field displays its missing styles.

```html
<link href="/themes/ee/cp/css/common.min.css" type="text/css" rel="stylesheet" media="screen">
```

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FnqzKzSe90gSbaUPvwgf6%2Fansel-cf-styled.png?alt=media&amp;token=d808d8bd-39d9-4fed-8b25-9ee8a5d89906" alt=""><figcaption></figcaption></figure>

If you're using the legacy grid view in Ansel, you will not need to include the `common.min.css` file, it still renders most of it's styles.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2Fb9cB40Gacl72aSoNf0Vo%2Fansel-cf-grid.png?alt=media&amp;token=c573b7a7-decd-489f-810d-b8b838ea64b7" alt=""><figcaption></figcaption></figure>

**Ultimately it is up to you how to render the Ansel field in a channel:form. A little extra work to add custom styles may be necessary if you do not want to include `common.min.css`.**

We do recommend hiding the meta edit button since the slide out modal normally find when inside of the ExpressionEngine control panel is not available in channel:form tags.

```css
.js-ansel-edit-meta { display: none; }
```


# Troubleshooting

If you're encountering an issue with Ansel take these steps before submitting a support ticket.

## Check the Developer logs

Ansel 3 does a bit more logging of potential error messages than previous versions of Ansel. If you're encountering an issue hopefully it is getting logged. Please share the log messages when reporting an issue if they are available.

### Missing rendered images

If an image is not rendering (it is returning a 404) but it has a URL there are a couple things you can check. In your're using the `:resize` modifier in your Ansel tags the  Ansel Saves folder should contain an `_ansel_image_cache` folder. If there is not then the Saves folder may not be writable or someone or something deleted the `_ansel_image_cache` folder. If the `_ansel_image_cache` folder exists but you do not see the resized image in that folder then you should check the `user/cache/ansel_persistent` directory. In an effort to reduce overhead and unnecessary image generation Ansel creates a simple text cache file in the `user/cache/ansel_persistent` directory that correlates to the usage of an Ansel field or module tag. Delete all the contents of the persistent directory, then try reloading the page containing your Ansel tags. If this was the issue you were experiencing then your resized images should be regenerated and display as expected.

### Using in Channel Form

Ansel does not currently work in `channel:form` tags. As far as we can tell this is not an an issue with the new 3.0 version. The old 2.3.4 version also does not work in `channel:form` tags in ExpressionEngine 7.5.x. It appears something has changed in ExpressionEngine that affects Ansel's usage in `channel:form` tags but we have not found a fix.


# Template Tags

The tag pair is how you output Ansel images. Whether you are using the custom field tag within a Channel Entry tag pair, a Grid field tag pair, a Bloqs field tag pair, a Low Variables tag pair, or the stand alone tag pair, usage is almost identical across the tags. In fact, the only real difference is with the stand alone tag pair, which has just a few extra parameters available to it.

```html
{exp:channel:entries channel="my_channel"}
    <!-- Number of images in field -->
    {my_ansel_field count="true"}

    <!-- Check if field has images -->
    {if "{my_ansel_field count='true'}" > 0}
        <!-- Do stuff if field has images -->
    {/if}

    <!-- Output images -->
    {my_ansel_field}
        {img:url}
        
        <!-- If using named crops output the entire <picture> tag -->
        {img:picture_tag}
    {/my_ansel_field}

    <!-- Output images from a grid field -->
    {grid_field}
        {grid_field:my_ansel_field}
            {img:url}
        {/grid_field:my_ansel_field}
    {/grid_field}
    
    <!-- Output images from a Relationship field -->
    {my_relationship_field}
        {my_ansel_field}
            {img:url}
        {/my_ansel_field}
        
        or
        
        {my_relationship_field:my_ansel_field}
            {img:url}
        {/my_relationship_field:my_ansel_field}
    {/my_relationship_field}
    
    <!-- Output images from nested Relationship fields across 2 entries -->
    {my_relationship_field}
        {my_relationship_field:child_relationship_field}
            {my_ansel_field}
                {img:url}
            {/my_ansel_field}
        {/my_relationship_field:child_relationship_field}
    {/my_relationship_field}

    <!-- Output images from a blocks field -->
    {blocks_field}
        {my_block}
            {my_ansel_block_field}
                {img:url}
            {/my_ansel_block_field}
        {/my_block}
    {/blocks_field}
{/exp:channel:entries}

<!-- Output images from a Low Variables Ansel field -->
{exp:low_variables:pair var="my_ansel_var"}
    {img:url:resize width="500"}
{/exp:low_variables:pair}

<!-- Output images with the stand alone tag -->
{exp:ansel:images content_id="102" field_id="33"}
    {if img:no_results}
        <!-- Do no results markup here -->
    {/if}
    {img:url:resize width="400"}
{/exp:ansel:images}
```

### Tag Pair Parameters

```php
image_id="104|105"
not_image_id="106|110"

site_id="2"
not_site_id="1"

file_id="2|3" // EE file id
not_file_id="18|27"

original_location_type="ee|assets" // Possible values: ee|assets|treasury
not_original_location_type="treasury|ee"

original_file_id="34|45"
not_original_file_id="45|56"

upload_location_type="assets|treasury" // Possible values: ee|assets|treasury
not_upload_location_type="ee|assets"

upload_location_id="34|45"
not_upload_location_id="45|56"

filename="my_file" // do not include extension
not_filename="foo"

extension="jpg"
not_extension="jpg"

original_extension="png"
not_original_extension="png"

filesize="182827"
filesize="< 182827"
filesize="> 182827"

original_filesize="182827"
original_filesize="< 182827"
original_filesize="> 182827"

width="300"
width="< 300"
width="> 300"

height="300"
height="< 300"
height="> 300"

title="my title|my other title"
not_title="foo|baz"

description="my description|my other description"
not_description="foo|baz"

member_id="23|24"
not_member_id="56|23"

position="2"
position="< 2"
position="> 2"

cover_only="true"
skip_cover="yes"
show_disabled="y"

namespace="my_namespace" // Access variables as {my_namespace:var}

limit="4"

offset="4"

order_by="date:desc|order:asc"

random="yes" // Overrides order_by parameter

cover_first="yes"

manipulations="true" // Include EE directory manipulations (requires extra queries)

host="https://cdn.domain.com/"
```

The following parameters are unique to the stand-alone image tag:

```php
not_source_id="45|46"

content_id="34|36" or entry_id="34|36" // Usually the entry id, but could be low variables id
not_content_id="46|68" or not_entry_id="34|36"

content_type="channel|grid" // Possible values: channel|grid|blocks|low_variables
not_content_type="low_variables|channel"

field_id="23|26" or field_name="foobar"
not_field_id="45|56" or not_field_name="foobar"

row_id="32|33" // Grid or Blocks
not_row_id="444|745"

col_id="43|44" // Grid or Blocks
not_col_id="5|10"
```

### Tag Pair Variables

Note all variable examples assume the default namespace of img. Use your custom namespace if you have set the namespace parameter.&#x20;

These variables are available in the tag pair:

```html
{img:id}
{img:site_id}
{img:content_id}
{img:field_id}
{img:content_type}
{img:row_id}
{img:col_id}
{img:file_id}
{img:original_file_id}
{img:upload_location_id}
{img:filename}
{img:original_filename}
{img:basename}
{img:original_basename}
{img:extension}
{img:original_extension}
{img:filesize}
{img:original_filesize}
{img:width}
{img:height}
{img:title}

<!-- 
This is the legacy variable, but {img:description} is preferred.
img:caption has it's value wrapped in htmlspecialchar(), meanting it will
output any html as encoded html, thus not rendering as html on output.
If you need the raw html to render, use the _raw alias.
-->
{img:caption}
{img:caption_raw}

<!--
This is the new variable that replaces caption.
img:description has it's value wrapped in htmlspecialchar(), meanting it will
output any html as encoded html, thus not rendering as html on output.
If you need the raw html to render, use the _raw alias.
-->
{img:description} 
{img:description_raw}

{img:member_id}
{img:position}
{img:cover}
{img:upload_date}
{img:modify_date}
{img:path}
{img:original_path}
{img:url}
{img:original_url}
{img:thumbnail_path}
{img:thumbnail_url}
{img:index}
{img:count}
{img:total_results}
{img:host}
{img:description_field}
{img:credit_field}
{img:location_field}
{img:original_title_field} or {img:original_title}
{img:original_description_field} or {img:original_description}
{img:original_credit_field} or {img:original_credit}
{img:original_location_field} or {img:original_location}
```

The variables starting with "original" reference the value from the source file. When Ansel saves an image, it creates a new instance of that image in your file manager (wether it was cropped or not). Each instance of a file can have it's own description, caption, title etc. If you want to always reference the original description, regardless of which cropped version you're displaying in the template, you can use a conditional. For example:

```
{if img:original_description}
    {img:original_description}
{if:else}
    {img:description}
{/if}
```

When you have made manipulations available with the parameter `manipulations="true"`, they will be available as follows:

```html
{img:manipulation_name:path}
{img:manipulation_name:url}
{img:manipulation_name:width}
{img:manipulation_name:height}
```

### No Results

If your tag pair has no results, you can use the `{if img:no_results}` conditional to output content.

### On the Fly Cropping and Resizing

Ansel can crop and resize images on the fly. Doing so is very efficient because the cropped/resized image is cached after the first request for it is made. Any request for that cropped/resized image thereafter comes straight from the cache on disk.

Cropping/resizing is available to the `{img:url}` variable.

```php
width="400"
height="100"
crop="true"
background="d34747"
force_jpg="true"
quality="80"
scale_up="true"
cache_time="86400" // Defaults to forever
```

```php
{img:url:resize
    width="400"
    height="100"
    crop="true"
    background="d34747"
    force_jpg="true"
    quality="80"
    scale_up="true"
}
```

### Get Total

Sometimes you would like to display the total number of images in a field. Ansel makes this really easy with a parameter on the custom field tag:

```php
{exp:channel:entries channel="my_channel"}
    {my_ansel_field count="true"}
{/exp:channel:entries}
```

### Picture Tag

#### `{img:picture_tag}` <a href="#imgpicture_tag" id="imgpicture_tag"></a>

Outputs a complete `<picture>` element. Only named crops with a **Media Query** value are included as `<source>` elements. The fallback `<img>` always uses the main crop (`{img:url}`).

Example output:

```html
<picture>
  <source media="(min-width: 1200px)" srcset="https://example.com/uploads/ansel_saves/photo-42-desktop-1234567890.jpg">
  <source media="(min-width: 768px)"  srcset="https://example.com/uploads/ansel_saves/photo-42-tablet-1234567890.jpg">
  <source media="(max-width: 767px)"  srcset="https://example.com/uploads/ansel_saves/photo-42-mobile-1234567890.jpg">
  <img src="https://example.com/uploads/ansel_saves/photo-42-1234567890.jpg"
       width="1200" height="675" alt="Photo caption" loading="lazy">
</picture>
```

**Tag Parameters**

| Parameter             | Default  | Description                                             |
| --------------------- | -------- | ------------------------------------------------------- |
| `picture_tag_loading` | `lazy`   | Value of the `loading` attribute on the `<img>` element |
| `picture_tag_class`   | *(none)* | CSS class applied to the `<picture>` element            |

Example:

```ee
{ansel_field picture_tag_loading="eager" picture_tag_class="hero-image"}
    {img:picture_tag}
{/ansel_field}
```

***

### Config Variable Substitution in Media Queries <a href="#config-variable-substitution-in-media-queries" id="config-variable-substitution-in-media-queries"></a>

Media query values in the field settings support `{config_key}` placeholders that are resolved at render time from EE's `$config` array. This avoids hard-coding breakpoints in the database and makes them easy to update site-wide.

**In `config/config.php`:**

```php
$config['bp_small']  = '(max-width: 767px)';
$config['bp_medium'] = '(min-width: 768px)';
$config['bp_large']  = '(min-width: 1200px)';
```

**In the field settings Media Query column:**

```
{bp_large}
{bp_medium}
{bp_small}
```

At render time these resolve to their config values. Unrecognized placeholders are left as-is so typos are immediately visible in the rendered HTML.

### Twig/Blade

If you're using [Coilpack](https://expressionengine.github.io/coilpack-docs/docs/), Ansel supports Twig and Blade templates. All of the same parameters from the EE Template examples above are also available in Twig or Blade.

{% tabs %}
{% tab title="EE Template" %}

```html
{exp:channel:entries
    channel="pages"
    dynamic="no"
}
    <h1>{title}</h1>
    
    {my_ansel_field}
        <img src="{img:url:resize width='600' height='200'}" />
        <!-- See list above for additional variables -->
    {/my_ansel_field}

{/exp:channel:entries}
```

{% endtab %}

{% tab title="Twig" %}

```twig
{% for entry in exp.channel.entries({
    channel: 'pages',
    dynamic: 'no'
}) %}
    <h1>{{ entry.title }}</h1>

    {% for row in entry.my_ansel_field %}
        {{ row.url }} <!-- or -->
        <img src="{{ row.url.resize({width: 400, quality: 60}) }}" />
        
        {{ row.title }}
        {{ row.description }}
        {{ row.file_id }}
        <!-- See list above for additional variables -->
    {% endfor %}
    
{% else %}
    <span>No entries found</span>
{% endfor %}
```

{% endtab %}

{% tab title="Blade" %}
Example coming...
{% endtab %}
{% endtabs %}


# Publisher Support

If you're using Ansel and Publisher, you will also need this extension: <https://boldminded.com/add-ons/publisher-ansel>


# FAQs

### Does Ansel support the Assets field type?

No, version 3.0 removed support for Assets. It only supports the native File Manager. Simply updating the Ansel field settings to point to the same EE file directory as an Assets folder will not work. The ExpressionEngine team has been working on an Assets to native File Manager migration tool, which will be released soon. For this reason, we've decided to end support for the Assets field and recommend using the migration tool when it becomes available. Assets, while being a great add-on, has become less important when building ExpressionEngine sites now that the native File Manager supports sub-folders, and remote (e.g. S3) file services. So in an effort to keep things moving forward, and to simplify the Ansel codebase, we've decided not to support Assets.

### Does Ansel support Treasury?

No, version 3.0 removed support for Treasury, for much the same reason as removing support for Assets.

### Does Ansel work in channel:form tags?

Ansel does not currently work in `channel:form` tags. As far as we can tell this is not an an issue with the new 3.0 version. The old 2.3.4 version also does not work in `channel:form` tags in ExpressionEngine 7.5.x. It appears something has changed in ExpressionEngine that affects Ansel's usage in `channel:form` tags but we have not found a fix.


# Docs


# Why Bloqs?

Bloqs is like a Fluid field, but on steroids. It gives you the freedom to build robust repeatable content modules using the [atomic design pattern](https://bradfrost.com/blog/post/atomic-web-design/). Functionally Bloqs shares more similarities with ExpressionEngine's native Grid field. Every row in the Grid field has the same custom field options because they all share the same columns. Think of Bloqs as a Grid field where each row has different columns, and you can place the rows in any order you want.&#x20;

If a Grid field has rows and columns, then a Bloqs field has bloqs and atoms. You create a bloq, then add one or more custom fields (aka atoms) to it, and then assign that bloq to one or more custom fields in a channel. Unlike the native Fluid field, since a single bloq contains multiple custom fields, those fields can work in conjunction with each other. One field can contain your text, the other one might be a select menu determining how that text is displayed. If you're using [nested bloqs](/bloqs/docs/features/nesting), the fields within a single bloq can be even more powerful as their values can cascade down to child bloqs.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLweK5wMFHpLui_2WQ6%2F-MLwhGQkAUzRAvXUEoZP%2Fbloqs-hoth.gif?alt=media\&token=f011894c-534b-48ab-baef-eaf81a38f7ba)


# Features

This section highlights some important features in Bloqs


# Add Bloq Menu

When managing a bloq definition, you can optionally assign the bloq to a group that you define. If a bloq is not assigned to a group, it is considered "Ungrouped", which is the default.

![The Group option when managing a bloq definition.](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFLy5oerhy1rSM-po3%2F-MKFM0eFbQS4OT2eZ3is%2Fbloqs-group-option.png?alt=media\&token=eef3d703-9f8e-4b9e-9a62-c1c8266fc698)

By default, the Add Bloq menu will display as a vertical list. If no additional Bloq Groups are defined, then bloqs will not be organized by group in the Add Bloq menu.

![The default view of the Add Bloq menu when using groups.](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFLy5oerhy1rSM-po3%2F-MKFO3NcqKyAaX50lJQx%2Fbloqs-add-list.png?alt=media\&token=aa464db9-ef01-462e-be60-4a41621acb90)

### Add Menu Grid

When editing the Bloqs field from the *Channel > Fields* menu there is an option to display the Add Bloq menu as a grid. This will organized bloqs by groups (if defined) to a 3 column layout, and also display a choosen image or icon with the bloq. The image or icon can be used to help visually identify what type of bloq it is.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFKT4ZBY2pqxUHgUu2%2F-MKFLHzWZy4sQIGSXUxE%2Fbloqs-grid-option.png?alt=media\&token=457e003d-9d0c-4fda-ae9c-1a7bc143b458)

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFLy5oerhy1rSM-po3%2F-MKFO7IpF736qBbDR5xv%2Fbloqs-add-grid.png?alt=media\&token=d3f724e4-edde-4b5a-a7cb-1558030349ba)

### Preview Image or Icon

When displaying the Add Bloq menu as a grid, you can assign an image or icon to the bloq. The Image field uses ExpressionEngine's native File Manager, so you can add any image you want. It is recommended to keep your image in the 50x50 pixel dimension. You can also use one of the existing Icons from the Font Awesome library that is available through ExpressionEngine. **The Preview Icon field is only available in ExpressionEngine 6 or greater.**

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFLy5oerhy1rSM-po3%2F-MKFPPgmcfnuGGlNn1iv%2Fbloqs-image-icon.png?alt=media\&token=7a829df9-e738-43ff-a321-1cb0102d088e)


# Nesting

With Bloqs you can create true heirarchial relationships with your content, and have them render as such in your templates. To start nesting just enable it on the field, and add an additional closing tag pair to each bloq as mentioned below.

Some basic nesting rules are also available that can be applied to each bloq.

* Can be nested at any level
* Can only be at the root
* Can't be root and must be a child of another bloq
* Set a bloq as must be a child of another bloq by selecting 1 or more possible parent bloqs
* Set a bloq is allowed to have child bloqs
* Set a bloq as having a minimum, maximum, or exact number of child bloqs

While editing your bloqs you may see warning messages when one of these rules is violated, however, Bloqs will not stop an entry from saving if a nestable tree is invalid based on the rules you defined. It is up to the content editors to ensure the nesting is valid before saving an entry (no warning messages means it is valid). If an invalid bloq tree is saved, the bloqs will still be rendered in your templates, but they may not display as you expect them to based on nesting rules and the HTML and CSS defined for the bloq.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MOhBCLIg-kziUnBYlcM%2F-MOhDwGzX4VSUyHF0QLr%2Fbloqs-nesting-error.png?alt=media\&token=d4fdbda3-5742-4aca-903f-704ba921130f)

When nesting is enabled you can choose to add a new bloq as an immediate child of the current bloq. Use the "Add bloq as child" to avoid excessive drag and drop of bloqs to build a nested structure. When nesting rules are applied, the menu respects those rules and will not let you insert a bloq where it does not belong.&#x20;

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MOhBCLIg-kziUnBYlcM%2F-MOhDXVBBVksVv55lwBA%2Fbloqs-add-child-small.png?alt=media\&token=b8b6acbe-8a41-41d2-8527-0efca43f160c)

### Enabling Nesting

All Bloqs fields default to the old sortable behavior. You must enable nesting on each field. If you want to use Bloqs, but don't need the nesting functionality, just don't enable the option and continue using Bloqs like you always have (you won't need the closing tag pair mentioned below).

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLsN6e6CCPJuq28cmFP%2F-MLsNUeeSRp-ZfPYv-lj%2Fbloqs-setting-nesting.png?alt=media\&token=02f591b4-5461-4f1e-90cf-066f41debae7)

### Variables

If nesting is enabled, each bloq contains a few new variables:

```
{bloqs:parent:id}

{bloqs:parent:shortname}

{bloqs:children:total_bloqs} or {bloqs:children:total_rows}

{bloqs:siblings:total_bloqs} or {bloqs:siblings:total_rows}
```

### Template Tags

When nesting is enabled, the templates expect that each bloq have its own `{bloqs:children}` tag. This tag will render all child bloqs of the current bloq recursively.

```
{bloqs_field}
    {section_bloq}
        <div class="section"}
            <h1>{heading_atom}</h1>
            {bloqs:children}
        </div>
    {/section_bloq}

    {row_bloq}
        <div class="row"}
            {another_atom}
            {bloqs:children}
        </div>
    {/row_bloq}
{/bloqs_field}
```

{% hint style="warning" %}
The following documentation about the `{close:*}{/close:*}` tag pairs was deprecated in version 4.6.0, and completely removed in version 5.0. For this reason, it is suggested to use the new `{bloqs:children}` syntax noted above.
{% endhint %}

When nesting is enabled, the templates expect that each bloq have its own closing tag. The contents of this tag will be moved to the appropriate location in the final template output to wrap child bloqs. If you create a "section" bloq that is the parent of several child bloqs, the closing content of the section bloq will be added after the child bloqs have been rendered to create a properly nested HTML structure. The closing tag is always the name of the bloq it is in, prefixed with "close:"

```html
{bloqs_field}
    {section_bloq}
        <div class="section"}
            <h1>{heading_atom}</h1>
        {close:section_bloq}
        </div>
        {/close:section_bloq}
    {/section_bloq}

    {row_bloq}
        <div class="row"}
            {another_atom}
        {close:row_bloq}
        </div>
        {/close:row_bloq}
    {/row_bloq}
{/bloqs_field}
```

If you have templates using the old {close} tag syntax, here is an example of how the previous example would be updated to use {bloqs:children}.

```html
{bloqs_field}
    {section_bloq}
        <div class="section"}
            <h1>{heading_atom}</h1>
            {bloqs:children}
        </div>
    {/section_bloq}

    {row_bloq}
        <div class="row"}
            {another_atom}
            {bloqs:children}
        </div>
    {/row_bloq}
{/bloqs_field}
```

### Bloq Variables

When a field is nestable, you have the option of prefixing your atom short names with `bloq_var_`. If an atom name has this prefix its value will be passed down to all child bloqs. The next bloq that contains the same atom at the same nesting depth, or higher, will have a new value, and thus pass it down to its children. In the example below the `{bloq_var_columns}` atom is defined on the Row bloq, which means its value is available in the Row bloq itself, and the Basic Content and CTA bloqs that are children of Row. Bloq variables only work with basic atom fieldtypes such as text, radio, or checkbox fields (or any fieldtype that outputs a simple string value). You can not use a Relationship field, or a 3rd party field such as Assets or Ansel as  bloq variable.

If you use a `bloq_var` in a conditional, you will need to quote it, e.g. `{if '{bloq_var_columns}' > 2}`

### Example Code

The following example code is used in [this video demo](https://vimeo.com/251749666).

```
{bloqs_field}
    {section}
        <div class="container">
            <h1>{heading}</h1>
            <p>{summary}</p>

            {bloqs:children}
        </div>
    {/section}
    {row}
        <h2>{heading}</h2>
        <div class="row">

            <div class="col row-footer"><small>Some footer content at the end of each row</small></div>
            {bloqs:children}
        </div>
    {/row}
    {basic_content}
        <div class="col col-{bloq_var_columns}">
            <div class="basic-content">
                <h3>{heading}</h3>
                <p>{body}</p>

                {bloqs:children}
            </div>
        </div>
    {/basic_content}
    {cta}
        <div class="col col-{bloq_var_columns}">
            <div class="cta">
                <h4>{heading}</h4>

                {bloqs:children}
            </div>
        </div>
    {/cta}
{/bloqs_field}
```

Another option to display child bloqs that was introduced in verison 5.0, is to use the `{bloqs:child:[my_bloq_name]}` tags instead of `{bloqs:children}`, which automatically iterates all the child bloqs and renders them. This is useful for when you need the output of children bloqs to display in a certain way, regardless of the order in which the children appear when editing the entry in the publish page. For example, the above code could also be the following:

```
{bloqs_field}
    {section}
        <div class="container">
            <div class="ctas">
                {bloqs:child:cta}
            </div>
            
            <h1>{heading}</h1>
            <p>{summary}</p>

            {bloqs:child:basic_content}
        </div>
    {/section}
    {row}
        <h2>{heading}</h2>
        <div class="row">

            <div class="col row-footer"><small>Some footer content at the end of each row</small></div>
            {bloqs:children}
        </div>
    {/row}
    {basic_content}
        <div class="col col-{bloq_var_columns}">
            <div class="basic-content">
                <h3>{heading}</h3>
                <p>{body}</p>

                {bloqs:children}
            </div>
        </div>
    {/basic_content}
    {cta}
        <div class="col col-{bloq_var_columns}">
            <div class="cta">
                <h4>{heading}</h4>

                {bloqs:children}
            </div>
        </div>
    {/cta}
{/bloqs_field}
```


# Cloning

Bloqs can be cloned when in nested and non-nested fields. If you're cloning a bloq in a nested field with child bloqs, all the child bloqs will also be cloned.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLsNYRMaXKEXhk7sTrr%2F-MLsNm52KefMj6_6kLUM%2Fbloqs-cloning-ani.gif?alt=media\&token=3ed8b93a-3399-428f-a7f9-79eefd55f069)

###


# Drafts

Bloqs can be set to draft, which means it will not display on the front-end when rendered in your templates. If you're marking a bloq as draft that has children, then all the children of the draft bloq will also not display on the front-end. In the animation below, the Section bloq is set to draft, but the child Basic Content bloqs are not. In this scenario the only bloq that would display on the front-end is the Slab bloq.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MTBq9RXI2monSwGp2P1%2F-MTBqF_W_C6Si9z0ai3g%2Fbloqs-drafts-4.6.0.gif?alt=media\&token=1c9a6888-ee24-47e9-948a-3af22c17457c)


# Bloq Usage

Starting in Bloqs 4.4.0 you can get more insight into how your bloqs are being used. The bloq definition list now includes a new column that displays the number of entries the bloq is used in. The number displayed in this list is the total usages of the bloq across all Bloqs fields it is assigned to. Clicking the link will display a list of entries and links to edit each entry.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLo3HY67G_y7x2FHg6G%2F-MLo4tDH6R2UAz2o_NH-%2Fbloqs-usage-report.png?alt=media\&token=b114c5d2-f557-437a-98cb-7eff8ec99860)

Also starting in version 4.4.0 is that once a bloq exists in any entry, the checkbox on the Bloqs edit field where you assign which bloqs are available to the field, are disabled. Prior to 4.4.0, if you uncheck a bloq on this page and save the custom field, any entries that contained that bloq had it's bloqs removed, thus causing a potentially unintended data loss. If you need to remove a bloq assignment from a field, it is best to use the usage links on the right to find the entries the bloq is used in, edit the entries, then remove the bloq assignment. The usage links on this page show the total entries the bloqs are used in this field only, not a cumulative count like on the bloq definition list above.&#x20;

If you want to disable this feature and return to pre-4.4.0 behavior, you can use the `bloqs_disable_strict_assignment` [hidden config variable](/bloqs/docs/configuration/hidden-config-options).

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLnyn_HynTp9FZ7XgFK%2F-MLnzVcYA4fDIXRLIol8%2Fbloqs-disabled-assignments.png?alt=media\&token=6fb1c8a4-adce-42b7-a3c3-0a97e0917c63)


# Deprecated Bloqs

Starting in Bloqs 4.4.0 you can assign the "Deprecated" status to a bloq. As your site evolves you may need to update certain bloqs, and in some cases retire them all together. Simply deleting the bloq may cause data loss and existing pages to break. To avoid that you can gracefully deprecate your bloqs. When you assign a deprecated status the bloq all existing instances of the bloq will continue to function as expected on the front-end, and in the control panel, but content editors will no longer see the bloq in the Add Bloq menu, thus they can't create new instances of a deprecated bloq. To make a bloq deprecated, just toggle the *Deprecated?* setting on the bloq. You can also add a short note that will appear in all existing instances of the bloq to inform content editors of its new status. If you do not add a deprecated note, Bloqs will simply display "This block is deprecated".

{% hint style="info" %}
Setting a bloq to deprecated **does not prevent saving an entry** that contains deprecated bloqs. Deprecation is just a visual warning meant to nudge content editors to update or remove the bloqs with whatever direction you provide them, and prevent them from adding additional deprecated bloqs to an entry.
{% endhint %}

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLquwr6N-1QsNgcRAkh%2F-MLqx4dSZ9bDW3bFHel5%2Fbloqs-deprecated-setting.png?alt=media\&token=a165aeeb-9c36-4403-9535-10c455780bc5)

When a bloq is deprecated, it will have a warning icon next to its name in the bloq definition list. This way you can easily tell which bloqs are deprecated.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLquwr6N-1QsNgcRAkh%2F-MLqx8VQaKApKlWYsJ7e%2Fbloqs-deprecated-listing.png?alt=media\&token=2064c40f-dc69-4010-af35-c08cf4b3b7a3)

When a Bloqs field has 1 or more deprecated bloqs, a warning appears above the field informing the content editor of the issue. Deprecated bloqs are again noted with a warning icon and a note.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLquwr6N-1QsNgcRAkh%2F-MLqwirwUsGYSvNEdnSa%2Fbloqs-deprecated-field.png?alt=media\&token=ce70fcd8-7f7b-43e8-ab28-50351f50744f)


# Caching

### Field Caching

Bloqs offers field caching, which will cache the final output of a single Bloqs field in a template. Enabling caching will decrease page load time by rendering cached content, thus avoiding the additional parsing necessary to render all the custom field data from the atoms inside each bloq.

Bloqs is a complicated add-on. If you have an entry with a Bloqs field that has 10 bloqs, and each bloq has 5 atoms (custom fields), you're effectively rendering 50 fields to in a template. If each of those 10 bloqs has a Relationship field, then you're rendering 10 Relationship fields on the front-end of your site. These complicated field types (any fieldtype that is not a basic text, textarea, wysiwyg, radio, checkbox etc) can add considerable overhead. Below is an example of a profiling output from a cached and uncached Bloqs field render:

```
# Uncached rendering time in seconds
Bloqs Replace Tag: 3.1513

# Cached rendering time in seconds
Bloqs Replace Tag: 0.0086
```

&#x20;When an entry is saved, the  corresponding cache item will be deleted. The next time the field is rendered on the front-end, a fresh cache item will be created.

```
$config['bloqs_cache_enabled'] = true;
```

When displaying a cached Bloqs field on the front-end, it will be wrapped in the following html comment tags:

```
<!-- Start cached Bloqs output -->... Your bloqs content here ...<!-- End cached Bloqs output -->
```

{% hint style="info" %}
The only way to clear a cached item is to re-save an entry, or clear the cache manually. If a site is actively being developed, it is probably best to not enable caching because making changes to template files will not clear the cache, thus you may not see the most up-to-date content. Enable cache in production sites only.
{% endhint %}


# Profiling

If ExpressionEngine's profiling is enabled, you will see additional entries in the Performance tab in the output profiler at the bottom of the page. Example output:

```
Bloqs Render Bloq #529874/Theme: 0.0002
Bloqs Render Bloq #529993/Locale: 0.0002
Bloqs Render Bloq #529993/Theme: 0.0002
Bloqs Render Bloq #529996/Locale: 0.0002
Bloqs Render Bloq #529996/Theme: 0.0002
Bloqs Render Bloq #530006/Locale: 0.0002
Bloqs Render Bloq #530009/Links: 0.0308
Bloqs Render Bloq #530010/Content: 0.2050
Bloqs Render Field #631: 2.6799
```

In this example, the pattern `#[bloq id]/[atom name]` is followed by the total time it took the server to parse the full value of the field to a string, which is inserted into the template. With this information you can identify potential bottlenecks in your Bloqs fields. The very last line `Bloqs Render Field #631` is an example of the total time it took Bloqs to render the output for the entire custom field.&#x20;


# Bloq Components

*Added in version 4.6.0 as Bloq Templates*\
*Renamed in version 4.8.0 to Bloq Components*

{% embed url="<https://youtu.be/MoFD9wZlqkI>" %}

![A non-editable component is distinguished by the dashed border around the bloqs.](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MS4PFrXj0gnpjfvNs2u%2F-MS4PXjuxQ3bVQhNOue8%2Fbloqs-templates.png?alt=media\&token=70144798-0cd4-49e3-83ff-eb6c474e7c87)

Bloq components are designed to do two things:

1. Speed up content management by building pages quicker.
2. Give you, the developer or designer, more control over how content is created.

As a developer, you can create a component that is collection of bloqs that can be added to an entry with a single click. Most sites have re-usable content patterns. Simple patterns can be built using a single bloq, but bigger, or more complex patterns, may require the uses of several bloqs. Some patterns may share bloqs as well. By using a bloq component, you can pre-build these re-usable patterns. Components do not allow pre-populating of content, just the definition of the component itself. If you're familiar with atomic design, each bloq is the equivalent of a [molecule](https://bradfrost.com/blog/post/atomic-web-design/#molecules). A single bloq consists of multiple atoms, or fields. Bloq components, on the other hand, let you define a [template](https://bradfrost.com/blog/post/atomic-web-design/#templates), or an [organism](https://bradfrost.com/blog/post/atomic-web-design/#organisms).&#x20;

{% hint style="info" %}
Bloq components do not require you to change your front-end template file code. You can start using bloq components on existing sites using bloqs you already have defined. Just create a new bloq and set it to a component, and assign bloqs that you want to use within the component. Those same bloqs can still be added separately to a field as long as they're selected in the field's settings.
{% endhint %}

### Editable Components

When an editable component is inserted into an entry, it inserts multiple bloqs just like if you had inserted each bloq individually, except it inserts them all at once. The content editor can than remove, clone, or re-order those bloqs just as they could any other bloq. This also means they can choose to not use the design pattern that you have put in place. In some cases this may be desired. An editable component is useful for common, but flexible, content patterns. Since they can be modified they do not have a unique visual appearance like a non-editable component.

### Non-Editable Components

A non-editable, which is the default state, component means that when the component is inserted into an entry, none of the bloqs within the component can be removed, cloned, or re-ordered. The component is fixed, and will always present itself that way. This is the strictest use of a bloq component. After all you created the component for a reason. It has a pre-defined and expected structure, and you always want it to appear this way without any modifications.

### Bloq Components and non-nested fields

If you create a bloq component and you assign it to a Bloqs channel field that is not set to nestable, the field will automatically be set to nestable. If you disable the nestable setting on the Bloqs channel field and save an entry containing bloq components, then the nesting structure will be lost and your bloqs will flatten out, thus affecting the presentation on the front-end and within the control panel.


# Bloq Groups

Bloq groups are a way to organize your bloqs into logical groups. If you have several bloqs that create social media content, *Social Media* may make a good group name. Groups are only used in the add bloq menu inside the control panel when editing an entry. If no groups are created, the bloqs will be displayed in the order you define on the channel field edit page where you assign bloqs to a field. If a bloq is added to a group, then the display order in the add bloq menu will be based on the *Order* field of the group, then the order field of the bloqs within that group. If no groups are defined, then no groups will be displayed in the add bloq menu. If groups are defined, but some bloqs are not assigned to a specific group, then they will be added to an *Ungrouped* group at the bottom of the menu.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MS4PaN9lavUfsTtRBOk%2F-MS4RkQfzYZ3dCC8scme%2Fbloqs-group-menu.png?alt=media\&token=4165f9ec-35d0-4ec9-bbc0-e636e5904a98)


# Import & Export

Bloqs 5.5 lets you export selected bloq definitions to a JSON file and re-import them into another ExpressionEngine site. Use it to move bloqs from local to production, between staging sites, or to share a set of bloqs with another project. Only the definitions are exported (the bloq types, their fields, settings, nesting rules, and component structure). Entry content is never exported; it stays in the entries on each site.

{% hint style="warning" %}
BoldMinded is not responsible for data loss when using this feature.&#x20;

**Backup your database before running imports.**
{% endhint %}

### Exporting

On the **Bloqs** listing screen, each row has a checkbox. Select one or more definitions, choose **Export** from the bulk-action bar that appears, and submit. Your browser downloads a JSON file named `bloqs-definitions-N.json`.

The export is self-contained and human-readable. Because the receiving site won't share the same database IDs, every cross-reference is rewritten into a portable form:

* **Bloq groups** are exported by name.
* **Nesting rules** ("can be a child of...") are exported as shortnames.
* **Component child bloqs** are referenced by shortname.

#### Example export

```json
{
  "bloqs_export_version": 1,
  "source": {
    "bloqs_version": "5.5.0"
  },
  "block_definitions": [
    {
      "shortname": "hero",
      "name": "Hero",
      "instructions": "A full-width hero banner.",
      "group": "Layout",
      "deprecated": 0,
      "deprecated_note": "",
      "preview_image": "",
      "preview_icon": "image",
      "is_component": 0,
      "is_editable": 0,
      "settings": {
        "nesting": {
          "root": "any",
          "child_of": ["section"],
          "no_children": "y"
        }
      },
      "atom_definitions": [
        {
          "shortname": "headline",
          "name": "Headline",
          "instructions": "",
          "order": 0,
          "type": "text",
          "settings": { "field_fmt": "none", "col_required": "y" }
        },
        {
          "shortname": "related",
          "name": "Related Entry",
          "instructions": "",
          "order": 1,
          "type": "relationship",
          "settings": { "channels": ["3"], "authors": [], "limit": "1" }
        }
      ]
    }
  ]
}
```

A **component** bloq also carries a `component_children` array listing the child bloqs by shortname along with their tree position (`order`, `depth`, `parent_id`, `lft`, `rgt`, `cloneable`).

The `bloqs_export_version` field is the format version. A site running an older version of Bloqs refuses a file whose version is newer than it understands.

### Importing

Use the **Import** screen to upload an exported file. You upload the file and choose options, then see a preview of exactly what will happen. Nothing is written until you confirm.

#### The import options

**Definition File** (required): the JSON file you exported.

**Preserve Site-Specific Settings** (off by default): when off, field settings that reference site-specific IDs are reset to safe defaults. Turn it on only when the target site shares the same channel, upload-directory, and field IDs as the source, such as a local-to-production sync of the same site. See "The site-specific settings rule" below.

**Update Existing Definitions** (off by default): when off, a matching shortname is imported as a renamed copy. When on, matching bloqs are updated in place. See "Adding vs. updating" below.

**Remove Atoms Not In File** (off by default, only applies when "Update Existing" is on): deletes fields on the target that aren't in the file. This is destructive, since deleting a field also deletes its content in every entry that uses it. The preview shows which fields would be deleted and how many entries would lose content.

#### The preview

Before committing, you get a dry-run summary:

* **Will be created**: definitions that don't exist on the target.
* **Will be updated**: existing definitions updated in place (only when "Update Existing" is on).
* **Renamed to avoid collisions**: definitions whose shortname already exists and will get a numeric suffix.
* **Fields needing reconfiguration**: fields whose site-specific settings will be reset.
* **Atoms to be deleted**: fields that will be removed, each with the count of affected entries.

### The rules: adding, updating, deleting

#### Adding (the default)

With "Update Existing" off, every definition is created new. If a shortname already exists on the target (whether from another bloq or a native channel field), the imported copy gets a unique suffix: `hero` becomes `hero_2`, then `hero_3`. An import can never overwrite existing data this way; the worst case is a renamed duplicate. References within the import (nesting, component children) are rewired to point at the renamed copies.

#### Updating (opt-in)

With "Update Existing" on, a definition whose shortname matches one on the target is updated in place. This updates the bloq and its fields:

* The bloq's own properties (name, instructions, group, deprecation, preview image/icon, component flags, nesting settings) are overwritten from the file.
* Each field is matched by shortname within that bloq:
  * A field on both is updated in place, keeping its existing database ID. Since entry content is tied to that ID, no content is lost.
  * A field in the file but not on the target is created.
  * A field on the target but not in the file is left untouched by default.

#### Deleting (explicit, destructive)

An import never deletes anything unless you ask. The only deletion path is **"Remove Atoms Not In File"**, available only in update mode. When enabled, fields on the target but absent from the file are deleted, and deleting a field cascades to its content in every entry. The preview lists each field with its entry-usage count. Whole definitions are never deleted by an import; to remove a bloq, use the delete action on the Bloqs listing (which is blocked while the bloq is in use).

### The site-specific settings rule

Some field types store references identified by site-local IDs: relationship fields point at channels, categories, statuses, and authors; file and Assets fields point at upload directories. These IDs are meaningless on a different site.

Unless "Preserve Site-Specific Settings" is on, the importer resets these references to safe empty defaults for the affected types (relationship, file, file\_grid, assets). The field is imported intact and usable, just unconfigured for those references, which you reconnect on the target. The preview lists every affected field. The rest of a field's settings (label, limits, required, formatting) are always preserved.

Turn this option on only when the target's IDs match the source's, typically a local-to-production sync of the same site.

### The `renamed_from` nuance

The importer matches definitions and fields by shortname. If you rename `headline` to `title` in the source and import with "Update Existing" on, the importer can't tell `title` is the old `headline`. It sees `headline` as gone and `title` as new, so it creates a new empty `title` and leaves the old field alone.

To handle a rename, add a `renamed_from` property by hand to that entry in the JSON before importing. It tells the importer: if you can't find my current shortname, look for this old one and update that record in place.

```json
{
  "shortname": "title",
  "name": "Title",
  "type": "text",
  "renamed_from": "headline",
  "settings": { }
}
```

With this and "Update Existing" on, the importer finds the existing `headline` field, updates it in place (keeping its ID and all entry content), and changes its shortname to `title`. The same works at the bloq level: `"renamed_from": "hero"` with `"shortname": "hero_banner"` updates the existing `hero` bloq and renames it.

Notes on `renamed_from`:

* It's matched only as a fallback. The current shortname is tried first, so it's harmless to leave in a file you re-import.
* It's supplied by hand. Bloqs doesn't auto-detect renames or write `renamed_from` into exports, because the database keeps no rename history.
* It works safely with "Remove Atoms Not In File": a field matched via `renamed_from` has its old shortname treated as accounted for, so the remove step won't delete it.

Within a single site you never need `renamed_from`. Just rename the field in the control panel; content is tied to the field's ID, so nothing is lost. `renamed_from` exists only for cross-site imports, where the two sites share no ID and the shortname is the only stable key.


# Installation

To install Bloqs move the `/addons/bloqs/` directory to your `/system/user/addons/` folder. Also copy the `/themes/user/bloqs/` folder to the `/themes/user/` directory in your ExpressionEngine install. Login to your control panel and visit the ***Developer > Add-ons*** page, and click the Install link for Bloqs.

### Updating Blocks (for EE 2) to Bloqs (for EE 3+)

As a result of the Add-on name change, updating from Blocks (EE2) to Bloqs (EE53) is a tad bit misleading during one part of the process, but rest assured all will work out and you will be up and running in no time. &#x20;

**Here’s what you’ll need to do:**

1. Upgrade your ExpressionEngine 2 installation to the latest version of ExpressionEngine 3. ([upgrading from EE2 to EE3](https://docs.expressionengine.com/v3/installation/upgrade_from_2.x.html))
2. [Download](https://devot-ee.com/add-ons/bloqs-blocks-for-ee3) the latest version of Bloqs
3. In the `/system/user/addons` directory of your (recently upgaded) ExpressionEngine installation, locate and delete the “blocks” folder. This is the old version of Blocks which is only compatible with EE2.
4. In the `/themes/user` directory of your ExpressionEngine installations, locate and delete the `blocks` folder.
5. Open up the Bloqs package you downloaded in Step 2, and move the `/system/user/addons/bloqs` folder into the `/system/user/addons` directory of your ExpressionEngine installation.
6. Move the `/themes/user/bloqs` folder into the `/themes/user` directory of your ExpressionEngine installation.
7. Now, login to your ExpressionEngine Control Panel and open up the Add-on Manager view.
8. (Here’s where things are a bit misleading).  Locate ‘Bloqs’ in the Third Party Add-Ons list. You’ll notice that it’s grayed out with a button that says “Install” next to it. Don’t panic! If Bloqs was already installed on your EE2 site, it’s still there - you haven’t lost anything. Just click the button that says ‘Install.’
9. The Bloqs update routine will run and handle the Blocks to Bloqs conversion process.
10. That’s it, you’re all set to go!


# Requirements & Compatibility

Requirements

* ExpressionEngine 6.4 or greater
* PHP 8.2 or greater
* PHP `max_input_vars` set to a minimum of `2000`.

{% hint style="info" %}
Sites using a lot of bloqs and custom fields within an entry may want to consider increasing PHP's `max_input_vars` setting. Usually the default is 1000. Increasing it to 2000 or more may be required.
{% endhint %}

### Compatibility

**Native ExpressionEngine Fieldtypes**

* Checkboxes
* Date
* Duration
* Email Address
* File
* Multi Select
* Number Input
* Notes
* Radio Buttons
* Relationships
* Rich Text Editor
* Select Dropdown
* Selectable Buttons
* Text Input
* Textarea
* Toggle
* Value Slider
* URL

**Native ExpressionEngine Features**

* Entry Manager custom column (Bloqs must be set to searchable)
* Search

{% hint style="info" %}
Bloqs is built using the same concepts as the Grid field, and because Grids within Grids are not allowed, a Grid (and similarly File Grid) may not be placed within a blo&#x71;*,* however, you can use our [Simple Grids & Tables](https://boldminded.com/add-ons/simple-grids-tables) add-on for basic Grid functionality within Bloqs.

The native Fluid field type is also not supported by Bloqs.
{% endhint %}

#### Third-party fieldtypes & add-ons opted in to Bloqs

* [Ansel](https://boldminded.com/add-ons/ansel)
* [Assets](https://eeharbor.com/assets) (by [EEHarbor](https://eeharbor.com))
* [Expresso](https://www.putyourlightson.net/expresso) (by [Put Your Lights On](https://www.putyourlightson.net/))
* [Publisher](https://boldminded.com/add-ons/publisher-2) (by [BoldMinded](https://boldminded.com/))
* [Wygwam](https://eeharbor.com/wygwam) (by [EEHarbor](https://eeharbor.com))
* [Wyvern](https://addons.reinos.nl/wyvern) (by [Reinos](https://addons.reinos.nl/))

If you are unsure if an add-on works in Bloqs, but it works in Grid, then chances are it already works in Bloqs. There are a vast number of 3rd party fieldtypes not listed above that already work in Bloqs.

If your add-on/fieldtype is compatible with Bloqs and you don't see it in the list above, [send us a message](https://eebloqs.com/contact) and let us know!

### Performance

Bloqs is a robust and powerful add-on that adds a lot of flexibilty when building out your site content. Since you can add any number of bloqs to a single entry, keep in mind that the more bloqs you add, the more template parsing is required, and the longer it may take for a control panel page to load when editing an entry. For example, if you add 10 bloqs to an entry, and if each bloq has 5 atoms in it it, you're effectivley adding 50 custom fields to the entry. To mitigate this on the front-end you can use Bloqs' [built in caching option](/bloqs/docs/configuration/hidden-config-options#field-caching), or add full featured caching with [Speedy](/speedy/docs). In most cases caching won't be required, just keep performance in mind when planning and building your site with Bloqs.


# Setup & Configuration

After installing Bloqs the first thing you will want to do is create some bloqs! Below is a screenshot of the main Bloqs settings page. As you can see we have 7 bloqs already created, but after a new installation of Bloqs, this list will be empty.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLt1LVGFcaRFCOYZfxy%2F-MLt1yug62VNAo4wdY6s%2Fbloqs-list.png?alt=media\&token=958f264f-c3a1-4416-8553-b24c75f12bb7)

Creating a bloq is easy. At the top of the page you give it a name and short name, as well as assign it to a group, preview icon or image, and set any nesting rules. The bottom of the bloq configuration page is where you create your atoms for the bloq. Creating atoms is exactly like creating columns in a Grid field.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLt1LVGFcaRFCOYZfxy%2F-MLt2urVFI8X4lTPZjLY%2Fbloqs-edit-atoms.png?alt=media\&token=30f1462a-21cf-4cec-a364-9aed1ad2f425)

Once you have created a few bloqs, it is time to create your Bloqs custom field and assign bloqs to it. Suppose we have a very simple "Pages" Channel in ExpressionEngine which contains 1 Bloqs field:

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLt0H9fG17_1Dnq1hjw%2F-MLt0_AEmQNHbg4t7WcE%2Fbloqs-ee-field.png?alt=media\&token=9454053f-43b5-4cff-ae6e-16e83788ce85)

At the bottom of this Edit Field form is a list of all the bloqs you have defined. From here you select which bloqs will be made available to content editors when editing an entry in a channel that the Bloqs field is assigned to. Since bloqs can belong to one or more custom fields, you need to select which bloqs are available on a per-field basis.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLnyn_HynTp9FZ7XgFK%2F-MLnzVcYA4fDIXRLIol8%2Fbloqs-disabled-assignments.png?alt=media\&token=6fb1c8a4-adce-42b7-a3c3-0a97e0917c63)

Once you have created your bloqs, assigned them to a custom field, and assigned that field to a channel you can edit an entry and start building robust and dynamic pages in ExpressionEngine.&#x20;

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MKFLy5oerhy1rSM-po3%2F-MKFO7IpF736qBbDR5xv%2Fbloqs-add-grid.png?alt=media\&token=d3f724e4-edde-4b5a-a7cb-1558030349ba)

Once you have your field, bloqs, and atoms defined, you will want to learn how to render them to the page. Learn about [Template Tags](/bloqs/docs/template-tags) next.

## Entry Manager Support

Bloqs includes basic support for the new Entry Manager in ExpressionEngine 6, but the Bloqs custom field, and atoms inside a bloq must be set to searchable. All HTML tags will be removed from the content for display in the Entry Manager column.


# Hidden Config Options

### Collapse bloqs when dragging

By default, when you are dragging and reordering bloqs on an entry page, if the bloq is expanded, then it will remain expanded when dragging it. To collapse all bloqs when dragging, add the following value to your config.php file. Note that this can cause a slight jump in the page scrolling, which is why this is disabled by default.

```
$config['bloqs_collapse_on_drag'] = true;
```

### Collapse Children

When enabled child blocks are hidden and only revealed when the parent is expanded. Disabled by default.

```
$config['bloqs_collapse_children'] = true;
```

### Disable Logging

If you want to reduce the noise in your log you can turn it off. This is not recommended by default or while your site is in development because it can reveal some important information.

```
$config['bloqs_disable_logging'] = true;
```

### Field Caching

You can optionally cache the output of a Bloqs field. [Read more about it](/bloqs/docs/features/caching#field-caching).

```
$config['bloqs_cache_enabled'] = true;
```

### Confirm removal

By default Bloqs will immediately remove a bloq from an entry when the trash/delete icon is clicked. Enabling this option will present a confirmation modal to confirm the deletion.

```
$config['bloqs_confirm_bloq_removal'] = true;
```

### Strict tree structure

By default Bloqs will only warn you when the nesting structure of a bloqs field is incorrect and will still let you save the entry. When this is enabled the warning turns into an issue with more visual indicators that the user needs to correct something, and also will not let you save the entry the entry until the nesting structure is correct. This option is really only necessary if you have nesting enabled on a field, and you have nesting rules defined for your bloqs.

```
$config['bloqs_enable_strict_tree_structure'] = true;
```

### Usage Reports

In the control panel where you manage the bloq definitions there is a Usage column, which indicates in which entries the bloq is used in. On a large site this may add a little bit of load time to this page due to the query that is executed to find the usages across all fields and entries. Displaying the Usage column is default behavior. You can disable this column and the queries by using the config option.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FgDyOyMX4WXmtHMA8NtQE%2Fblocks-usage-report.png?alt=media\&token=754f3970-2cbc-41b3-8ab8-244e4f83bbcb)

```
$config['bloqs_disable_usage_reports'] = true;
```

### Bloq Field Assignments

Beginning in Bloqs 4.4.0, when a bloq is assigned to a field, and is used in a saved entry, you can not remove the bloq's assignment from that field. Removing the assignment will cause the bloq to disappear from existing entries (this has always been the behavior). **To prevent data loss**, and to make it more clear how bloqs are being used, the checkboxes are now disabled by default if the bloq is used in an existing entry. Below is a screenshot of what it may look like. The "*Used in X entries*" informs you how many entries the bloq is being used in for the current field. You can click the link to view the entries the bloq is assigned to. If the bloq is assigned to multiple fields, then the usage report when you click the link may report a different value than what you see on this screen. That is because the usage report is cumulative across all fields.

{% hint style="warning" %}
Note that this config option is not officially supported because it alters the intended default behavior of Bloqs and can cause data loss. If you experience visual or functional issues with this option disabled you can submit a support ticket, but there are no guarantees it will be addressed.
{% endhint %}

![The default behavior as of 4.4.0](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MLnyn_HynTp9FZ7XgFK%2F-MLnzVcYA4fDIXRLIol8%2Fbloqs-disabled-assignments.png?alt=media\&token=6fb1c8a4-adce-42b7-a3c3-0a97e0917c63)

To prevent disabling of these checkboxes and return to the pre-4.4.0 behavior, add the following value to your config.php file.

```
$config['bloqs_disable_strict_assignment'] = true;
```


# Creating Bloqs

When creating a bloq you are required to add at least 1 atom (field) to the bloq. Atom names and short names can be reused. For example every bloq you create could have an atom with the Name of "Heading", and Short Name of "heading". However, you can't repeat short names in the same bloq. This behavior mimics that of a Grid field.

If you give an atom the short name of `__hidden` then it will not be displayed in the user interface when editing a bloq in an entry. An atom of this name has no other purpose than to fulfill the requirement that every bloq must contain at least 1 atom. Why would you need to do this you might ask? Glad you asked. This can be helpful if you need to create a bloq that has no configuration options or content values, but you might want to render something in a template when that bloq is added to an entry. This can be especially helpful if you're using the nested bloqs and want to give a group of bloqs a parent container to assist with layout rendering. For example:

```html
{bloqs_field}
    {section}
        <h1>{heading}</h1>
        <div>
            {bloqs:children}
        </div>
    {/section}
    
    {container}
        <div>
            {bloqs:children}
        </div>
    {/container}
    
    {content}
        <h2>{heading}<h2>
        {body}
    {/content}
{/bloqs_field}
```

In that scenario you might have 3 bloqs in your entry nested like this:

> section
>
> &#x20;   container
>
> &#x20;       content
>
> &#x20;       content
>
> &#x20;   container
>
> &#x20;       content


# Template Tags

{% hint style="info" %}
Bloqs supports the Template Generator feature of ExpressionEngine 7.5. It'll output all the necessary tags for your Bloqs field. You will of course need to add your html around the tags, but the generated template will give you a jump start.
{% endhint %}

At the most basic level, templating with Bloqs is nothing more than the standard EE tag pair/tag process. But tag pairs in Bloqs go a bit deeper than what you might normally expect to encounter in ExpressionEngine. Let's take a look at an example so you can see how the logic plays out.

Lets say we have a single bloq type defined with 2 atoms:

* **Bloq Name:** Content Bloq
* **Bloq Short Name:** `{content_bloq}`
* **Bloq Atom Definitions:**
  * **Atom Name:** Heading
  * **Atom Short Name:** `{heading}`
  * **Atom Name:** Body Content
  * **Atom Short Name:** `{body_content}`

And this bloq type is added to a Bloqs custom field assigned to the Pages channel.

* **Field Type:** Bloqs
* **Field Name:** My Bloqs Field
* **Field Short Name:** `{my_bloqs_field}`
  * **Selected Bloq Types:** Content Bloq

Given our base definitions, to output the content in a Pages Channel Entry, your template code would look similar to the following:

```html
{exp:channel:entries channel="page"}
    {my_bloqs_field}
        <!-- Note that any html here will not render. Only content within
        the tag pairs below will render. -->
        <div class="container">For example, this will not be output</div>
        
        {content_bloq}
            <div class="container">
                <h2>{heading}</h2>
                {body_content}
            
                <!-- If using nested bloqs, you'll want to add the {bloqs:children}
                tag within each bloq that might have one or more children -->
                <div class="children-container">
                    {bloqs:children}
                </div>
            </div>
        {/content_bloq}
        
        <!-- Working with file field examples -->
        {media_bloq}
            <div class="container">
                {media_bloq:file_field}
                    <img src="{url}" />
                {/media_bloq:file_field}
                
                <!-- Or -->
                
                <img src="{media_bloq:file_field:resize width='90' height='90'}" />
                
                <!-- Or -->
    
                {media_bloq:file_field:webp}
                     <img src="{url}" />
                {/media_bloq:file_field:webp}
                
                <!-- Or -->
                
                <img src="{file_field}" />
            </div>
        {/media_bloq}
        
    {/my_bloqs_field}
{/exp:channel:entries}
```

{% hint style="info" %}
Note the file fields in the above example are prefixed with the bloq name. If you have *any custom field that uses a tag pair inside of a bloq* (e.g. File, Relationship, Ansel, or Simple Grid & Tables), you will likely have to prefix the field name with the bloq name. This is simply an issue with the native EE template parser.

```html
{my_bloq}
    {my_bloq:my_ansel_field}
        <img src="{img:url}" />
    {/my_bloq:my_ansel_field}
{/my_bloq}
```

You may also want to avoid prefixing your bloqs with the same name as the bloqs field, and atom's with the same name as the parent blog's field name. Prefixing is handy in most cases, but the ExpressionEngine template parser can get confused. If you run into any template parsing issues, this should be the first thing you look into.

```html
{exp:channel:entries channel="page"}
    {bloqs_field}
        {bloqs_field_hero}
            {hero_heading}
            {hero_content}
        {/bloqs_field_hero}
    {/bloqs_field}
{/exp:channel:entries}
```

{% endhint %}

A slight more advanced example including a *Feature Component* bloq:

```html
{exp:channel:entries channel="page"}
    {my_bloqs_field}
        {content_bloq}  
            {heading}
            {body_content}
        {/content_bloq}
        
        {feature_component}
            {if bloqs:is:first_child}
                <div class="features">
            {/if}
            <div class="feature">
                <div class="feature-title {bloqs:switch='yellow|purple|green'}">
                    <img src="{feature_icon}">
                    {feature_name}
                </div>
                <a class="cover-link" href="{feature_link}"></a>

                {bloqs:children}

            </div>
            {if bloqs:is:last_child}
                </div>
            {/if}
        {/feature_component}
    {/my_bloqs_field}
{/exp:channel:entries}
```

At the bottom of a Bloqs custom field settings page will be a live basic example template based on the bloqs you've created for your project you can use to get started. Once you've created your basic field tag pairs, you may need to use Context Variables to assist in the rendering of the output. Read more about [Context Variables](/bloqs/docs/context-variables) next.

If you are using the bloq nesting feature, then [more template examples are available](/bloqs/docs/features/nesting#template-tags).


# Twig/Blade

The following is a basic example of how to render a Bloqs field with nested bloqs using Twig. If you're using Blade, the concepts are identical, just slightly different syntax for Blade.

Note the include statements are passing an array of possible template names to include in the brackets. If the first template file is not found, it will skip it and attempt to render the next one. For example, if you have a bloq named "summary" and you want that bloq to render entirely different HTML, just create a template named `_bloq_summary.html` All other bloqs will render in the `_bloq.html` template. These template names are completely arbitrary and you can choose any names you want. In this example I have a `coilpack.group` folder with the `_bloq.html.twig` and `_atom.html.twig` templates inside.

```twig
{# entry.html.twig #}

{% for entry in exp.channel.entries.parameters(
    {'channel': 'pages', 'entry_id': '4'}
) %}
    <h1>{{ entry.title }}</h1>

    {% for bloq in entry.bloqs %}
        {% include [
            ('ee::coilpack._bloq_' ~ bloq.getDefinition().getShortName()),
            'ee::coilpack._bloq'
        ] with {
            'bloq': bloq
        } %}
    {% endfor %}

{% else %}
    <span>No blog entries</span>
{% endfor %}
```

```twig
{# _bloq.html.twig #}

{% set bloqName = bloq.getDefinition().getShortName() %}
{% set isComponent = bloq.getDefinition().isComponent() %}

<div class="{% if isComponent %}component{% else %}basic{% endif %}">
    {% for atom in bloq.getAtoms() %}
        {% include [
            ('ee::coilpack._atom_' ~ atom.getDefinition().getShortName()),
            'ee::coilpack._atom'
        ] with {
            'atom': atom
        } %}
    {% endfor %}

    {% for child in bloq.getChildren() %}
        {% include [
            ('ee::coilpack._bloq_' ~ child.getDefinition().getShortName()),
            'ee::coilpack._bloq'
        ] with {
            'bloq': child
        } %}
    {% endfor %}
</div>
```

```twig
{# _atom.html.twig #}

{# If your atom contains HTML, use the raw modifier, e.g. {{ atom | raw }} #}
{{ atom }}
```

If in this scenario there was an atom with the short name of `content_with_image`, and a template file named `_atom_content_with_image.html.twig` existed, it would be loaded instead of the `_atom.html.twig` template.

Twig is extremly flexible and there are probably a number of ways to approach building out a Bloqs field. This is a basic example of how to use an include to render a deeply nested Bloqs field.

Once you have a bloq variable you can access it's atoms and atom values directly via the dot notation:

```twig
{% for entry in exp.channel.entries.parameters(
    {'channel': 'pages', 'entry_id': '4'}
) %}
    <h1>{{ entry.title }}</h1>

    {% for bloq in entry.bloqs %}
        {{ bloq.heading }}
        {{ bloq.summary }}
    {% endfor %}
{% endfor %}
```

### Relationships

Accessing relationship field data in Twig or Blade is the same as accessing any other array.

```twig
{% for entry in exp.channel.entries.parameters(
    {'channel': 'pages', 'entry_id': '4'}
) %}
    <h1>{{ entry.title }}</h1>

    {% for bloq in entry.bloqs %}
        {% for relationship in bloq.relationship_field %}
            {{ relationship.title %}
        {% endfor %}
    {% endfor %}
{% endfor %}
```


# Examples

### Creating a tabbed content group

Creating a tabbed content group in Bloqs is pretty easy when using nested bloqs. You can put as many bloqs or components inside of a tab. In this example there is a single component in each tab.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FWJKpbCEFXeXpUxYzlduI%2Fbloqs-tab-group-example.png?alt=media\&token=0629940f-cfc4-41da-bc2c-e2b384bbe5cd)

The template markup is where the magic happens. Since Bloqs lets you repeat the same tag pair more than once in the page, you'll create two different loops. The first loop is to render all of your tabs, either in a list, or in this example a series of buttons. And the second loop will render the actual tab panel itself, which will render the contents, or children of each tab, through the `{bloqs:children}` variable.

```
{block_tab}
    {if bloqs:is:first_child}
        <div role="tablist">
    {/if}

        <button
            role="tab"
            aria-controls="{tab_id}"
            tabindex="0"
            aria-selected="{if bloqs:is:first_child}true{if:else}false{/if}"
            data-text="{tab_label:attr_safe}"
        >
            {tab_label}
        </button>

    {if bloqs:is:last_child}
        </div>
    {/if}
{/block_tab}

{block_tab}
    {if bloqs:is:first_child}
        <div class="tab-group__panel-container">
    {/if}

        <div
            id="{tab_id}"
            role="tabpanel"
            aria-expanded="{if bloqs:is:first_child}true{if:else}false{/if}"
        >
            {bloqs:children}
        </div>

    {if bloqs:is:last_child}
        </div>
    {/if}
{/block_tab}
```

### Displaying Assets fields in Bloqs

```
{bloqs_field}
    {asset_field}
        {!-- Looping over multiple files in an Assets field --}
        {asset_field:file}
            <img src="{url}" />
        {/asset_field:file}
        
        {!-- Displaying a single file in an Assets field --}
        {asset_field}
            <img src="{file:url}" />
        {/asset_field}
    {/asset_field}
{/bloqs_field}
```


# GraphQL

Below is an example of querying Bloqs field data using GraphQL.&#x20;

```graphql
query GetBloqsEntry {
  exp_channel_entries(entry_id: "123", limit: 1) {
    data {
      entry_id
      title
      url_title

      bloqs {
        ... on Bloqs__bloqs__basic_content {
          _id
          _block_name
          _depth
          _order
          _parent_id
          heading
          content
          note
        }
        
        ... on Bloqs__bloqs__ansel_test {
          _id
          _block_name
          _depth
          ansel {
            id
            url
            thumbnail_url
            title
            width
            height
          }
        }

        ... on Bloqs__bloqs__media {
          _id
          _block_name
          _depth
        }

        ... on Bloqs__bloqs__related_content {
          _id
          _block_name
          related_entries {
            entry_id
            title
            url_title
          }
        }

        ... on Bloqs__bloqs__section {
          _id
          _block_name
          _depth
        }

        ... on Bloqs__bloqs__video {
          _id
          _block_name
          reel
        }

        ... on Bloqs__bloqs__single_relationship {
          _id
          _block_name
          related {
            entry_id
            title
          }
        }
      }
    }
  }
}
```


# Context Variables

All Bloqs fields have context variables available. Regardless of what your field or atom names are, the following special variables can be used in your Bloqs field tag pairs. These variables are **only available inside of a bloq tag pair** (the same level as an atom). For example:

```
{exp:channel:entries channel="page"}
    {my_bloq_field} <!-- This is the Bloqs custom field --> 
        
        {content_bloq} <!-- This is a bloq tag pair --> 
            {heading} <!-- This is an atom inside of a bloq -->
            {bloqs:count} <!-- This is one of the context variables below -->
        {/content_bloq}
        
        {featured_article_bloq} <!-- Another bloq tag pair -->
            {bloqs:count} <!-- This will also work -->
        {/featured_article_bloq}
        
        {bloqs:count} <!-- This will not work here -->
        
    {/my_bloq_field}
{/exp:channel:entries}
```

{% hint style="info" %}
Please note that as of version 4.4.0, there was a universal change in the Bloqs UI, documenation, and variables of all references of "block" and "blocks" to "bloq" and "bloqs". If upgrading from a version prior to 4.4.0, your old`{blocks:total_blocks}` variables will still work. For every variable listed below, there is an `{blocks:*}` alias for it. If you are using a version prior to 4.4.0, then you must use the old variable prefixes, e.g. `{blocks:*}`
{% endhint %}

`{bloqs:all_vars}`\
List out all of the variables below, and their values in your template for debugging purposes.

`{bloqs:id}`\
The unique ID of the current bloq being displayed.

`{bloqs:shortname}`\
The short name of the current bloq being displayed.

`{bloqs:template:shortname}`\
If the current bloq is a template wrapper, this will display the template's short name from it's definition data (`{bloqs:shortname}` will always be `__template_wrapper` , this variable helps provide a unique name to reference)

`{bloqs:count}`\
The count of the current bloq being displayed. The first bloq will have a value of "1", the next "2", etc.

`{bloqs:index}`\
The zero-based index of the bloq being displayed. The first bloq will have a value of "0", the next "1", etc.

`{bloqs:total_bloqs}`\
The total number of bloqs to display.

`{bloqs:total_rows}`\
The same as `{blocks:total_bloqs}` but for people who feel more comfortable with Grid terminology

`{bloqs:count:of:type}`\
The count of the current bloq being displayed for this particular type. The first bloq of this type will have a value of "1", the next "2", etc.

`{bloqs:index:of:type}`\
The zero-based index of the bloq being displayed for this particular type. The first bloq of this type will have a value of "0", the next "1", etc.

`{bloqs:total_bloqs:of:type}`\
The total number of bloqs of the current type to display.

`{bloqs:total_rows:of:type}`\
The same as `{bloqs:total_bloqs:of:type}` but for people who feel more comfortable with Grid terminology

`{bloqs:previous:shortname}`\
The shortname of the bloq that precedes the current bloq, or "" if this is the first bloq.

`{bloqs:next:shortname}`\
The shortname of the bloq that follows the current bloq, or "" if this is the last bloq.

`{bloqs:switch="one|two|three"}`\
Works the same way as the {switch=""} variable in a channel:entries tag, but with bloqs. Note that this takes into account *all* bloqs of the same type within a specific field. If using nested bloqs or components the context/depth of a bloq will not reset the switch value.&#x20;

#### The following context variables are useful when using a nested bloqs field.

`{bloqs:is:first_child}`\
Boolean - If this bloq is the first child of a parent bloq.

`{bloqs:is:last_child}`\
Boolean - If this bloq is the last child of a parent bloq.

`{bloqs:root:id}`\
If a bloq is a child of another bloq, this will be the value of the nearest parent bloq at the root level.

`{bloqs:is:root}`\
Boolean - If this bloq is at the root level (it doesn't have any parents).

`{bloqs:parent:id}`\
The id of the parent bloq

`{bloqs:parent:ids}`\
Similar to `{bloqs:parent:id}`, but when the bloq is a child of another bloq, this will report all of it's parent bloq IDs up to the root level. For example: '123|124|125'

`{bloqs:parent:shortname}`\
The short\_name of the parent bloq

`{bloqs:children}`\
This will render all child bloqs of a current bloq. You do not need to do anything special when using this tag. Bloqs will automatically grab the content of other `{my_bloq}{/my_bloq}` tag pairs and  insert them into their proper location.

`{bloqs:child:my_bloq}`\
**New in Bloqs 5.0** - Instead of using the :children tag, you can selectively choose which children of a bloq get rendered, and where in the parent bloq tag pair they get rendered. Note, however, that if your template markup does not include a child tag of a particular bloq, but the bloq exists in the Entry data, it will not render on the front-end. These child tags and the `{bloqs:children}` tag are mutually exclusive.

`{bloqs:children:total_bloqs}`\
If a bloq has children, display how many children it has.

`{bloqs:children:total_rows}`\
The same as `{bloqs:children:total_bloqs}` but for people who feel more comfortable with Grid terminology

`{bloqs:siblings:total_bloqs}`\
How many siblings, at the same nesting level, the current bloq has.

`{bloqs:siblings:total_rows}`\
The same as `{bloqs:siblings:total_bloqs}` but for people who feel more comfortable with Grid terminology

`{bloqs:depth}`\
**New in Bloqs 5.0** -Use this variable to output an integer value of how deep the current bloq is inside of a nested bloq structure.

`{bloqs:count:at:depth}`\
**New in Bloqs 5.0** -Just like the {count} variable, this will output an integer value of the total bloqs at the same depth as the current bloq. If you have a single root level bloq with 3 children, the first child will have a {bloqs:count:at:depth} value of 1, and the last will have a value of 3, but the {bloqs:count} value of the 3rd child will be 4, since it is the 4th total bloq displayed.

#### Variable Modifiers

`:total_bloqs`\
&#x20;`{your_field_name:total_bloqs}` returns the number of total bloqs that the field contains. This may be useful for conditionally including the field. E.g., `{if "{your_field_name:total_bloqs}" > "0"}`.

`:total_bloqs` - with "type" parameter (optional)\
&#x20;`{your_field_name:total_bloqs type="shortname"}` will return the number of bloqs defined for a particular type, as specified by the bloq type shortname.

### Bloq Variables

Bloq variables are an advanced feature. If you add a parameter to your Bloqs field tag pair with the prefix `bloq_var_`, then the value of that parameter will be made available within your bloq tag pairs. This parameter can override, or workin tandem with the `bloq_var`'s when [nesting bloqs](/bloqs/docs/features/nesting).

{% hint style="info" %}
As of version 4.9, `{bloq_var_[name]}` and `{bloqs:get:[name]}` are interchangable.
{% endhint %}

```
{exp:channel:entries channel="page"}
    {my_bloq_field bloq_var_some_key="some-value"} 
        {content_bloq}  
            {heading}
            {body_content}
            {if bloq_var_some_key == "some-value"}
                Do something
            {/if}
        {/content_bloq}
    {/my_bloq_field}
{/exp:channel:entries}
```

### Bloq Variable Setters & Getters

As of version 4.9, Bloqs now has variable setters and getters, which are very similar to ExpressionEngine's [Layout Variables](https://docs.expressionengine.com/latest/templates/layouts.html#layout-variables). So similar infact that it uses the same parsing code. For example you can set a variable with a value and later reference the variable in a child bloq, or the next bloq in the sequence if not using the nesting feature.

```
{exp:channel:entries channel="page"}
    {my_bloq_field} 
        {section_wrapper_bloq}
            {bloqs:set name="foo" value="bar"}
        {/section_wrapper_bloq}
        
        {content_bloq}  
            {bloqs:get:foo}
        {/content_bloq}
    {/my_bloq_field}
{/exp:channel:entries}
```

#### Bloq Variable Scoping

You can also set a variable in a child bloq, and pass a value back up to it's immediate parent bloq. Scoping currently only supports bloq IDs as values. Text based named scoping is not currently supported.

```
{exp:channel:entries channel="page"}
    {my_bloq_field} 
        {section_wrapper_bloq}
            {bloqs:get:scoped:foo}
        {/section_wrapper_bloq}
        
        {content_bloq}  
            {bloqs:set name="foo" value="bar" scope="{bloqs:parent:id}"}
        {/content_bloq}
    {/my_bloq_field}
{/exp:channel:entries}
```

If you're not using the nesting feature, and rendering bloqs sequentially, you can send a variable value to a previous bloq.

```
{exp:channel:entries channel="page"}
    {my_bloq_field} 
        {section_wrapper_bloq}
            {bloqs:get:scoped:foo}
        {/section_wrapper_bloq}
        
        {content_bloq}  
            {bloqs:set name="foo" value="bar" scope="{bloqs:previous:id}"}
        {/content_bloq}
    {/my_bloq_field}
{/exp:channel:entries}
```

Finally, you can also set a variable and reference it from *all* of it's parent bloqs (note the variable modifier is `:ids`, not `:id`).

```
{exp:channel:entries channel="page"}
    {my_bloq_field} 
        {section_wrapper_bloq}
            {bloqs:get:scoped:foo}
        {/section_wrapper_bloq}
        
        {row_wrapper_bloq}
            {bloqs:get:scoped:foo}
        {/row_wrapper_bloq}
        
        {content_bloq}  
            {bloqs:set name="foo" value="bar" scope="{bloqs:parent:ids}"}
        {/content_bloq}
    {/my_bloq_field}
{/exp:channel:entries}
```


# Add-on Developers

Working on some custom functionality, or building out a new fieldtype? Bloqs was built with you in mind and comes packaged with some features to make your life as easy as possible.

### Adding Fieldtype Support

Getting your field type working with Bloqs is a 3 step process, all of which are pretty straight forward.&#x20;

#### **Step 1 - Your Field Type must be compatible with Grid**

You’re going to start out the process by first getting your field type to be compatible with Grid. If you haven’t done this before, EllisLab has put together some excellent documentation to help you out with the process which you can find here: <https://docs.expressionengine.com/latest/development/fieldtypes.html#grid-fieldtype-development>&#x20;

Once your field type is compatible with Grid, you're just about there!

#### **Step 1.b - Depending on the complexity of your fieldtype**

If your fieldtype creates new HTML elements, similar to adding new rows in a table, and stores the new row HTML template in a `<script>` or `<template>` tag you might need to add a little extra JavaScript to the `onDisplay()` method in your grid.js file. Since the HTML is inside of a `<script>` or `<template>` tag, Bloqs can't perform these updates for you. For example, in the `onDisplay` handler you may need to ensure the form fields in your newly added markup have the correct block ID value. The following is an example to get you started:

```
var onDisplay = function(cell) {
    var isBloqs = cell.hasClass('blocksft-atom');

    if (isBloqs) {
        var $myFieldtype = $('.my-fieldtype');

        $myFieldtype.find('[name]').each(function() {
            var $field = $(this);
            var blockId = $field.closest('.blocksft-block').data('id');
            var eleName = $field.attr('name');

            $field.attr('name', eleName
                .replace(
                    /blocks_new_block_\d+/gm,
                    blockId
                ));
        });
    }
};
```

#### Step 2 - Opt in for Bloqs Support

The second step in the process is configuring your field to “opt in” to Bloqs support. This sounds tricky, but it’s really not. To do this, just modify the `accepts_content_type` method in your field type so that it also accepts content for “bloqs/1”.

Here’s a simple and straight forward example you can reference:

```
public function accepts_content_type( $name )
{
    return ($name == 'channel' || $name == 'grid' || $name == 'bloqs/1');
}
```

{% hint style="info" %}
You may see some fieldtypes referencing "blocks/1" as the accepts content type, and that is perfectly valid. Bloqs version 4.4.0 made the nomenclature switch from "blocks" to "bloqs" in the documentation, UI, and references within it's own codebase.
{% endhint %}

#### Step 3 - Test

The third and final step in the process is to test everything out and make sure it's running smoothly.&#x20;

That's all there is to it!

### Hooks

#### `blocks_discover_fieldtypes`

The `blocks_discover_fieldtypes` hook is called when Bloqs needs to determine which fieldtypes support Bloqs. Extensions may add or remove fieldtypes from the list of valid fieldtypes.

`blocks_discover_fieldtypes` is called with a single PHP array of stdClass objects, and should return that same PHP array, potentially modified.

The format of the objects within the array is as follows:

* type – ExpressionEngine's identifier for the fieldtype. E.g. 'wygwam'.
* version – The version of the fieldtype. E.g., '1.2'.
* name – The human-readable name of the fieldtype. E.g. 'Wygwam'.
* adapter (optional) – An object that serves as an adapter to a fieldtype. If the adapter is set, before Bloqs calls a method on the fieldtype, it first checks if the adapter has the same method, and calls it instead. If the adapter has a `setFieldtype` method, it will be called with the actual fieldtype that is being shimmed.

#### `blocks_post_save`

The `blocks_post_save` hook is called after Bloqs has finished saving.

The parameters that are passed to the hook are as follows:

* blocks – The blocks that now belong to the field.
* context – The context of the field, as an array.
  * $context\['site\_id']
  * $context\['entry\_id']
  * $context\['field\_id']


# FAQs

### Does Bloqs support ExpressionEngine's conditional fields?

No it does not, and probably never will. You can already create multiple bloqs with different atom/field variations, so adding conditionals fields into a bloq further complicates things (UI and code wise) for little gain. If ExpressionEngine does gain the ability to use conditional fields in complex custom field types we'll reevaluate the possibility.

### Does Bloqs work with ExpressionEngine's database Find and Replace feature?

No it does not, but you can use the following queries as a guide to perform a find and replace on any Bloqs data.

```sql
# Replace content across all Bloqs content
UPDATE exp_blocks_atom SET `data`=(replace(`data`, 'old_value','new_value'));

# Replace content in a specific Atom
UPDATE exp_blocks_atom SET `data`=(replace (`data`, 'old_value','new_value')) where atomdefinition_id = "##";
```

### I added a new atom to a bloq but existing entries don't reflect the change.

If you have entries using a bloq, then you update the atoms in that bloq, the existing instances of that bloq in entries might not reflect the addition or removal of the atom. This is easily fixed. Save the entry again (you don't need to make any changes to it, just click Save), and when the entry reloads it should display the modifications to the bloq that you made changes to.

### Does Bloqs work with Fluid?

No, Bloqs and Fluid are similar in their capabilities, but a Bloqs field can not be placed inside of a Fluid field.


# Docs


# Installation

To install Carson, unzip the zip file and place the `user/addons/cason` folder into your `system/user/addons` folder. Then login to the Control Panel and go to the ***Add-ons*** page and click the Install button next to Carson. Three new fieldtypes will be available for selection when creating a new field:

1. Carson Omni
2. Carson Assistant
3. Carson SEO


# Requirements

## Required

* ExpressionEngine 7.3 or greater
  * *ExpressionEngine 7.4 or greater if using the image description generator.*&#x20;
* PHP 8.2 or greater
* [OpenAI API key](https://platform.openai.com/api-keys) and available credits.

{% hint style="warning" %}
Due to the nature of AI models, BoldMinded makes no claim that the responses from AI will be perfect. **Please do not report bugs for content correctness or quality**. For instance if you modify the frequency, temperature, or presence settings to the extremes you could see some very strange results and sometimes jibberish.
{% endhint %}


# Configuration

The following variables can be added to your config.php file to modify the display and/or behavior of the Omni field.

### carson\_fieldtypes

By default the Omni field attaches itself to all Text, Textarea, and Rte fields. It will not attach itself to an Email field, even though it is rendered on the page as a, `<input>`. If you have a custom field that *is* a text or textarea type field that you want to bind an Omni field to, just add the class name to the `carson_fieldtypes` array. For example:

```php
$config['carson_fieldtypes'] = ['Wyvern'];
```

### carson\_open\_ai\_models

Currently the default model is `gpt-4o` . If you want to use a different model you can create this config variable and the array options will be appened to the selectable list in Carson's settings. For example:

```php
$config['carson_open_ai_models'] = [
    'some-model' => 'some-model',
]
```

## Menu Options

The top half of the menu that includes "Improve Writing" is the default section. The middle section that includes the heading "Change Tone" is the tone section, and the last section with the heading "Translate to" is the language section. With the following options you can choose to hide certain sections that may not be applicable to your site.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F6hEnesFaRvnOEA2ymMuR%2FScreenshot%202024-02-13%20at%205.01.30%E2%80%AFPM.png?alt=media&amp;token=690e6b9c-0ced-464e-9edb-04ba93568dcc" alt="" width="165"><figcaption></figcaption></figure>

### carson\_menu\_include\_tone

For example:

```php
$config['carson_menu_include_tone'] = 'n'; // Default is 'y'
```

### carson\_menu\_include\_default

For example:

```php
$config['carson_menu_include_default'] = 'n'; // Default is 'y'
```

## Language Options

By default, if Publisher or Transcribe is installed, all actions in the Omni menu will be contextual to the current language. English is assumed to be the default language. You can modify the language behaviors with the following config values.

### carson\_menu\_use\_language\_context

If you want to ignore the current language context when using Publisher or Transcribe in prompts. Disabling will default the responses to the default language, English. For example:

```php
$config['carson_menu_include_language'] = 'n'; // Default is 'y'
```

### carson\_menu\_include\_language

Not to be confused with `carson_languages` config option. If Publisher or Transcribe is installed, and you're viewing an entry in a non-default language, Carson will automatically add a "Translate to" menu option for the current language, which will let you translate whatever content is in that field to the current language. For example:

```php
$config['carson_menu_include_language'] = 'n'; // Default is 'y'
```

### carson\_languages

If you are not using Publisher or Transcribe but still need translated content, don't worry Carson has you covered. If you're using another method of managing translations on your site, you can define an array of which languages Carson can translate content to.

<pre class="language-php"><code class="lang-php"><strong>$config['carson_languages'] = [
</strong>    'Japanese',
    'Traditional Chinese',
    'Dutch',
];
</code></pre>

### carson\_menu\_include\_image

```php
$config['carson_menu_include_image'] = 'n'; // Default is 'y'
```

Allows you to use Omni without the Image Description, or use just the Image Description without the rest of the fields getting an Omni menu as well. If you want to use Carson just for generating descriptions for images and not display the Omni menu in text fields, then you will want to enable this option, and disable all other options, e.g.

```php
$config['carson_menu_include_tone'] = 'n';
$config['carson_menu_include_default'] = 'n';
$config['carson_menu_include_language'] = 'n';
$config['carson_menu_include_image'] = 'y';
```

## Hooks

Hi-jack and replace the whole menu if you want, just make sure you're re-building it correctly, and if used we can't guarantee support if something does not behave as you expect it to.

### carson\_modify\_menu

<pre class="language-php"><code class="lang-php">public function carson_modify_menu(Menu $menu, PromptModifierHelper $promptModifier): Menu
{
    // Inspect the ft.carson_omni.php file for a working 
    // example of using the Menu object.
<strong>    return $menu;
</strong><strong>}
</strong></code></pre>


# Fields

Carson comes with 3 custom field types:

* [Carson Omni](/carson/docs/fields/omni)
* [Carson Assistant](/carson/docs/fields/assistant)
* [Carson SEO](/carson/docs/fields/seo)

Assistant and SEO are configurable, but the main objective of each is to add AI helpers to your entry publish page. They use the entry's content as the context when making requests to OpenAI.


# Omni

The Omni field, when added to a channel, will add a menu next to every Text, Textarea, and Rich Text field (this includes CKEditor and Redactor fields, and fields within Fluid, Grid, and Bloqs).

Each option is meant to improve, change, or just find the right words for your content on the fly.

<figure><picture><source srcset="/files/sWvJMVE42Ix3eZQLv7Yg" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FHAWtLtqb2ajgZL4uxbE9%2Fcarson-omni-example-lt.png?alt=media&amp;token=7634c012-621d-4742-95f1-206ebaec815b" alt=""></picture><figcaption></figcaption></figure>

{% hint style="info" %}
The Omni field only displays help text (seen in the screenshot above), it's functionality comes from JavaScript added to the publish page. If you do not wish to see the help text, we recommend editing the Publish Layout for the channel and moving the field to another tab.

Hiding the field entirely will prevent it from working. With this in mind, if you want to restrict which roles are able to use the Omni field you can create multiple Publish Layouts and hide the field for user roles that should not be able to use the Omni functionality.
{% endhint %}

You can even create descriptions of images through the image edit modal.&#x20;

<figure><picture><source srcset="/files/fRGv2KBXDPQTaLxZbPxC" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F8khvsjEsUIsmloAjzsrI%2Fcarson-omni-image-example-lt.png?alt=media&amp;token=3d85974f-9987-4f40-9799-1ef65ab4f655" alt=""></picture><figcaption></figcaption></figure>

{% hint style="warning" %}
The Omni field will only attach itself to the description field in ExpressionEngine 7.4 or greater (when the file meta data slide out panel was added), and if the file is actually an image. Lastly, it will only be able to create descriptions from files that are publicly accessible. If you are working with locally uploaded files on your personal computer, or a remote development environment that is not publicly accessible, then it will not be able to read the image to generate a description.
{% endhint %}

You can see the image description feature in action on Vimeo:

{% embed url="<https://vimeo.com/910601523>" %}


# Assistant

The following screenshot is an example of a Carson Assistant field configured with the most strict options. The prompt, which defaults to "Summarize this page." is hidden to the user and can not be changed. Using the field settings (see screenshot below) you can change the default prompt value. For example you could change the button label to "Make this blog post sound more professional and academic." It will use the content from the current entry (by using the Live Preview output if Live Preview is enabled, or the form field values) as the context, and place the response into a chosen text, textarea, or RTE field.

<figure><picture><source srcset="/files/huCztnISrvgH6IeizBW6" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FCLKVPVTUYxvLY4Z7c1Ks%2Fcarson-assistant-field-small-lt.png?alt=media&amp;token=3a156691-ed5c-4bcd-be0f-4f7b47010de4" alt=""></picture><figcaption></figcaption></figure>

The following screenshot is of the Assistant field configured to display the prompt field, but *not* use itself for the target of the output. Meaning the result from OpenAI will be placed in another targeted text, text area, or RTE field. The user can change the default prompt value to anything they want. For example, they could use it to generate a blog post, or improve the writing of the current entry if Force Context is enabled.

<figure><picture><source srcset="/files/H1yjyejT4Ec5ROzCSb9h" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FNJe3wA0sMEzo4pgbwUAc%2Fcarson-assistant-field-prompt-lt.png?alt=media&amp;token=29745daa-89f4-4ecc-9887-151724076b01" alt=""></picture><figcaption></figcaption></figure>

This screenshot is an example of the Assistant field configured to use itself as a target of the output. This can be useful if you are adding Carson to an entry and need to copy and paste to split up the results of the response into separate fields, such as fields inside of Fluid, Bloqs, or Grid. Think of it is an AI clipboard directly in the entry that uses the entry's content to supply additional context to the prompt.

<figure><picture><source srcset="/files/fSaY8DmtyOjGy35qfa7g" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FMkZYt3HA6YzkhvmeDQ71%2Fcarson-assistant-field-full-lt.png?alt=media&amp;token=8ac61fcc-f95f-4a03-9094-640092d03389" alt=""></picture><figcaption></figcaption></figure>

The Carson Assistant field settings. As noted in the previous screenshots the options presented to the user and how the field functions can be configured here. Since this is a fieldtype, you can create multiple assistants and add them to the same entry, or an assistant field to each channel.

<figure><picture><source srcset="/files/ZbRzaOD43KJLylGymKEq" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F9GbqrgyAhAYary3Uvzrv%2Fcarson-assistant-settings-lt.png?alt=media&amp;token=a2f42cb2-32e6-4efd-bf5b-0683b4fc6af6" alt=""></picture><figcaption></figcaption></figure>


# SEO

Quickly generate an entry's SEO title, description, and keywords using AI.

Unlike the Assistant field, the SEO field has no display options other than changing the button label and working text (see the screenshot below). The prompt is hard-coded to use the entry's Live Preview output, or the publish page form fields as the context in which to generate SEO meta data from. The SEO field will work with ExpressionEngine custom fields of your choosing, or if you are using the SEO Lite, or SEEO add-ons, it will place the generated title, description, and keywords into the corresponding fields for those add-ons.

In order to generate SEO data from a fully rendered page you need to have the Live Preview feature enabled and configured for the channel that the SEO Assistant field is added to. If Live Preview is not enabled it will simply use the plain text from all of the form fields within the entry. It is recommended to use the Live Preview feature as it will generate a full HTML page with heading and other contextual tags that will help OpenAI generate meta data for the page. It is also recommended to hide your `<meta>` tags using the `{if is_live_preview_request == false}<meta> tags here{/if}` , otherwise the existing meta tag values will be sent to OpenAI and regurgitated back to you.

<figure><picture><source srcset="/files/2YB1uWhEnX2J6Mk5zaLL" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FY7fsiJyhjVmQt0gdfB1q%2Fcarson-seo-field-lt.png?alt=media&amp;token=45ec5641-09ea-4297-b1e5-9bd639bee2fd" alt=""></picture><figcaption></figcaption></figure>

The settings for the SEO field are fairly minimal. You can change the button label and working text, as well as which character to use as the meta title. The prompt text (what is sent to OpenAI) to generate the meta data is hard-coded and can not be changed for this field. This field is designed to do 1 thing and 1 thing only.

<figure><picture><source srcset="/files/oYHtshzBJWp5MjVQl2E5" media="(prefers-color-scheme: dark)"><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FwdiZECNzR1cr3E21G7O3%2Fcarson-seo-settings-lt.png?alt=media&amp;token=530bba40-99c7-44f2-883b-75f6c3fb6965" alt=""></picture><figcaption></figcaption></figure>


# Template Tags

Carson 1.3 added an assistant tag, which lets you perform AI prompts in a template. It can be used as a single tag, or a tag pair that will append whatever content you place into it to the prompt instructions.

```json
{exp:carson:assistant instructions="Do something..."}
```

Used as a tag pair:

```json
{exp:carson:assistant instructions="Do something with this tag content..."}
   ... anything here ...
{/exp:carson:assistant}
```


# Troubleshooting

If Carson is not responding with any data, or acting suspicious, inspect the JavaScript console for errors, and inspect the Ajax requests in the Network tab. There is a very high probability if Carson is not responding as expected, *something* will be revealed in either of those places. If so, also make sure to include this in any support ticket you create.&#x20;

If you're using the SEO field, and it is not working see the previous tip, but also make sure you're Live Preview feature is enabled and working in the channel, and also make sure your environment has cURL enabled, and you're not running into a self-referential request failure (e.g. [making requests to URLs on the same server](https://stackoverflow.com/questions/5412069/can-i-do-a-curl-request-to-the-same-server))


# Docs

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-MUPXQzpPsEh7sBvgafx%2F-MUPXgzpE9fxfIBP2rTW%2Ficon-csm.gif?alt=media\&token=f4eada4b-5757-43e7-a398-0f47c099f2f2)


# Installation

To install Custom System Messages, unzip the zip file and place the `system_messages` folder into your `/system/user/addons` folder. Then login to the Control Panel and go to ***Developer > Add-ons*** and install Custom System Messages.


# Requirements

* ExpressionEngine 6.4 or greater
* PHP 8.2 or greater


# Variables

The following list of variables is not unique to Custom System Messages. They are native variables available in ExpressionEngine's [System Message Templates](https://docs.expressionengine.com/latest/cp/design/system/index.html).

* `{meta_refresh}` Renders the meta refresh tag for your HTML head
* `{heading}` Message header
* `{content}` Message details, usually a sentence of text or possibly an unordered list of error messages.
* `{link}` Link to return to the previous page.
* `{action}` will return the ACT ID which you can use it in your templates to show messages specific to the action the user performed. This will usually be found in your URL as `?ACT=[id]`.
* `{error}` Boolean: If the action is an error message or not. Use this in conditionals to change the behavior of your error template. By default, ExpressionEngine uses the same ACT ID for success or error messages, only internally does it know which message to display (e.g. what the `{content}` variable is set to).
* `{referrer}` The URL prior to the error message.

Custom System Messages also provides aliases for each each variable that are parsed early, thus they can be used in conditional statements and other template tags. Each variable above has an alias prefixed with `csm:`. For example `{content}` is also available with `{csm:content}`. If you are having trouble with the original variables, try the aliases.

Even though the `{action}` and `{error}` variables are available, you may want to use entirely different templates for each ACT and on their success or error results instead of conditionals. You can do this by selecting which template each action in the drop down is assigned to, there is also an "(On Error)" version for each action.


# Template Tags

Given the following settings, Custom System Messages will deliver all ExpressionEngine's action and form messages inline with your normal templates. In this case the `messages.group/error.html` file will serve as the default template for all global error messages. The template code below is an example of a basic sign in form in the `/account.group/sign-in.html` file. It is also defined in the Custom Actions section as the template to use for success *and* error messages when submitting the sign in form. All other non-login related errors in this example will be displayed in the `/messages.group/error.html` template. Note that both Submission Error and General Error fields are required.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-M7xAcCFQg5Ap5W1Ck5z%2F-M7xAknRy7RBjq1xnN5t%2Fcsm-docs-settings.png?alt=media\&token=c7483173-50fb-4c34-a918-c184e7a92ea5)

{% hint style="info" %}
If you add a {redirect=""} tag in any of your templates defined in CSM, then the error messags will not be displayed in the template, even if the template you're redirecting to is defined in CSM to handle errors. This is because the messages are not saved during a full http request/redirect. All redirections need to be handled by CSM as defined in it's settings.
{% endhint %}

## account/sign-in Template

```
{if csm:error == true && csm:action == 11}
    <div class="form-error-section">
        <div class="error-message">
            <strong>The following errors were encountered:</strong>
            <ul class="error-fields">
                {csm:content}
            </ul>
        </div>
    </div>
{/if}

<div>
    <h3>SIGN IN</h3>
    <p>Don’t have an account yet? <a href="{path='account/register'}">Create an account.</a></p>
    <p><a href="{path='account/forgot-password'}">Forgot your password?</a></p>
</div>

<div>
    {exp:member:login_form return="account/sign-in"}
        <div>
            <label for="signin-email">Email Addresss <span class="input-desc">(Username)</span><span class="input-required">*</span></label>
            <input type="text" id="signin-email" name="username" />
        </div>
        <div>
            <label for="signin-password">Password <span class="input-required">*</span></label>
            <input type="password" id="signin-password" name="password" />
            <button type="submit" name="submit" value="SIGN IN">SIGN IN</button>
        </div>
    {/exp:member:login_form}
</div>
```

## Result

The result of this configuration is the error messages are shown above the login form, instead of on a separate template, and at the same URL. You can still use a separate template if you like. The combination of settings in Custom System Messages gives you the flexibility to use whichever template you prefer to display your sites error or success messages.&#x20;

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ly1eNBG1JvS5rivVtg3%2F-Lyoy9NQTeAWjUqKK8UX%2F-Lyp-78isomi9h2XEbCy%2Fcsm-inline-error.png?alt=media\&token=466964ec-e73f-4bd1-829d-cdfefc3167d1)


# Docs


# Installation & Upgrading

### New installation

To install DataGrab, unzip the zip file and place the `user/addons/datagrab` folder into your `system/user/addons` folder. Then login to the Control Panel and go to the ***Add-ons*** page and click the Install button next to DataGrab.

{% content-ref url="/pages/zbtFp9CZU72xTGh7EwIQ" %}
[Upgrading from 3 to 4](/datagrab/docs/installation-and-upgrading/upgrading-from-3-to-4)
{% endcontent-ref %}

{% content-ref url="/pages/1X1pNjlqiQYKELvSbVUH" %}
[Upgrading from 4 to 5](/datagrab/docs/installation-and-upgrading/upgrading-from-4-to-5)
{% endcontent-ref %}

{% content-ref url="/pages/Cw3S3GiWeAxzLsaOi8jp" %}
[Upgrading from 5 to 6](/datagrab/docs/installation-and-upgrading/upgrading-from-5-to-6)
{% endcontent-ref %}


# Upgrading from 3 to 4

To install DataGrab 4, unzip the zip file and place the `user/addons/datagrab` folder into your `system/user/addons` folder. Then login to the Control Panel and go to the ***Add-ons*** page and click the Install button next to DataGrab.

{% hint style="warning" %}
It is suggested to upgrade to DataGrab 3.0.4 before upgrading to 4.0. If you can't get a copy of 3.0.4, read the instructions at the bottom of this page.
{% endhint %}

If you have version of DataGrab prior to 4, probably version 3.0.3 or 3.0.4 so, then do the following:

1. Keep the existing `system/user/addons/ajw_datagrab` folder, DO NOT delete it.

2. Copy the `datagrab` folder from the zip file into your `system/user/addons` folder.

3. Yes you will have 2 DataGrab add-ons displaying in your ExpressionEngine control panel. You will see the following **uninstalled** DataGrab 4.x version that has an icon:\
   \
   ![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FGGTxopkak1dmCXHjlNAX%2Fdatagrab-install-step1.png?alt=media\&token=9c5b6049-01f5-4aa6-b49d-31932f9eac02)\
   \
   And the currently **installed** 3.x version that does not have an icon: \
   \
   ![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F7VEBbkdrGa9igfrJxZi9%2Fdatagrab-install-step2.png?alt=media\&token=610b7b15-bfef-40bd-91bc-c34bdf76cca7)

4. Click on the Install button for DataGrab 4.x.

5. Once the install is complete, click the gear icon on the DataGrab 3.x version, and click Uninstall, and then you can remove the`addons/ajw_datagrab` folder. All your previously saved imports and Import URLs will remain the same.

6. If you were using the `{exp:ajw_datagrab:run_action}` or `{exp:ajw_datagrab:run_saved_import}` template tags you will need to rename them to `{exp:datagrab:run_action}` and `{exp:datagrab:run_action}`

#### Upgrading from DataGrab 3.x

If DataGrab 3.0.3 is the latest version you have, you may experience a database SQL error on step #5 above. To avoid the error you will need to edit the upd.ajw\_datagrab.php file. Go to line 121 of the file and you will see: `ee()->db->delete('module_member_groups');` You can choose to comment out this line and the one above it, then uninstall, or change `'module_member_groups'` to `'module_member_roles'`, then uninstall.

### Playa, Matrix, and Pixel & Tonic fieldtype Support

{% hint style="warning" %}
**Playa, Matrix, and any Pixel & Tonic fieldtypes are deprecated**. If you have a site with an import configured to use them, DataGrab may continue to work, but no future development will be performed on those fieldtypes, and support will not be provided for them if there is an issue importing values into those fields. Playa and Matrix are antiquated and should be switched to Relationship and Grid fields respectively. The specific files no longer supported are:

* datagrab\_matrix.php
* datagrab\_playa.php
* datagrab\_fieldpack\_checkboxes.php
* datagrab\_pt\_multiselect.php
  {% endhint %}


# Upgrading from 4 to 5

To install DataGrab 5, unzip the zip file and place the `user/addons/datagrab` folder into your `system/user/addons` folder. Then login to the Control Panel and go to the ***Add-ons*** page and click the Install button next to DataGrab.

Version 5.0 of DataGrab brings a fairly significant change to how DataGrab works. DataGrab 5 introduces the [Laravel Queue package](https://laravel.com/docs/8.x/queues). This means that DataGrab now supports the producer/consumer model. Since DataGrab's initial release in 2010 it has relied on reading a JSON, XML, or CSV file and interating the contents of that file to perform the updates. Users with large imports often ran into server timeout responses or PHP memory issues. Simply put DataGrab was never built to handle large imports.

A lot has changed under the hood, but the actual methods that perform the entry importing have remained unchanged, but everything leading up to the actual import process has received an overhaul. Overall the code is simpler and DataGrab doesn't have to perform as much gymnastics to read and iterate an import file as it used to. When an import file is read, it inserts the import items into a queue (this is the "producer"). The items, or entries, remain in the queue until a consumer enacts upon them and completes the import. If you are running the imports manually within the control panel not much has changed for you. Initiating an import will run the producer to read the import file and create the queue, then immediately start consuming the queue.&#x20;

### Notable Changes

If you had imports configured with a "limit" value below 50, upon upgrading to DataGrab 5 it will change the limit to 50. This is because the queue does a much better job at managing it's own resources and we don't have to set a "limit" of 1 (the previous default) to stay within the any PHP or server based timeout settings. You can still adjust this value when configuring an import but we recommend starting at 50 and seeing how the imports peform based on your server's configurations. You maybe able to set it to 0, which means the consumers will import as many entries as possible until it decides to self terminate the consumer and start a new one.

When importing within the control panel and you have configured your import to delete non-imported entries, you will see a second red progress bar. The first purple progress bar is the consumer that is importing the entries, and the second progress bar is the consumer that deletes the other entries. The deletions to be included in the same request, but since we're using queues we're taking advantage of them and split up the work. The second red progress bar is the indicator that the initial entries were imported, and it started a new consumer to delete the entries that should be deleted.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FNUBN6InAh1Ta65AappUz%2Fimport-queue.png?alt=media&amp;token=ce97a870-a6cb-417d-9bae-cef9d0582761" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FzCZrjjhaZZjjePP4MKNe%2Fimport-delete.png?alt=media&amp;token=bc0b9b2c-854e-4b84-a550-ddfd95b055d8" alt=""><figcaption></figcaption></figure>

#### Deletions

A new "Soft delete" option was added. If you checked the "Delete old" option to delete old entries from a channel that were not included in the import you can optionally soft delete them, which will set it's status to Closed instead of removing the entry entirely from the database.

#### Improved Cartthrob Order Items fieldtype

The Cartthrob Order Items fieldtype support had been horribly neglected and did not work with more recent versions of Cartthrob. It has been updated to support importing variable column values, but it needs to follow a specific format. Your import file must contain an "extra" node that contains a JSON object.

```
...
<quantity>3</quantity>
<price>$100.00</price>
<extra><![CDATA[
  {
      "discount": 1,
      "price_plus_tax": "$20",
      "product_color": "Blue",
      "product_code": "WIDGET123"
  }
]]></extra>
```

If your import file is a JSON file, then the "extra" node needs to contain a JSON string:

```
"quantity": 3,
"price": "$100.00"
"extra": "{\"discount\": 1,\"price_plus_tax\": \"$20\",\"product_color\": \"Blue\",\"product_code\": \"WIDGET123\"}""
```

#### CLI Commands

The existing CLI commands will continue to work as they did before. If no additional arguments are defined it will produce and immediately consume the entries from the queue.

For more information on the CLI commands please visit the [Importing with cron](/datagrab/docs/automatic-imports/importing-with-cron) documentation.

#### Queue Drivers

By default DataGrab uses the database for it's queue. No changes are needed to your config files to support this. You can optionally use Redis as a queue driver as well. You'll need to have Redis installed and configured on your server, and add the following to your ExpressionEngine config.php file.

```
$config['datagrab'] = [
    'driver' => 'redis',
    'redis_config' => [
        'host' => 'redis',
        'port' => '6379',
        'timeout' => '0',
        'password' => null,
    ],
];
```

{% hint style="info" %}
When using the Database queue driver, which is the default, it is best to only run 1 consumer at a time. Running multiple consumers at the same time may result in database locking issues and all items in the queue may not be imported. If you want to run more than 1 consumer at a time try the Redis queue driver.
{% endhint %}


# Upgrading from 5 to 6

To install DataGrab 6, unzip the zip file and place the `user/addons/datagrab` folder into your `system/user/addons` folder. Then login to the Control Panel and go to the ***Add-ons*** page and click the Install button next to DataGrab.

{% hint style="warning" %}
Version 6 is the first version of DataGrab to introduce some breaking changes. The most notable change is after upgrading you may need to review your existing imports, reconfigure the field mappings, and re-save it. DataGrab 6 does attempt to migrate these settings for you but in some cases it will require human intervention.
{% endhint %}

{% hint style="warning" %}
**Other noteable changes:**

Support for Cartthrob and Store are not currently available. We hope to add support for those add-ons in the fall of 2025.

Officially dropped support for Rel (old EE2 style relationship field), Assets, Matrix, Playa, PT Multiselect, Field Pack Checkboxes, MX Google Map, and VMG Chosen Member. These were deprecated in DataGrab 5.x, but the fieldtype files were still included to support any legacy imports. The files are no longer included with DataGrab 6.
{% endhint %}

Version 6 introduces several new features. One of them is the ability to import into Bloqs and Fluid field groups. Some fieldtypes such as Simple Grid, Simple Table, and the native Grid field all allow for repeated rows of content to be imported. Importing into Grid, Simple Grid, or Simple Table as a standalone field works the same way as it did in previous versions of DataGrab. You simply choose a column and select from the dropdown list of properties in the import file and the data will be imported into that column.

To get this same behavior to work when one of these fields is inside of Bloqs, Grid, or Fluid you will need to set one additional value from the dropdown. The "Parent Node" option will only present itself in this scenario. *DataGrab assumes that data imported into these fields will be grouped together and at the same level in your import file(1).* The value of the Parent Node will need to be the top most parent of that group. It's going to be the property that ends with `__parent__ - [includes 1 children]`In this example that would be `fluid_field/{n}/fluid_group_3/{n}/grid/parent - [includes 1 children]`. This tells DataGrab to grab all the values for that field in one pass, as opposed to performing excessive loops. It's also important because with Bloqs, Grid, and Fluid, it's possible to have more than 1 instance of of basically the same field nested inside of it. This helps keep the contents of the field assigned to the appropriate instance of the nested field when there are more than 1.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FKeUcpERIpt3BXT4enNoR%2Fdatagrab-v6-check-settings.png?alt=media&amp;token=45f1c85f-6560-4950-b6ef-9bd8d6a7283f" alt=""><figcaption></figcaption></figure>

This is an example of a Simple Grid field configuration inside of a Bloqs field.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FPOcfrixBdbddXSx2ltzT%2Fdatagrab-v6-sub-grids-1.png?alt=media&amp;token=8b937483-94ad-4c82-9397-97676d7ddf1b" alt=""><figcaption></figcaption></figure>

This is an example of a Simple Table field inside of a Bloqs field.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FNNpw5NwESugJ9NUEOvUJ%2Fdatagrab-v6-sub-grids-2.png?alt=media&amp;token=55e6304d-b96f-4007-8fc4-79734de5cc2c" alt=""><figcaption></figcaption></figure>

(1) If you're trying to pluck data from varying locations and depths of your XML or JSON file into a Grid, Simple Grid, or Simple Table field, it will not work. The data should be grouped how you want it presented in the field. For example:

```xml
<grid>
    <row>
        <related>
            Han Solo, Luke Skywalker
        </related>
        <date>
            2026/08/25
        </date>
        <desc>
            This is some text
        </desc>
    </row>
    <row>
        <related>
            Darth Vader
        </related>
        <date>
            2026/08/26
        </date>
        <desc>
            This is some more text in row 2text
        </desc>
    </row>
</grid>
```

### New parameters

When running an import through the command line there are 3 new options:

`--filename="path/to/file.json"`

This will let you override the import file name or url value configured in your import at runtime. You can also add this parameter to an import when running it through the ACT url, e.g. `https://mysite.com/?ACT=10&filename=path/to/file.json`

`--params="foo=bar&fizz=bazz"`&#x20;

Allows you to pass additional GET parameters to your import filename. The script processing your import filename will need to be setup to accept these parameters. You can use this to configure multiple imports through crontab entries all using the same import configuration, but customizing each one to process the import feed differently.

`--force_update=yes`&#x20;

When used this will override the "Update existing entries" option in the import configuration, thus it will look at the matching fields to see if an entry containing the title value in your import matches an existing entry (if you set the Title field as one of the matching fields), then this will override the "Update existing entries" toggle. In the following scenario the TItle field is set to check for duplicates, but executing the import without the `--force_update` override it will not update an existing entry that matches the Title if it exists.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FREI1CBUBWmb1u3s1hub1%2Fdatagrab-force-update.png?alt=media&amp;token=b76053dc-1642-41a1-bdb4-ab0ed15d88eb" alt=""><figcaption></figcaption></figure>


# Requirements

* ExpressionEngine 6.4 or greater
* MySQL 5.7 - 8.0 or MariaDB 10.2+ (InnoDB engine preferred. Install may fail if database engine is MyISAM)
* PHP 8.2 or greater
* php-xml module installed if importing from an XML file. If you see an error such as "Call to undefined function xml\_parser\_create()", [then this is likely the issue](https://stackoverflow.com/questions/45912836/call-to-undefined-function-xml-parser-create).
* cURL extension installed and enabled
* PCNTL extension installed and enabled

DataGrab will also attempt to set some PHP variables to improve it's performance and hopefully avoid errors. Below is a list of the variables it will set. For this reason, it is best to make sure `ini_set()` is not in PHP's `disabled_functions` directive (some shared hosting providers will disable `ini_set()`)

### **error\_log**

DataGrab will log its import status. This is used for self debugging, and when reporting support issues. [Read more about the logging options](/datagrab/docs/troubleshooting).

### **auto\_detect\_line\_endings**

When using the CSV import type, DataGrab will attempt to enable the `auto_detect_line_endings` option.

### **pcre.backtrack\_limit**

When using the Wordpress import type, DataGrab will attempt to set the `pcre.backtrack_limit` to `250000`.

### **memory\_limit**

When the import process runs, DataGrab will attempt to set the PHP `memory_limit` to `1G` for the duration of the import to help ensure the import is successful.&#x20;


# Troubleshooting

{% hint style="warning" %}
If you see an error similar to the one below, **there is a 99.9% chance that this is&#x20;*****not*****&#x20;a DataGrab issue**. Please do not submit a ticket until you have fully read this guide and exhausted all other posibilities.
{% endhint %}

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FxgrZWWWF0pAS4sfA9jLz%2Fdatagrab-xml-error-example.png?alt=media&amp;token=84204747-f989-44a7-9cd1-66fe172aa6ca" alt=""><figcaption></figcaption></figure>

### File validation

Make sure your import files are valid. Use an [online validation checker](https://codebeautify.org/xmlvalidator) to ensure there are no errors with the file formatting. DataGrab can not read invalid XML, JSON, or CSV files.

{% hint style="info" %}
There have been multiple bug reports of CSV file imports failing that have boiled down to a character encoding issue. Either a special or hidden character in the file that breaks the import. DataGrab 6 will attempt to convert the encoding to UTF-8.  If DataGrab still can't read the file because of a character encoding issue then you will need to find and resolve the issue yourself. When this happens it will not show up in the DataGrab log file as a clearly identified character issue. It'll likely just stop importing after a certain point in the file (e.g. it stops after importing 27 entries), and that is where you will likely find the issue, on entry #28.
{% endhint %}

### Import Configuration

Most of the issues encountered with DataGrab occur when configuring an import. Usually it's because the import file is invalid or inaccessible.

&#x20;Check the following:

* The path in the config is correct
* Try running `curl https://site.com/path/to/feed.xml`in your command line and see what the result is. Often times this might return something like a CloudFlare error page, or some other HTML based page that is not the expected JSON or XML response, which means there is an issue with your server that needs to be resolved before DataGrab can read the file. Sometimes loading the URL in a browser window appears fine making it appear as if it is a DataGrab issue, but using `curl`will reveal the real issue.
* Make sure your environment has cURL enabled, and make sure you're not running into a self-referential request failure (e.g. [making requests to URLs on the same server](https://stackoverflow.com/questions/5412069/can-i-do-a-curl-request-to-the-same-server))
* Make sure `$config['datagrab_verify_peer'] = 'y';` is not in your config.php file, or it is set to `'n'`
* Open the import feed in a separate browser window, if it does not load and you do not see the XML or JSON data, neither can DataGrab.
* If you are importing from a local file or URL and using basic auth to block access to your site while in development, then be sure to include the credentials in your import URL as well. The import process does not use the same authenticated session as your browser. For example: `http://user:pass@mystagingsite.com/import-file.json`
* If you are importing from a remote file or URL that is behind a basic auth, make sure it is correct, and make sure the username and password don't contain an @. If you open the feed with the username and password in a separate browser window and you receive a login prompt, then DataGrab sees the same thing and can not read the file. This likely indicates your username or password is wrong, or something else is misconfigured on the server.
* Make sure your site and feed protocols match. Don't load a feed from https if your site is running from http.

### Check the logs

If your import file is valid and everything is configured and you are  imports checking the log file is the best place to go to monitor the import activity. By default DataGrab will log to the `system/user/cache/DataGrab-import.log` file. Each time an import starts, it will purge old logged items from the file to avoid compounding the file size. You can change the logging type by adding the following option to your `config.php` file:

&#x20;`$config['datagrab_log_type'] = 'file|developer|php|off';`

`'file'` is the default value. You do not need to add this to your config file unless you want to use one of the other options.

If you set the log type to `'developer'` it will log to ExpressionEngine's Developer Log that is accessible in the control panel. Go to **Tools > Logs >Developer Log** and review the log for any error messages. It may be helpful to fully clear the entire log, then run an import again to ensure you have up-to-date log messages.

If you set the log type to `'php'` it will write to your server's PHP log file. Depending on your system and how you have PHP configured, this file path may not be in a standard location such as `/var/log/.` If you do not want to log to your database (e.g. ExpressionEngine's Developer Log) or the `user/cache` directory, then you should use the php log option. If you choose '`php'` as your logging option you can also change the path if you use the following config option:

`$config['datagrab_log_file'] = '/var/www/log/DataGrab-import.log';`

Note that if you change the log file make sure the directory is writable. We do not provide support for system configuration settings if the log is not outputting to the value you've defined. This option will not set your entire php error logging to this file. It will only log DataGrab events and errors to this file. Other ExpressionEngine or similar PHP errors will continue to be logged to your system defined log file.

If you do not set this option in your config file DataGrab will continue logging to the `DataGrab-import.log` file in your `user/cache` directory as mentioned above. You can turn off all logging by setting the log type to `'off'`;

{% hint style="danger" %}
If you disable logging and create a support ticket, you will be asked to re-enable and provide a log output of the import issue you're experiencing. **The php or default DataGrab-import.log file is preferred for support requests** since I can not read your Developer log without control panel access.
{% endhint %}

### Try the Import URL

If you attempt to run an import and the Status column does not render a progress bar, but instead renders an error message, or something that looks like the screenshot below, then use the **Import URL** option and attempt to run the import in a new browser tab. From there, the full error message will be revealed. It is also highly possible the error will be in your import log. Please include the full error message if you're creating a support ticket.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FSXgi0VCkYp6SR5sUD1oN%2Fdatagrab-iframe-error.png?alt=media\&token=94e6d3fa-029e-41b4-8b46-77b0a75b3aa1)


# DataGrab-import.log Rotation

Starting in version 6.0.5 of DataGrab you can now add `$config['datagrab_rotate_log'] = 'y';` to your ExpressionEngine `config.php` file. If enabled the log file from the previous import will be backed up with a timestamp in it's name. This will happen every time a new import is started.

Now that you have log rotation enabled, you might want to keep the log files from piling up. Create a file in the `system/user` folder called `cleanup-log.sh` . Make it executable with `chmod +x cleanup-log.sh` . Add this code to that file.

```bash
#!/usr/bin/env bash
set -euo pipefail

# Path to the log file you rotate
LOG_FILE="cache/DataGrab-import.log"

# Directory where the logs live
LOG_DIR=$(dirname "$LOG_FILE")

# The base name of the log (without directory)
LOG_BASE=$(basename "$LOG_FILE")

# Delete rotated logs (with timestamp suffix) older than 7 days
find "$LOG_DIR" -type f -name "${LOG_BASE}.*" -mtime +7 -print -delete

```

You will need to add an entry to your crontab to run this script periodically. You can run it as often as you like. Every 15 minutes, once a day, or once a week. The following will run every hour.

```
0 * * * * /path/to/system/user/cleanup-log.sh
```

{% hint style="info" %}
For more information about crontab, [see the docs](https://man7.org/linux/man-pages/man5/crontab.5.html).
{% endhint %}


# Configuring Imports

From the ExpressionEngine Control Panel, go to Add-Ons > Modules and choose the DataGrab module.

DataGrab has the option to import different types of data. XML, CSV, JSON, and WordPress. Before configuring your import double check the valididty of your import file.&#x20;

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FuShUtkKfnHdtGveDXKgX%2Fdatagrab-choose-import-type.png?alt=media\&token=44b6e227-42a3-4ff0-a42e-16a278a0cbdb)

{% hint style="info" %}
For additional security you can add `$config['datagrab_verify_peer'] 'y';` to your config file. This will verify the SSL certificate of the host for the URL of the file you are importing.
{% endhint %}

### **Import Settings**

Once you have choosen your data type, you are presented with options determined by that data type. Every data type requires a Filename or URL, and most require a path to each entry in the document. This could be the XML path or the node in a JSON file.

The Filename or URL field can also be an environment variable. For example, if you have the following in your `.env.php` file, you can use `$MY_IMPORT_FILE_URL` as the field value. Using an environment variable means you can have different import files in a local or dev environment vs production, and not have to change the DataGrab settings when moving between environments.

```
MY_IMPORT_FILE_URL=https://domain.com/some-data.xml
```

{% hint style="info" %}
DataGrab will support [basic authentication](https://http.dev/authentication), but if your import URL requires additional authentication, like OAuth, to a private REST API, you will need to create a custom mediator script to handle the authentication, then use that mediator as the URL in your DataGrab configuration. For example: `https://mysite.com/api-authenticator.php`
{% endhint %}

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FF5Netta44HEfx4zTQN8e%2Fdatagrab-import-settings.png?alt=media\&token=f0e205b2-5178-48fb-8642-21f11ad66c62)

### Check Settings

If your file is valid and readable, you will be presented with an example of the data found in the file. If no data was found, then it means there was probably an error on the previous Import Settings page.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FypYbTFFTgS9ZnDyM4kjK%2Fdatagrab-check-settings..png?alt=media\&token=cd7b75f2-81f3-4903-8539-3efcb7346d3b)

If everything looks correct, click on “Configure Import” to continue to the next step.

{% hint style="warning" %}
If you see an error on this page, usually something to do with cURL, **there is a 99.9% chance that this is&#x20;*****not*****&#x20;a DataGrab issue**. For some reason, DataGrab can not read your import file. Check the following:

* The path in the config is correct
* Make sure `$config['datagrab_verify_peer'] = 'y';` is not in your config.php file, or it is set to `'n'`
* Open the import feed in a separate browser window, if it does not load and you do not see the XML or JSON data, neither can DataGrab.
* If you are importing from a local file or URL and using basic auth to block access to your site while in development, then be sure to include the credentials in your import URL as well. The import process does not use the same authenticated session as your browser. For example: `http://user:pass@mystagingsite.com/import-file.json`
* If you are importing from a remote file or URL that is behind a basic auth, make sure it is correct, and make sure the username and password don't contain an @. If you open the feed with the username and password in a separate browser window and you receive a login prompt, then DataGrab sees the same thing and can not read the file. This likely indicates your username or password is wrong, or something else is misconfigured on the server.
* Make sure your site and feed protocols match. Don't load a feed from https if your site is running from http.
  {% endhint %}

### Configure Import

This documenation will not cover every option on the Configure Import page, because depending on the data type chosen the options will vary based on your ExpressionEngine configuration. The configuration screen is self documenting. Every option has a description explaining what it does.&#x20;

If the channel you are importing into has category groups assigned to it you will be able to choose which values in your import file map to which category. The main options are the Title field, and the Custom Fields section. You will see an option for every custom field assigned to your channel. Just like the categories you choose which values in your import file map to which custom field.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FRTB8ANw37g66KnJYWr8E%2Fdatagrab-configure-import.png?alt=media\&token=afd8b0de-9088-4875-b0c3-282e27f03ff2)

Additional options let you tell DataGrab what to do if an entry already exists. This is especially useful if we want to periodically re-run the import. The best way to do this is to import a unique value into a custom field (often referred to as a GUID). This unique value could be an id, a url specific to the entry, a stock control number, or just something as simple as the title.

If the module encounters a duplicate entry, we can tell it ignore it and not import it, or we can get it to check to see if anything has changed and updates the existing record.

Sometimes, we want to delete entries that are not in the current import file. By adding a timestamp we can see when a record was last updated. We can then delete older entires if desired.

### Save Import

After you save the configuration you will have the chance to give the import a name, short description of what it does, and optionaly give it a passkey.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FIHmfrk8f3w4od6TNVnT6%2Fdatagrab-save-import.png?alt=media\&token=cb8de62d-9c70-4d18-9495-8353477e32df)


# Running Imports

Imports can be executed directly in the ExpressionEngine control panel, or by requesting the Import URL. Note that the Import URL does not require the user to be logged in to execute. Therefore, if a random user discovers your Import URL, they can run a import for you without you knowing! For this reason, it is recommended to use a passkey, which you can set when configuring the import.

If you encounter any issues running an import, the first step you should take is look at your import log, as described in the [troubleshooting section](/datagrab/docs/troubleshooting). DataGrab has pretty extensive logging, so there is a good chance you will find what you are looking for in the logs.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F1dEVnTKxdLm8iWnrlWla%2Fdatagrab-imports-list.png?alt=media\&token=41e05095-cd04-481e-bde8-6c1677e6c010)

Since DataGrab operates using a queue, there may be times where you view the DataGrab imports list and a status changes, such as displaying a loading bar, on one of the imports without you taking action. This may happen if an item from that import is in the queue and DataGrab takes action on it. When this page is open in the browser the it is always checking for status updates of imports.

### Toolbar Actions

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FbZiLwP3r4ltih3sh1wnv%2Fdatagrab-icon-start.png?alt=media\&token=1286850c-1898-4509-ace1-616045b07799) starts, or continues an import.

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FMxm1wI4qlXjiiotYZXkS%2Fdatagrab-icon-restart.png?alt=media\&token=14d62dc4-4de2-480a-b226-ce1bea9e7ece) restarts an import from the beginning. *This button is only visible if an import is WAITING.*

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F1qi1zmgGd22DninikpMj%2Fdatagrab-icon-edit.png?alt=media\&token=91fa3b88-f8aa-4cfa-bea4-55a4a97a5ee5) *lets you edit the import settings, such as it's name, description, and passkey.*

![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FG0lnmhJTMa4dg7EeWQml%2Fdatagrab-icon-config.png?alt=media\&token=d580df3a-026e-4b24-9c6f-7f11b025399d) *lets you configure the import itself, e.g. mapping import file properties to fields in ExpressionEngine.*

***Import URL** simply gives you the URL to the import so you can run it in a cron, or outside of the control panel.*

If an import was halted for any reason, such as reloading the control panel page while it was in progress, it will switch to the WAITING status. You can resume the import at any time by clicking on the ![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FbZiLwP3r4ltih3sh1wnv%2Fdatagrab-icon-start.png?alt=media\&token=1286850c-1898-4509-ace1-616045b07799) icon.

### Memory Management and timeout issues

DataGrab uses import queues. When an import initiated it reads your import file and creates records in the queue database table for all the items it needs to import. This could be 5 or 5000 entries. It then uses consumer processes to import the entries in batches, defined by your batch import size setting (defaults to 50). Consumers will stop importing when it hits that limit, or if it reaches a PHP script max execution time. DataGrab will start new consumers when the import is executed through the control panel (see the screenshot above), or if using the ACT url to import. However, if you are importing with the CLI, you will need to manually start new consumers. Fortunately there are things like cron and supervisord that will automatically start new consumers on whatever interval you define. Read more about this in the [Automatic Imports](/datagrab/docs/automatic-imports/importing-with-cron) section of the docs.


# Automatic Imports

To run imports automatically, you can use one of three methods:

### 1. You can use the ExpressionEngine CLI tool (Recommended):

<pre><code><strong>php system/ee/eecli.php import:run --import_id=2 --limit=1
</strong></code></pre>

To run automatically, you will need to set up a cron to call the command. For more details see: [Running imports using cron](/datagrab/docs/automatic-imports/importing-with-cron).

### 2. You can create a template, and include the tag:

```
{exp:datagrab:run_saved_import id="2"}
```

(this will run the import with the id of 2). The import will run when this template is viewed in the browser.

### 3. You can directly call the URL:

```
http://yoursite.com/index.php?ACT=n&id=2
```

Where n is the action id assigned to DataGrab. This is displayed on the module’s homepage.

###


# Importing with cron

New method of importing using ExpressionEngine's CLI

Starting with DataGrab 4.2, this is the preferred and supported way to import data with cron.

If you are not familiar with the ExpressionEngine CLI tool you should start with [their documenation](https://docs.expressionengine.com/latest/cli/intro.html).

```bash
php system/ee/eecli.php import:run --id=2
```

When the command is executed it will read from your import file, put items into the queue (produce), then enact on that queue (consume) with a single command. This single command is usually fine for small imports where you've configured your import limit to be the same or greater than the number of entries in your import file. The limit is set to 50 by default. If you're having issues with importing your entire dataset with this command you may need to run the producer and consumer commands separately (see "Importing large data sets" below).

An example of the output when running the `import:run` command:

```bash
/var/www/html$ php system/ee/eecli.php import:run --id=2
Starting: Simple Import Test... 
Queueing...
Consuming...

Worker Stopped
```

You can see the options and parameters by using the `-h` flag.

```bash
php system/ee/eecli.php import:run -h
```

If an import gets stuck or stops prematurely, or you need to reset it to a new import status run the following command.

```bash
php system/ee/eecli.php import:reset --id=2
```

## Importing large data sets

Verision 5 introduces 2 new arguments to the CLI commands.

```bash
php system/ee/eecli.php import:run --id=27 --producer
```

Using the producer flag with the  command will only read entries from your import file and put them into the queue (produce) where they will reside until enacted upon (consume). If you want to run your import once a day, then you should run the command with the producer flag once a day. Once the producer command is in place you will need something to enact upon the items that the producer puts into the queue. This is what the consumer flag is for.

```bash
php system/ee/eecli.php import:run --id=27 --consumer
```

Running this command will create a single worker to consume entries from the queue. If your import is configured with a "limit" of 50, then it will only import 50 entries then stop. This is indicated in the DataGrab import log file as "WORKER STOPPED". A consumer can also stop if it reaches the PHP script max execution time. If a consumer stops because it reached its limit, or timed out early, you'll need to run the `--consumer` command again. The best way to do this is to setup a crontab on a schedule to run the command every 1, 3, 5, or 30 minutes (or use [supervisord](http://supervisord.org)). Choose any interval that works for you. If the queue is empty and there is nothing to consume, then the consumer will start, find that there is nothing in the queue to enact on, then immediately stop. At the next interval, it will start another consumer, check the queue and if something exists it will enact upon it. Rinse and repeat.

If you want to run a single consumer that will import all items in the queue then set the limit to 0. Using a limit of 0 on a large import will likely run into server memory or request timeout limits, therefore it is only recommended to use a limit of 0 on smaller imports. If you set a limit and find that the import is not finishing, then you know that 0 is not a viable option for your import size and server settings, and you'll have to to define a limit value and run the consumer periodically with a cron.

```bash
php system/ee/eecli.php import:run --id=27 --consumer --limit=0
```

To setup a consumer to run every 5 minutes your cron entry will look similar to the following:

```bash
*/5 * * * *    php system/ee/eecli.php import:run --id=27 --consumer --limit=50
```

It is perfectly fine to configure the DataGrab consumer to execute every X minutes, even if there is nothing to import. If there is nothing in the queue, then it will simply abort and try again a few minutes later. To learn more about cron visit [cron.guru](https://cronitor.io/guides/linux-cron-jobs#cron-reference?utm_source=crontabguru\&utm_campaign=cron_reference).

Conclusion, your crontab might look like this:

```sh
# This will run once every morning at 5am to read your import file and fill up the queue
0 5 * * * php /var/www/mysite.com/system/ee/eecli.php import:run --id=27 --producer

# This will run every 5 minutes and will check and pull from the queue if anything exists
# it will only grab 50 entries to import, then 5 minutes later it will grab the next 50 etc
# If the queue is empty, this will run and do nothing, then run again in 5 minutes etc
*/5 * * * * php /var/www/mysite.com/system/ee/eecli.php import:run --id=27 --consumer --limit=50
```

{% hint style="info" %}
Help configuring crontab or supervisord is not included as part of DataGrab's support. [Adequate documentation](/datagrab/docs/automatic-imports/importing-with-cron) is available, and this generally requires direct access to the server.
{% endhint %}


# Endpoints

New in DataGrab 5.3 are Endpoints. DataGrab Imports are designed to seek out and pull data from an external file or remote URL. Endpoints are a way for DataGrab to receive incoming data from an external application. Usually this is referred to as a webhook. An event happens in an application and it triggers a webhook, which POSTs data to a URL you've configured. Now with DataGrab you can accept incoming data to create or update entries in ExpressionEngine.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FAa7qIKSJ1UWbgCz83N17%2Fendpoint-list.png?alt=media&amp;token=3d193a41-797f-4686-b2fe-99e6c2fbee0e" alt=""><figcaption></figcaption></figure>

Endpoints are always paired with a previously configured Import. Imports usually handle many entries at a time from a remote file. Endpoints can also handle many entries, but generally the payload will be much smaller, such as 1 entry at a time. Regardless of the number of entries, the format of the payload sent by the external application must match the format of an already configured import.

### To use endpoints:

1. Create an Import: Begin by creating an import using a static or remote JSON or XML file that matches the sender’s data format. Run the import through the DataGrab interface and verify it works as expected.
2. Create the Endpoint: Assign the import to a new endpoint. The sender will POST data to the Endpoint URL, including proper authentication.

### Endpoints require:

1. A configured JSON or XML-based import.
2. POST requests must include body content that matches the import’s format.
3. A [crontab process running](https://docs.boldminded.com/datagrab/docs/automatic-imports/importing-with-cron) with the `--consumer` flag to process data received by the endpoint. If you have multiple endpoints with different imports, each import requires its own dedicated consumer process.

{% hint style="warning" %}
That last step is very important. You must configure a crontab to consume the data from an endpoint. Otherwise it will sit in the queue and never become an ExpressionEngine entry.
{% endhint %}

### Creating an Endpoint

Creating an Endpoint is very easy. Give it a name and associate it to an existing Import. You must also provide some method of authentication. Endpoints can accept authentication via GET variables, or HTTP Headers. You can add more than 1 parameter, though 1 or 2 is all that is usually required. If the parameters do not match exactly what is sent the request will fail.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F69Fu5tDErRYvZqMhUPab%2Fendpoint-config.png?alt=media&amp;token=0ddfa1e0-6453-4c0a-90dd-ac977f32e9f0" alt=""><figcaption></figcaption></figure>

To test an endpoint you can use an application such as [Bruno](https://www.usebruno.com/), [Insomnia](https://insomnia.rest), or any other rest client. In these screenshots we're using Bruno. First start with defining the query parameters to the Endpoint  URL, which is generated after you create an Endpoint.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FgU1Vpi5W7hjdYtZLPKKI%2Fendpoint-bruno-1.png?alt=media&amp;token=4f3143f6-b1e2-448c-a1a4-3067364fb901" alt=""><figcaption></figcaption></figure>

Add the POST body content. This should match the value of a full import, except it will contain data for 1 entry.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FPrS7dDr26j8VG905ZMJ6%2Fendpoint-bruno-2.png?alt=media&amp;token=7b2b528d-ab80-45f3-9882-bcc515b86a85" alt=""><figcaption></figcaption></figure>

Most importantly, add your authentication values. You don't want to openly accept any data sent to your endpoint.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FxpatZ6NpwwPeGGgwD5zp%2Fendpoint-bruno-3.png?alt=media&amp;token=44dfc098-72ef-4a1d-9798-f61fd7fa05ca" alt=""><figcaption></figcaption></figure>

After submiting the test request from your preferred rest client you should see a row in your exp\_datagrab\_jobs table (if you are using the database queue driver, which is the default). If you do not see anything in this table, and you have the DataGrab module page open in a browser tab it is possible that DataGrab already picked up the job from the queue. In this case you should see the new entry in ExpressionEngine. If it still does not appear to be working, [troublehoot it](/datagrab/docs/troubleshooting) with the log file.


# Configuration Options

#### Url Title <a href="#urltitle" id="urltitle"></a>

DataGrab’s default behaviour is to generate an entry’s url title from the entry’s Title field (as it would if you were publishing an entry manually). If you need the entry to have a specific url title then you can specify which field of the data source to use.

Note, that URL Titles cannot be made entirely from numbers. If your titles are numeric, then you can use the Publish Page Customization options under the Channel Preferences to add a [URL Title Prefix](http://expressionengine.com/user_guide/cp/admin/channels/channel_edit_preferences.html#url-title-prefix).

#### Date <a href="#date" id="date"></a>

DataGrab will attempt to read your date in whatever format you provide. For best results use YYYY-MM-DD hh:mm:ss. Try to avoid ‘ambiguous’ formats, such as DD/MM/YY or MM/DD/YY.

#### Duplicate Entries <a href="#duplicateentries" id="duplicateentries"></a>

DataGrab can check whether entries already exists and if they do it can skip or attempt to update the existing entry.

To check for an existing or duplicate entry, DataGrab will test whether the potential new entry has a field value that matches an existing entry.&#x20;

For example, an RSS feed often has a `guid` field that uniquely identifies a feed element, or a store product usually has a stock id number.

If you save this value into a custom field, you can then, on subsequent imports, check this field for an existing entry and update it or ignore it as appropriate.

You can use the **Use this field to check for duplicates** setting to select which field to check. If you leave this field blank it will always add a new entry.

if you check the **Update existing entries** option then DataGrab will attempt to update the entry’s values from the new record. Otherwise, it will ignore it (ie, not update it, but not add a new entry).

#### Add a timestamp to this field <a href="#addatimestamptothisfield" id="addatimestamptothisfield"></a>

DataGrab can add a timestamp to a field that it adds or updates. This can help in tracking which entries have been updated or deleting old entries.

#### Delete old entries <a href="#deleteoldentries" id="deleteoldentries"></a>

If you check this, DataGrab will delete any entries in the channel that have not been added or updated by this import. This can be useful for maintaining an up-to-date product list, for example, but must be used with care.&#x20;

{% hint style="info" %}
If you enable this option, make sure you have the **Limit** parameter set to a value higher than the number of entries you are importing, or use the CLI commands with separate `--producer` and `--consumer` flags. If you are importing 50 entries with a single CLI `import:run` comand without separate `--producer` and `--consumer` flags, or importing through the ExpressionEngine control panel interface, and have the limit set to 10, it will not be able to properly record and delete entries. The deletion must occur at the very end of the import process. Setting a very high limit, or using separate `import:run --producer` and `import:run --consumer` commands ensures this happens. [Learn more about setting up cron based imports](/datagrab/docs/automatic-imports/importing-with-cron).
{% endhint %}

#### Default Author <a href="#defaultauthor" id="defaultauthor"></a>

This allows you to set the author of any new entries.

#### Author <a href="#author" id="author"></a>

You can also assign the author from a field in the data source. The **Author Field Value** tells the system what format the author is stored as (eg, email address, username) and **Author** specifies which data field it is in.

#### Status <a href="#status" id="status"></a>

The Status field allows you to specify a data source field to set the entry’s status or you can use Open, Closed or the [channel’s default](http://expressionengine.com/user_guide/cp/admin/channels/channel_edit_preferences.html#administrative-preferences).

#### Offset <a href="#offset" id="offset"></a>

Sometimes you need to adjust the time to take into account localization. The offset option allows you to add or subtract any number of seconds from the date/time values.

**Limit**

The number of entries to import in a single request. By default, this is set to 1 to prevent server timeouts when performing large imports. After the limit is reached, DataGrab will refresh the page and start importing the next batch of entries.


# Config File Options

If you're making a request to an external import file/url and it requires additional validation you can use the datagrab\_custom\_headers config option. In this example 6 is the import ID to assign these headers to.

```php
$config['datagrab_custom_headers'] = [
    6 => [
        'Content-Type: application/json',
        'Authorization: Bearer YOUR_ACCESS_TOKEN'
    ]
];
```

If you want to disable verifying peers when making a cURL request you can  set this to 'n'.

```php
$config['datagrab_verify_peer'] = 'n';
```

By default DataGrab will write to the `system/user/cache/DataGrab-import.log` file each time an import is executed, and erase the log contents from the previous import and replace it with the currently executing import. If you want to keep a backup of the log files you will want to enable log rotate, which will append a timestamp to the end of of the file. If you do this you may want to consider deleting old backups, which will [require a simple bash script](/datagrab/docs/troubleshooting/datagrab-import.log-rotation).

```php
$config['datagrab_rotate_log'] = 'y';
```

Starting with DataGrab 6.1, remote import files are fetched through a new URL guard that only allows `http` and `https` URLs and refuses to connect to hosts that resolve to private, loopback, or otherwise reserved IP addresses. This prevents a saved import from being pointed at internal infrastructure or cloud metadata endpoints. The following config options let you tune that behavior.

If you import from a local development domain (such as a `.ddev.site` or `.test` address) or from a trusted host on your internal network, those requests would normally be blocked because they resolve to private IPs. You can allow specific hosts through with the `datagrab_allowed_hosts` config option. It accepts an array of host names, and a leading `*.` will match any subdomain. Your site's own domain (from `base_url`) is always trusted automatically, so same-host imports keep working without any configuration.

{% code overflow="wrap" %}

```php
$config['datagrab_allowed_hosts'] = [
    'localhost',
    '*.ddev.site',
    '*.test',
];
```

{% endcode %}

If you'd rather not maintain a host list and simply want to allow requests to private and reserved addresses everywhere, you can disable the private-address block entirely by setting `datagrab_allow_private_urls` to `'y'`. The `datagrab_allowed_hosts` option above is the safer, more targeted choice and should be preferred on production sites.

{% code overflow="wrap" %}

```php
$config['datagrab_allow_private_urls'] = 'y';
```

{% endcode %}

If you fetch import files from a URL that issues redirects, DataGrab now follows each redirect manually and re-checks the destination against the same security rules before continuing. The number of redirects it will follow is capped, and you can adjust that limit with the `datagrab_max_redirects` config option. This only applies when redirect following is already enabled via `datagrab_follow_redirects`; the default is 5.

{% code overflow="wrap" %}

```php
$config['datagrab_max_redirects'] = 5;
```

{% endcode %}

If your import is run from outside the Control Panel using its `?ACT=` URL, the passkey can now be sent as an HTTP request header instead of as a query-string parameter, which keeps the secret out of server logs, browser history, and `Referer` headers. Send the passkey in the `X-Datagrab-Passkey` header; the existing `passkey=` query-string parameter continues to work for backwards compatibility.

<br>


# Increasing PHP memory limit

#### How to find the current memory limit

You can find the current PHP memory limit from the ExpressionEngine 2 Control Panel

Go to: **Tools > Utilities > PHP Info** and search for the line “memory\_limit”

￼![](https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FTtTdgNMtAsdltwHIskQ9%2Ftools-php-info.png?alt=media\&token=55af00c8-fa16-484c-9446-937e0f8bbe04)

<div align="center"><img src="https://brandnewbox.co.uk/images/uploads/memory_limit.png" alt=""></div>

the first value (in the “Local” column) will tell you the current memory limit

#### Increasing the memory limit

Depending on your web hosting, you may be able to increase your memory limits. Some hosts will not allow this, or will limit the maximum amount of memory you can set. I’ve found some hosts are willing to set quite high limits if you explain it will be for a one-off data import and that it can be lowered again afterwards.

Your web hosting service will be able to tell you the appropriate way to increase you memory limit, but here is a summary of the options.

**.htaccess**

This is my preferred method as it makes it easy to set values of a site-by-site basis, but will not work under all hosting configurations. In a .htaccess file in the root of your website, add the line:

```
php_value memory_limit 128M
```

This will give you a memory limit of 128M.&#x20;

**php.ini**

If you are able to edit the server’s php.ini file you can modify the the memory limit there. Look for the line:

```
memory_limit = 64M
```

and adjust the value accordingly. This will apply to all sites.

You may also be able to create/modify a php.ini in the website’s root folder.


# Import Types


# CSV

A quick walkthrough of a CSV import

#### Prepare the CSV file

![](https://brandnewbox.co.uk/images/DataGrab_an_introduction/media_1273161798072.png)

CSV ([Comma-separeated values](http://en.wikipedia.org/wiki/Comma-separated_values)) files can created manually using a text editor, or is a common export format for many other applications, especially spreadsheets.

####


# Importing into Grid or Matrix field

Importing data into a field that allows repeating data is a bit trickier using a CSV file but can be done if the data is formmated correctly

#### Data format

To import multiple rows of Grid (or Matrix) data, you need to have multiple rows of data in the CSV file. These rows should be identical and contain all the data for the entry you want to import, and also have a way of identifying that the rows all belong to the same entry. The Title field may work here but a unique id (such as a product or stock id is better).

A simple example is:

```
ID,TITLE,IMAGE,LABEL,DATE
1,Rhubarb,https://brandnewbox.co.uk/images/uploads/bnb1.png,"Image 1","2011-01-01"
1,Rhubarb,https://brandnewbox.co.uk/images/uploads/bnb2.png,"Image 2","2011-10-01"
2,Armadillo,https://brandnewbox.co.uk/images/uploads/bnb3.png,"Image 3","2011-01-01"
3,Bananas,https://brandnewbox.co.uk/images/uploads/bnb4.png,"Image 4","2011-01-02"
3,Bananas,https://brandnewbox.co.uk/images/uploads/bnb5.png,"Image 5","2011-01-03"
```

Here, each entry has some common data (ID, TITLE, DATE) and some data that changes for each row of the Grid/Matrix (IMAGE, LABEL).

#### DataGrab configuration

You can then set up the Grid/Matrix as normal. You probably want to delete any existing rows if you are doing an update. For a one-time import, you can ignore the Action to take when an entry is updated option.

We are also importing the unique ID field to the GUID custom field.

![DataGrab configuration](https://brandnewbox.co.uk/images/importing-data-into-a-grid-or-matrix-field-from-a-csv-file/datagrab-configuration.png)

#### DataGrab configuration 2

To let DataGrab know which CSV rows are related to which entry you need to set up the “Use this field to check for duplicates” option to a field (custom or Title). If DataGrab finds an entry that has a matching value in this field (in our case the GUID custom field) it will update and append the new Grid rows to the existing entry.

You also need to check the “Update existing entries” to get it to update (rather than overwrite or ignore).

![DataGrab configuration 2](https://brandnewbox.co.uk/images/importing-data-into-a-grid-or-matrix-field-from-a-csv-file/datagrab-configuration-2.png)


# JSON


# Example file

{% hint style="warning" %}
Ensure your JSON file is valid, otherwise the import or import configuration will fail. If you are unsure if it is a valid file, run it through a [validator](https://jsonlint.com).
{% endhint %}

This is just an example. You can name your properties anything you want. Just make sure each entry you want to import is a different object in an array. For best results, avoid deeply nested objects. Try to stick to 2-3 levels deep if you can.

```json
{
   "root": {
      "entry": [
         {
            "id": "A",
            "title": "Dummy 1",
            "date": "1973-12-11 13:00:00",
            "summary": "Adds a related field"
         },
         {
            "id": "B",
            "title": "Dummy 2",
            "date": "1973-11-04 23:59:00",
            "summary": "Adds another related field"
         }
      ]
   }
}
```

For this example, the configuration **path** will be

```
/root/entry
```

\ <br>


# WordPress

A brief introduction on how to use DataGrab to import data from a WordPress WXR XML file.

{% hint style="warning" %}
The Wordpress import type is currently undergoing improvements. When completed these out of date docs will be updated accordingly.
{% endhint %}

Before running an import from a Wordpress export file, it might be helpful to run it through the [wp:scrubber.](https://wpscrubber.com/)

#### Set up the import

Upload the WordPress export file to you webserver. In the example screenshots, I’ve uploaded the file theme-unit-test-data.xml to my site’s /feeds/wordpress folder (ie, <http://ee2.dev/feeds/wordpress/theme-unit-test-data.xml>)

Select the channel you wish to import to, and enter the URL to your WordPress file in the "Filename or URL" input. You will probably want to ‘tick’ the "Just import posts?" checkbox (unless you know you want to import other types of posts).

Click on "Check settings"

![Set up the import](http://media.screensteps.me/brandnewbox/2w6clc/set-up-the-import.png?1363260330)

#### Check Settings

This should display a quick summary of the fields available from the WordPress file.

Click on "Configure Import"

![Check Settings](http://media.screensteps.me/brandnewbox/2w6clc/check-settings.png?1363260332)

#### Configure Import (1/3)

The WordPress datatype will automatically configure the import as much as possible but there are a few fields you will need to set-up.

Most importtantly, you’ll need to decide which field to put the post content in. In the example, I’ve used the "Body" field by selecting the "Content…" option in the pull-down menu.

![Configure Import (1/3)](http://media.screensteps.me/brandnewbox/2w6clc/configure-import-13.png?1363260334)

#### Configure Import (2/3)

You may wish to import the categories. If so, decide which category group to add them to.

![Configure Import (2/3)](http://media.screensteps.me/brandnewbox/2w6clc/configure-import-23.png?1363260335)

#### Configure Import (3/3)

You can set up a default author for the posts, or you can try and get DataGrab to assign posts to existing members.

See: <http://brandnewbox.co.uk/support/details/assigning\\_authors\\_to\\_entries\\_with\\_datagrab> for more details.

Click on "Do import" to start the import process.

![Configure Import (3/3)](http://media.screensteps.me/brandnewbox/2w6clc/configure-import-33.png?1363260337)

#### Results

DataGrab should import the WordPress posts and comments, retaining the posts’ original dates and statuses where possible.

![Results](http://media.screensteps.me/brandnewbox/2w6clc/results.png?1363260342)


# XML


# Example file

DataGrab can handle most XML formats. Here’s some examples.

{% hint style="warning" %}
Valid XML files must have all ampersands, e.g. &, escaped, otherwise the import or import configuration will fail. If you are unsure if it is a valid file, run it through a [validator](https://www.w3schools.com/xml/xml_validator.asp).
{% endhint %}

#### An XML file

```
<root>
	<entry>
		<id>A</id>
		<title>Dummy 1</title>
		<date>1973-12-11 13:00:00</date>
		<summary>Adds a related field</summary>
	</entry>
	<entry>
		<id>B</id>
		<title>Dummy 2</title>
		<date>1973-11-04 23:59:00</date>
		<summary>Adds another related field</summary>
	</entry>
</root>
```

For this example, the path will be

```
/root/entry
```

#### An RSS feed

```
<?xml version="1.0"?>
<rss version="0.91">
  <channel>
    <title>Channel</title>
    <link>http://test.com/</link>
    <description>Description</description>
    <item>
      <title>Item #1</title>
      <description>Description #1</description>
      <link>http://test.com/1</link>
      <guid>1</guid>
    </item>
    <item>
      <title>Item #2</title>
      <description>Description #2</description>
      <link>http://test.com/2</link>
      <guid>2</guid>
    </item>
  </channel>
</rss>
```

The path for this file would be

```
/rss/channel/item
```

\ <br>


# Creating your own import type

DataGrab has the ability to import other kinds of data than default types that it comes bundled with (CSV, JSON, Wordpress, and XML).

To add a custom data type add add the folder "datagrab\_datatypes" in the addons directory at the same level as your "datagrab" add-on folder. For example:

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FmoEB9MmsVDbUX2VpSGhz%2Fdatagrab-custom-types.png?alt=media&amp;token=8c9d3530-b82a-4c79-a9ee-a1c5eb0cb54c" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
If you create a file with the same name as an existing file in the datagrab/datatypes directory, it will be loaded first, effectively giving you the ability to override core DataGrab functionality. With great power comes great responsibility. While we provide the ability to do this, we do not offer support for debugging or assisting creating these override files. You're on your own should you choose this path.
{% endhint %}

Your datatype class will need to extend the AbstractDataType class. At minimum it will need to implement the fetch() and next() methods.

```php
<?php

class Datagrab_twitter extends AbstractDataType
{
    public function fetch()
    {
    }

    public function next()
    {
    }
}
```

Now when you go to the DataGrab module you should have an extra option in the “Create new import” menu.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2F0clX17VB9skcsVkHry9V%2Fdatagrab-custom-types-import.png?alt=media&amp;token=7f560f01-11e0-4ffb-8591-d9b7cdd0978d" alt=""><figcaption></figcaption></figure>


# Field Types


# Assets

Currently the Assets fieldtype only supports assigning existing images in Assets to a field value through an import file. It does not upload an image from a remote URL or local file path.

When defining a file you can provide just the file name, or the file name prefixed with the file upload directory variable if you know it. When the import is executed Assets will see if a file of that name already exists, and if so it will assign the `file_id` value to the Assets field in the entry being imported. If the file does not exist no file will be added to the Assets field when the entry is imported.

```
<root>
    <entry>
        <title>Load a file from a upload folder</title>
        <files>
            <file>bnb5.png</file>
        </files>
    </entry>

    <entry>
        <title>Load a file from a URL</title>
        <files>
            <file>{filedir_1}newmoon.pdf</file>
        <files>
    </entry>
</root>
```


# Ansel

Importing into an Ansel field is basically the same as importing a native File field. The main difference is Ansel can import multiple images into the same field, thus it has a multi-file delimiter option.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FkYIdfKiuZpseztNJivO1%2Fansel-import-options.png?alt=media&amp;token=d1307758-4ec6-49fc-917b-2c9311497aa5" alt=""><figcaption></figcaption></figure>


# Bloqs

Below is an example XML import file of what it might look like to import data into a Bloqs field. The property key names do not have to match the field names, but it helps for clarity. You can also refer to the [Fluid](/datagrab/docs/field-types/fluid) example since both fields import complex data in a similar fashion.

```xml
<root>
    <entry>
        <title>Advanced Bloqs Import</title>
        <bloqs>
            <section>
                <section_heading>
                    This is Section 1
                </section_heading>
                <basic_content>
                    <heading>
                        Heading 1
                    </heading>
                    <summary>
                        ABC
                    </summary>
                </basic_content>
                <basic_content>
                    <heading>
                        Heading 2
                    </heading>
                    <summary>
                        XYZ
                    </summary>
                </basic_content>
            </section>
            <section>
                <section_heading>
                    This is Section 2
                </section_heading>
                <basic_content>
                    <heading>
                        Heading 3
                    </heading>
                    <summary>
                        123
                    </summary>
                </basic_content>
                <basic_content>
                    <heading>
                        Heading 4
                    </heading>
                    <summary>
                        456
                    </summary>
                </basic_content>
            </section>
        </bloqs>
    </entry>
</root>
```


# Calendar

The following is an example file that imports multiple entries/events.&#x20;

Note the last entry contains complex rules for repeating events. Due to the complexity of the frequency selector, the values must be a JSON object. We have tried to illustrate possible values for the objects, but for further assistance on all the possible values, we suggest reaching out to the Solspace support team. These values mimic the POST data when saving an event in an entry.&#x20;

If you are using the JSON import format (as in the example below), make sure to encode the values as a string (escape the quotes). If you are using the XML or CSV import formats you should not need to encode the string, but it must still be a JSON object. For example: `{"byday": ["SU", "TH"]}`

```json
[
  {
    "title": "Event #1",
    "startTime": "03-01-2025 4:00 PM",
    "endTime": "03-01-2025 5:00 PM"
  },
  {
    "title": "Event #2",
    "startTime": "01-01-2026",
    "endTime": "01-02-2026"
  },
  {
    "title": "Event #3",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "exclude": "01-02-2026, 01-03-2026, 01-05-2026"
  },
  {
    "title": "Event #4",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "repeats": true,
    "freq": "daily",
    "interval": 1,
    "until": "01-01-2027",
    "exclude": "01-02-2026, 01-03-2026, 01-05-2026"
  },
  {
    "title": "Event #5",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "repeats": true,
    "freq": "dates",
    "interval": 1,
    "until": "01-01-2027",
    "include": "05-15-2026, 05-16-2026, 05-17-2026"
  },
  {
    "title": "Event #7",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "repeats": true,
    "freq": "weekly",
    "weekly": "{\"byday\": [\"FR\", \"SA\"]}",
  },
  {
    "title": "Event #8",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "repeats": true,
    "freq": "monthly",
    "monthly": "{\"bymonthdayorbyday\": \"bymonthday\", \"bydayinterval\": \"1\"}",
  },
  {
    "title": "Event #8",
    "startTime": "01-01-2026",
    "endTime": "01-25-2026",
    "allDay": true,
    "repeats": true,
    "freq": "yearly",
    "yearly": "{\"bydayinterval\": \"1\"}"
  }
]

```


# Channel Images

DataGrab does not officially support Channel Images

However, you can try this 3rd party fieldtype file. Your mileage may vary, and BoldMinded does not provide support for this package.

<https://github.com/dougblackjr/datagrab_channelimages>


# Date

Coming soon...


# File

DataGrab allows you to import files and images into the ExpressionEngine File field.

DataGrab allows you to import files and images into the ExpressionEngine File field. File fields within [Grid](http://ellislab.com/expressionengine/user-guide/modules/channel/grid.html) fields are also supported.

{% hint style="warning" %}
If you're importing multiple files per entry, then the import process will be a little slower than normal. Importing files is fairly intensive as there are a lot of checks and operations to perform.
{% endhint %}

#### Prerequisites <a href="#pre-requisites" id="pre-requisites"></a>

You will need to have set up a [File Upload area](http://ellislab.com/expressionengine/user-guide/cp/content/files/file_upload_preferences.html#create-edit-upload-destination).

You will also need to ensure that your PHP is configured to allow plenty of memory and not to time out.

#### Configuration options <a href="#configuration-options" id="configuration-options"></a>

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FR9AAbz8leJldjhUG6Ctp%2Ffile-import-options.png?alt=media&amp;token=a910b8df-1c9b-435f-9946-2e3e6b654ab7" alt=""><figcaption></figcaption></figure>

The first pulldown menu selects the data source field that contains the file/image.

The second field allows you to choose a default upload folder.

The third field selects whether DataGrab should try and fetch images/files from specified URLs.

If fetching files from a remote url, you can optionally automatically create the same sub-directory path when importing the file. This will only work if using ExpressionEngine 7.x with the sub-folder option enabled. This can be handy when importing from a Wordpress site, and you want to import files from the `wp-content/uploads` directory, and keep the dated directory structure intact in ExpressionEngine.

#### Field formatting options <a href="#field-formatting-options" id="field-formatting-options"></a>

DataGrab can accept data in 3 formats:

`https://mysite.com/images/uploads/filename.png`

This will tell DataGrab to use the file filename.png stored in the file upload area with an id of 1. It won’t check to see if the file exists - this lets you do the import and then copy the file into place afterwards. Note, if you do copy the files manually you will probably need to [resynchronise the folder](http://ellislab.com/expressionengine/user-guide/cp/content/files/sync_files.html) afterwards.

`filename.png`

If you do not specify a file upload folder, DataGrab will use the default folder select in the configuration options.

`http://mysite.com/image/filename.png`

If you specifiy a URL and set the "Fetch files from urls:" option to "Yes" then DataGrab will attempt to read the file from the specified URL and copy it to the default upload folder you specified in the configuration options. It will also create any additional sizes that you have specified for that folder and resync the folder.

#### Example file <a href="#example-file" id="example-file"></a>

```
<root>

    <entry>
        <title>Load a file from a upload folder</title>
        <files>
            <file>https://mysite.com/images/uploads/bnb5.png</file>
        </files>
    </entry>

    <entry>
        <title>Load a file from a URL</title>
        <files>
            <file>http://ee2.dev/images/newmoon.jpg</file>
        <files>
    </entry>

    <entry>
        <title>Load a file from its filename</title>
        <files>
            <file>es.jpg</file>
        </files>
    </entry>
    
    <entry>
        <title>Assign a file already in the EE File Manager and you know the ID.</title>
        <files>
            <file>{file:123:url}</file>
        </files>
    </entry>
    
    <entry>
        <title>If using the File Manager in compatibility mode, assign a file already in the EE File Manager and you know the name.</title>
        <files>
            <file>{filedir_123}some-file.jpg</file>
        </files>
    </entry>

</root>
```


# File Grid

Coming soon...


# Fluid

DataGrab includes basic support Fluid Field support.

Below is an example JSON import file of what it might look like to import data into a Fluid field. The property key names do not have to match the field names, but it helps for clarity.

**Example file**

```json
{
  "root": {
    "entry": [
      {
        "title": "Fluid Test Entry 1 JSON",
        "fluid_field": {
          "summary": [
            "This is some summary text for entry #1",
            "This is more summary text, the 4th node, for entry #1"
          ],
          "body": [
            "This is some of the longer body text for entry #1",
            "This is some of the longer body text for entry #1 body field 2"
          ],
          "file_basic": "https://media.jamf.com/images/solutions/app-mashup.jpg?q=80&w=1600",
          "fluid_group_1": [
            { "body": "A" },
            { "body": "B" },
            { "body": "C" },
            { "body": "D" }
          ],
          "related": "Luke Skywalker,Darth Vader"
        }
      },
      {
        "title": "Fluid Test Entry 2 JSON",
        "fluid_field": {
          "summary": [
            "This is some summary text for entry #2",
            "This is some summary text for entry #2 summary field 2"
          ],
          "body": [
            "This is some of the longer body text for entry #2"
          ],
          "related": "Han Solo"
        }
      },
      {
        "title": "Fluid Test Entry 3 JSON",
        "fluid_field": {
          "summary": [
            "This is some summary text for entry #3 in Group 1",
            "This is some summary text for entry #3 summary field 2 in Group 1"
          ],
          "body": [
            "This is some of the longer body text for entry #3 in Group 1"
          ],
          "fluid_group_2": [
            {
              "url": "https://google.com",
              "number": 20
            },
            {
              "url": "https://apple.com",
              "number": 100
            }
          ],
          "fluid_group_1": [
            {
              "body": "This is another body field in entry #3 in second Group 1"
            }
          ]
        }
      },
      {
        "title": "Fluid Test Entry 4 (with sub-Grid) JSON",
        "fluid_field": {
          "summary": [
            "This is some summary text for entry #3 in Group 1",
            "This is some summary text for entry #3 summary field 2 in Group 1"
          ],
          "fluid_group_3": [
            {
              "grid": [
                {
                  "related": "Han Solo",
                  "date": "2026/01/01",
                  "desc": "ABC"
                }
              ]
            },
            {
              "grid": [
                {
                  "related": "Han Solo, Luke Skywalker",
                  "date": "2026/08/25",
                  "desc": "This is some text"
                },
                {
                  "related": "Darth Vader",
                  "date": "2026/08/26",
                  "desc": "This is some more text in row 2"
                }
              ]
            }
          ],
          "body": [
            "This is some of the longer body text for entry #3 in Group 1"
          ]
        }
      }
    ]
  }
}

```


# Grid

{% hint style="info" %}
If you're using Publisher, and the Publisher DataGrab Connector, unfortunately Relationship fields *inside* of a Grid field do not import. We have not been able to find a working solution and apologize for the inconvenience.
{% endhint %}

#### Configuration options

![Grid configuration](https://brandnewbox.co.uk/images/uploads/datagrab-grid-field-config.png)

You should get a field for each column of the Grid. The example above has 2 columns: Name and File

Select the datasource field that you want to import into each column. If your data source has repeated data (eg, see the example file below), DataGrab will try and be clever enough to work this out and import one row for each set of data.

The final option “Action to take when an entry is updated” allows you to select which option DataGrab should take when import is re-run on an existing entry.

`Delete all existing rows` will empty the Grid and re-fill it with new values from the data source.

`Keep existing rows` will keep the existing entry’s data and add any new rows to the end.

You can also get DataGrab to try and update a row of the Grid if a value is the same.

#### Example file

```
<root>

	<entry>
		<id>F1</id>
		<title>File entry #1</title>
		<summary>Load a file from a upload folder</summary>
		<files>
			<file>https://mysite.com/images/uploads/bnb5.png</file>
			<label>BNB5</label>
		</files>
		<files>
			<file>https://mysite.com/images/uploads/bnb3.png</file>
			<label>BNB3</label>
		</files>
		<status>open</status>
	</entry>

	<entry>
		<id>F2</id>
		<title>File entry #2</title>
		<summary>Load a file from a URL</summary>
		<files>
			<file>https://mysite.com/images/newmoon.jpg</file>
			<label>New Moon</label>
		<files>
		</files>
			<file>https://mysite.com/images/es.jpg</file>
			<label>Elliott Smith</label>
		<files>
		</files>
			<file>https://mysite.com/images/basement.jpg</file>
			<label>From a Basement on the Hill</label>
		</files>
		<status>closed</status>
	</entry>

	<entry>
		<id>F3</id>
		<title>File entry #3</title>
		<summary>Load a file from its filename</summary>
		<files>
			<file>es.jpg</file>
			<label>Elliott Smith</label>
		</files>
		<status>Pending</status>
	</entry>
```


# Low Events

Coming soon...


# Relationships

{% hint style="info" %}
If you're using Publisher, and the Publisher DataGrab Connector, unfortunately Relationship fields *inside* of a Grid field do not import. We have not been able to find a working solution and apologize for the inconvenience.
{% endhint %}

In order for imports into a relationship field to work, it will need to be able to find existing entries to relate to, meaning importing relationships is usually a 2 step process. This usually means configuring 2 different imports with 2 different files. The first import contains all of the entries you're expecting to be assigned to a Relationship field, and the 2nd import would contain the configuration for the Relationship field itself to create the assignments.

When importing multiple relationships into a single Relationship field, you must use a single XML node or JSON property to assign all the relationships, for example:

```xml
<entries>
  <entry>
    <related>Entry One, Entry Two</related>
  </entry>
  ...
</entries>
```

The following formats will not work:

```xml
<entries>
  <entry>
    <related>
      <title>Entry One</title>
    </related>
    <related>
      <title>Entry Two</title>
    </related>
  </entry>
  ...
</entries>

<entries>
  <entry>
    <related>
      <title>Entry One</title>
      <title>Entry Two</title>
    </related>
  </entry>
  ...
</entries>
```

There are multiple characters you can choose as the separator.

<figure><img src="https://3476660910-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ly1eNBG1JvS5rivVtg3%2Fuploads%2FIi1Z1JxLlvkUGFBOixkW%2Fdatagrab-relationship-config-2.png?alt=media&amp;token=63b07535-1a98-4910-9b3f-1e7bd0ee148f" alt=""><figcaption></figcaption></figure>

The example above illustrates importing by title, but you can also use IDs if you know what they are. For example if `3|10|23` is the value of one of the CSV columns, JSON or XML nodes, and you mapped that field to a Relationship field in ExpressionEngine, then the imported entry will be assigned 3, 10, and 23 assigned as a relationship.




---

[Next Page](/llms-full.txt/1)

