> For the complete documentation index, see [llms.txt](https://developer.celigo.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.celigo.com/cli/commands/datasets.md).

# datasets

Choose which tables or objects a sync replicates, and how their records land in the destination. Each dataset selects one source table or object and sets an ingestion mode: `append`, `replace`, or `merge`. A dataset exists only as a child of a sync, so most subcommands require `--sync`.

The subcommands form one workflow. `available` lists what a connection could replicate. `list` shows what the sync replicates today. `upsert` changes that selection. `fields` inspects the columns of a single table or object.

**REST API**: [Syncs](https://developer.celigo.com/api/api-reference/data-ingestion/syncs)

```
celigo datasets <subcommand> [args] [flags]
```

Supports all [global flags](/cli/getting-started/global-flags.md).

***

## Subcommands

| Subcommand                            | Purpose                                                                                                         |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `list`                                | List the datasets configured on a sync — the tables and objects it replicates.                                  |
| `get <datasetId>`                     | Fetch one dataset, including its column selections (`dataElements`).                                            |
| `upsert`                              | Batch create-or-update a sync's datasets from a JSON array (`--file <path>` or stdin).                          |
| `available <connectionId>`            | List every table or object a connection exposes for syncing, plus its exports usable as export-backed datasets. |
| `fields <connectionId> <datasetName>` | Read column-level detail for one table or object: data types, constraints, and delta cursor candidates.         |

***

## `celigo datasets list`

List the datasets configured on a sync. Results are sorted by name, and the CLI follows the response's `Link` cursor to collect every page.

**Signature**

```bash
celigo datasets list --sync <syncId>
```

**Arguments**

None.

**Flags**

| Flag              | Type   | Default | Description                                |
| ----------------- | ------ | ------- | ------------------------------------------ |
| `--sync <syncId>` | string | —       | **Required.** Sync whose datasets to list. |

Default table columns: `_id`, `name`, `externalId`, `enable`, `ingestionMode`, `userActionRequired`.

**Example**

```bash
celigo datasets list --sync 5f83a9b2c7d3e8f1a2b3c4d5 --format table
```

**Corresponds to**: [`GET /v1/syncs/{_syncId}/datasets`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `listDatasetsForSync`)

***

## `celigo datasets get <datasetId>`

Fetch one dataset by ID, including its per-column selections in `dataElements`.

**Signature**

```bash
celigo datasets get <datasetId> --sync <syncId>
```

**Arguments**

| Argument      | Type   | Required | Description                                  |
| ------------- | ------ | -------- | -------------------------------------------- |
| `<datasetId>` | string | Yes      | Dataset ID (the `_id` from `datasets list`). |

**Flags**

| Flag              | Type   | Default | Description                                                                          |
| ----------------- | ------ | ------- | ------------------------------------------------------------------------------------ |
| `--sync <syncId>` | string | —       | **Required.** Sync the dataset belongs to. Datasets have no unscoped by-ID endpoint. |

**Example**

```bash
celigo datasets get 6a559b547885d8f93921c2ba --sync 5f83a9b2c7d3e8f1a2b3c4d5 --jq '.dataElements[] | select(.enable)'
```

**Corresponds to**: [`GET /v1/syncs/{_syncId}/datasets/{_id}`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `getDataset`)

***

## `celigo datasets upsert`

Create and update a sync's datasets in one batch `PUT`. Read the body from a file with `-f, --file`, or pipe it on stdin.

The batch is all-or-nothing: one invalid entry rejects the whole request. Datasets you leave out of the array keep their saved configuration. `upsert` merges — it never deletes.

**Signature**

```bash
celigo datasets upsert --sync <syncId> --file <path>
celigo datasets upsert --sync <syncId> < datasets.json     # or pipe on stdin
```

**Arguments**

None.

**Flags**

| Flag                | Type   | Default | Description                                                                    |
| ------------------- | ------ | ------- | ------------------------------------------------------------------------------ |
| `--sync <syncId>`   | string | —       | **Required.** Sync to upsert datasets on.                                      |
| `-f, --file <path>` | string | —       | Read the JSON body from a file instead of stdin (`--file -` also means stdin). |

**Request body**

A JSON **array** (not an object) of dataset entries, matching the `PUT /v1/syncs/{_syncId}/datasets` request schema. Each entry carries one identifier — `externalId` to create, `_id` to update:

| Field                    | Purpose                                                                                                                                                                                |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_id`                    | Updates the saved dataset with that ID. Omit when creating.                                                                                                                            |
| `externalId`             | Creates a dataset for that source table or object — the `name` from `datasets available` — or the export's ID for an export-backed dataset. Omit when updating and send `_id` instead. |
| `name`                   | Display name of the dataset.                                                                                                                                                           |
| `enable`                 | Whether the sync replicates the dataset on each run.                                                                                                                                   |
| `ingestionMode`          | `append`, `replace`, or `merge`. Required whenever `enable` is `true`.                                                                                                                 |
| `exportProperties`       | Extraction behavior — `type: delta` or `all`, plus `delta.dateField` for the cursor. Source-table datasets only.                                                                       |
| `isExport` / `tableName` | Mark an export-backed dataset and name its destination table. `tableName` must be unique across the sync's datasets.                                                                   |
| `dataElements`           | Per-column selections. Omit to replicate every column.                                                                                                                                 |
| `driftPolicy`            | Per-dataset override of the sync's schema drift policy.                                                                                                                                |

One entry creating a dataset, one entry disabling a saved one:

```json
[
  {
    "name": "Account",
    "externalId": "Account",
    "enable": true,
    "ingestionMode": "merge",
    "exportProperties": { "type": "delta", "delta": { "dateField": "LastModifiedDate" } },
    "dataElements": [
      { "name": "Id", "enable": true, "isPrimaryKey": true },
      { "name": "Name", "enable": true }
    ]
  },
  { "_id": "6a559b547885d8f93921c2ba", "enable": false }
]
```

**Example**

```bash
celigo datasets upsert --sync 5f83a9b2c7d3e8f1a2b3c4d5 --file ./datasets.json
```

**Corresponds to**: [`PUT /v1/syncs/{_syncId}/datasets`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `upsertDatasetsForSync`)

***

## `celigo datasets available <connectionId>`

List every table or object the connection's application exposes for syncing. The listing also includes the connection's exports, which can back export-backed datasets. This is the whole candidate pool, not a list of what remains unselected.

The API returns two catalogs, and the CLI merges them into one list. A `_catalog` column tags each row: `dataset` for a source table or object, `export` for an export resource.

**Signature**

```bash
celigo datasets available <connectionId> [--sync <syncId>] [--type <datasets|exports|all>] [--refresh]
```

**Arguments**

| Argument         | Type   | Required | Description                                 |
| ---------------- | ------ | -------- | ------------------------------------------- |
| `<connectionId>` | string | Yes      | Source connection to read the catalog from. |

**Flags**

| Flag              | Type    | Default | Description                                                                                                                                                                  |
| ----------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--sync <syncId>` | string  | —       | Merge the catalog with this sync's saved dataset selections, so saved and unsaved entries appear side by side. Saved entries carry an `_id`.                                 |
| `--type <type>`   | string  | `all`   | Restrict the catalog to `datasets` (source tables/objects), `exports` (export resources), or `all`. Sent as `type`; an unknown value is rejected locally before any request. |
| `--refresh`       | boolean | `false` | Bypass the platform's cached catalog and re-read it from the source application. Sent as `refreshCache=true`.                                                                |

**Example**

```bash
# Everything the connection could replicate, annotated with what this sync already selected
celigo datasets available 5f8d43a1b9e5a80011a35f2e --sync 5f83a9b2c7d3e8f1a2b3c4d5 --format table

# Only the source tables, re-read live from the application after a schema change
celigo datasets available 5f8d43a1b9e5a80011a35f2e --type datasets --refresh
```

**Corresponds to**: [`GET /v1/di/metadata/connections/{_connectionId}/datasets`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `listConnectionDatasets`), or with `--sync`, [`GET /v1/di/metadata/sync/{_syncId}/connections/{_connectionId}/datasets`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `listSyncConnectionDatasets`)

***

## `celigo datasets fields <connectionId> <datasetName>`

Read column-level detail for one table or object on a connection. Each column reports its data type, length, precision, and constraints. The response also carries `deltaFields`: the columns usable as the delta export cursor (`exportProperties.delta.dateField`). Use `fields` to build a `dataElements` selection before calling `upsert`.

The API wraps its answer in a `{ "dataset": … }` envelope. The CLI prints the inner object.

**Signature**

```bash
celigo datasets fields <connectionId> <datasetName> [--sync <syncId>] [--is-export] [--refresh] [--record-type <type>] [--display-name <name>]
```

**Arguments**

| Argument         | Type   | Required | Description                                                                                                                                                                                                         |
| ---------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<connectionId>` | string | Yes      | Source connection to read the columns from.                                                                                                                                                                         |
| `<datasetName>`  | string | Yes      | Table or object name from `datasets available`. With `--is-export`, supply an export-backed identifier instead — see the gotchas below. The CLI URL-encodes the value, so names containing spaces are safe to pass. |

**Flags**

| Flag                    | Type    | Default | Description                                                                                                                                |
| ----------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `--sync <syncId>`       | string  | —       | Merge the source columns with this sync's saved dataset state and per-column selections.                                                   |
| `--is-export`           | boolean | `false` | Treat `<datasetName>` as an export-backed entry and read the export's record structure instead of a source table. Sent as `isExport=true`. |
| `--refresh`             | boolean | `false` | Bypass the cached column list and re-read it from the source application. Sent as `refreshCache=true`.                                     |
| `--record-type <type>`  | string  | —       | NetSuite record type backing a saved-search dataset. Sent as `recordType`; omit for other sources.                                         |
| `--display-name <name>` | string  | —       | NetSuite saved-search display name. Sent as `displayName`, and only meaningful alongside `--record-type`.                                  |

**Example**

```bash
# Column catalog for one Salesforce object, with the delta-cursor candidates
celigo datasets fields 5f8d43a1b9e5a80011a35f2e Account --jq '{deltaFields, columns: [.dataElements[].name]}'

# The same object merged with what this sync has already selected
celigo datasets fields 5f8d43a1b9e5a80011a35f2e Account --sync 5f83a9b2c7d3e8f1a2b3c4d5

# A NetSuite saved-search dataset
celigo datasets fields 5f8d43a1b9e5a80011a35f2e "Sales Orders" \
  --record-type salesorder --display-name "Sales Orders"
```

**Corresponds to**: [`GET /v1/di/metadata/connections/{_connectionId}/datasets/{datasetName}/details`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `getConnectionDatasetDetails`), or with `--sync`, [`GET /v1/di/metadata/sync/{_syncId}/connections/{_connectionId}/datasets/{datasetName}/details`](https://developer.celigo.com/api/api-reference/data-ingestion/syncs) (operationId: `getSyncConnectionDatasetDetails`)

***

## Gotchas

* **`--sync` is required on `list`, `get`, and `upsert`.** Datasets exist only as children of a sync. There is no unscoped `/v1/datasets` collection and no by-ID endpoint that works without the sync. Omitting `--sync` fails locally, before any request is sent.
* **`upsert` merges; it never deletes.** Datasets missing from the array keep their saved configuration. Removing an entry does not unselect the table. To stop replicating one, send it with its `_id` and `enable: false`.
* **Every entry needs an identifier.** Send `externalId` (the source table or object name) to create a dataset, and `_id` to update one that already exists. An entry with neither rejects the whole batch with `400 Bad Request` and the error code `dataset_externalId_id_required`. To update, read the saved `_id` from `datasets list --sync <syncId>` and send that — it identifies the dataset unambiguously.
* **`dataElements` replaces the whole column selection.** The array you send becomes the dataset's complete selection. Omit it to replicate every column. Read the current selection first with `celigo datasets get <datasetId> --sync <syncId>`.
* **`enable: true` requires `ingestionMode`, and `merge` requires a primary key.** Enabling a dataset without an ingestion mode fails validation. So does `merge` with no enabled `isPrimaryKey` column.
* **`available` is the candidate pool, not a comparison.** It lists everything the connection exposes, whether or not the sync already replicates it. With `--sync`, only saved datasets carry an `_id` — that column separates saved from unsaved. The merged view omits `dataElements` to keep the payload small, so use `fields` to inspect columns.
* **`--refresh` costs a live round trip to the source application.** It re-reads the catalog or column list from the connected system instead of the platform cache. Expect a slower response, and a failure if the source is offline. Use it after a schema change in the source, not by default.
* **`fields` identifies export-backed datasets differently.** Pass `--is-export`, then supply the export's ID as `<datasetName>` for the connection-scoped lookup. When you also pass `--sync`, supply the saved dataset's `_id` instead.
* **`mismatchSyncConnection: true` means the export drifted.** The flag appears in `list` output when an export-backed dataset's export no longer uses the sync's source connection. Upserting that dataset fails with the error code `connection_mismatch`. Repoint the export at the sync's connection, or remove the dataset.
* **`upsert` returns no body.** The `PUT` succeeds with an empty payload, so the CLI prints a confirmation message instead of a resource. Read the saved state back with `datasets list --sync <syncId>` to confirm what landed.
* **Datasets live only inside a syncs-type integration.** An integration holds either flows or syncs, never both. Create the parent integration with `syncs` enabled before attaching datasets to a sync inside it.
* **`upsert` requires `full` mode.** It creates and reconfigures datasets, so it is gated like `create` and `update`. `list`, `get`, `available`, and `fields` work in `read` mode. See [Profiles & regions](/cli/getting-started/profiles.md#permission-modes).

## Related

* [syncs](/cli/commands/syncs.md) — the sync that owns these datasets, and where the source connection and destination are configured.
* [sync-jobs](/cli/commands/sync-jobs.md) — the runs a sync produces, showing whether the selected datasets replicated.
* [connections](/cli/commands/connections.md) — resolve the `<connectionId>` that `available` and `fields` read their catalogs from.
* [exports](/cli/commands/exports.md) — the exports that back export-backed datasets.