> For the complete documentation index, see [llms.txt](https://developer.celigo.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.celigo.com/cli/commands/datasets.md).

# datasets

Choose which tables or objects a sync replicates, and how their records land in the destination. Each dataset selects one source table or object and sets an ingestion mode: `append`, `replace`, or `merge`. A dataset exists only as a child of a sync, so most subcommands require `--sync`.

The subcommands form one workflow. `available` lists what a connection could replicate. `list` shows what the sync replicates today. `upsert` changes that selection. `fields` inspects the columns of a single table or object.

**REST API**: [Syncs](https://developer.celigo.com/api/api-reference/syncs)

```
celigo datasets <subcommand> [args] [flags]
```

Supports all [global flags](/cli/getting-started/global-flags.md).

***

## Subcommands

| Subcommand                            | Purpose                                                                                                         |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `list`                                | List the datasets configured on a sync — the tables and objects it replicates.                                  |
| `get <datasetId>`                     | Fetch one dataset, including its column selections (`dataElements`).                                            |
| `upsert`                              | Batch create-or-update a sync's datasets from a JSON array (`--file <path>` or stdin).                          |
| `available <connectionId>`            | List every table or object a connection exposes for syncing, plus its exports usable as export-backed datasets. |
| `fields <connectionId> <datasetName>` | Read column-level detail for one table or object: data types, constraints, and delta cursor candidates.         |

***

## `celigo datasets list`

List the datasets configured on a sync. Results are sorted by name, and the CLI follows the response's `Link` cursor to collect every page.

**Signature**

```bash
celigo datasets list --sync <syncId>
```

**Arguments**

None.

**Flags**

| Flag              | Type   | Default | Description                                |
| ----------------- | ------ | ------- | ------------------------------------------ |
| `--sync <syncId>` | string | —       | **Required.** Sync whose datasets to list. |

Default table columns: `_id`, `name`, `externalId`, `enable`, `ingestionMode`, `userActionRequired`.

**Example**

```bash
celigo datasets list --sync 5f83a9b2c7d3e8f1a2b3c4d5 --format table
```

**Corresponds to**: [`GET /v1/syncs/{_syncId}/datasets`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `listDatasetsForSync`)

***

## `celigo datasets get <datasetId>`

Fetch one dataset by ID, including its per-column selections in `dataElements`.

**Signature**

```bash
celigo datasets get <datasetId> --sync <syncId>
```

**Arguments**

| Argument      | Type   | Required | Description                                  |
| ------------- | ------ | -------- | -------------------------------------------- |
| `<datasetId>` | string | Yes      | Dataset ID (the `_id` from `datasets list`). |

**Flags**

| Flag              | Type   | Default | Description                                                                          |
| ----------------- | ------ | ------- | ------------------------------------------------------------------------------------ |
| `--sync <syncId>` | string | —       | **Required.** Sync the dataset belongs to. Datasets have no unscoped by-ID endpoint. |

**Example**

```bash
celigo datasets get 6a559b547885d8f93921c2ba --sync 5f83a9b2c7d3e8f1a2b3c4d5 --jq '.dataElements[] | select(.enable)'
```

**Corresponds to**: [`GET /v1/syncs/{_syncId}/datasets/{_id}`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `getDataset`)

***

## `celigo datasets upsert`

Create and update a sync's datasets in one batch `PUT`. Read the body from a file with `-f, --file`, or pipe it on stdin.

The batch is all-or-nothing: one invalid entry rejects the whole request. Datasets you leave out of the array keep their saved configuration. `upsert` merges — it never deletes.

**Signature**

```bash
celigo datasets upsert --sync <syncId> --file <path>
celigo datasets upsert --sync <syncId> < datasets.json     # or pipe on stdin
```

**Arguments**

None.

**Flags**

| Flag                | Type   | Default | Description                                                                    |
| ------------------- | ------ | ------- | ------------------------------------------------------------------------------ |
| `--sync <syncId>`   | string | —       | **Required.** Sync to upsert datasets on.                                      |
| `-f, --file <path>` | string | —       | Read the JSON body from a file instead of stdin (`--file -` also means stdin). |

**Request body**

A JSON **array** (not an object) of dataset entries, matching the `PUT /v1/syncs/{_syncId}/datasets` request schema. Each entry carries one identifier — `externalId` to create, `_id` to update:

| Field                    | Purpose                                                                                                                                                                                |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_id`                    | Updates the saved dataset with that ID. Omit when creating.                                                                                                                            |
| `externalId`             | Creates a dataset for that source table or object — the `name` from `datasets available` — or the export's ID for an export-backed dataset. Omit when updating and send `_id` instead. |
| `name`                   | Display name of the dataset.                                                                                                                                                           |
| `enable`                 | Whether the sync replicates the dataset on each run.                                                                                                                                   |
| `ingestionMode`          | `append`, `replace`, or `merge`. Required whenever `enable` is `true`.                                                                                                                 |
| `exportProperties`       | Extraction behavior — `type: delta` or `all`, plus `delta.dateField` for the cursor. Source-table datasets only.                                                                       |
| `isExport` / `tableName` | Mark an export-backed dataset and name its destination table. `tableName` must be unique across the sync's datasets.                                                                   |
| `dataElements`           | Per-column selections. Omit to replicate every column.                                                                                                                                 |
| `driftPolicy`            | Per-dataset override of the sync's schema drift policy.                                                                                                                                |

One entry creating a dataset, one entry disabling a saved one:

```json
[
  {
    "name": "Account",
    "externalId": "Account",
    "enable": true,
    "ingestionMode": "merge",
    "exportProperties": { "type": "delta", "delta": { "dateField": "LastModifiedDate" } },
    "dataElements": [
      { "name": "Id", "enable": true, "isPrimaryKey": true },
      { "name": "Name", "enable": true }
    ]
  },
  { "_id": "6a559b547885d8f93921c2ba", "enable": false }
]
```

**Example**

```bash
celigo datasets upsert --sync 5f83a9b2c7d3e8f1a2b3c4d5 --file ./datasets.json
```

**Corresponds to**: [`PUT /v1/syncs/{_syncId}/datasets`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `upsertDatasetsForSync`)

***

## `celigo datasets available <connectionId>`

List every table or object the connection's application exposes for syncing. The listing also includes the connection's exports, which can back export-backed datasets. This is the whole candidate pool, not a list of what remains unselected.

The API returns two catalogs, and the CLI merges them into one list. A `_catalog` column tags each row: `dataset` for a source table or object, `export` for an export resource.

**Signature**

```bash
celigo datasets available <connectionId> [--sync <syncId>] [--type <datasets|exports|all>] [--refresh]
```

**Arguments**

| Argument         | Type   | Required | Description                                 |
| ---------------- | ------ | -------- | ------------------------------------------- |
| `<connectionId>` | string | Yes      | Source connection to read the catalog from. |

**Flags**

| Flag              | Type    | Default | Description                                                                                                                                                                  |
| ----------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--sync <syncId>` | string  | —       | Merge the catalog with this sync's saved dataset selections, so saved and unsaved entries appear side by side. Saved entries carry an `_id`.                                 |
| `--type <type>`   | string  | `all`   | Restrict the catalog to `datasets` (source tables/objects), `exports` (export resources), or `all`. Sent as `type`; an unknown value is rejected locally before any request. |
| `--refresh`       | boolean | `false` | Bypass the platform's cached catalog and re-read it from the source application. Sent as `refreshCache=true`.                                                                |

**Example**

```bash
# Everything the connection could replicate, annotated with what this sync already selected
celigo datasets available 5f8d43a1b9e5a80011a35f2e --sync 5f83a9b2c7d3e8f1a2b3c4d5 --format table

# Only the source tables, re-read live from the application after a schema change
celigo datasets available 5f8d43a1b9e5a80011a35f2e --type datasets --refresh
```

**Corresponds to**: [`GET /v1/di/metadata/connections/{_connectionId}/datasets`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `listConnectionDatasets`), or with `--sync`, [`GET /v1/di/metadata/sync/{_syncId}/connections/{_connectionId}/datasets`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `listSyncConnectionDatasets`)

***

## `celigo datasets fields <connectionId> <datasetName>`

Read column-level detail for one table or object on a connection. Each column reports its data type, length, precision, and constraints. The response also carries `deltaFields`: the columns usable as the delta export cursor (`exportProperties.delta.dateField`). Use `fields` to build a `dataElements` selection before calling `upsert`.

The API wraps its answer in a `{ "dataset": … }` envelope. The CLI prints the inner object.

**Signature**

```bash
celigo datasets fields <connectionId> <datasetName> [--sync <syncId>] [--is-export] [--refresh] [--record-type <type>] [--display-name <name>]
```

**Arguments**

| Argument         | Type   | Required | Description                                                                                                                                                                                                         |
| ---------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<connectionId>` | string | Yes      | Source connection to read the columns from.                                                                                                                                                                         |
| `<datasetName>`  | string | Yes      | Table or object name from `datasets available`. With `--is-export`, supply an export-backed identifier instead — see the gotchas below. The CLI URL-encodes the value, so names containing spaces are safe to pass. |

**Flags**

| Flag                    | Type    | Default | Description                                                                                                                                |
| ----------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `--sync <syncId>`       | string  | —       | Merge the source columns with this sync's saved dataset state and per-column selections.                                                   |
| `--is-export`           | boolean | `false` | Treat `<datasetName>` as an export-backed entry and read the export's record structure instead of a source table. Sent as `isExport=true`. |
| `--refresh`             | boolean | `false` | Bypass the cached column list and re-read it from the source application. Sent as `refreshCache=true`.                                     |
| `--record-type <type>`  | string  | —       | NetSuite record type backing a saved-search dataset. Sent as `recordType`; omit for other sources.                                         |
| `--display-name <name>` | string  | —       | NetSuite saved-search display name. Sent as `displayName`, and only meaningful alongside `--record-type`.                                  |

**Example**

```bash
# Column catalog for one Salesforce object, with the delta-cursor candidates
celigo datasets fields 5f8d43a1b9e5a80011a35f2e Account --jq '{deltaFields, columns: [.dataElements[].name]}'

# The same object merged with what this sync has already selected
celigo datasets fields 5f8d43a1b9e5a80011a35f2e Account --sync 5f83a9b2c7d3e8f1a2b3c4d5

# A NetSuite saved-search dataset
celigo datasets fields 5f8d43a1b9e5a80011a35f2e "Sales Orders" \
  --record-type salesorder --display-name "Sales Orders"
```

**Corresponds to**: [`GET /v1/di/metadata/connections/{_connectionId}/datasets/{datasetName}/details`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `getConnectionDatasetDetails`), or with `--sync`, [`GET /v1/di/metadata/sync/{_syncId}/connections/{_connectionId}/datasets/{datasetName}/details`](https://developer.celigo.com/api/api-reference/syncs) (operationId: `getSyncConnectionDatasetDetails`)

***

## Gotchas

* **`--sync` is required on `list`, `get`, and `upsert`.** Datasets exist only as children of a sync. There is no unscoped `/v1/datasets` collection and no by-ID endpoint that works without the sync. Omitting `--sync` fails locally, before any request is sent.
* **`upsert` merges; it never deletes.** Datasets missing from the array keep their saved configuration. Removing an entry does not unselect the table. To stop replicating one, send it with its `_id` and `enable: false`.
* **Every entry needs an identifier.** Send `externalId` (the source table or object name) to create a dataset, and `_id` to update one that already exists. An entry with neither rejects the whole batch with `400 Bad Request` and the error code `dataset_externalId_id_required`. To update, read the saved `_id` from `datasets list --sync <syncId>` and send that — it identifies the dataset unambiguously.
* **`dataElements` replaces the whole column selection.** The array you send becomes the dataset's complete selection. Omit it to replicate every column. Read the current selection first with `celigo datasets get <datasetId> --sync <syncId>`.
* **`enable: true` requires `ingestionMode`, and `merge` requires a primary key.** Enabling a dataset without an ingestion mode fails validation. So does `merge` with no enabled `isPrimaryKey` column.
* **`available` is the candidate pool, not a comparison.** It lists everything the connection exposes, whether or not the sync already replicates it. With `--sync`, only saved datasets carry an `_id` — that column separates saved from unsaved. The merged view omits `dataElements` to keep the payload small, so use `fields` to inspect columns.
* **`--refresh` costs a live round trip to the source application.** It re-reads the catalog or column list from the connected system instead of the platform cache. Expect a slower response, and a failure if the source is offline. Use it after a schema change in the source, not by default.
* **`fields` identifies export-backed datasets differently.** Pass `--is-export`, then supply the export's ID as `<datasetName>` for the connection-scoped lookup. When you also pass `--sync`, supply the saved dataset's `_id` instead.
* **`mismatchSyncConnection: true` means the export drifted.** The flag appears in `list` output when an export-backed dataset's export no longer uses the sync's source connection. Upserting that dataset fails with the error code `connection_mismatch`. Repoint the export at the sync's connection, or remove the dataset.
* **`upsert` returns no body.** The `PUT` succeeds with an empty payload, so the CLI prints a confirmation message instead of a resource. Read the saved state back with `datasets list --sync <syncId>` to confirm what landed.
* **Datasets live only inside a syncs-type integration.** An integration holds either flows or syncs, never both. Create the parent integration with `syncs` enabled before attaching datasets to a sync inside it.
* **`upsert` requires `full` mode.** It creates and reconfigures datasets, so it is gated like `create` and `update`. `list`, `get`, `available`, and `fields` work in `read` mode. See [Profiles & regions](/cli/getting-started/profiles.md#permission-modes).

## Related

* [syncs](/cli/commands/syncs.md) — the sync that owns these datasets, and where the source connection and destination are configured.
* [sync-jobs](/cli/commands/sync-jobs.md) — the runs a sync produces, showing whether the selected datasets replicated.
* [connections](/cli/commands/connections.md) — resolve the `<connectionId>` that `available` and `fields` read their catalogs from.
* [exports](/cli/commands/exports.md) — the exports that back export-backed datasets.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://developer.celigo.com/cli/commands/datasets.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.