Upload a large dataset (CLI)
Upload thousands of files with the command-line client so that an interrupted upload resumes
Required permission: Write datasetsDEVELOPER · OPERATOR · ADMIN
Once you have more than a few hundred files, the command-line client geo-mlops-client works better than the browser. It gathers files into uncompressed ZIP bundles (16 MiB by default) and sends them as resumable pieces (chunks), so it does not hit the request size or response time limits of a front proxy, and an interrupted upload resumes.
First create the dataset on screen (Tutorial step 2), and find the dataset ID (ds-…) in the address bar or on the detail screen.
- Sign in. Enter the password when prompted, and have the client remember the tenant as well.
geo-mlops-client login you@example.com \ --base-url https://mlops.example.com --tenant DEMO - Upload a whole folder. The folder is scanned recursively, and hidden files are skipped.
geo-mlops-client datasets upload-archive ds-3263ed3c333b ~/data/helmet/ \ --skip-existing 2>&1 | tee -a ~/upload.log - When it finishes, a summary JSON is printed. On the dataset's Files (파일) tab on screen, check the count and Valid (유효) / Error (오류).
If the upload is interrupted
- If one piece fails, the command resends it by itself (3 times per piece).
- If the command still stops (a laptop going to sleep, for example), run the same command again, unchanged.
--skip-existingskips file names already in the dataset and carries on. - Progress logs go to standard error and the summary JSON goes to standard output, so you can pipe it straight into
| jq.
{ "dataset_id": "ds-…", "chunks": 2, "created": 85,
"skipped_by_server": 0, "skipped_existing": 0, "failed_chunk": 3 }Good to know
| Item | Details |
|---|---|
| Two commands | upload-archive (default, ZIP bundles + chunks) / upload (one or two files, one file = one request) |
| Bundle size | --chunk-size 16M (default). Most of the time you do not need to change it |
| File kinds | Decided by extension. .png/.jpg/… images, .json/.xml annotations, .ply point clouds, .csv/.tsv/.parquet tabular and time series, everything else is other |
| File names | Only the file name is kept, without folders. Same names in different folders overwrite each other (the command warns you in advance) |
| Only some extensions | geo-mlops-client datasets upload-archive ds-… ~/data/*.jpg ~/data/*.json |
| Temporary space | TMPDIR needs free space for one bundle. If it runs short, prefix the command with TMPDIR=/path/with/space |
| Tenant | In this order: --tenant → environment variable GEO_MLOPS_CLIENT_TENANT → the value remembered at sign-in |
| Server address | --base-url or GEO_MLOPS_CLIENT_BASE_URL (default http://localhost:10000) |
Installing the client and its other commands are covered in the Integration chapter.