Vector Transfer guide

Try a local transfer

Choose your platform and run these commands in an empty directory. The binary includes the dashboard, a 1,000-record sample source and a local JSON destination. No Go installation, account or database credentials are required.

curl -fSLO https://transfer.polign.com/downloads/vtransfer-darwin-arm64 &&
curl -fSLO https://transfer.polign.com/downloads/SHA256SUMS &&
shasum -a 256 --ignore-missing --check SHA256SUMS &&
chmod +x vtransfer-darwin-arm64 &&
./vtransfer-darwin-arm64 serve -demo -data ./transfer-demo

Open http://127.0.0.1:23005, choose New transfer, then Start transfer. The source, destination and four-dimensional vector setting are prefilled. When it succeeds, inspect transfer-demo/demo-vectors/: it contains 1,000 JSON files with IDs, vector values and metadata.

The demo runs only on loopback and makes no cloud database calls. Stop with Ctrl+C. Rerun the same command to reopen the saved history; use the same data folder to retain checkpoints. The This server execution option means the process you just started on your machine.

Run a worker in your infrastructure

A customer worker is a program on your Mac or Linux machine that reads the source and writes the destination. It connects outbound to the Polign control plane over HTTPS. Database credentials, vectors, record metadata and source cursors stay on your worker and databases. The dashboard receives worker identity, connection names and capabilities, approved source/destination pairs, job settings, status, counts and audit information.

1. Sign in, then open Set up a worker. Enter a worker name, source and destination settings, and vector dimension.
2. Download and review the generated setup script. Settings remain in the browser until downloaded; only the worker name is submitted when registering it.
3. Run the script on a machine that can reach both databases. It verifies the worker download, enrolls the worker, and prompts for database credentials locally. S3 Vectors can use the machine’s AWS workload identity.
4. Keep the setup page open. Once the worker is online, choose Create transfer, review the prefilled settings and choose Start transfer.

The enrollment token expires after 10 minutes. Delete the setup script after enrollment. The generated script does not save database credentials; supply them at the prompt or through your process environment. Use a supervisor for long-running workers.

Worker identity and checkpoints are stored under ~/.polign-transfer/WORKER_ID/. Preserve that folder and restart with its run-worker.sh script. A worker runs one job at a time and only accepts its locally approved connection pairs and dimensions. Jobs are not automatically moved to another worker. Back up the state directory; a missing or stale checkpoint is rejected rather than silently starting over.

Run on Polign

In New transfer, select Polign hosted. Create source and destination connections, enter database credentials, match the vector dimension and submit. Endpoints must be publicly reachable with verified TLS; use your own worker for private network endpoints.

In this mode, vectors and record metadata pass through the Polign server. Saved credentials are encrypted at rest with AES-256-GCM and decrypted by the service to connect to the databases. Polign’s runtime and administrators are trusted with those credentials and data. Hosted checkpoints are saved on the server.

To rotate a saved credential, open the job’s Source credentials or Destination credentials, save the new value, then resume. A database or resource change requires a new connection and job. Temporary AWS credentials must be replaced when they expire.

Run the whole server yourself

The same downloadable binary can run the dashboard and transfer engine in your infrastructure. Create a connection file such as this Qdrant-to-pgvector example, prepare the databases, then inject the named credential environment variables:

{
  "connections": {
    "qdrant-source": {
      "kind": "qdrant", "endpoint": "https://YOUR-QDRANT-HOST",
      "collection": "documents", "api_key_env": "QDRANT_READ_KEY", "read_only": true
    },
    "pgvector-sink": {
      "kind": "pgvector", "endpoint": "postgresql://YOUR-POSTGRES-HOST:5432",
      "database": "documents", "collection": "embeddings",
      "id_field": "id", "vector_field": "vector", "metadata_field": "metadata",
      "username_env": "PGVECTOR_USER", "password_env": "PGVECTOR_PASSWORD", "write_only": true
    }
  }
}

Save it as connections.json. Using the binary you downloaded, replace the demo flags with serve -config connections.json -data ./transfer-data. The server listens on 127.0.0.1:23005 by default. In this mode, database data and credentials stay with your own server; a Polign account is not required.

Operator mode shares jobs and connections among anyone with access to that server. Before listening beyond loopback, configure VECTOR_TRANSFER_TOKEN and put the service behind HTTPS. Enter that token through Access token in the dashboard. The program uses your configured token; it does not generate or print one. Database credentials referenced by environment variables are loaded at startup.

Checkpoints, retries and ID collisions

When does a checkpoint advance?

Only after the destination acknowledges an entire batch. Temporary failures retry automatically. After a failure, fix the cause and choose Resume transfer. A crash after a write but before its checkpoint may replay that batch, so delivery is not exactly once.

What happens to an existing destination ID?

Transfers upsert using source IDs. Matching destination records are replaced; records with unrelated IDs are left in place. IDs are not automatically renamed to avoid collisions. Provider-specific ID formats and metadata rules still apply.

What does cancellation do?

Cancellation stops at an operation boundary. An in-flight write may finish, and completed writes are not rolled back. Resuming continues from the saved checkpoint.

Can the source change during a transfer?

Keep the source unchanged through execution, retries and resume. Preserve the original worker identity, state directory and resource settings. Avoid concurrent transfers writing the same destination IDs.

What does “succeeded” guarantee?

Every source page was acknowledged by the destination. It does not independently verify destination readback, index readiness or search visibility. Reported counts can include repeated IDs from providers such as Redis SCAN.

What is outside the transfer contract?

Sparse vectors, multiple vector fields, automatic schema creation, embedding generation, index definition copying and automatic conversion of incompatible metadata. Prepare the destination’s dimensions, ID format, distance metric and metadata schema first.

Database compatibility

All 15 connectors implement source reads and destination writes within these limits. This is not a claim that every provider pair, version or schema has been tested together.

Most provider checks use protocol fixtures. S3 Vectors → Polign was verified through the hosted dashboard with a customer worker, 1,200 synthetic records, worker crash/restart, cancel/resume and destination point-read comparison. Polign, pgvector and FAISS integrations have also been tested locally. This is functional verification, not a throughput benchmark; validate your own database version and schema before migration.

Use resource-scoped credentials. Some destinations also require read permission to implement replacement semantics. Application read-only/write-only settings do not replace database permissions.