Manual worker setup

A worker is a program you run on your own server to move embeddings and metadata between databases. Use it to keep database credentials and checkpoints in your infrastructure while you create and monitor transfers in the Polign dashboard. It connects to Polign over HTTPS and reports progress; vector data does not pass through the control plane.

Create connection generates a script that downloads, configures, enrolls, and starts your worker. Use the commands below for manual installation.

Download

Choose your platform. These commands download the worker, verify its checksum, and save it as vtransfer. Then enroll and configure it below.

Linux x64

Download vtransfer-linux-amd64

curl -fSLO https://transfer.polign.com/downloads/vtransfer-linux-amd64 &&
curl -fSLO https://transfer.polign.com/downloads/SHA256SUMS &&
sha256sum --ignore-missing --check SHA256SUMS &&
chmod 755 vtransfer-linux-amd64 &&
mv vtransfer-linux-amd64 vtransfer

Linux ARM64

Download vtransfer-linux-arm64

curl -fSLO https://transfer.polign.com/downloads/vtransfer-linux-arm64 &&
curl -fSLO https://transfer.polign.com/downloads/SHA256SUMS &&
sha256sum --ignore-missing --check SHA256SUMS &&
chmod 755 vtransfer-linux-arm64 &&
mv vtransfer-linux-arm64 vtransfer

macOS Apple Silicon

Download vtransfer-darwin-arm64

curl -fSLO https://transfer.polign.com/downloads/vtransfer-darwin-arm64 &&
curl -fSLO https://transfer.polign.com/downloads/SHA256SUMS &&
shasum -a 256 --ignore-missing --check SHA256SUMS &&
chmod 755 vtransfer-darwin-arm64 &&
mv vtransfer-darwin-arm64 vtransfer

Enroll

The generated setup script handles enrollment. For manual setup, send POST /v1/workers with {"name":"production"} using your authenticated session and CSRF token. Save the returned enrollment_token privately as enrollment.token on the worker machine. It expires in 10 minutes.

chmod 600 enrollment.token
./vtransfer worker enroll -url https://transfer.polign.com -data ./worker-data -token-file ./enrollment.token

Remove the token file after enrollment succeeds. Keep worker-data private and persistent. Use one process per worker identity.

Configure connections

Create connections.json locally. This example reads Pinecone and writes to a Polign collection:

{
  "connections": {
    "production-source": {
      "kind": "pinecone",
      "endpoint": "https://YOUR-PINECONE-INDEX-HOST",
      "namespace": "documents",
      "api_key_env": "SOURCE_API_KEY",
      "read_only": true
    },
    "polign-destination": {
      "kind": "polign",
      "endpoint": "https://YOUR-POLIGN-DATABASE",
      "collection": "documents",
      "api_key_env": "DESTINATION_API_KEY",
      "write_only": true
    }
  }
}

Inject the referenced environment variables through your secret manager or process supervisor. Use verified TLS and resource-scoped database permissions. Restart the worker after changing API keys.

Create worker.json to approve the connection pair and dimension:

{
  "connections_file": "connections.json",
  "pairs": [
    {"source": "production-source", "sink": "polign-destination", "dimension": 1536}
  ]
}

Run and submit

./vtransfer worker run -data ./worker-data -config ./worker.json

Wait for online in the dashboard. Choose New transfer, select your worker and approved connections, then submit. One worker runs one job at a time. Configure a supervisor to start it after a machine reboot.

Recovery

• Keep the source unchanged during transfers and resumes. Matching destination IDs are replaced.
• After a restart, reuse the same data directory and connection settings. Encrypt and back up that directory.
• For a failed job, fix the cause and choose Resume transfer. Missing checkpoints require restoring the original worker data.
• Connection loss stops execution. Cancellation or revocation cannot retract a database request already in flight. A crash can replay a batch.

The control plane can request only locally approved transfer pairs. You control the worker's installation, database access, and updates.

← Return to transfers