Skip to Content
RigFabTech docs
Operate

Object storage

Products keep files, such as uploads and exports, in S3-compatible object storage. By default the base chart runs one SeaweedFS for every product. You can point the platform at your own S3-compatible store instead.

How products get a bucket

Each product that stores files has a BucketClaim in its namespace. The base chart's controller gives it its own bucket. It then writes a Secret and a ConfigMap named after the claim, <product>-bucket, in the product's namespace. They have the shape of an ObjectBucketClaim:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  • BUCKET_NAME, BUCKET_HOST, BUCKET_PORT, BUCKET_REGION, BUCKET_ENDPOINT and BUCKET_PATH_STYLE
  • BUCKET_PUBLIC_ENDPOINT, the address browsers use

To list the claims:

kubectl get bucketclaims --all-namespaces

Uninstalling a product keeps its bucket and data.

The bundled store

objectStorage:
  mode: bundled
  bundled:
    size: 50Gi
    storageClass: ""
    host: s3
    public: true
  • It runs as one pod with one volume, seaweedfs in rft-system. Helm doesn't delete the volume.
  • It's a single instance. For high availability, use your own store.
  • Each product's key reaches only its own bucket. When a product's key changes, the controller restarts the product's pods onto it.
  • Inside the cluster, the store answers over plain HTTP on port 8333 to the controller and to pods labelled rigfab.com/object-storage. Any pod may carry that label, so the keys, not the network, keep one product out of another's bucket.
  • Browsers reach it on s3.<domain>. See Browsers.

Back it up

The volume holds the only copy of the products' files. Back it up.

Back it up with your volume snapshots, or copy every bucket off the cluster with any S3 tool. Use the admin key in Secret rft-system/rft-object-storage. For example, to copy a bucket to a directory you then back up:

kubectl --namespace rft-system port-forward service/seaweedfs 8333 &
export AWS_ACCESS_KEY_ID="$(kubectl --namespace rft-system get secret rft-object-storage --output jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)"
export AWS_SECRET_ACCESS_KEY="$(kubectl --namespace rft-system get secret rft-object-storage --output jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)"
aws --endpoint-url http://localhost:8333 --region us-east-1 s3 sync s3://<BUCKET> ./backup/<BUCKET>

To restore, run the same command the other way round, into the same bucket names. rclone sync copies straight to another store, as both ends are S3 remotes.

Browsers

People's browsers upload and download files straight to the store, so files never pass through the products' pods. Each URL is short-lived, and the product signs it for one object.

The bundled store answers browsers on s3.<domain> through your ingress or Gateway. global.rigfab.hosts.s3 gives it another name. Browsers reach a second listener, port 8334:

  • It knows only the products' keys. It has no admin key and no IAM API.
  • With the Gateway API, its route admits only GET, HEAD, PUT and OPTIONS.
  • It's open to whatever global.rigfab.networkPolicy.ingressFrom names, or to every pod when that's empty, as the products' own ports are.

The controller lets each product's own https://<product>.<domain> reach its bucket from a browser (CORS). It does this in a rule named rft-browsers, beside any others on the bucket. No other site's pages can reach the bucket.

Ingress limits

Large files go up in parts, each in its own request, so the ingress's default limits hold. Your ingress must still pass request bodies of at least 100 MB to s3.<domain> and give each request 60 seconds:

  • ingress-nginx refuses bodies over 1 MB by default. Set nginx.ingress.kubernetes.io/proxy-body-size: 100m in global.rigfab.ingress.annotations, which every route takes.
  • Traefik reads a whole request within its entry point's respondingTimeouts.readTimeout, 60 seconds by default.
  • A Gateway's limits are its implementation's. Check them against the same numbers.

Turn off browser access

To keep the store off your ingress, set public: false. The products then answer that browser access to files is off, and uploads and downloads that need it fail.

Your own store

Use Amazon S3, Ceph RGW, MinIO or any S3-compatible store:

objectStorage:
  mode: external
  external:
    endpoint: https://s3.us-east-2.amazonaws.com
    publicEndpoint: ""
    region: us-east-2
    pathStyle: false
    existingSecret: site-s3

Browsers reach your store at endpoint. When the address the cluster uses isn't one browsers can reach, such as http://minio.storage.svc:9000, set publicEndpoint to its HTTPS name. Whatever answers there must:

  • forward the Host header unchanged
  • add no path
  • have a certificate browsers trust

One key for every product

existingSecret is a Secret in rft-system with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. With it, the controller:

  • creates each product's bucket, rft-<domain>-<product>, when it's missing. namePrefix changes the first part.
  • copies that one key into every product's namespace, so every product can reach whatever that key can.

A key per product

Prefer a key per product, scoped to its bucket, under objectStorage.apps.<product>:

objectStorage:
  apps:
    inspection:
      existingSecret: inspection-s3
      bucketName: acme-inspection

existingSecret is then a Secret in the product's namespace with the same keys. bucketName is an existing bucket.

With workload identity (EKS, AKS, GKE), set workloadIdentity: true and each product's serviceAccountAnnotations, such as eks.amazonaws.com/role-arn, under objectStorage.apps.<product>. You create the buckets.

objectStorage.apps sits in the same values file as everything else, so one file still serves the base chart and every product.

Bucket CORS

Browsers need CORS on each bucket: GET, HEAD and PUT from the product's https://<product>.<domain>, exposing the ETag header.

The controller sets it when the key may. When the store refuses, the BucketClaim stays ready, its message says so, and you set CORS yourself. For example, on Amazon S3:

aws s3api put-bucket-cors --bucket <BUCKET> --cors-configuration '{"CORSRules":[{"AllowedOrigins":["https://<product>.<domain>"],"AllowedMethods":["GET","HEAD","PUT"],"AllowedHeaders":["*"],"ExposeHeaders":["ETag"],"MaxAgeSeconds":3600}]}'

MinIO has no bucket CORS. Set MINIO_API_CORS_ALLOW_ORIGIN to the products' addresses on the MinIO server.

Products and network access

Give the products the same objectStorage.mode as the base chart. With rft config's one values file, this is automatic.

The products reach the store on HTTPS through the internet. When it's on your network, or internetEgress is none, they reach it at global.rigfab.site.destinations.objectStorage.