8 minute read

Introduction

You have, or plan to buy, StorageGRID and do something more with it - such as some AI use cases. To do that, you need embeddings for your data.

You can offload some of that work by using managed OpenSearch by Instaclustr. It takes 15 minutes to start, after which you have a properly configured OpenSearch with TLS access.

StorageGRID has search integration that's the same as it was 10 years ago. It works, but it's not good enough for AI.

We can create embeddings ourselves, but we can also auto-generate embeddings using OpenSearch.

This post shows how this simpler auto-generated approach works. It also shows why that probably isn't enough and how OpenSearch can help you.

Auto-generate embeddings in Instaclustr OpenSearch

Note that you can do it at scale by provisioning a "vectorization worker" if you have a lot of stuff to vectorize.

For smaller scale, meaning simple text, small documents and tiny small images, maybe you can get away by using CPU. Don't go too crazy with "benchmarking" if you also search and index at scale.

Instead of trying to get creative, we just RTFM and copy-paste their stuff.

First, we enable generic OpenSearch nodes for vectorization workload:

PUT _cluster/settings
{
  "persistent": {
    "plugins.ml_commons.only_run_on_ml_node": "false",
    "plugins.ml_commons.native_memory_threshold": "85"
  }
}

If you use dedicated nodes, don't use the above - configure a dedicated node instead.

Second, join OpenSearch Confusion Fest at this page. Take your time picking through the 100 choices!

We'll create a "local" group. It can be any - this is a tag for your own groupings.

POST /_plugins/_ml/model_groups/_register
{
  "name": "local_model_group",
  "description": "A model group for local models"
}

Use the returned ID in model_group_id below (or otherwise drop the key below) and pick one of the models from that pre-trained list (also get the model version right).

NOTE: this one generates 384-d vectors. Maybe you want a different model or version. DYOR.

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
  "version": "1.0.3",
  "model_group_id": "1sVSMaEBxbvMjUqWqPfr",
  "model_format": "TORCH_SCRIPT"
}

Response:

{
  "task_id": "-sVWMaEBxbvMjUqWG_dc",
  "status": "CREATED"
}

Get the sucker:

GET /_plugins/_ml/tasks/-sVWMaEBxbvMjUqWG_dc

Model ID is needed later so copy this into some "notepad" of sorts:

{
  "model_id": "hqRWMaEB62QuBhwmJD7e",
  "task_type": "REGISTER_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "COMPLETED",
  "worker_node": [
    "akGpd1uwQxiIVho1qp1c9Q"
  ],
  "create_time": 1791829089110,
  "last_update_time": 1791829124033,
  "is_async": true
}

Deploy that ID:

POST /_plugins/_ml/models/hqRWMaEB62QuBhwmJD7e/_deploy

Done?

{
  "task_id": "JcVaMaEBxbvMjUqWjfjP",
  "task_type": "DEPLOY_MODEL",
  "status": "CREATED"
}

Check what happened:

GET /_plugins/_ml/tasks/JcVaMaEBxbvMjUqWjfjP

Response:

{
  "model_id": "hqRWMaEB62QuBhwmJD7e",
  "task_type": "DEPLOY_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "COMPLETED",
  "worker_node": [
    "-oALb9-DTSCiT_gmhMYeaQ",
    "akGpd1uwQxiIVho1qp1c9Q",
    "u8KOQW00RwSpeVcAkr2iig"
  ],
  "create_time": 1791829380508,
  "last_update_time": 1791829408386,
  "is_async": true
}

We're supposedly "ready" now. Let's give it a try!

POST /_plugins/_ml/_predict/text_embedding/hqRWMaEB62QuBhwmJD7e
{
  "text_docs":[ "train"],
  "return_number": true,
  "target_response": ["sentence_embedding"]
}

This took around 3 seconds on a small CPU node.

Result (trimmed):

{
  "inference_results": [
    {
      "output": [
        {
          "name": "sentence_embedding",
          "data_type": "FLOAT32",
          "shape": [
            384
          ],
          "data": [
            -0.023314947,
            0.089756876,
            -0.08136316,
            0.045115348
          ]
        }
      ]
    }
  ]
}

Since this works, our next step would be to doing something useful with it.

OpenSearch has an automated pipeline approach described here.

If that doesn't work for you, you can create a "manual", or rather "custom", workflow.

Using your model ID (ours is hqRWMaEB62QuBhwmJD7e):

POST /_plugins/_flow_framework/workflow?use_case=semantic_search&provision=true
{
    "create_ingest_pipeline.model_id" : "hqRWMaEB62QuBhwmJD7e",
    "text_embedding.field_map.output.dimension": "384",
    "text_embedding.field_map.input": "passage"
}

Response:

{
  "workflow_id": "-MVvMaEBxbvMjUqW6PgE"
}

Check that sucker:

GET /_plugins/_flow_framework/workflow/-MVvMaEBxbvMjUqW6PgE/_status

Response:

{
  "workflow_id": "-MVvMaEBxbvMjUqW6PgE",
  "state": "COMPLETED",
  "resources_created": [
    {
      "resource_id": "nlp-ingest-pipeline",
      "workflow_step_name": "create_ingest_pipeline",
      "workflow_step_id": "create_ingest_pipeline",
      "resource_type": "pipeline_id"
    },
    {
      "resource_id": "my-nlp-index",
      "workflow_step_name": "create_index",
      "workflow_step_id": "create_index",
      "resource_type": "index_name"
    }
  ]
}

Now we can use some Wikipedia content from the S3 storage page to create entries:

PUT /my-nlp-index/_doc/1
{
  "passage": "Object storage (also known as object-based storage) is a computer data storage approach that manages data as blobs or objects ."
}

PUT /my-nlp-index/_doc/2
{
  "passage": "Jim Starkey coined the term blob working at Digital Equipment Corporation to refer to opaque data entities ."
}

PUT /my-nlp-index/_doc/3
{
  "passage": "Other related work includes the Coda filesystem project at Carnegie Mellon, which started in 1987, and spawned the Lustre file system ."
}

These resulted in three responses like this one:

{
  "_index": "my-nlp-index",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Let's try to find something. Provide your model ID (hqRWMaEB62QuBhwmJD7e) and a search term. Say, Coda appears in the third document we indexed.

GET /my-nlp-index/_search
{
  "_source": {
    "excludes": [
      "passage_embedding"
    ]
  },
  "query": {
    "neural": {
      "passage_embedding": {
        "query_text": "coda",
        "model_id": "hqRWMaEB62QuBhwmJD7e",
        "k": 1
      }
    }
  }
}

k=1 because I want just one answer.

{
  "took": 278,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.4139904,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "3",
        "_score": 0.4139904,
        "_source": {
          "passage": "Other related work includes the Coda filesystem project at Carnegie Mellon, which started in 1987, and spawned the Lustre file system ."
        }
      }
    ]
  }
}

The score (0.4139904) is relative. The absolute value doesn't significantly matter. You want k top results first and maybe massage that set from another aspect to re-rank or pick based on some hybrid evaluation approach. You could get k=N for top N, but drop results below a certain limit, etc.

At the end, let's take a look at that auto-index:

GET my-nlp-index

Response:

{
  "my-nlp-index": {
    "aliases": {},
    "mappings": {
      "properties": {
        "passage": {
          "type": "text"
        },
        "passage_embedding": {
          "type": "knn_vector",
          "dimension": 384,
          "method": {
            "engine": "lucene",
            "space_type": "l2",
            "name": "hnsw",
            "parameters": {}
          }
        }
      }
    },
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "2",
        "provided_name": "my-nlp-index",
        "merge": {
          "policy": {
            "floor_segment": "2mb",
            "max_merge_at_once": "10"
          }
        },
        "default_pipeline": "nlp-ingest-pipeline",
        "knn": "true",
        "creation_date": "1791830780927",
        "unassigned": {
          "node_left": {
            "delayed_timeout": "30m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "ZHb2INLQT6W0c84k1E07cA",
        "version": {
          "created": "137277827"
        },
        "knn.derived_source": {
          "enabled": "true"
        }
      }
    }
  }
}

That's different from indexes generated by StorageGRID storage integration. Example (objects with tags haven't been ingested, but "metadata" is present):

{
  "sgi-index": {
    "aliases": {},
    "mappings": {
      "properties": {
        "accountId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "bucket": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "key": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "md5": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "metadata": {
          "properties": {
            "key1": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "region": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "size": {
          "type": "long"
        },
        "versionId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "1",
        "provided_name": "sgi-index",
        "creation_date": "1791751427057",
        "unassigned": {
          "node_left": {
            "delayed_timeout": "30m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "h2NLvXJWSeWjvlo2epC9dQ",
        "version": {
          "created": "137277827"
        }
      }
    }
  }
}

Other approaches

You can create embeddings manually on your side, without using OpenSearch for it. OpenSearch would be used only for search - i.e. comparison between stored embeddings with embeddings you send in queries.

I blogged about that in this post, so I won't repeat that.

Will you need "other approaches"? Almost certainly, yes.

StorageGRID integration

StorageGRID search integration for OpenSearch isn't useful for AI, because its format is fixed.

You're gonna need a bigger boat index. This is where "custom" indexes come in:

  • Create one that has embeddings and other details, like in the linked post above
  • Or, create several (one basic based on StorageGRID search integration, one automated for Gen AI, etc.), so that you can search some for one thing and another for other

The issue with creating several - also described in the linked post, is that you need to work with notifications and touch them on their way to OpenSearch, which also means you shouldn't drop 'em while you do that, i.e. you need a reliable message store or database of some kind to prevent data loss.

Alternatively, run periodic object scans and refresh OpenSearch. This doesn't work well for large buckets, but utilities like SGII may be able to help you create these lists more effectively.

Because embeddings require content (you could create it from an object's metadata and tags, but that probably wouldn't be useful), you also need to download these objects and use their content to create embeddings.

Remember that these models have limits - you can't create a vector for an entire movie or book. You need to "chunk" them, as shown in the search section of that post above.

Most of these things are in the previous post so I won't spend much time on rehashing them.

Conclusion

It takes minutes to get your vectorization pipeline ready with Instaclustr OpenSearch. It's simple, but it gets complicated quickly.

If you're trying to meet deadlines, you probably don't want to add OpenSearch to your list of worries.

With Instaclustr OpenSearch, you can run OpenSearch worry-free in your own cloud account. If you're up for custom indexes and pipelines, you probably want Instaclustr Kafka as well. These are fully Open Source-based, and you can replicate them 100% on premises. There's value in using the same exact software on-premises and in the cloud when complexity of your stack isn't trivial.

Related to StorageGRID, you will likely not want to use the built-in search integration feature.

You can build your own indexes based on StorageGRID Event Notification and object reads or use something like Kompromise Lite described in the previous post (full-featured version is complicated, I'm still working on it). If you don't need near-real-time search, you can generate updated object lists with SGII or your own tool, and batch-upload to OpenSearch.

If you want to generate embeddings yourself, then create manual indexes and read, chunk and vectorize before uploading data to OpenSearch. This is where the value of "100% reproducible" comes in play: you want same models, same dimensions and same workflows on-premises (development, for example) and in the cloud (production). Instaclustr's OpenSearch and Kafka give you that, and more.

You may also enjoy