Feed StorageGRID 12 data to Elasticsearch 9 indexes with Kompromise

Introduction

StorageGRID has two integrations with Elasticsearch. I blogged about them here: there's Search and there's (indirect, via Logstash, but it's part of ELK stack) Logging.

This post is about Search (not Logging). Let's see if we can integrate StorageGRID 12.1 and Elasticsearch 9 better using Kompromise.

This was already demonstrated in an early prototype of Kompromise called "Go NATS!" last year, but this is a more detailed version that are unlikely to change by the time I share Kompromise Webhook binaries.

From Kompromise to Elasticsearch

As I wrote in the recent post on Kompromise Webhooks, there are Simple and Enhanced.

Here's an example of an index entry created from Enhanced Notifications in Kompromise:

There are details such as date, tags, metadata and the source IP that created the object.

Simple Notifications will have the usual (bare) details - nothing from kompromise.* keys above. Simple Notifications are fine if you search for things like object names, prefixes and dates, but you probably realize that this is useless for most practical purposes (especially anything AI-related).

So:

• At the most basic level, you'd use Enhanced (aka "Rich") as in the screenshot above. At least, you can search by tags and metadata which you can't with "some other integrations"
• Sometimes, you'd just skip this approach altogether and use something that applies to this time and age, such as vector embeddings and full text search

We can do basic (Simple or Rich), we can do lexical, we can do vectors. And we can do all three kinds for the same bucket if we need to, as shown here (the assumed bucket).

Search with Simple/Rich notifications

For basic search (that only stores data from Simple or Rich notifications), you don't have to do anything extra.

Pick 'em up from NATS topics and submit to Elasticsearch.

With Kompromise, indexing works better than StorageGRID's own Search Integration in 12.1 and that is reasonable, given that we use more resources to do it - Kompromise should better do it better.

With Kompromise, Simple notifications are persisted in NATS queues and batched to Elasticsearch, so we consume IOPS on NATS (3x, at that) but spare Elasticsearch (also RF3, if you do it right) from being bombarded with 100 byte requests every few seconds - this is much better for Elasticsearch, especially if you pay for it! Rich Kompromise notifications may be lost in rare circumstances (heavy load on Kompromise or StorageGRID) but, as I explained elsewhere, if those are critical you can use Simple and create own jobs for reliable Rich that work off Simple.

Search also works better with Kompromise if you use Rich Notifications (where tags, metadata are included).

Here's an example of default StorageGRID Search Integration entry (bucket versioning is off, unlike in the Kompromise Rich example above):

Let's just say this won't get your AI initiatives very far.

Full text search with Kompromise

We need to read all object content and store it in body or similar index field.

This is impractical for large documents which need to be broken down in shorter segments. For e-commerce product descriptions and even small to medium documents, it's fine.

I used the content from free books (Alice in Wonderland and Moby Dick) in my early testing. There were just a couple of KB each, so I did not need to prefetch them to AIS cache.

Document conversion and OCR may be required before one can even begin, but we used text files (markdown, CSV, and similar formats would also work) since data preparation is an extra step (although some commercial tools have it included to provide end-to-end, fully self-contained, workflows).

S3 access isn't a problem because this works out of the tenant's Kubernetes namespace (which maps 1:1 to the bucket name on StorageGRID). Our "worker" needs GET access to read the object from the notification message.

Vector search with Kompromise

Here, too, we must read full content of the object.

The way embeddings usually work is we "cut" an object in pieces ("chunks") and create embeddings for each piece. Then we store one or more embeddings under the S3 object's key in Elasticsearch.

Imagine that's a movie: 30 frames per second times 1,800 seconds. That's over 50,000 images.

We could probably run image analysis on just one compute node, but that would be much slower than if we did it on two or 12 Kubernetes workers.

That's where step 3A and the integrated AIS come in play: we can pre-fetch to cache (3A) and then compute. Or just compute (3B) using own client that connects to NATs or a curated function if we deal with lightweight content.

Here's a screenshot from the same "Alice in Wonderland" document, but with embeddings. This is the assume-vectors index, focused on chunk #3 which is one of the places which have information related to the question we'll ask.

Unlike with Full Text Search, here I have a bunch of smaller chunks, each of which has a fraction of overall text, and a vector field for semantic search.

The query shows just one chunk, which is one of the chunks where "ten soldiers" (the highest number) are mentioned (the same sequence also appears in chunk #4). The challenge - for a human reader, at least - is there are many places where different soldier numbers are mentioned.

Community vs. commercial Elasticsearch

The free edition can't run own pipelines with vectors. This is why I create them on the client.

The commercial edition can run vectorization on Elasticsearch, using their own curated models which are very good.

Likewise, when searching using the free edition, I can't search embeddings for words. I compute queries into vectors on the client, and search for best-matching vectors on Elasticsearch. It works, but it's less convenient.

OpenSearch, on the other hand, is "free" but always has bugs and ends up frustrating me much more than the limitations in Elasticsearch'es Community Edition.

Search results

With lexical search, we search for terms, words or combinations. For an example, "ten soldiers" AND queen. This does appear in the document we have, so that document gets found.

With embeddings, we look for stuff that may have matches in multiple chunks. We can search with vectorized questions, but in the free version that doesn't work well because - as mentioned above - we're not talking to a chatbot, you're comparing vectors and shouldn't be sending question-style queries.

To demonstrate this, the question was:

"Which of the two groups, gardeners or soldiers, was more numerous?"

"Query as a a question" doesn't work well because "which" doesn't help with semantic comparison, so results are likely to suck more than if we tried "there were five soldiers" (as "five" is at least a number).

Also, this question is hard, if you read that chapter from the book.

Let's see what 768-dimensional vectors have to say (don't read the text, it's enough to check score and #<number>):

1. score=0.8049 assumed/alice01.txt#10
   d she put them into a large flower-pot that stood near. The three soldiers wandered about for a minute or two, looking for them, and then quietly marched off after the others.  "Are their heads off?" ...
2. score=0.7954 assumed/alice.txt#3
   es, to—" At this moment Five, who had been anxiously looking across the garden, called out "The Queen! The Queen!" and the three gardeners instantly threw themselves flat upon their faces. There was a...

Actually, this isn't bad, because chunk #3 is in fact one of the two chunks that have the information we're looking for (the other is #4, but there are other relevant chunks, too). But you don't get a chatbot answer in vector search like this - you get chunks that best match the vectors you sent.

There are different techniques to identify better results (other than simple ranking), but normally we'd work with agents or LLMs which take care of that for us; there's no need to try and turn this into a chatbot experience. That comes later.

We can build one or both index types (lexical and semantic) or even something hybrid, depending what we need to achieve.

If you have the both kinds of indexes, you can do fancy searches or have an agent or LLM do them for you. So, our work on getting StorageGRID contents to agents and AIs has been completed!

• Documents get uploaded to a bucket
• Kompromise processes notifications and workers - that possibly require extra steps for conversion, extraction, normalization - store documents in Elasticsearch or OpenSearch (focus of this post, although we could use other indexers)
• Other steps: if documents are deleted, we need to decide what to do. Kompromise subjects wil have delete events, but you may or may not want to delete index documents - that depends on use case
• We need to make sure permissions on each index are open only to a tenant that has ownership of the bucket. But that's harder than it looks, because who would that be? The tenant administrator for the bucket? One of 100 tenant accounts? Who exactly?

Agents and LLMs

I'll leave this for another post related to Kompromise, but I've written about this before and created some examples with indexes on S3. This is the same thing except indexes are stored on block devices (e.g. EF-Series) and tiered to S3 only later.

But let's see what we expect to happen later:

• We have one or two indexes
• We have an chatbot or agent that can use these in many different ways
• Now we, or agents that work with agents that have access to these indexes, can ask questions, not just enter words or throw vectors at it

YMMV, as they say. Earlier, we've established there were ten soldiers. At least, that's what I thought.

When an LLM comes into the picture, it searches (like we did above) and does its own reasoning from and with the results. There's a bunch of techniques and models and each can produce different results.

To illustrate this point, I used two approaches to answer the question - now in the right "chat form" - with LLMs that had access to the same indexes. The LLM model was the same, but the technique was not.

Example answer from approach A:

[answer] The story mentions ten soldiers carrying clubs in passages [3] and [4], which describe them as part of a procession. While passages [1] and [2] reference three soldiers, these are likely part of the same group of ten, as the narrative context suggests a cohesive scene. Therefore, the total number of soldiers in the story is ten.

Example answer from approach B:

[answer] The story mentions three soldiers in passages [3] and [4], and ten soldiers in passage [5]. These groups are distinct, as the ten soldiers are part of a procession during the Queen's arrival, while the three soldiers are part of the court. Thus, the total number of soldiers is 3 + 10 = 13.

That's intriguing!

But it's also not our problem: we are in just charge of making sure they have data to work with.

Chunk details (like [3], [4]) and document URIs is what you'd get as URLs to the sources, as usual in RAG or similar applications - the so-called "grounded answers".

Comparison vs…

Kompromise isn't a product and, if it gets published, it will be distributed as (binary) freeware, so it shouldn't be compared with shipping or supported products: you can't get support for it and you'd probably build a similar tool if you wanted to use it in production in an enterprise environment.

But since the topic is of interest to StorageGRID users in general, I'll share what I currently know because it's hard to find StorageGRID-specific information elsewhere.

What else is out there

As of now, your choices in terms of near real-time integration are limited and you have to build own integrations.

Maybe Komprise can receive StorageGRID notifications, but their documentation isn't detailed (or publicly available), so it's hard to tell. They have excellent workflows for hybrid cloud and AI/analytics workflows and solutions.

There are data migration products such as Datadobi, which support StorageGRID. I think they may subscribe to Notifications as that would be helpful in migrations, but I don't know if they do it and how their indexes look like (I'm sure they have them, but do they store information AI agents need, or does it need to be re-created by the user?). Kompromise could be used in migrations, but it primarily targets event-driven unstructured data processing use cases.

Then there's Starfish Software with explicit StorageGRID support. It appears they use bucket scans as well. Positioning-wise, it's a cross-vendor storage management tool with support for storage-related jobs and workflows, so similar to the Komprise (it can drive workflows) and Datadobi (it is used for migrations). Starfish includes own database (standard PostgreSQL) and lets you access it their API or CLI or even PostgreSQL clients to query the database and dispatch jobs and workflows to other places.

Among the more recent solutions, two-three days ago Blocks & Files mentioned NetApp partnered with Diskover. I don't know if Diskover works with StorageGRID, but presumably it does and I would hope it works the same way Kompromise does (based on Notifiations). But it could be doing full bucket enumeration instead.

Diskover has many valuable plugins that come useful in those pre-indexing steps I mentioned earlier, while Kompromise doesn't have any and can't parse complex document formats. Although I could develop those or use existing open source tools like I did in this post, that's out of scope for Kompromise. One last thing about Diskover: it integrates well with Elasticsearch, so if ES is what you already use that alone makes it a strong contender.

Finally, earlier this week NetApp also announced next version of AIDE would support StorageGRID. As of October 4, the software is not yet available (based on the Web site it seems its GA is planned for October 23), but there are documentation pages like this page and this page which indicate that indexes are built using bucket scans (ObjectListV2 every time you update an index), so it may not be directly comparable to Kompromise as far as the "how indexes are updated" question is concerned.

How other AIDE things compare, I do not know, but it seems this month's release is more of a "storage capacity monitoring and management" tool - much closer to Starfish Software than Kompromise.

NetApp also has an existing product called Cloud Sync, which is a data replication tool which may (that part is unclear to me) receive S3 event notifications, but doesn't do any processing except synchronization (copying).

Long story short: there are very few tools with explicit support for StorageGRID Notifications as far as near real-time processing for AI is concerned. Maybe some of these can consume StorageGRID Notifications, but that's unlikely to be their strong point.

Full re-scan vs notifications

There's one valid question here, though: if we have TBs of data that's already in a bucket, how to index it for AI or other workflows?

• Please do not worry. It's just one CLI command with AWS CLI or MinIO client. You can output the result to a file and loop through the list to get the data you need. See this example for finding "version hogs". One interesting pattern is I've seen users with NAS background who expect "the storage admin" to do these things for them, but that's not how it should work on S3. Who's going to do something for the user whose files are encrypted client-side, or who creates ephemeral COSI bucket claims? "Storage admin" has no clue if a chatbot or Elasticsearch instance may be linking to the stuff in your bucket. Clearly, data management issues should be fixed in workflows, not by "storage admin" actions like it used to be done in 2003
• If you need to create embeddings or a lexical index, that may be worth automating separately using own tool
• If you use a "brute-force lister" to enumerate bucket contents for "storage administration", and another tool to do it for AI, that becomes very expensive quickly

But wait, why even use Notifications when we can simply re-index buckets?

• You can't re-enumerate an entire bucket once every minute, but you can receive Notifications of new objects every second and business may not be able to wait until next weekend
• Brute-force listing doesn't work well at scale. Imagine enumerating all objects across hundreds of TB sized buckets every week. I don't know if that's what others do or not, but I do know that's unlikely to work well. This isn't done for one-off migration, mind you
• "Scanning performance" doesn't play a role in tool selection. ListObjectsV2 is a metadata query that costs the S3 client almost nothing, but consumes non-insignificant resources the object store (which is why AWS charges for such API calls). Most users should throttle these rather than try to do "scan faster". They'll be constrained by object storage performance, not by client performance.
• Bottom line is, there's no substitute for Notifications. It doesn't matter how you get 'em, Webhook or Kafka, every consumer and every software gets the same information at the same time. Kompromise Webhooks won't have less information than any other software, so I expect Notifications to remain one of the least dirty shirts in the closet. You can supplement Notifications with full re-scan once a week to pick objects you missed and maybe reconcile with Elasticsearch (Kompromise can do that as well, just read the list from a file rather than from NATS and the rest is identical), but that should be an exception, not a pattern
• You'd also have to update all Elasticsearch indexes, not just S3 buckets. Why? Because you probably want to remove stale objects from indexes, not just add new ones. Or, maybe you want mark stale index entries as deleted, if you have to keep indexes in sync with S3

As far as getting full listing (and building a search index from that content) of a bucket is concerned, StorageGRID already has a solution for that: Snapshot Leases with sg-cosi:

• Create a Snapshot Lease using my StorageGRID COSI driver. Let's say you create a 48 hour snapshot lease
• That gives you 48 hours (this can be extended up to 7 days) to access the bucket from anywhere it's accessible to S3 clients by using ephemeral S3 keys generated by sg-cosi
• The Snapshot Lease function in my sg-cosi was created for bucket backup workflows, but you can use it to run ListObjectsV2 for 36-hours non-stop if you'd like (not the greatest idea because such scanning competes with ILM and all workloads - you should throttle it or have limits on the StorageGRID load balancer to control this behavior)
• Then, we can send those results to any, or several, places to perform index maintenance

That's available and easy to use (create a Github issue if it doesn't work), but it should be used only in exceptional situations, not as the main method to keep track of your S3 data. It can't scale well and it's very taxing on object store.

I see a lot of AI-related examples that are based on full bucket scans of buckets with dozens to thousands images. "You just re-scan". That's fine - you can do that. But why would you have a bucket with just a few thousand images?

It must be a consequence of copying. You have the same 7,500 images in another bucket, together with 750 thousand other images, but rather than organizing that data in a datalake (which lets you use SQL and permissions) or using OpenSharing, you choose to copy objects to an app-specific bucket for every application, and then re-scan and re-index every time anything changes. Well, that can work, but it doesn't sound right.

People also forget that vector embeddings aren't cheap to recreate at scale, so if you don't have a database or a "diff" utility, how can you even tell which new or updated documents need to be processed and updated? If you are lucky and have "just 7,500 documents", rebuilding the entire Elasticsearch index every day won't be expensive. There's a chance!

Kompromise, as a reference stack for data pipelines, does not aim for prescriptive integrations because it's supposed to be able to work with anything on either input (matter of implementation; I hope to add Versity S3 Gateway with E-Series later) or output (matter of user's choice) side.

Even with Notifications, you can still re-scan and rebuild indexes every day (or week), but at least you'll get updates seconds later and avoid daily reconciliations.

Notifications are resource-cheap, while data churning is expensive and with Kompromise I can do both and it scales out. After processing, data can be stored on StorageGRID - a scale-out platform itself - and indexes on any scale out database (such as Elasticsearch), so Kompromise plays well here - it focuses on specific areas where it can add value. It doesn't make you change or fragment your workflows - it just makes existing work better.

The only hard-coded dependency is NATS, which is there for a reason. I could support two event stores ("never say never"), but NATS is integrated and doesn't require the user to come up with own NATS cluster. Additionally, any qualified user (with requirements matching what Kompromise does) will likely need AIS as well, and NATS and AIS have the same requirements storage-wise - there are no unnecessary overheads.

Kompromise Lite

Kompromise Lite is s simplified binary version of the Webhook component that only requires Docker, but provides only basic features sufficient for non-production demonstrations of Webhook forwarding (see in the repo).

It also gives a way to forward events to Kafka, as an alternative for Kafka users, since full Kompromise stack will continue using NATS.

Conclusion

In 2026, keeping your data out of reach for agents is almost as bad as keeping it on tapes.

Kompromise makes StorageGRID-parked data more valuable because with it, you and your agents can actually find your stuff.

Kompromise Webhooks let you plugin StorageGRID in existing workflows, send data to existing databases and easily add new and even multiple targets (e.g. add databases specialized not in generic search, but in agentic AI, so that you don't have to use cascading replication where there's no need for it).

Another benefit of this database-neutral approach is your database can be anywhere - in the cloud, hybrid, or in several locations at the same time (YugabyteDB). If you have several federated StorageGRIDs at different sites, each can send data to the same distributed YugabyteDB which has read-write replicas in multiple sites.

I feel I need to mention this in almost every Kompromise post: the Webhook is the only new part here, while the rest is off the shelf software, but I perhaps shouldn't, because much of commercial software is not any better in that regard.

Today's post doesn't include AIS examples (I showed them in the previous posts, though), but it does show that simply with the Webhook and per-tenant NATS queues, I can easily deliver any kind of search integration we need - including heavy media processing steps, if any - without spending days on reinventing the wheel.

While the scripts to used to populate Elasticsearch indexes aren't part of Kompromise, I may add their key parts to the repo where Kompromise Webhook is expected to appear when ready. These are out of scope (downstream from Kompromise) and everyone does things slightly differently, so I'd rather let everyone consume StorageGRID events the way they prefer - existing compute workflows and existing databases/indexes.