Skip to content

Adapt search pipeline to prefer semantic results #26771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lautel wants to merge 7 commits into
base: main
Choose a base branch
from
+19 −25
Open

Adapt search pipeline to prefer semantic results #26771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a87cdf5
Adapt search pipeline to prefer semantic results
lautel Mar 25, 2026
70f2af1
Merge branch 'main' into improve-search-recall
lautel Mar 25, 2026
fe1a1aa
Merge branch 'main' into improve-search-recall
lautel Mar 26, 2026
ec1b1e0
Merge branch 'main' into improve-search-recall
lautel Mar 26, 2026
ca9ccb8
Update CLAUDE.md with environment setup instructions (#26714)
harshach Mar 26, 2026
382276a
Merge branch 'main' into improve-search-recall
lautel Mar 27, 2026
ec60120
Merge branch 'main' into improve-search-recall
lautel Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...tegration-tests/src/test/java/org/openmetadata/it/tests/VectorEmbeddingIntegrationIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,11 +253,11 @@ void testBatchFingerprintOperations() throws Exception {

@Test
void testEnsureHybridSearchPipelineCreatesAndUpdates() throws Exception {
vectorService.ensureHybridSearchPipeline(0.6, 0.4);
vectorService.ensureHybridSearchPipeline(0.4, 0.6);

Map<String, Object> pipeline = getSearchPipeline(OpenSearchVectorService.HYBRID_PIPELINE_NAME);
assertNotNull(pipeline, "Pipeline should exist after creation");
assertWeightsInPipeline(pipeline, 0.6, 0.4);
assertWeightsInPipeline(pipeline, 0.4, 0.6);

vectorService.ensureHybridSearchPipeline(0.3, 0.7);

Expand Down
4 changes: 2 additions & 2 deletions openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,9 +453,9 @@ public void ensureHybridSearchPipeline() {

ElasticSearchConfiguration cfg = getSearchConfiguration();
NaturalLanguageSearchConfiguration nlConfig = cfg.getNaturalLanguageSearch();
double keywordWeight = nlConfig.getKeywordWeight() != null ? nlConfig.getKeywordWeight() : 0.6;
double keywordWeight = nlConfig.getKeywordWeight() != null ? nlConfig.getKeywordWeight() : 0.4;
double semanticWeight =
nlConfig.getSemanticWeight() != null ? nlConfig.getSemanticWeight() : 0.4;
nlConfig.getSemanticWeight() != null ? nlConfig.getSemanticWeight() : 0.6;

try {
SearchSettings ss =
Expand Down
2 changes: 1 addition & 1 deletion ...service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ public void ensureHybridSearchPipeline(double keywordWeight, double semanticWeig
MAPPER
.createObjectNode()
.put("technique", "rrf")
.put("rank_constant", 60)
.put("rank_constant", 30)
Copy link
Copy Markdown
Contributor

@tomasmontielp tomasmontielp Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

motivation behind reducing denominator constant? overall curious about this variable, 60 as default also intriguing

Copy link
Copy Markdown
Contributor Author

@lautel lautel Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

60 is the default rank constant. Everywhere you see RRF, the default is 60. Thing is, it makes scores quite uniform and masks very high ranking items. I halved it to maintain clearer differences in scores between rank positions.

.set("parameters", MAPPER.createObjectNode().set("weights", weights));
var scoreRanker =
MAPPER
Expand Down
2 changes: 1 addition & 1 deletion ...a-service/src/test/java/org/openmetadata/service/search/SearchRepositoryBehaviorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1474,7 +1474,7 @@ void initializeVectorSearchServiceInitializesOpenSearchVectorSupport() throws Ex
assertSame(embeddingClient, spyRepository.getEmbeddingClient());
assertSame(vectorService, spyRepository.getVectorIndexService());
assertNotNull(spyRepository.getVectorEmbeddingHandler());
verify(vectorService).ensureHybridSearchPipeline(0.6, 0.4);
verify(vectorService).ensureHybridSearchPipeline(0.4, 0.6);
}

@Test
Expand Down
6 changes: 3 additions & 3 deletions ...ice/src/test/java/org/openmetadata/service/search/vector/OpenSearchVectorServiceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ void testEnsureHybridSearchPipelineSendsCorrectRequest() throws IOException {
ArgumentCaptor<os.org.opensearch.client.opensearch.generic.Request> captor =
ArgumentCaptor.forClass(os.org.opensearch.client.opensearch.generic.Request.class);

vectorService.ensureHybridSearchPipeline(0.6, 0.4);
vectorService.ensureHybridSearchPipeline(0.4, 0.6);

verify(mockGenericClient).execute(captor.capture());
os.org.opensearch.client.opensearch.generic.Request captured = captor.getValue();
Expand All @@ -448,9 +448,9 @@ void testEnsureHybridSearchPipelineSendsCorrectRequest() throws IOException {

String body =
new String(captured.getBody().get().bodyAsBytes(), java.nio.charset.StandardCharsets.UTF_8);
assertTrue(body.contains("\"weights\":[0.6,0.4]"));
assertTrue(body.contains("\"weights\":[0.4,0.6]"));
assertTrue(body.contains("\"technique\":\"rrf\""));
assertTrue(body.contains("\"rank_constant\":60"));
assertTrue(body.contains("\"rank_constant\":30"));
assertTrue(body.contains("\"collapse\""));
assertTrue(body.contains("\"parentId\""));
}
Expand Down
18 changes: 6 additions & 12 deletions openmetadata-spec/src/main/resources/elasticsearch/indexMapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,7 @@
"parentAliases": [
"storageService",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": []
},
Expand All @@ -169,8 +168,7 @@
"parentAliases": [
"driveService",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": [
"file",
Expand All @@ -185,8 +183,7 @@
"driveService",
"directory",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": []
},
Expand All @@ -198,8 +195,7 @@
"driveService",
"directory",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": [
"worksheet"
Expand All @@ -213,8 +209,7 @@
"driveService",
"spreadsheet",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": []
},
Expand Down Expand Up @@ -331,8 +326,7 @@
"parentAliases": [
"classification",
"all",
"dataAsset",
"dataAssetEmbeddings"
"dataAsset"
],
"childAliases": []
},
Expand Down
4 changes: 2 additions & 2 deletions ...etadata-spec/src/main/resources/json/schema/configuration/elasticSearchConfiguration.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,12 +141,12 @@
"keywordWeight": {
"description": "Weight for BM25 keyword search results in hybrid RRF pipeline (0.0-1.0)",
"type": "number",
"default": 0.6
"default": 0.4
},
"semanticWeight": {
"description": "Weight for semantic vector search results in hybrid RRF pipeline (0.0-1.0)",
"type": "number",
"default": 0.4
"default": 0.6
},
"embeddingProvider": {
"description": "The provider to use for generating vector embeddings (e.g., bedrock, openai).",
Expand Down
4 changes: 2 additions & 2 deletions openmetadata-spec/src/main/resources/json/schema/configuration/searchSettings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,12 @@
"keywordWeight": {
"description": "Weight for BM25 keyword search in hybrid RRF pipeline (0.0-1.0)",
"type": "number",
"default": 0.6
"default": 0.4
},
"semanticWeight": {
"description": "Weight for semantic vector search in hybrid RRF pipeline (0.0-1.0)",
"type": "number",
"default": 0.4
"default": 0.6
}
},
"additionalProperties": false
Expand Down
Loading