feat: add new falkordb integration#3158
feat: add new falkordb integration#3158ghassenzaara wants to merge 23 commits intodeepset-ai:mainfrom
Conversation
…e and native vector search
…ith neo4j-haystack
c580421 to
d380366
Compare
bogdankostic
left a comment
There was a problem hiding this comment.
Thanks so much for this PR, @ghassenzaara! Great work so far. You'll see quite a few comments below, but they are mostly just minor formatting improvements for the docstrings.
| - "Test / dspy" | ||
| - "Test / elasticsearch" | ||
| - "Test / faiss" | ||
| - "Test / falkor_db" |
There was a problem hiding this comment.
For consistency (for example with arcadedb) let's use falkordb throughout this integration instead of falkor_db, so changing for example also integrations/falkor_db -> integrations/falkordb.
| - any-glob-to-any-file: ".github/workflows/faiss.yml" | ||
|
|
||
|
|
||
| integration:falkor-db: |
There was a problem hiding this comment.
| integration:falkor-db: | |
| integration:falkordb: |
| documented_only: true | ||
| skip_empty_modules: true | ||
| renderer: | ||
| description: FalkorDB integration for Haystack — GraphRAG document store, embedding retriever, and Cypher retriever |
There was a problem hiding this comment.
| description: FalkorDB integration for Haystack — GraphRAG document store, embedding retriever, and Cypher retriever | |
| description: FalkorDB integration for Haystack |
| def to_dict(self) -> dict[str, Any]: | ||
| """ | ||
| Serialise this component to a dictionary. | ||
|
|
||
| :returns: Dictionary representation of this retriever's configuration. | ||
| """ | ||
| data = default_to_dict( | ||
| self, | ||
| custom_cypher_query=self._custom_cypher_query, | ||
| ) | ||
| data["init_parameters"]["document_store"] = self._document_store.to_dict() | ||
| return data | ||
|
|
||
| @classmethod | ||
| def from_dict(cls, data: dict[str, Any]) -> "FalkorDBCypherRetriever": | ||
| """ | ||
| Deserialise this component from a dictionary. | ||
|
|
||
| :param data: Dictionary previously produced by :meth:`to_dict`. | ||
| :returns: A new :class:`FalkorDBCypherRetriever` instance. | ||
| """ | ||
| init_params = data.get("init_parameters", {}) | ||
| if "document_store" in init_params: | ||
| init_params["document_store"] = FalkorDBDocumentStore.from_dict(init_params["document_store"]) | ||
| return default_from_dict(cls, data) |
There was a problem hiding this comment.
These methods shouldn't be needed, see our documentation on default serialization behavior.
| """ | ||
| Retrieve documents by executing an OpenCypher query. | ||
|
|
||
| If a ``query`` is provided here, it overrides the ``custom_cypher_query`` |
There was a problem hiding this comment.
We use single backticks inside docstrings for inline code.
| If a ``query`` is provided here, it overrides the ``custom_cypher_query`` | |
| If a `query` is provided here, it overrides the `custom_cypher_query` |
| Translate a Haystack filter dict into an OpenCypher ``WHERE`` sub-expression. | ||
|
|
||
| Supports the full Haystack filter DSL: | ||
|
|
||
| - Logical: ``AND``, ``OR``, ``NOT`` | ||
| - Comparison: ``==``, ``!=``, ``>``, ``>=``, ``<``, ``<=`` | ||
| - Membership: ``in``, ``not in`` | ||
|
|
||
| All values are passed as named query parameters to prevent injection. | ||
|
|
||
| :param filters: A Haystack filter dictionary. | ||
| :returns: Tuple of ``(where_clause_string, params_dict)``. | ||
| :raises ValueError: If an unsupported operator or malformed filter is provided. |
There was a problem hiding this comment.
| Translate a Haystack filter dict into an OpenCypher ``WHERE`` sub-expression. | |
| Supports the full Haystack filter DSL: | |
| - Logical: ``AND``, ``OR``, ``NOT`` | |
| - Comparison: ``==``, ``!=``, ``>``, ``>=``, ``<``, ``<=`` | |
| - Membership: ``in``, ``not in`` | |
| All values are passed as named query parameters to prevent injection. | |
| :param filters: A Haystack filter dictionary. | |
| :returns: Tuple of ``(where_clause_string, params_dict)``. | |
| :raises ValueError: If an unsupported operator or malformed filter is provided. | |
| Translate a Haystack filter dict into an OpenCypher `WHERE` sub-expression. | |
| Supports the full Haystack filter DSL: | |
| - Logical: `AND`, `OR`, `NOT` | |
| - Comparison: `==`, `!=`, `>`, `>=`, `<`, `<=` | |
| - Membership: `in`, `not in` | |
| All values are passed as named query parameters to prevent injection. | |
| :param filters: A Haystack filter dictionary. | |
| :returns: Tuple of `(where_clause_string, params_dict)`. | |
| :raises ValueError: If an unsupported operator or malformed filter is provided. |
| build-backend = "hatchling.build" | ||
|
|
||
| [project] | ||
| name = "falkor-db-haystack" |
There was a problem hiding this comment.
| name = "falkor-db-haystack" | |
| name = "falkordb-haystack" |
There was a problem hiding this comment.
These changes should be reverted.
There was a problem hiding this comment.
Let's use our DocumentStoreBaseTests for testing the document store as described in our docs.
There was a problem hiding this comment.
Let's add a sentence here saying that in order to run the integration tests, a docker container needs to be run, similar to how we do for example for opensearch
|
I re-requested a review by accident. |
No worries, let me know if there's anything you're unsure about. |
… or falkor-db to falkordb for consistency, remove useless implementation, fix other small issues
…ssenzaara/haystack-core-integrations into feature/falkordb-integration
…ix and compatibility failures
|
Hi @bogdankostic, I've addressed all the feedback from the previous review. The changes should now align with Haystack's integration conventions and requirements. Please let me know if anything else needs to be adjusted. Happy to iterate further! |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
I've created a follow up issue 3219 to add the extended operations to |
Related Issues
Proposed Changes:
FalkorDBDocumentStoreto connect Haystack with FalkorDB graph databases.FalkorDBEmbeddingRetrieverfor standard vector searches.FalkorDBCypherRetrieverfor running custom GraphRAG Cypher queries.vecf32()in Cypher queries.How did you test it?
hatch run test:unit).hatch run test:integration).hatch run test:types,hatch run fmt).Notes for the reviewer
vecf32()explicit cast in theUNWINDcypher queries. This is specifically required by FalkorDB to parse vector embeddings correctly.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.