Skip to content

Issue #39 | ClickhouseTable: integration tests#43

Open
IceKhan13 wants to merge 19 commits intoneuralinkcorp:mainfrom
IceKhan13:chore/ci-integration-tests
Open

Issue #39 | ClickhouseTable: integration tests#43
IceKhan13 wants to merge 19 commits intoneuralinkcorp:mainfrom
IceKhan13:chore/ci-integration-tests

Conversation

@IceKhan13
Copy link
Copy Markdown
Contributor

@IceKhan13 IceKhan13 commented Aug 3, 2025

Summary

Integration tests for Clickhouse table

Details

  1. Integration tests for Clickhouse
  2. Separate folder for functional and integration tests
  3. This PR bring new dependency to project which is https://ibis-project.org/. It supports ~20 different backends, which will give ability to expand datarepo tables to more engines like spark, druid, postgres, etc. Moreover it has nice SQL query builder which might cover needs for Use sql builder to construct queries #9 (I can cover SQL builder and more backends in follow up PRs). And it's pretty lightweight.

P.S.
Connectionx was also pain to work with as polars + connectionx + clickhouse has a lot of conflicting dependencies (and it looks like it was working through mysql anyway).

@IceKhan13 IceKhan13 marked this pull request as ready for review August 8, 2025 02:32
@houqp
Copy link
Copy Markdown
Collaborator

houqp commented Aug 12, 2025

@IceKhan13 is this PR ready for review?

@IceKhan13
Copy link
Copy Markdown
Contributor Author

IceKhan13 commented Aug 12, 2025

@houqp yes, it is!
Sorry, I do not have rights in this repo to request reviews :)

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces integration tests for ClickHouse table functionality and reorganizes the test structure. The implementation adds a new dependency (ibis-framework) to replace connectionx for database connectivity, which provides better compatibility and broader backend support.

  • Adds comprehensive integration tests for ClickHouseTable with Docker Compose setup
  • Reorganizes test structure separating functional and integration tests
  • Replaces polars.read_database_uri with ibis for ClickHouse connectivity

Reviewed Changes

Copilot reviewed 8 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
test/integration/test_clickhouse_table_int.py New integration test suite with comprehensive ClickHouse table testing scenarios
test/integration/resources/docker-compose.yml Docker Compose configuration for ClickHouse test environment
test/functional/test_core.py Updated import paths to reflect new test structure
test/functional/tables/test_clickhouse_table.py Updated mocks to use ibis instead of polars.read_database_uri
test/functional/roapi/test_roapi.py Updated import paths for new test structure
src/datarepo/core/tables/clickhouse_table.py Replaced polars.read_database_uri with ibis.connect for database queries
pyproject.toml Added ibis-framework and clickhouse-driver dependencies
.github/workflows/build_and_publish.yml Added integration test workflow with Docker Compose setup

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants