Skip to content

213 : ExtractionModule & ExtractionChain implementation and integrations#633

Open
ibrodkin wants to merge 37 commits intodevelopfrom
213-dual-rdma-extractor
Open

213 : ExtractionModule & ExtractionChain implementation and integrations#633
ibrodkin wants to merge 37 commits intodevelopfrom
213-dual-rdma-extractor

Conversation

@ibrodkin
Copy link
Copy Markdown
Collaborator

@ibrodkin ibrodkin commented May 5, 2026

this PR contains the code changes for the suite of issues grouped under issue #213

  1. chronolog::StoryChunkExtractionModule
  • new implementation replaced packed argument list with the variant array of templated extractors. the change allows for better performance with continouis memory access and easier maintenance
  • ExtractionChain is now a template
  • initialize method takes ExtractionConfiguration as generic input
  1. chronolog::ExtractionChainConfiguration is implemented in generic way so that it can carry individual extractors json objects without knowing any details about them
  2. test/unit/chrono-common contain unit tests for ExtractionConfiguration and StoryChunkExtractionModule
  3. ChunkLoggingExtractor, ChunkExtractorCSV, & ChunkExtractorRDMA reworked to fit the template expected by new StoryChunkExtractionModule. They are used by several chronolog processes so they are part of chrono_common
  4. DualEndpointChunkExtractoRDMA implements RDMA based StoryChunk extraction to both grapher and player consumers, intended for use by only the chrono-keepers
  5. ChronoKeeperExtractionChain implementation allows to combine LoggingExtractor, ChunkExtractorCSV, ChunkExtractorRDMA, and DualEndpointChunkExtractorRDMA
  6. ChronoKeeperInstance.cpp integrates StoryChunkExtractionModule into the chrono-keeper process
  7. HDF5FileChunkExtractor modified to fit the template expected by the new StoryChunkExtractionModule
  8. ChronoGrapherExtractionChain implementation allows to combine LoggingExtractor, ChunkExtractorCSV, HDF5FileChunkExtractor
  9. ChronoGrapher.cpp integrates StoryChunkExtractionModule into the chrono-grapher process
  10. default_conf.json.in was modified to replace old blocks with the new ExtractionModule blocks for ChronoKeeper and ChronoGrapher
  11. deploy_local.sh and deploy_cluster.sh were modified to generate keeper and grapher configuration files with the new ExtractionModule json blocks
  12. added test/functional section for testing chrono-keeper and chrono-graphrer extraction modules

Please, note that there's a follow-up #635 that will have a separate PR

ibrodkin and others added 29 commits November 20, 2025 14:23
…ialization

- Added a destructor to ConfigurationManager to ensure proper cleanup of JSON configuration objects.
- Refactored the constructor to initialize all configuration pointers to nullptr and load configurations from a JSON file.
- Updated condition checks in LoadConfFromJSONFile to use logical AND instead of OR for better validation of JSON object types.
- Updated the return values in the parseJsonConf method across multiple configuration files (ChronoGrapher, ChronoKeeper, ChronoPlayer, ChronoVisor) to use the chronolog::CL_SUCCESS constant instead of a hardcoded value of 1 for improved clarity and consistency in success indication.
- Adjusted the deployment script to reflect changes in configuration paths for better maintainability.
- Replaced assert statements with conditional checks in the parseJsonConf methods across ChronoGrapher, ChronoKeeper, ChronoPlayer, and ChronoVisor configuration files to improve error handling.
- Added error messages to standard output for invalid configuration types, ensuring clearer feedback for configuration issues.
- Updated return values to use chl::CL_ERR_INVALID_CONF for consistency in error reporting.
- Added conditional checks in the parseJsonConf methods for CLOCK_CONF and AUTH_CONF to handle parsing errors more gracefully.
- Introduced error messages to standard output for invalid configurations, enhancing feedback for users.
- Ensured consistent exit behavior on parsing failures by utilizing chronolog::CL_ERR_INVALID_CONF.
…, ExtractionChain is using variant array instead of argument pack; new KeeperExtractionConfiguration; new KeeperExtractionChain implementation; new GrapherExtractionConfiguration & new GrapherExtractionChain implementation
… and Grapher; added tests/functional/chrono-grapher
@ibrodkin ibrodkin marked this pull request as draft May 5, 2026 00:37
@grc-iit grc-iit deleted a comment from github-actions Bot May 5, 2026
@github-actions

This comment was marked as resolved.

@ibrodkin ibrodkin requested review from fkengun and iameneko May 6, 2026 21:48
@ibrodkin ibrodkin marked this pull request as ready for review May 6, 2026 21:49
Copy link
Copy Markdown
Contributor

@iameneko iameneko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me know your thoughts on my comments

Comment thread ChronoKeeper/src/DualEndpointChunkExtractorRDMA.cpp Outdated
Comment thread ChronoKeeper/src/DualEndpointChunkExtractorRDMA.cpp Outdated
Comment thread ChronoKeeper/include/DualEndpointChunkExtractorRDMA.h Outdated
Comment thread ChronoKeeper/include/ChronoKeeperConfiguration.h Outdated
@iameneko iameneko self-requested a review May 7, 2026 15:25
Comment thread chrono_common/include/StoryChunkExtractionModule.h Outdated
bool is_active() const { return (nullptr != rdma_sender); }

private:
tl::engine& sender_tl_engine; // local tl::engine
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tl::engine& cannot be rebound, so the copy-assignment at ChunkExtractorRDMA.cpp:75

sender_tl_engine = other.get_sender_engine();

calls tl::engine::operator= on the externally-owned engine itself, not on the local member. Thallium's copy-assign explicitly runs ~engine() on the LHS before swapping m_impl (thallium/engine.hpp:276-281) — if that's the last reference, finalize() runs on the live process engine; otherwise the upstream engine silently points at a different impl. Either way the "borrowed reference" contract is broken.

Recommend changing the member to tl::engine* sender_tl_engine; so copy-assign rebinds the pointer instead of mutating the engine. Same fix needed in DualEndpointChunkExtractorRDMA.h:40 / DualEndpointChunkExtractorRDMA.cpp:157.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see . so the engine is silently finalized .. nice...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make ExtractionChain composition configurable Implement Dual Endpoint RDMA Story ChunkExtractor

3 participants