Data model with LinkML

[Silvanoc]

The Margo specification needs a single source of truth from which documentation, validation schemas, and tooling get automatically generated. This SUP proposes using LinkML to create the data model and leverage its tooling for all derived artifacts.

Any single source of truth approach increases complexity, but is necessary given the specification's inherent complexity. The success of the approach depends on the available tooling. Any approach will be perceived as wrong if the tooling is not good. Some effort is needed on good tooling.

[phil-abb]

@margo/technical-wg A new SUP has been started by @Silvanoc for having a single source of truth using LinkML data modeling.

[phil-abb]

@margo/technical-wg - This SUP to use LinkML for single-source data modeling is ready for final review. Please review the proposal and add any questions/comments to the PR.

All TWG participants have 2 weeks to start commenting.

If no comments are added or all comments are addressed within 2 weeks, we'll schedule a meeting for voting members to approve the SUP. If the comments have not been addressed within these 2 weeks, the voting will be scheduled once all comments are addressed.

[phil-abb]

I have nothing to add beyond what was commented on in the draft PR in the spec repo.

It adds complexity, but it's worth it.

[Silvanoc]

It adds complexity, but it's worth it.

I does add complexity, no doubt.

But I'm also convinced that it's worth it. Otherwise I wouldn't be investing my time on demonstrating its need and its feasibility. And honestly, I don't see an option to having some source of truth and generating out of it. For me the open point is which tooling. IMO LinkML is the best approach.

As I argue in the SUP, the key to tackle that complexity is having appropriate tooling.

[matlec]

@Silvanoc thanks for your effort to address the inconsistencies in the spec we saw prior to and during the Plugfest in March. I see the need for managing the inherent complexity of the spec, and I think LinkML is the right category of tool. But we must be careful to pair the introduction of LinkML with strong tooling UX and examples. Otherwise, I'm concerned that the learning curve associated with LinkML will slow down community contributions.

[Silvanoc]

@Silvanoc thanks for your effort to address the inconsistencies in the spec we saw prior to and during the Plugfest in March. I see the need for managing the inherent complexity of the spec, and I think LinkML is the right category of tool.

Glad to see that we agree on this.

But we must be careful to pair the introduction of LinkML with strong tooling UX and examples. Otherwise, I'm concerned that the learning curve associated with LinkML will slow down community contributions.

I don't know what you mean with "examples" apart from the partial implementation available in margo/specification#158 and the whole LinkML documentation (which provides also examples).

I fully agree on the need of tooling to lower the entry barrier. My assumption is that we all agree on the need of a single source of truth. Any of the possible alternatives that I've considered would need such tooling, in most cases the lack of tooling is much bigger than in the LinkML case (one of the main reasons why I chose LinkML).

We must differentiate between the tools needed by the people working on the content of the specification (who might need to modify the model) and those shaping the presentation (HTML, OpenAPI,...).

Nowadays I'd say that the entry barrier to modify the model is very low with the use of AI. Easy generation helps confirming that the expected output is obtained.

Shaping the presentation might be a bit tough, but this should affect much less people than those modifying the model.

[matlec]

I don't know what you mean with "examples" apart from the partial implementation available in margo/specification#158 and the whole LinkML documentation (which provides also examples).

It would be good to have a brief guide / how-to for spec authors that explains the steps and prerequisites (in tooling and knowledge) that are needed to make a change to the spec, to validate and preview the change, ... so that we lower the entry barrier to contribute - maybe even as part of an AGENTS.md.

[Silvanoc]

I don't know what you mean with "examples" apart from the partial implementation available in margo/specification#158 and the whole LinkML documentation (which provides also examples).

It would be good to have a brief guide / how-to for spec authors that explains the steps and prerequisites (in tooling and knowledge) that are needed to make a change to the spec, to validate and preview the change, ... so that we lower the entry barrier to contribute - maybe even as part of an AGENTS.md.

Fully agree on that. All of that belongs IMO to tooling to be developed if this SUP gets approved.

[phil-abb]

@margo/technical-wg - This SUP has been voted on and approved. #53