Simplify data setup

[maedoc]

This adds a simple way to get the demo dataset from Zenodo w/o the wget/unzip stuff or missing data with the pip package. It's currently in the library part but could be anywhere, @liadomide thoughts?

[i-Zaak]

This might be helpful: https://github.com/fatiando/pooch

[liadomide]

This adds a simple way to get the demo dataset from Zenodo w/o the wget/unzip stuff or missing data with the pip package. It's currently in the library part but could be anywhere, @liadomide thoughts?

I appreciate the suggestion; I think this is a good idea.
Maybe we put it inside tvb-library package so that after a pip install tvb-library we have it at hand in a user's env for other modules depending on tvb-library (e.g. tvb-framework) to use ?

[i-Zaak]

100% agree with @liadomide - tvb-library would be the most intuitive place.

[liadomide]

This might be helpful: https://github.com/fatiando/pooch

@i-Zaak what exactly do you have in mind? to publish also with pooch, thus keep the data replicated in 2 places or replace Zenodo entirely?

[i-Zaak]

I'd use pooch to fetch the zip file from zenodo, verify checksum and unzip.

I recently started using it for the EBRAINS datasets:

_ = pooch.retrieve(
    url='https://object.cscs.ch/v1/AUTH_227176556f3c4bb38df9feea4b91200c/hbp-d000059_Atlas_based_HCP_connectomes_v1.1_pub/200-Schaefer17Networks.zip',
    known_hash='5086f4b3405acff84ffe132cee17c67a90000a3fae98da50d4e14fb55d7f5d57',
    path='some/path/where/the/data/lives',
    processor=pooch.Unzip(extract_dir='.')
)

[maedoc]

I'm onboard for pooch but I'd just like to get #633 done first so that it's easier to add the dependency.

[maedoc]

keep the data replicated in 2 places or replace Zenodo entirely?

On a longer term view, it could be of interest to keep the Zenodo for the DOI but have the primary source be a Datalad repository which stores significant demo datasets, simulated data etc, that could help jumpstart users' projects. For instance, having 100 HCP subjects preprocessed, with some basic simulations done could be a huge benefit.

On short term, having an easy fetcher for our Zenodo but also Ebrains is very nice.

[i-Zaak]

We could start incorporating various data sources in the TVB data (datalad, zenodo, EBRAINS) similarly to how MNE is handling datasets: https://mne.tools/stable/overview/datasets_index.html

In the end, there should be minimal data contained in the tvb-data package itself, but it should have code to get those. The condition for priority of inclusion would be for me public availability to avoid dealing with tokens and other auth interfaces (exception for EBRAINS HDG, that is known beast).

Another candidate to be merged in: https://gitlab.ebrains.eu/fousekjan/tvb-ebrains-data

[maedoc]

@liadomide do you think this PR is still relevant, given the work in #691 ?

(btw do you have a link to the issue TVB-1999? I cannot find it at req.thevirtualbrain.org)

[romina1601]

@liadomide do you think this PR is still relevant, given the work in #691 ?

(btw do you have a link to the issue TVB-1999? I cannot find it at req.thevirtualbrain.org)

@maedoc here is the link for TVB-1999: https://tvb-projects.atlassian.net/browse/TVB-1999

[liadomide]

@maedoc I think it is safe to close this, and leave the #691 for a final review and integration.
This was a good base for #691

[maedoc]

close in favor of #691