Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
ce3e658
Delete method=API from mastr.py #622
FlorianK13 Apr 24, 2025
bce5810
Delete deprecated mirror class #622
FlorianK13 Apr 24, 2025
fe466df
Delete MaStRDownload Class #622
FlorianK13 Apr 24, 2025
257dde0
Delete API from main script #622
FlorianK13 Apr 24, 2025
39af5cf
Delete empty file #622
FlorianK13 Apr 24, 2025
7a6a10a
Change docstring of Mastr class #622
FlorianK13 Apr 24, 2025
d5dbfc0
Delete unused test file #622
FlorianK13 Apr 24, 2025
a1490f9
Delete API parts from test #622
FlorianK13 Apr 24, 2025
af78c23
Commenting out the metadata functions #622
FlorianK13 Apr 24, 2025
b05431e
Repair broken test #622
FlorianK13 Apr 24, 2025
9bae59e
Reintroduce test file #622
FlorianK13 Apr 24, 2025
ee4c3f3
Fix parameter testing #622
FlorianK13 Apr 24, 2025
97c8962
Fix invalid parameter test #622
FlorianK13 Apr 24, 2025
88f9674
Add small comment to function #622
FlorianK13 Apr 24, 2025
43f2008
Delete broken test #622
FlorianK13 Apr 29, 2025
2270ff1
Delete outdated return values #622
FlorianK13 Apr 29, 2025
2e0e912
Delete unused test requirements file #622
FlorianK13 Apr 29, 2025
224030c
Delete API Mirror scripts #622
FlorianK13 Apr 29, 2025
416e6f8
Delete outdated orm.py #622
FlorianK13 Apr 29, 2025
bd8aa54
Delete Soap API download from docs #622
FlorianK13 Apr 29, 2025
1c738d7
Repair broken link in docs #622
FlorianK13 Apr 29, 2025
83ed65a
Delete Soap API download references from docs #622
FlorianK13 Apr 29, 2025
8014719
Delete unmaintained release-notes #622
FlorianK13 Apr 29, 2025
376b349
Delete outdated `sphinx` file #622
FlorianK13 Apr 29, 2025
d0cab9b
Update Changelog #622
FlorianK13 Apr 29, 2025
99d7652
Delete API specific tables #622
FlorianK13 Aug 19, 2025
2821d82
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Aug 19, 2025
c4b2911
Delete outdated parameter that survived the merge #622
FlorianK13 Aug 19, 2025
9dd939c
Repair docstrings #622
FlorianK13 Aug 19, 2025
4a21f83
Make xml download work in tests #660
FlorianK13 Aug 26, 2025
8e47564
Hopefully repair test #622
FlorianK13 Aug 27, 2025
a90d488
Dispose engine after usage #622
FlorianK13 Aug 27, 2025
da77a15
Create tranlated db directly in function #622
FlorianK13 Aug 27, 2025
3bfb683
Deactivate _translate testing #622
FlorianK13 Aug 27, 2025
808ef94
Re-add class BasicUnit to orm #622
nesnoj Oct 20, 2025
c78024c
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Oct 27, 2025
4be6399
Revert changes for delayed testing strategy #622
FlorianK13 Oct 27, 2025
3819953
Revert changes at test_mastr.py #622
FlorianK13 Oct 27, 2025
ba2291d
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Dec 5, 2025
6f229c4
Remove outdated postprocessing folder #649
FlorianK13 Dec 8, 2025
b689f61
Update docs/advanced.md #622
FlorianK13 Dec 8, 2025
82624cd
Update docs/getting_started.md #622
FlorianK13 Dec 8, 2025
bef07f0
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
e1216b6
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
5e786db
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
bb46dc5
Update open_mastr/mastr.py #622
FlorianK13 Dec 8, 2025
337e3bb
Delete unused function to check credentials #622
FlorianK13 Dec 8, 2025
6dd79dc
Update docs/advanced.md #622
FlorianK13 Dec 8, 2025
b533a05
Remove API specific constants #622
FlorianK13 Dec 8, 2025
99c8e33
Run black #622
FlorianK13 Dec 8, 2025
fab3783
Delete file with reference to postprcessing #649
FlorianK13 Dec 9, 2025
4ec3115
Delete test_requirements file #649
FlorianK13 Dec 9, 2025
dfd0cf2
Add interactive download functionality for MaStR date selection #696
fharookshaik Dec 16, 2025
e4cddae
Add tests for interactive download functionality
fharookshaik Dec 17, 2025
0f2876f
Merge branch 'develop' into feature-696-add-interactive-download
fharookshaik Dec 17, 2025
ecd8ba0
Refactor date parsing in download_xml_Mastr to improve robustness and…
fharookshaik Dec 17, 2025
192a3e8
Update Changelog with unreleased section
FlorianK13 Dec 18, 2025
ed9e31c
Merge pull request #703 from OpenEnergyPlatform/release-v0.16.1
FlorianK13 Dec 18, 2025
5a3d70e
Delete if statement that stops manual publishing
FlorianK13 Dec 18, 2025
fb3fd49
Merge pull request #707 from OpenEnergyPlatform/release-v0.16.1-manua…
FlorianK13 Dec 18, 2025
8c4e56c
Fix prompt message in select_download_date function to reflect valid …
fharookshaik Dec 21, 2025
e01cd1d
Remove manual trigger #712
FlorianK13 Jan 2, 2026
444a34c
Try to fix publishing error on test pypi #712
FlorianK13 Jan 6, 2026
c7c6c57
Use seperate environment for test pypi #712
FlorianK13 Jan 6, 2026
2531391
Change to regular env for test pypi #712
FlorianK13 Jan 6, 2026
c7b31e7
Add changelog entry #712
FlorianK13 Jan 6, 2026
4c34c39
Merge pull request #713 from OpenEnergyPlatform/712-remove-outdated-T…
FlorianK13 Jan 6, 2026
f3840df
Merge pull request #686 from OpenEnergyPlatform/649-delete-unused-files
FlorianK13 Jan 9, 2026
87e2c41
Merge branch 'develop' into 622-deprecate-api-download
FlorianK13 Jan 13, 2026
acabbda
Delete unused import #622
FlorianK13 Jan 13, 2026
329add6
Use correct date parameter #622
FlorianK13 Jan 13, 2026
2a8e26b
Merge pull request #635 from OpenEnergyPlatform/622-deprecate-api-dow…
FlorianK13 Jan 13, 2026
bf440cb
Add two more translations #714
Aimmig Jan 13, 2026
c291b37
Merge branch 'develop' into feature-696-add-interactive-download
FlorianK13 Jan 15, 2026
07226b3
Update constants.py
Aimmig Jan 20, 2026
490c1d1
Update CHANGELOG.md
Aimmig Jan 20, 2026
2993658
Merge pull request #715 from Aimmig/translation_missing
FlorianK13 Jan 21, 2026
0631a94
Merge pull request #697 from fharookshaik/feature-696-add-interactive…
FlorianK13 Feb 5, 2026
ac5e103
Add market_actors_and_roles table and update mappings in ORM
philipp-c Feb 27, 2026
ffd7126
Remove Marktrollen as per review feedback
philipp-c Mar 2, 2026
c57a8ff
Update dataset documentation to rename 'market_roles' to 'market_acto…
philipp-c Mar 3, 2026
bc7a8a1
Update references from 'market_roles' to 'market_actors_and_roles' in…
philipp-c Mar 3, 2026
297cd59
Merge pull request #722 from pclasen-eb/feature-721-adding-MarktAkteu…
FlorianK13 Mar 3, 2026
5e92130
Remove duplicated code #726
FlorianK13 Mar 26, 2026
ef8111c
Zero fill specific date strings #719
pt-kkraemer Mar 26, 2026
7107830
Add change to CHANGELOG.md
pt-kkraemer Mar 30, 2026
d148294
Fix: only pad strings with zeros and skip None
pt-kkraemer Mar 30, 2026
ecd2816
Merge pull request #728 from pt-kkraemer/719-fix-isoformat-string
pt-kkraemer Apr 2, 2026
09e3485
Update changelog #726
FlorianK13 Apr 2, 2026
b69f06b
Merge pull request #727 from OpenEnergyPlatform/726-prepare-new-minor…
FlorianK13 Apr 2, 2026
9b920b0
Bumversion commit
pt-kkraemer Apr 2, 2026
916e708
Choose different columns for test
pt-kkraemer Apr 2, 2026
ba41dec
FInal pushes for release
pt-kkraemer Apr 2, 2026
f07d640
Test fix
pt-kkraemer Apr 2, 2026
db3b363
Update changelog
pt-kkraemer Apr 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.16.1
current_version = 0.17.0
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>(a|na))+(?P<build>\d+))?
serialize =
{major}.{minor}.{patch}{release}{build}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
- name: create package
run: python -m build --sdist
- name: import open-mastr
run: python -m pip install ./dist/open_mastr-0.16.1.tar.gz
run: python -m pip install ./dist/open_mastr-0.17.0.tar.gz
- name: Create credentials file
env:
MASTR_TOKEN: ${{ secrets.MASTR_TOKEN }}
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/pypi-publish.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
name: ⚠️ Build production package and release on pypi

on:
workflow_dispatch:
release:
types: [published]

Expand Down Expand Up @@ -38,5 +37,4 @@ jobs:
--wheel
--outdir dist/
- name: Publish distribution 📦 to PyPI
if: startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@release/v1
uses: pypa/gh-action-pypi-publish@release/v1
2 changes: 1 addition & 1 deletion .github/workflows/test-pypi-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/open-mastr
url: https://test.pypi.org/p/open-mastr
permissions:
id-token: write
steps:
Expand Down
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,27 @@ For each version important additions, changes and removals are listed here.
The format is inspired from [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [v0.17.0] Full download via SOAP API deprecation and interactive download functionality - 2026-04-02
### Added
- Add interactive download functionality for MaStR date selection
[#696](https://github.com/OpenEnergyPlatform/open-MaStR/pull/696)
- Add trusted publishing for test releases
[#713](https://github.com/OpenEnergyPlatform/open-MaStR/pull/713)
- Update list for english column translation
[#715](https://github.com/OpenEnergyPlatform/open-MaStR/pull/715)
### Changed
- Replace Marktrollen with MarktakteureUndRollen
[#722](https://github.com/OpenEnergyPlatform/open-MaStR/pull/722)
- Zero fill specific date strings
[#728](https://github.com/OpenEnergyPlatform/open-MaStR/pull/728)

### Removed
- Deprecate and remove Soap API Download and Mirror code
[#635](https://github.com/OpenEnergyPlatform/open-MaStR/pull/635)
- Remove duplicated code in Mastr.download
[#727](https://github.com/OpenEnergyPlatform/open-MaStR/pull/727)


## [v0.16.1] Hotfix - 2025-12-17

### Changed
Expand Down
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ authors:
title: "open-MaStR"
type: software
license: AGPL-3.0
version: 0.16.1
version: 0.17.0
doi:
date-released: 2025-12-17
date-released: 2026-04-02
url: "https://github.com/OpenEnergyPlatform/open-MaStR/"
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ In particular, ``open-mastr`` facilitates access to the daily provided MaStR dum
No. ``open-mastr`` is a wrapper around the MaStR data and does not edit or change the data. It is intended to be used as a tool for working with the MaStR data.

Benefits provided by ``open-mastr``
==================================
===================================

.. list-table::
:widths: 30, 70
Expand Down
40 changes: 6 additions & 34 deletions docs/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,14 +155,6 @@ Moreover, the datatypes of different entries are set in the data cleansing proce

If needed, the tables in the database can be obtained as csv files. Those files are created by first merging corresponding tables (e.g all tables that contain information about solar) and then dumping those tables to `.csv` files with the [`to_csv`][open_mastr.Mastr.to_csv] method.

=== "Advantages"
* No registration for an API key is needed
* Download of the whole dataset is possible

=== "Disadvantages"
* No single tables or entries can be downloaded
* Download takes long time (you can use the partial download though, see [Getting Started](getting_started.md#bulk-download))

**Note**: By default, existing zip files in `$HOME/.open-MaStR/data/xml_download` are deleted when a new file is
downloaded. You can change this behavior by setting `keep_old_downloads`=True in
[`Mastr.download()`][open_mastr.Mastr.download].
Expand All @@ -179,12 +171,8 @@ via its API a [registration](https://www.marktstammdatenregister.de/MaStRHilfe/f
To download data from the MaStR API using the `open-MaStR`, the credentials (MaStR user and token) need to be provided in a certain way. Three options exist:

1. **Credentials file:**
Both, user and token, are stored in plain text in the credentials file.
For storing the credentials in the credentials file (plus optionally using keyring for the token) simply instantiate
[`MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] once and you get asked for a user name and a token. The
information you insert will be used to create the credentials file.

It is also possible to create the credentials file by hand, using this format:
Both, user and token, are stored in plain text in the credentials file. The file is located at
'~/.open-MaStR/config/credentials.cfg'. Fill in your user and token like this:

```
[MaStR]
Expand All @@ -194,8 +182,6 @@ To download data from the MaStR API using the `open-MaStR`, the credentials (MaS

The `token` should be written in one line, without line breaks.

The credentials file needs to be stored at: `$HOME/.open-MaStR/config/credentials.cfg`

2. **Credentials file + keyring:**
The user is stored in the credentials file, while the token is stored encrypted in the [keyring](https://pypi.org/project/keyring/).

Expand Down Expand Up @@ -419,33 +405,19 @@ For API calls, models and optional parameters refer to the
identifier (`EinheitMastrNummer`) and their power plant type (`Einheitentyp`). You can then sort them by power
plant type and use the power plant type specific API query to retrieve information about it. <br>

Cumbersome? <br>
Luckily, `open-MaStR` has you covered and provides methods to just query for all units of a power
plant type.




### MaStRDownload

The class `MaStRDownload` builds upon methods provided in the class `MaStRAPI`. <br>

It provides methods to download power plant unit types and additional information
for each unit type, such as extended unit data, permit data, chp-specific data, location data
or eeg-specific data. <br>

The class handles the querying logic and knows which additional data for each unit type is available
and which SOAP service has to be used to query it.
!!! warning "MaStRDownload has been removed"
In versions > `v0.16.0` the `MaStRDownload` class cannot be used anymore.


### MaStRMirror

The class `MaStRMirror` builds upon methods provided in the class `MaStRDownload`. <br>

The aim of the class has been to mirror the Marktstammdatenregister database and keep it up-to-date.
Historically, `open-mastr` has been developed before the owner of the dataset, BNetzA, offered the `bulk` download.
The class can still be used for use-cases where only the most recent changes to a local database are of interest.
For downloading the entire MaStR database we recommend the bulk download functionalities by specifying `donwload(method="bulk")`.
!!! warning "MaStRMirror has been removed"
In versions > `v0.16.0` the `MaStRMirror` class cannot be used anymore.



64 changes: 0 additions & 64 deletions docs/conf.py

This file was deleted.

31 changes: 0 additions & 31 deletions docs/data-release-notes.md

This file was deleted.

10 changes: 5 additions & 5 deletions docs/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Most unit information is openly accessible and is published under an open data l

For units with a net capacity of less than 30 kW, some location information is restricted from publication. This includes street names, house numbers, parcel designations, and exact coordinates of units. The most detailed location information accessible for all units is the postal code or the municipality.

In our paper titled [Analyzing Data Reliability in Germany's Energy System: A Validation of Unit Locations of the Marktstammdatenregister](https://arxiv.org/abs/2304.10581), we provide further insights into the content and quality of the dataset.
In our paper titled [Monitoring Germany's Core Energy System Dataset: A Data Quality Analysis of the Marktstammdatenregister](https://doi.org/10.1145/3717413.3717421), we provide further insights into the content and quality of the dataset.

## Content

Expand All @@ -17,12 +17,12 @@ The German Federal Network Agency regularly updates the dataset and adds new tab

## Difference between `bulk` and `API` dataset

As you may have noticed, we distinguish between `bulk` and `API` datasets. The `bulk` dataset refers to the data obtained from the zipped XML files downloaded from [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) using the [`Mastr.download`][open_mastr.Mastr.download] function. The `API` data is obtained by requesting information via the SOAP-API and the [`soap_api.download.MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] module.
As you may have noticed, we distinguish between `bulk` and `API` datasets. The `bulk` dataset refers to the data obtained from the zipped XML files downloaded from [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) using the [`Mastr.download`][open_mastr.Mastr.download] function. The `API` data is obtained by requesting information via the SOAP-API and the [`soap_api.download.MaStRAPI`][open_mastr.soap_api.download.MaStRAPI] module.

??? question "Why is the table structure in the open-mastr database as it is?"

The structure of the database is historically determined by the data retrieved via API. (open-mastr existed before the XML-dump was provided).
<br> See [MaStR data model](#Mastr-data-model)
<br> See [MaStR data model](#mastr-data-model)


## Tables in the database
Expand Down Expand Up @@ -77,7 +77,7 @@ After downloading the MaStR, you will find a database with a large number of tab
| grids | *Does not contain geoinformation* |
| locations_extended | *Connects units with grids - to get coordinates of units use the _extended tables*|
| market_actors | |
| market_roles | |
| market_actors_and_roles | |
| permit | |
| storage_units | |
| kwk | *short for: Combined heat and power (CHP)* |
Expand All @@ -86,7 +86,7 @@ After downloading the MaStR, you will find a database with a large number of tab


### MaStR data model
A useful overview of the MaStR data model can be found [here (in german)](https://www.marktstammdatenregister.de/MaStRHilfe/files/webdienst/Objektmodell%20-%20Fachliche%20Ansicht%20V1.2.0.pdf). A translated version using the names from the tables you can find in your local database is presented here:
A useful overview of the MaStR data model can be found at the MaStR [help page](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/faq.html). A translated version using the names from the tables you can find in your local database is presented here:

=== "translated image (english)"
![Data model of the MaStR](images/DetailAnlagen_english.PNG)
Expand Down
12 changes: 2 additions & 10 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,17 +50,9 @@ More detailed information can be found in the section [bulk download](advanced.m

API download
-----------------------------------
When using `download(method="API")`, the data is retrieved from the MaStR API. For using the MaStR API, credentials
are needed (see [SOAP API download](advanced.md#soap-api-download)).
!!! warning "API download has been removed"
In versions > `v0.16.0` the `download(method="API")` cannot be used anymore. The default behaviour changes `method="API"` to `method="bulk"`. You can still use the basic SOAP API functionalities from the [`MaStRAPI`][open_mastr.soap_api.download.MaStRAPI] class.

```python
from open_mastr import Mastr

db = Mastr()
db.download(method='API')
```

The default settings will save retrieved data into the sqlite database. The function can be used to mirror the open-MaStR database using the SOAP API. Note that the data provided by the bulk download and the SOAP API is similar, but not exactly the same.

## Accessing the database

Expand Down
2 changes: 0 additions & 2 deletions docs/reference/advanced.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# Advanced functions to use the MaStR SOAP-API

::: open_mastr.soap_api.download.MaStRAPI
::: open_mastr.soap_api.download.MaStRDownload
::: open_mastr.soap_api.mirror.MaStRMirror
41 changes: 0 additions & 41 deletions main.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
open-MaStR - Main file

Bulk: Download XML-Dump and fill in local SQLite database.
API: Download latest entries using the SOAP-API.

SPDX-License-Identifier: AGPL-3.0-or-later
"""
Expand Down Expand Up @@ -45,34 +44,6 @@
"permit",
]

# API download
# for parameter explanation see: https://open-mastr.readthedocs.io/en/latest/advanced/#soap-api-download

api_date = "latest"
api_chunksize = 10
api_limit = 10
api_processes = None

data_api = [
"biomass",
"combustion",
"gsgk",
"hydro",
"nuclear",
"solar",
"storage",
"wind",
]

api_data_types = ["unit_data", "eeg_data", "kwk_data", "permit_data"]

api_location_types = [
"location_elec_generation",
"location_elec_consumption",
"location_gas_generation",
"location_gas_consumption",
]

if __name__ == "__main__":
# instantiate Mastr class
db = Mastr()
Expand All @@ -81,18 +52,6 @@
# bulk download
db.download(method="bulk", data=data_bulk, date=bulk_date, bulk_cleansing=True)

# API download
db.download(
method="API",
data=data_api,
date=api_date,
api_processes=api_processes,
api_limit=api_limit,
api_chunksize=api_chunksize,
api_data_types=api_data_types,
api_location_types=api_location_types,
)

## export to csv
"""
Technology-related tables are exported as joined, whereas additional tables
Expand Down
Loading
Loading