Error converting comma-separated values "Katalogwerte" in cleansing bulk

[pclasen-eb]

Description of the issue

Running the default Mastr download() currently results in an error processing certain files - as of March 8th

• Error processing file 'Netze.xml': 'could not convert string to float: '334, 335''
• Error processing file 'EinheitenVerbrennung.xml': 'could not convert string to float: '2442, 2442''

As a result, the corresponding sqlite tables are empty.
The issue seems to be caused by the dtype check for "O" in replace_mastr_katalogeintraege. This check does not match the pandas string dtype introduced in pandas>=3.0

If the suggested solution outlined below aligns with your expectations, I would be happy to prepare a pull request.

Please let me know if there are any additional implications or edge cases that I may have overlooked.

Steps to Reproduce

1. run default db.download()
2. check if grids or combustion_extended tables in sqlite are empty

Ideas of solution

• adjust the if statement in replace_mastr_katalogeintraege (utils_cleansing_bulk.py) to include pandas string_dtype, i.e. instead of if df[column_name].dtype == "O":
• use if (pd.api.types.is_string_dtype(df[column_name]) or pd.api.types.is_object_dtype(df[column_name])):
• as the object dtype check is still included it should work the same as before with older pandas versions

Context and Environment

• Version used: v0.16.1 and latest commit on develop 297cd59 - specifically with pandas 3.0.1
• Operating system: unix/mac os
• Environment setup and (python) version: python 3.12, pandas 3.0.1

Workflow checklist

• I have checked the documentation and confirmed this issue is not already addressed there.
• I am aware of the workflow in CONTRIBUTING.md

[FlorianK13]

Hi @pclasen-eb - in #718 there is a complete rework of the database initialization and data loading. The problem with comma seperated values is known and if I'm correct, it is fixed in #718

I think the best way is to use all available resources to get #718 merged as soon as possible and then check again if this error still comes up.