Remove flatdict dependency by replacing fiboa-cli with lightweight parquet utilities

[Copilot]

flatdict has build issues with setuptools. It's pulled in as a transitive dependency from fiboa-cli==0.7.

Changes

New module: ftw_tools/parquet_utils.py

• features_to_dataframe(): Converts GeoJSON features to GeoDataFrame
• create_parquet(): Writes GeoDataFrame to Parquet with fiboa metadata injection

Updated imports

• ftw_tools/inference/utils.py
• ftw_tools/postprocess/polygonize.py

Dependencies

• Removed fiboa-cli==0.7 from pyproject.toml

Approach

The codebase only uses two functions from fiboa-cli: create_parquet and features_to_dataframe. Newer fiboa-cli versions (≥0.20.0) dropped flatdict but underwent a major rewrite with breaking API changes.

Rather than migrating to the incompatible API, implemented focused replacements (~80 LOC) that:

• Use geopandas' native parquet writer for geometry serialization
• Inject fiboa metadata via PyArrow schema metadata post-write
• Maintain exact API compatibility with existing call sites

# Replacement implementation
def create_parquet(data, columns, collection, output_file, config=None, compression=None):
    data[columns].to_parquet(output_file, compression=compression or "zstd", index=False)
    
    table = pq.read_table(output_file)
    metadata = table.schema.metadata or {}
    metadata[b"fiboa"] = json.dumps(collection).encode("utf-8")
    pq.write_table(table.replace_schema_metadata(metadata), output_file, compression=compression)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.lightly.ai
  • Triggering command: /home/REDACTED/work/ftw-baselines/ftw-baselines/.venv/bin/pytest pytest tests/ -k not integration and not instance_segmentation -v (dns block)
  • Triggering command: /home/REDACTED/work/ftw-baselines/ftw-baselines/.venv/bin/pytest pytest tests/ -k not integration and not instance_segmentation --ignore=tests/test_eval.py --ignore=tests/test_model_registry.py --ignore=tests/test_models.py -v (dns block)
• astral.sh
  • Triggering command: /usr/bin/curl curl -LsSf REDACTED (dns block)
• data.source.coop
  • Triggering command: /home/REDACTED/work/ftw-baselines/ftw-baselines/.venv/bin/pytest pytest tests/ -k not integration and not instance_segmentation --ignore=tests/test_eval.py --ignore=tests/test_model_registry.py --ignore=tests/test_models.py -v (dns block)
• planetarycomputer.microsoft.com
  • Triggering command: /home/REDACTED/work/ftw-baselines/ftw-baselines/.venv/bin/pytest pytest tests/ -k not integration and not instance_segmentation --ignore=tests/test_eval.py --ignore=tests/test_model_registry.py --ignore=tests/test_models.py -v (dns block)
• sentinel-cogs.s3.us-west-2.amazonaws.com
  • Triggering command: /home/REDACTED/work/ftw-baselines/ftw-baselines/.venv/bin/pytest pytest tests/ -k not integration and not instance_segmentation --ignore=tests/test_eval.py --ignore=tests/test_model_registry.py --ignore=tests/test_models.py -v (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Remove flatdict dependency</issue_title>
<issue_description>flatdict seems to have build issues with setup tools. We should consider removing this dependency with another convenience method.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Remove flatdict dependency #234

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[isaaccorley]

@copilot can we instead just upgrade the fiboa-cli which doesn't have flatdict dependency anymore?

[m-mohr]

No, this doesn't work yet.
We just removed flatpack today and the ftw-baseline still uses a very old version of fiboa-cli, we need to update to the latest, but it has a lot of breaking changes. It's on my todo, but not for today. ;-)

[Copilot]

@copilot can we instead just upgrade the fiboa-cli which doesn't have flatdict dependency anymore?

fiboa-cli >= 0.20.0 removed flatdict but underwent a complete rewrite that removed the create_parquet() and features_to_dataframe() functions we use. The new version is built on vecorel-cli with an incompatible object-oriented API.

Upgrading would require refactoring both call sites in ftw_tools/inference/utils.py and ftw_tools/postprocess/polygonize.py to use the new architecture.

The current lightweight implementation (~80 LOC) maintains exact API compatibility while removing the flatdict dependency. Would you prefer I refactor to use fiboa-cli 0.20.3's new API instead?

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/search/issues
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[m-mohr]

This will not create fiboa-compliant files anymore. fiboa CLI has some issues yet with being imported as library, so needs upstream work first.

[isaaccorley]

This will not create fiboa-compliant files anymore. fiboa CLI has some issues yet with being imported as library, so needs upstream work first.

Sounds good, saw the build errors in other PRs was just looking to debug but thanks for the heads up

[m-mohr]

I'll try to get to it next week.

[m-mohr]

Colsing in favor of #241