Skip to content

Added OpenMeteo Data Fetcher for gridded weather data#204

Open
Sidharth1743 wants to merge 2 commits intoopenclimatefix:mainfrom
Sidharth1743:openmeteo-data-integration
Open

Added OpenMeteo Data Fetcher for gridded weather data#204
Sidharth1743 wants to merge 2 commits intoopenclimatefix:mainfrom
Sidharth1743:openmeteo-data-integration

Conversation

@Sidharth1743
Copy link
Copy Markdown
Contributor

Pull Request

Description

This PR implements the OpenMeteoWeatherDataFetcher to enable fetching gridded weather data from the OpenMeteo API
This fetcher is designed to integrate seamlessly with the Graph Weather pipeline by returning properly structured in xarray format
Dataset objects : Latitude, Longitude, Time

Generates lat/lon meshgrids and aggregates batched API responses into a single 3D xarray.Dataset
Supports both Forecast and Historical OpenMeteo endpoints
Includes a token bucket RateLimiter (respecting free-tier limits) and exponential backoff retries
Includes Zarr based caching and threaded parallel fetching
Supports custom NWP models, resolution, and hourly parameters

Fixes #90

How Has This Been Tested?

test suite in tests/data/test_openmeteo_fetcher.py

Grid Verification: Verified fetch_forecast returns data with multiple latitude/longitude points (gridded) and correct dimensions
Structure: Verified output is an xarray.Dataset with latitude, longitude, and time_utc coordinates
Logic: Verified rate limiting, batching (50 coords/request), and date validation logic
[x] Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?
[x] Yes

Checklist:

  • [ x] My code follows OCF's coding style guidelines
  • [ x] I have performed a self-review of my own code
  • [ x] I have made corresponding changes to the documentation
  • [ x] I have added tests that prove my fix is effective or that my feature works
  • [ x] I have checked my code and corrected any misspellings

@Sidharth1743
Copy link
Copy Markdown
Contributor Author

@jacobbieker i have made some changes and includes only some of the parameters from openmeteo api. is that okay? or need to include all of the params

@Sidharth1743
Copy link
Copy Markdown
Contributor Author

also in the test i have only mocked the api. is this approach OK here or would you prefer an integration test with the real API as well?

@Sidharth1743
Copy link
Copy Markdown
Contributor Author

@jacobbieker pls review this

Copy link
Copy Markdown
Member

@jacobbieker jacobbieker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! As for testing the real API, I think its fine mocking it for now, as we don't want to hit the API too much in tests.

) -> None:
"""Validate latitude and longitude ranges."""
if lat_range[0] >= lat_range[1]:
raise ValueError("lat_range[0] must be less than lat_range[1]")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the values in the error, in something like the following

Suggested change
raise ValueError("lat_range[0] must be less than lat_range[1]")
raise ValueError(f"lat_range[0] {lat_range[0]} must be less than lat_range[1] {lat_range[1]}")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for using OpenMeteo Open Dataset for training/inference

2 participants