Skip to content

Add URL support to DataAccessor #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
whyjz wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add URL support to DataAccessor #245

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4eb36a9
Add URL support to DataAccessor (.vrt in a local folder if filename i…
whyjz Mar 3, 2021
53c492e
Update components/iscesys/ImageApi/DataAccessor/DataAccessorPy.py
whyjz Mar 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion components/iscesys/ImageApi/DataAccessor/DataAccessorPy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,12 @@ def createAccessor(self):
caster = '' or self.caster
filename = self.filename
scheme = self.scheme
self.extraFilename = self.filename + '.' + self._extra_reader
#if the filename is a URL, the extraFilename should indicate the file from the local machine
#instead of from the remote server.
if self.filename.startswith('http'):
Copy link
Copy Markdown
Member

@rtburns-jpl rtburns-jpl Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if self.filename.startswith('http'):
if self.filename.startswith('http://') or self.filename.startswith('https://'):

It may be possible someone's filename starts with http, but I wouldn't expect a local filename to contain //

Copy link
Copy Markdown
Contributor

@piyushrpt piyushrpt Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not create a .vrt and .xml and have source in vrt to point to remote? That way you have a record of what resource was actually used? That way you are not restricted to something thats just http / https. You can throw in anything vsis3 / vsiaz/ vsicurl etc. This approach still seems to need a .vrt. Just using basename of .vrt should be enough when it points to other things?

Copy link
Copy Markdown
Author

@whyjz whyjz Mar 4, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piyushrpt I did create a .vrt that points to a remote geotiff. In my use case, the XML is not needed. Here's an example of what would happen if running this snippet using the original code version:

import isce
from isceobj.Image.Image import Image
obj = Image()
obj.setFilename("https://download.osgeo.org/geotiff/samples/GeogToWGS84GeoKey/GeogToWGS84GeoKey5.tif")
obj.setWidth(101)
obj.setDataType('SHORT')
obj.setCaster('read', 'FLOAT')
obj.createImage()
# ... (continue to some Ampcor code)

GDAL open (R): https://download.osgeo.org/geotiff/samples/GeogToWGS84GeoKey/GeogToWGS84GeoKey5.tif.vrt
ERROR 1: HTTP error code : 404
Error. Cannot open the file https://download.osgeo.org/geotiff/samples/GeogToWGS84GeoKey/GeogToWGS84GeoKey5.tif.vrt in read mode.
Error in file /home/whyj/Software/isce2-2.4.2-build/isce/components/iscesys/ImageApi/InterleavedAccessor/src/GDALAccessor.cpp at line 77 Exiting

You can see that the ISCE's default behavior is to read a VRT that is at the same location as the original file, and it does not matter whether you have created a VRT on your local machine or not. If we replace the geotiff file name with the local vrt, ISCE will try to gdal open .vrt.vrt, which does not exist.

self.extraFilename = os.path.basename(self.filename) + '.' + self._extra_reader
else:
self.extraFilename = self.filename + '.' + self._extra_reader

if self._accessor is None:#to avoid creating duplicates
selection = self.methodSelector()
Expand Down