Work around NLTK v3.9.3 regression (#2042)

In NLTK v3.9.3, a new security check was added for the Zip Slip issue:
nltk/nltk#3468

This change unfortunately contained a bug that causes false positive
security errors when packages are downloaded to a symlinked path,
since the new check doesn't use `abspath` vs `realpath` consistently:
nltk/nltk#3509

This causes errors like:

```
-----> Downloading NLTK packages: punkt punkt_tab
       <frozen runpy>:128: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour
       [nltk_data] Downloading package punkt to
       [nltk_data]     /app/.heroku/python/nltk_data...
       [nltk_data]   Unzipping tokenizers/punkt.zip.
       [nltk_data] Zip Slip blocked: punkt/
       Error installing package. Retry? [n/y/e]
       Traceback (most recent call last):
         File "<frozen runpy>", line 198, in _run_module_as_main
         File "<frozen runpy>", line 88, in _run_code
         File "/app/.heroku/python/lib/python3.12/site-packages/nltk/downloader.py", line 2631, in <module>
           rv = downloader.download(
                ^^^^^^^^^^^^^^^^^^^^
         File "/app/.heroku/python/lib/python3.12/site-packages/nltk/downloader.py", line 773, in download
           choice = input().strip()
                    ^^^^^^^
       EOFError: EOF when reading a line
```

See also:
https://github.com/heroku/heroku-buildpack-python/actions/runs/22545824206/job/65308000022#step:5:539

Until upstream fix this regression, we can work around it by always
passing the raw path for Python home instead of the symlinked path.
(We generally prefer using the symlinked path where possible, to
ensure the paths used/displayed match between build time and
run time, given the build directory is different from the run directory.)

Longer term I will be deprecating and that removing support for `nltk.txt`
since it's a Heroku-proprietary invention that's unnecessary given most
users instead either commit the NLTK corpora or use the post_compile
hook feature instead.

Fixes #2037.
GUS-W-21410908.

Author: edmorley

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 
+- Added a workaround for `nltk.txt` package downloader errors caused by an upstream regression in NLTK v3.9.3. ([#2041](https://github.com/heroku/heroku-buildpack-python/pull/2041))
 - Changed the S3 URL used to download Python to use AWS' dual-stack (IPv6 compatible) endpoint. ([#2035](https://github.com/heroku/heroku-buildpack-python/pull/2035))
 
 ## [v335] - 2026-02-10

bin/steps/nltk

@@ -25,10 +25,11 @@ if is_module_available 'nltk'; then
 		readarray -t nltk_packages <"${nltk_packages_definition}"
 		output::step "Downloading NLTK packages: ${nltk_packages[*]}"
 
-		nltk_data_dir="/app/.heroku/python/nltk_data"
-
+		# Note: We have to use the raw build directory path here and not the symlinked `/app` path,
+		# otherwise it will cause a false positive in NLTK v3.9.3's new Zip Slip security check,
+		# which doesn't handle symlinked paths correctly: https://github.com/nltk/nltk/issues/3509
 		# TODO: Does this even need user-provided env vars, or can we remove the sub_env usage here?
-		if ! sub_env python -m nltk.downloader -d "${nltk_data_dir}" "${nltk_packages[@]}" |& output::indent; then
+		if ! sub_env python -m nltk.downloader -d "${BUILD_DIR}/.heroku/python/nltk_data" "${nltk_packages[@]}" |& output::indent; then
 			output::error <<-EOF
 				Error: Unable to download NLTK data.
 
@@ -41,7 +42,9 @@ if is_module_available 'nltk'; then
 			exit 1
 		fi
 
-		set_env NLTK_DATA "${nltk_data_dir}"
+		# Since this will be used at runtime, we must use the symlinked `/app` path and not
+		# the raw build directory path.
+		set_env NLTK_DATA "/app/.heroku/python/nltk_data"
 	else
 		build_data::set_string "nltk_downloader" "skipped-no-nltk-file"
 		echo "       'nltk.txt' not found, not downloading any corpora"

spec/hatchet/nltk_spec.rb

@@ -13,10 +13,10 @@
           remote: -----> Downloading NLTK packages: city_database stopwords
           remote:        .*: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour
           remote:        \\[nltk_data\\] Downloading package city_database to
-          remote:        \\[nltk_data\\]     /app/.heroku/python/nltk_data...
+          remote:        \\[nltk_data\\]     /tmp/build_.+/.heroku/python/nltk_data...
           remote:        \\[nltk_data\\]   Unzipping corpora/city_database.zip.
           remote:        \\[nltk_data\\] Downloading package stopwords to
-          remote:        \\[nltk_data\\]     /app/.heroku/python/nltk_data...
+          remote:        \\[nltk_data\\]     /tmp/build_.+/.heroku/python/nltk_data...
           remote:        \\[nltk_data\\]   Unzipping corpora/stopwords.zip.
         REGEX