ci: add release branch compatibility check to PR validation

[smamindl]

Adds a ReleaseBranchCompat job to the ADO pipeline that catches release branch breakage at PR time, before changes land in master.

What it does

On every PR to master, for each release branch (starting with spark4.0):

1. Rebase — replays the release branch's patches onto the PR HEAD
2. Compile — runs sbt compile test:compile on the rebased code
3. Unit tests — runs 13 non-service-dependent test packages (core, automl, causal, featurize, image, isolationforest, stages,
   recommendation, nn, train, vw, opencv, exploratory)

If rebase conflicts or compile/tests fail, the PR author sees a warning while they're still in context.

Design decisions

• Non-blocking (continueOnError: true) — doesn't gate PR merges
• Matrix-based — easy to add spark3.5, spark4.1 later
• Java version aware — spark4.0 uses JDK 17 via matrix variable
• Master-only — skips PRs targeting release branches
• ~36 min total, runs in parallel — no impact on overall pipeline duration

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Briefly describe the changes included in this Pull Request.

How is this patch tested?

CI validated on builds 215268893 (all steps passed) and 215256519.

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.
• Yes. Make sure you have added samples following below steps.

1. Find the corresponding markdown file for your new feature in website/docs/documentation folder.
   Make sure you choose the correct class estimators/transformers and namespace.
2. Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
3. Make sure the DocTable points to correct API link.
4. Navigate to website folder, and run yarn run start to make sure the website renders correctly.
5. Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
6. Make sure the WebsiteSamplesTests job pass in the pipeline.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) require an authorized user to comment /azp run to run.

[github-actions]

Hey @smamindl 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[smamindl]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Adds a new Azure DevOps PR-validation job to check whether release branches (starting with spark4.0) can be cleanly replayed onto incoming PR changes and still build, aiming to catch release-branch breakage before merging to master.

Changes:

• Adds a ReleaseBranchCompat job that fetches a release branch and attempts to rebase its unique commits onto the PR HEAD.
• Runs sbt compile test:compile after the rebase.
• Adds additional setup and unit-test execution steps for multiple packages, and publishes test results.

[Copilot]

The unit-test loop invokes sbt once per package, which repeatedly pays sbt startup/resolve costs and increases the chance of hitting the 30/60 minute timeouts. Consider running tests in fewer sbt invocations (e.g., a single sbt session with multiple testOnly commands) or using the existing UnitTests matrix approach instead of a loop.

Suggested change

	FAILURES=0
	for pkg in core automl causal featurize image isolationforest stages recommendation nn train vw opencv exploratory; do
	echo "=== Testing $pkg ==="
	if ! timeout 10m sbt $(SBT_JAVA_OPTS) "testOnly com.microsoft.azure.synapse.ml.$pkg.**"; then
	echo "##vso[task.logissue type=warning]$pkg tests failed on $(RELEASE_BRANCH)"
	FAILURES=$((FAILURES + 1))
	fi
	done
	if [ $FAILURES -gt 0 ]; then
	echo "##vso[task.logissue type=warning]$FAILURES package(s) failed on $(RELEASE_BRANCH)"
	PACKAGES=(core automl causal featurize image isolationforest stages recommendation nn train vw opencv exploratory)
	SBT_COMMANDS=()
	for pkg in "${PACKAGES[@]}"; do
	echo "=== Queueing tests for $pkg ==="
	SBT_COMMANDS+=("testOnly com.microsoft.azure.synapse.ml.$pkg.**")
	done
	if ! timeout 25m sbt $(SBT_JAVA_OPTS) "${SBT_COMMANDS[@]}"; then
	echo "##vso[task.logissue type=warning]Unit tests failed on $(RELEASE_BRANCH)"