Skip to content

Fix continuous reconciliation loop for NetworkConnectivityInternalRange - #12615

Merged
ldanielmadariaga merged 8 commits into
GoogleCloudPlatform:masterfrom
barney-s:12500
Aug 28, 2026
Merged

ldanielmadariaga merged 8 commits into
GoogleCloudPlatform:masterfrom
barney-s:12500

Conversation

@barney-s

@barney-s barney-s commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #12500

Root Cause Analysis

When a NetworkConnectivityInternalRange resource was created with spec.allocationOptions or spec.targetCIDRRange, subsequent reconcile loops immediately failed with:
googleapi: Error 400: Unsupported update mask path: "allocation_options": invalid argument

The issue was caused by:

  1. convertAPIToProto Deserialization Failure:

    • convertAPIToProto in pkg/controller/direct/networkconnectivity/utils.go used standard json.Unmarshal(j, &v) instead of protojson.Unmarshal.
    • The Go client library api.InternalRange serializes JSON with camelCase tags (allocationOptions, prefixLength, targetCidrRange), whereas protobuf-generated structs have snake_case tags (allocation_options, prefix_length, target_cidr_range).
    • Consequently, json.Unmarshal failed to deserialize these fields into a.actual, leaving a.actual.AllocationOptions, a.actual.TargetCidrRange, and a.actual.PrefixLength unpopulated (nil/0).
  2. Spurious Update Mask Diff on Create-Only / Immutable Fields:

    • In Update() (networkconnectivityinternalrange_controller.go), the controller diffed create-only/immutable fields (allocationOptions, targetCidrRange, ipCidrRange, peering, usage) against a.actual.
    • Because a.actual lacked these fields, !reflect.DeepEqual(...) evaluated to true on every single reconcile loop, appending "allocationOptions" and "targetCidrRange" to the update mask and issuing a PATCH request.
    • GCP's Patch API does not support allocation_options or target_cidr_range in the update mask, rejecting the request with Error 400.
  3. Missing Re-Reconciliation Check in E2E Tests:

    • NetworkConnectivityInternalRange was listed on the exemption list in tests/e2e/ratcheting.go, bypassing the steady-state re-reconciliation check during E2E tests.

Changes Made

  • Fixed convertAPIToProto (pkg/controller/direct/networkconnectivity/utils.go) to use protojson.UnmarshalOptions{DiscardUnknown: true} for proper proto unmarshalling.
  • Scoped Update() (pkg/controller/direct/networkconnectivity/networkconnectivityinternalrange_controller.go) to only diff mutable fields (description, labels, prefixLength).
  • Enabled Re-Reconciliation Testing (tests/e2e/ratcheting.go) by removing NetworkConnectivityInternalRange from the exemption list.
  • Added Test Fixture (pkg/test/resourcefixture/testdata/basic/networkconnectivity/v1alpha1/networkconnectivityinternalrange/networkconnectivityinternalrange-firstsmallestfitting) reproducing the customer configuration (allocationStrategy: FIRST_SMALLEST_FITTING and targetCIDRRange).
  • Updated Golden Files for existing internal range fixtures.
Fix continuous reconciliation loop and 400 error on NetworkConnectivityInternalRange when auto-allocation fields are set.

@barney-s

Copy link
Copy Markdown
Collaborator Author

Field Mutability Analysis for InternalRange

For reference and clarity on the controller's Update() diff logic, here is the breakdown of all fields in NetworkConnectivityInternalRangeSpec and the GCP InternalRange resource:

KRM Spec Field GCP Proto Field Status in GCP Supported in updateMask? Notes / API Semantics
description description Mutable ✅ Yes (description) Can be updated at any time.
labels labels Mutable ✅ Yes (labels) Standard GCP resource labels.
prefixLength prefix_length Mutable ✅ Yes (prefixLength) Used to resize IPv4 reservations ("Can also be used during updates to change the range size").
allocationOptions allocation_options Mutable ✅ Yes (allocationOptions) Mutable for RANDOM_FIRST_N_AVAILABLE strategy (e.g. updating firstAvailableRangesLookupSize).
overlaps overlaps Mutable ✅ Yes (overlaps) Allows updating overlap permissions (ROUTE_OVERLAP, OVERLAP_EXISTING_SUBNET_RANGE).
networkRef network Immutable ❌ No GCP docs explicitly specify: "Immutable. The URL or resource ID of the network in which to reserve the internal range."
ipCIDRRange ip_cidr_range Immutable ❌ No The allocated IP range cannot be changed in-place. Modifying IP address requires recreating the range (resizing is done via prefixLength).
targetCIDRRange target_cidr_range Immutable / Create-Only ❌ No Search space used only during initial auto-allocation. Not part of the live patch mask.
usage usage Immutable ❌ No Core purpose (FOR_VPC, EXTERNAL_TO_VPC, FOR_MIGRATION). Cannot be altered after creation.
peering peering Immutable ❌ No Peering mode (FOR_SELF, FOR_PEER, NOT_SHARED). Cannot be altered after creation.
migration migration Immutable ❌ No Only used during create when usage: FOR_MIGRATION.
projectRef / location Parent path Immutable ❌ No Resource identity hierarchy.

The direct controller's Update() diff check covers all 5 mutable fields (description, labels, prefixLength, allocationOptions, overlaps) and excludes the immutable / create-only fields from the update mask.

Comment on lines +199 to +203
// Only mutable fields supported by GCP InternalRange Patch are included in updateMask.
// Other fields (ipCIDRRange, targetCIDRRange, networkRef, usage, peering)
// are immutable / create-only parameters and cannot be modified in-place via Patch.
// prefixLength can be updated to resize the range size in IPv4 reservations.
// allocationOptions (e.g. firstAvailableRangesLookupSize for RANDOM_FIRST_N_AVAILABLE) can be updated.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the correct logic. We should not hide the users' invalid changes by dropping them. We should pass whatever the users want to change to the API so that the API decides whether it is acceptable or not.

And in a no-change re-reconciliation, there shouldn't be any diffs found.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #12620

@ldanielmadariaga
ldanielmadariaga added this pull request to the merge queue Aug 27, 2026
@google-oss-prow google-oss-prow Bot added the lgtm label Aug 27, 2026
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ldanielmadariaga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Merged via the queue into GoogleCloudPlatform:master with commit 31cf5fd Aug 28, 2026
248 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NetworkConnectivityInternalRange: every reconcile fails with 'Unsupported update mask path: "allocation_options"' after initial creation

3 participants

⌂ Proxy