[Bug]: Performance regression in Python pipelines using side inputs in 2.52.0-2.54.0 SDKs #30360

What happened?

In #28770 we have enabled caching for state api, which improves performance of iterable side inputs. Unfortunately, this change can result in a performance regression for certain pipelines that use fully materialized side inputs, such as side inputs that use AsDict, AsList views.

As a precaution, we recommend users of 2.52.0, 2.53.0 and 2.54.0 Beam SDK to supply an additional pipeline option --max_cache_memory_usage_mb=0 to restore the prior behavior, particularly if the pipelines use large materialized side inputs.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner