Generalization in machine learning focuses on finding invariances in the input space. This definition forces you to think that the agent should be endowed with a function that maps those two invariant inputs to the same output. The problem, then, is to figure out what that function is.
This is very problematic. First, we don’t know (and I think we can’t know) beforehand how many and what types of invariances that the agent could encounter in the future. Yet, seeing only a few examples, especially in puzzles, humans can quickly figure out what the input is about (which is what the ARC was about in the first place). Second, this means that the agent has to learn all kinds of invariances whenever it observes one, and should update all its previously learned functions to accommodate for this new type of invariance.
Learning shouldn’t be about a search for a function (especially a smooth function, as is the case with neural nets) that’ll give you the same output when the input varies in invariant directions. Not that such a function or a process doesn’t exist; on the contrary, humans must have something like that to be able to deal with all kinds of invariances in the environment. Rather, this function should be built up in an incremental manner, with maps and shortcuts. In other words, the function exists, but we must not search for it; we should just build it on the fly.
This problem is a bit awkwardly positioned because it's about the architecture, not the problem definition, and as such, it feels unimportant and arbitrary to those who focus more on the problem definition. For those who care about the architecture, you are again badly positioned, because now you are in an uphill battle against the one true smooth end-to-end function school of thought.