Turns out there's already a good open source console based on the RK3326 Arm CPU. It's the R36H by Boyhom. It's marketed as an emulator with support for GBA, PS2, etc, but it can be soft upgraded to run arbitrary games.

I have an idea of building a handheld console that's easy for indie developers to put their own games on. It's based on Linux. Here's some concept art for it.