I have lingering setup and I can still access the container but for what ever reason Podman seems to be unable to access the GPU for no apparent reason.
I think this may be an issue with systemd but I’m not entirely sure.
Solution: you need to be logged in for it to work. I accomplished this on a separate VM with autologin to icewm.
I already did that so that podman could access the device. (Podman runs as a local user). What was strange was that podman couldn’t access it without a graphical session running but my local user could.
No idea then :( AFAIK the logind mechanism I mentioned originally is based only on permissions, but I had never really needed to look into it further.