
For programs needing to take different code paths / behavior depending upon whether running on a traditional X.Org Server or rather running under XWayland in a modern Wayland desktop stack, the xisxwayland program can be called for helping to determine the state. Calling xisxwayland will exit with a status code of 0 when the running server is XWayland. It's that simple.
Version two of the xisxwayland program that is out today adds support for checking against the new XWayland extension . The XWAYLAND extension added this month to the X.Org Protocols is a "dumb" extension that is just implemented for reporting whether it's an XWayland server or not. Checking the XWAYLAND extension is now a definitive way for determining XWayland or not, compared to the prior error-prone approaches of checking different X.Org attributes for trying to determine the configuration.
So should you need a way from simple scripts to check for XWayland or not, xisxwayland 2 has been released.