Most of the time if the driver is wrong for given hardware, then the driver will fail to load. This is only a wild guess and very likely wrong, but it “feels” like you are using software which expects different hardware…in other words, the driver is probably correct for the hardware present, but the end user software is expecting something else.
It could just be another kernel feature has to be added, and what follows is just a general idea and not specific to your case…
Often there are drivers for access to the specific hardware. Sometimes there are optional functions which not everyone will use, and so the optional functions might be an “add-on” kernel feature which can only load if the actual hardware driver is first loaded. This is a dependency example, where a software function is dependent upon a hardware prerequisite.
I don’t know if any such “add-on” software feature is possible for that hardware. The given driver works, but so far I’ve only seen the UART used with RS232. Perhaps there is an add-on RS485 kernel feature required before the IOCTL would be recognized. The software using the UART is apparently expecting such a feature.
Not all UARTs will be able to use RS485, and I do not know if the particular UART can work with RS485 or not, but it is apparent that if the hardware is capable of working with RS485, then the kernel needs “something more”. I don’t use RS485, but basically your next step is to find out what Linux kernel config might be needed in support of RS485. If it is supported, then this will be a feature on top of the UART driver (think in two phases: Drivers to access hardware function, and a feature which understands the protocol.
If you find such a feature, then you would start with kernel source configured to match the current running system, and then enable that extra feature in addition to the current configuration. If you find this feature, and if the feature is available for build as a module, then compiling and installing the module is simple. If the feature must be integrated into the kernel, then you can work with this, but life is a bit more complicated for that case. If an RS485 feature depends on hardware not present, then you are out of luck. If you cannot find RS485 driver/feature information, then you must research the end software you are using and find out what it wants (which isn’t a bad idea anyway to see what prerequisites this end user software wants).