http://www.tweakguides.com/UT3_5.html
"Antialiasing Support: UT3 uses a technique called Deferred Lighting, which doesn't allow for the use of normal Anti-Aliasing, at least in DX9 mode. Epic has stated that when running in DX10 mode in Vista, the full version of the game will allow the use of AA. For the moment there are only two ways to force Anti-Aliasing in the game:
If you run a GeForce 8 series card you can rename the UT3Demo.exe file found under the \Program Files\Unreal Tournament 3 Demo\Binaries\ directory to BioShock.exe, and force Antialiasing through the Nvidia Forceware Control Panel using the existing Bioshock profile.
You can apply a brute force method called Supersampling - which works on pretty much any game - using a utility such as nHancer to set 'AA Compatibility mode 45' for the game profile. This is not recommended, as it has a major performance impact and can also cause visual glitches.