If it is a umod file, just run it. If not, you probably have a ZIP file containing *.u and *.int files which should by unzipped into the UnrealTournament\Systems\ directory.
In general, with some rare exceptions, for any "manual install" mod, unzip the files into the UnrealTournament subdirectory that already has many files with the extension of the file being installed.
I like some aspects of DMUT very much. However, the HUD display and certain spells, i.e., Olympian, Camoflage, and regeneration seem to be incompatable with most other mutators. Also, Stasis has a bug that sometimes makes players and bots immobile after the spell is supposed to wear off. You can combine the spellbook with other weapons by using the mutator, W.O.R.M.. Another mutator that works well with DMUT is Piniata which causes a dead player to drop all weapons, in this case, the spellbook which contains mana which is normally in short supply.