/Library/Extensions/ instead of
/System/Library/Extensions/

Just send all the files listed below to the Trash.

/Applications/Hammerfall DSP Mixer
/Applications/Hammerfall DSP Settings
/Library/Audio/MIDI Drivers/HDSP MADI MIDI.plugin
/System/Library/Extensions/HDSP MADI.kext
/Users/username/Library/Preferences/Hammerfall DSP folder
/Users/username/Library/Preferences/com.rme.HDSPeMixer.plist
/Users/username/Library/Preferences/com.rme.HDSPeSettings.plist

When using the new Snow Leopard driver also delete:
/Library/LaunchAgents/de.rme-audio.hdspAgent.plist

Under the latest Mac OS the User/Libraries folders are hidden and not visible in the Finder. To unhide them start terminal and enter:

"chflags nohidden ~/Library/"

Update 2013: Under the latest Mac OS the User/Library folder is not visible in the Finder. To unhide it start Finder, click on the menu item Go. Hold down the option (alt) key, then click on Library.