The SM57 is a dynamic mic to my knowledge so better switch off 48V.
The gain setting is pretty high, but thats good for this mic, it should work.

There is a lot of things in the overall Totalmix, I would do a total reset.
menu Options / Reset Mix / Total Reset
Then only set the input gain of AN1 up to 40 or so.
Assign the Main 1 to AN1/2
Then test.

A structured approach, especially if not only the equipment is unfamiliar, but also the topic of recording is new territory for you, would be really strongly recommended here as Mr. Carstens pointed out.

I would give you the following recommendations from my own experience:

1. Please watch the RME tutorial videos, both the new ones and the older ones. They are very short, clear and "to the point". Here, in a very pleasant and professional manner, RME has really only limited themselves to the essentials. So it doesn't "waste" any time.
This would be a very good introduction and overview of the possibilities that the very sophisticated software TotalMix FX offers you. I have compiled some video sources here in my blog articles for you:
https://www.tonstudio-forum.de/blog/ind … al-Videos/

2. As a next step, I would completely reset the mixer, as already suggested, and first make sure to get a valid base setup. For this, I have created a step by step guide for a "TotalMix FX first time setup" in a blog article that can help you with this. Over the course of time, more and more points have been added here as well. But from the order of the steps you have here something like a guide, with which steps you can come to a very good basic setup.
This is practically the essence of watching tutorial video and reading the product manual. Here the link to my blog article for you:
https://www.tonstudio-forum.de/blog/ind … rnal-equi/

3. I fully agree with Mr. Carstens that you should really read the manual first. But I also know that some people have a hard time with it and possibly only looked for the driver and firmware installation part. That's why I'm listing this step last, because I think it might be easier for you to read the manual if you're already a little more familiar with the basic operation.
I think then you'll also be more curious about what this little marvel of technology can do beyond that.

These are the up to date manuals of your Babyface and it contains already information about TotalMix FX (the newer version of TotalMix):
https://www.rme-audio.de/download/bface_d.pdf
https://www.rme-audio.de/download/bface_e.pdf

A few thoughts / tips on reading the manual. The manual is really very well and structured written, It brings things without redundancies very quickly to the point. You first get an overview of what the device can do, how to operate it, how to install drivers and perform flash updates if necessary. After that comes in a structured way information about inputs and outputs, technical information, information about the operation of TotalMix FX.

The beginning should be read thoroughly and the remaining things should be skimmed first, unless things are covered that you yourself also have in use. But at least you read all information even if you only browse through the topic, so that you simply remember, what information is in there and to look later in deep, if you need something new.

Regarding Gain settings for your dynamic Mic

Maybe one additional thing regarding your Mic settings / choice. Dynamic microphones are not so sensitive compared to condenser microphones. Also with condenser mics there are two major types with less/more sensitivity: small/large diaphragm condensor mics.

You need louder signals and higher preamp settings to get a good recording level with a less sensitive dynamic microphone. By this you get more information from the vocalist or instrument, but less sound information from the background and the room itself (by reflections...). As a rule of thumb try to avoid recording at full preamp gain levels as this might introduce a little bit higher noise floor.

I made some test recordings for you with my RME UFX+ using the Autoset feature to get automatically a proper gain level / optimum recording level and by using the same method, to make the gain settings better compareable.
The mics have been placed approx 25cm from the strings of the guitar, I took this distance to have it easier to find a good position for mic placement on my current work desk where I am just typing..
The Mics are aimed properly towards the f-holes of the guitars, where most volume is to be expected.
So this quick demo with a Jazz Guitar should be roughly comparable to your situation with a acoustic guitar.

The Results of the gain settings of an UFX+ using Autoset (max gain and preamp amplification is here 75dB):
SM57 dynamic mic:                                  gain 63
Rode NT5   small diaphragm condenser: gain 39
AKG C3000 large diaphragm condenser: gain 36

So a gain setting of 45/48 (from your TM FX screenshot) might be much too low for your guitar recordings using a SM57.
The 48V phantom power will not harm your SM57 and phantom power had been designed not to destroy dynamic microphones. But there are some "vintage" Ribbon Microphones out there, which can be destroyed by phantom power, see for example this information from Royer Labs: https://royerlabs.com/ribbon-mics-and-phantom-power/

More information about different Mic types you can find here:
https://musicianshq.com/whats-the-diffe … crophones/