This is the first time I try composting. It smell very uncomfortable. How do you tell if my compost is composting or just rotting? I am very worried.

Compost is rotting. It's a matter of how.
If you see mold, usually white, it isn't getting enough air.
Composting varied vegetable matter smells pretty good.
One of the worst smelling vegetation composts, however, is plain grass clippings. Too dense! Compost needs AIR. Dry leaves, fluffing with a pitchfork or rake, etc.
We can't begin to tell you what you might be doing wrong.
Do many hours of reading again, or tell us step by step what you put in it and how you handle it.