Failing Battery or Starter in Car?

My car has trouble turing over right away sometimes, usually after it sits for a while. I thought it was a drained battery starting to die. I got a multimeter and tested its voltage. It meets all the recommended voltages. Then I thought it might be the starter. I ran a quick diagnostic, I turned my lights on and then cranked it over. Apparently if the lights dim, you need a new battery. If they don't you need a starter. They dimmed so I guess it should be the battery but, that threw me because of the normal voltages I got. I next tried to test the battery by turning my lights on and opening and closing some windows and things. They power output decreases very fast. The lights dimmed and the windows closed very slowly after a matter of 20 or 30 seconds. I am now leaning more toward a battery than the starter, am I making the right choice?