Just as there are good and bad answers (and thus, voting up and down is allowed), is there a good reason why comment (not answer) downvote is not allowed?
Sometimes the comments are very much out there, so perhaps downvoting them should be allowed?
I believe downvoting comments would mislead users from its original purpose:
They are helpful in positively keeping the workflow toward good Questions and Answers, which is the SE main objective.
Whatever reason a comment is posted (asking for clarification, suggesting an improvement, stating a critic or complement),
upvotes already play the role to differentiate which comments community thinks are the most relevant. The asker/answerer can concentrate on these ones to prioritize actions on the thread or providing an explanation reply.
On the other hand a comment which is not upvoted already tends to be irrelevant.
In case a person does not agree with a comment, he/she can post another comment stating the reason of the disagreement. If the reason is a good one, the comment will be upvoted too.
Moreover, downvoting a comment could begin a discussion about a specific comment (which is not desired because takes the attention away from questions or answers).
This line of reasoning also applies to Q&A style (e.g. an answer with zero vote is probably not a good one), but in this case a negative score alerts the reader: do not follow this information to solve your problem, because it is wrong or is ineffective even if another solution/approach was not provided.
Lastly, there are exceptions like an highly upvoted funny and irrelevant comment or a zero score good answer, but they are a minor portion of total.
Other exceptions I can think of are answers embedded in comments. For this ones I share the opinion users should encourage each other to post: answers as answers and questions as questions.
Up votes on Comments do not appear to increase a user's reputation. So voting down on a comment wouldn't have much of an effect.