How is the expression the long walk misleading

According a general english dictionary the expression take a long walk is defined as idiomatic, pejorative colloquial usually used to tell someone to go away or that their request will not be meet.

And the word misleading if you describe something as misleading you mean that it gives you a wrong idea or impression.

This expression is referred when many people think differently a short walk to me might be a long walk to you.