In the New York Times, Michiko Kakutani says as much in her review of Jonathan Safron Foer's Eating Animals:
"It’s arguments like this that undermine the many more valid observations in this book, and make readers wonder how the author can expend so much energy and caring on the fate of pigs and chickens, when, say, malaria kills nearly a million people a year (most of them children), and conflict and disease in Congo since the mid-1990s have left an estimated five million dead and hundreds of thousands of women and girls raped and have driven more than a million people from their homes."
The problem is that this sort of logic--that we shouldn't "expend [...] energy and caring" on animals when there are still human problems--is that this logic can rightly be applied to most human activities and endeavours in the developed world. Why is Kakutani devoting any energy at all to reviewing the fiction of Nabokov, or Irving, or Ishiguro and Roth, when she could be devoting her energy to solving the world's human problems? What is reading literature doing to stop malaria or war or oppression of women? Why does she care about novels, when human beings are suffering?
Vegetarians are expected to get their priorities straight, and worry about the problems of human beings first. That all sorts of people are devoting all sorts of time, energy, and resources to all sorts of things that do nothing to assuage human suffering around the world is left aside.