Louie Mantia:
"Consider walking down a candy aisle at a drugstore. If you have a craving for Reese’s Peanut Butter Cups, you know how to find it. Of course, it’s the bright orange wrapper with a bubbly yellow word on it. Instant. Most candy bars have very distinct brands which can be instantly recognizable at a glance of the entire aisle. This should be precisely how you approach your app icon design."
He goes on to point out some fantastic examples of app icon design, including some groups of apps that fit under a single brand and have been designed to make this fact prominent, such as Nike.
When I got to this part...
"If you take that same logic to app icons, any style you have in your app should be applied to your icon."
...I immediately thought of Check the Weather, whose icon does not match a single part about the app's interface at all. It's one of my favorite apps, but that icon is about as generic-looking as you can get. Which is a shame, since the app itself has a very unique design aesthetic.
Go read the rest of Louie's post, it's pretty fascinating.