How (And Why) We Review Apps

Last week, the Internet was abuzz with talk about a single tweet which caused quite a stir. The head of a PR firm tweeted: “#AlwaysBetOnDuke too many went too far with their reviews…we r reviewing who gets games next time and who doesn’t based on today’s venom.” Working hand in hand with PR firms is something we at AppStorm have to do pretty much every day, but it’s not often we’ve seen one of them speak out like this.

Because of that issue, it got me thinking about how we do our reviews here at AppStorm, and it made me wonder if some of our readers think we might have a bias towards the positive side of things, and therefore, we don’t give “real” reviews. So to address that issue, I figured I’d peel back the curtain a bit and talk about how we at AppStorm review an app, and what that means for you, the reader.

The Input

We get lots of submissions at AppStorm, mostly from PR firms and developers who want to get the word out about their product. There’s a few different ways we take in the stream: There’s our Quick Look feature, email, and Twitter is another popular option. When we get these submissions, I’ll typically mark down the name on a spreadsheet I keep, and put it under a “potential review” section. We won’t review all of these apps (there’s just not enough time, frankly), but this way if a writer is looking for an assignment, we have something in the bank.

The Actual Review Process

As the editor, it’s my job to make sure that each writer is on task and writing about an app that either they’ve chosen themselves or that I’ve sent their way. To make it fair and accurate, I want them to use the app for a few days and get the feel of it. Sometimes, an app looks great on day one, but by day five, it’s delegated to the “Unused Apps” folder on their iPhone. This gives us the most fair and unbiased opinion on each app.

If the app works, then the writer puts up a review, with a rating at the bottom that designates how good the app is on a 1-10 scale. Typically, our reviews are between six to ten points, with most of them being in the eights, nines and tens. So why not anything below a six?

Peering Behind the Curtain

I bought an app the other day, with really high hopes for it. I expected it to replace two or three of my other go-to apps, and I figured that it was absolutely perfect to my workflow. Back then (before I was editor), I emailed the boss and called dibs on the review. He slotted me in, and I started playing with the app. As disappointment crept in, I realized that this app wasn’t going to be a 6 or above, so I told the editor as such. The response? “Fine with me, let’s just skip it and move on to the next one.”

We like to keep things positive here on the site, so we’re not going to publish scathing reviews that tear each developer up with each word — we’re just not that site. What we want to do is collect the best apps that you can get, and put them out for our readers to find. It’s those diamonds in the rough or hidden gems that we want to showcase alongside the big names, because we don’t want you to clutter up your iPhone with junk the way we do. The App Store is a pretty big place, so we like to think we’re helping you narrow down the process.

What We Don’t Review

Which, at the end of this all, means that we often review more apps than we actually publish. I know that between all of the AppStorm sites that I write for,  I personally turned down at least four last month because they just weren’t good enough. I spent hours of my time trying to determine if the applications were worth our reader’s time, and in the end, they weren’t. So instead of writing something fake just to get paid, I opted instead to move on to something else, that way the quality of the site was kept up.

And now, as editor, that’s something I continue to push. If one of our writers doesn’t like an app, don’t review it. For me, I find the best type of review is written by someone who uses — and hopefully loves — the program. For example, I’m a huge fan of Instapaper for the iPad. I use it every day — usually multiple times a day — and I really think it’s not only helped my productivity, but also the way I consume websites. As such, I wrote a review about it for our sister site, and gave it a 10 out of 10. You may not agree with the score, but at least you know that I have a strong opinion about it, which means that at least one person out there thinks it’s worth the money. Why not listen to someone who’s passionate about the program, instead of hunting and pecking in the App Store to find something that just works OK?

The Results

Because of that, we won’t get into an argument on the web with a PR firm the way other sites will. If a PR company pushes a product our direction and we don’t like it, then it’s just not going to get reviewed. That way, the PR company doesn’t get a negative hit on the product, and our readers don’t waste their time reading a review on a less-than-stellar app. It’s win-win in our book.

This isn’t how most businesses run review sites, and we understand that it’s not for everyone. But this system works out pretty well over here, and I think it keeps a positive spin on things. And frankly, if you’re here reading this now, chances are pretty good you agree.

  • Jesse

    This is probably the main thing I love about this site. You guys typically post only 1-2 articles/reviews a day per subject (i.e. Mac, iPhone, Android), but I know when something’s posted it’s worth looking into.

    Keep up the god work.

    • Jessed

      Keep up the good* work. (really would like the ability to edit a post though)

  • Connor Turnbull

    Hey Kevin,

    I didn’t realise you were editor now!

  • Mikael Konutgan

    I really like this about the site, when I’m going through my RSS Feeds I often dismiss many many articles but I almost always fully read every article from one of the AppStorm sites, because they are all worthwhile.

  • Matthew Guay

    Thanks for writing this, Kevin! You definitely summed up how I feel about reviewing apps. I’ll be saving this article to share with new writers in the future!

    There’s so many awesome apps out there, and it’s so nice to be able to share the stuff that will really be useful to readers. Developers, keep bringing on beautifully designed and useful iOS, Mac, Web, Android, and Windows apps … make them great and we’d love to review them!

  • Paul Mayne

    Thanks for sharing this concept and process. I appreciate the honesty and the way you handle reviews. Keep up the great work!

  • Jorge Rodriguez

    This is why I love writing for Appstorm. Everything is so transparent and honest, it’s just a dream to be a writer here.

  • Oren Todoros

    It’s always great to hear the reviewer’s side of the story. From our standpoint, not a day goes by where we don’t hear “We sent out 100 review requests, and didn’t hear back.” I think there needs to be a clear channel of communication between developers and reviewers. Good post. I shared it with the AppsMarketing Facebook fans.

  • Nick Cornaglia

    While it’s great that you only review the items you think are good enough to recommend, I want to know the apps you’ve tried and DON’T like, as well.

    While your method of choosing what to publish makes it easy for me to think at a glance, “well if it’s on appstorm…it must be good”. But there are 1000’s of apps. I may choose one that you have tried, think that it really sucks, and wouldn’t take the time to post your opinion because it would have been a negative review.

    That’s information I’m missing.

  • hugo

    I agree with Nick, it’s exactly the point I was going to make. Sometimes I go into the appstore, see an app and think “is it worth it?”, and I check this website to see a review on it. If there isn’t one, is it because you never reviewed the app, or because it isn’t good? Maybe some users will only be spoon-fed the apps you review, but there are others who search for apps on their own, and would like a professional opinion before buying them.
    I believe that actually makes you a little biased, and it could be resolved simply by putting the score on the top of the review, instead that on the bottom, color coding the bad apps, or even making separate rss feeds (for those who check them). What if you review an app and don’t like it, but reading the description of the app I want to know more and actually find that it fits my needs perfectly? Your point is not entirely valid, since I’ve seen reviews on this website that speak so well of an app and I completely disagree, what guarantees me the opposite does not happen? I want you to review the app and tell me what it does good and what it does bad, no to pick for me if an app is worth my time.

  • Pingback: Best of AppStorm in June | Freak Mac()

  • Pingback: Your iPhone Editor: Meet Kevin Whipps | iPhone.AppStorm()

  • Pingback: Your iPhone Editor: Meet Kevin Whipps | iPhone Unlock Jailbreak Guides and Apps, Games Review()