Can You Trust Amazon Vine Reviews?

If you love bananas and online shopping, you may have stumbled upon something on Amazon called Barnana Banana Brittle, a salty vegan treat made of “upcycled bananas.” If you wanted to know what an upcycled banana tasted like, you may have scrolled down and noticed that 20 of the 21 Amazon reviews at the time were labeled “Vine Customer Review of Free Product.” And if you’re like most people, you may have wondered what exactly that meant.

According to Amazon, the Vine program offers a way for vendors to provide free samples to select reviewers to increase the quantity and quality of customer reviews of those products. This program sounds similar to an “incentivized review” arrangement, in which reviewers receive free samples in exchange for a review of the sample—a practice Amazon explicitly prohibits on its platform outside of the Vine program. The ban makes sense given that ReviewMeta’s June 2016 analysis of 7 million Amazon reviews suggests that the average rating in incentivized reviews skews more positive, by a 0.38-star difference.

That doesn’t sound like much, but Columbia University business school professor Assaf Zeevi, who is not affiliated with Amazon but has in the past researched product recommendations and Amazon data, noted that in a scale that functionally ranges from three to five stars, even that slight bump can have a huge impact. The difference is often enough to push a product from a middling 50th percentile rating to the coveted 90th percentile. And to outside observers, it’s not clear why the Vine program shouldn’t also be subject to some suspicion of bias. In other words, does Barnana Banana Brittle live up to its four-star hype?

To reduce bias in this kind of system, Zeevi theorizes, Amazon must oversee a large base of reviewers who each review a small number of products. In an interview, Zeevi explained that a larger population of reviewers would ensure a level of averaging—meaning reviews likely wouldn’t skew too high or too low—that would help the program remain impartial. And limiting the number of products would theoretically increase the time each reviewer spends using the product and writing the subsequent review.

To figure out whether Vine met those criteria, I interviewed longtime members of the program to learn how it worked, what they liked and disliked about it, and what it took to give an honest review.

How Vine works

At its core, Vine is simply an incentivized review program: Reviewers receive free samples in exchange for reviews of said samples. But in practice, checks and balances at every step of the way help ensure that the reviews produce actually useful information for their readers.

For starters, not just anyone can become a Vine Voice member. If you take Amazon at its word, Vine reviewers are the cream of the crop, people who have taken the time to review products they’ve bought and have already submitted higher-quality, more-useful reviews as judged by the “helpful” votes system—which is to say that they were already good reviewers before joining the program.

And although manufacturers get to choose what they offer for review and how many units to distribute, Amazon has greater control over who does the reviewing. Products remain up for grabs for 28 days, or until the manufacturer-designated supply runs out, according to a FAQ page about the Vine program (PDF) published in 2017. Vine offers each member a rolling, targeted queue of Amazon items based on their review history and the demographic questionnaires Vine members fill out upon joining the program.¹ “This allows time to read the product description and consider if this is something that I can properly review,” said Vine member Sean Logue. In theory, it helps ensure that the reviews that members write are relevant to their areas of expertise. In practice, the targeting could work better: “As a single guy in his 40's I often get offered dog food (I have no dogs), women's underwear, sanitary napkins, bras," an anonymous Viner told me in an interview.

Most Vine members said they chose items based on what they would legitimately use in their life. Adam Shields, who is a stay-at-home parent, often reviews toys, baby clothes, and diapers. Gina Pera decides which items to review based on four things: if she needs it, if she could use it, if she would enjoy it, and how much she would pay in tax.

As it turns out, the free samples received through Vine are often not actually free. In a previous iteration of the Vine program, members never technically owned the items they reviewed, instead “borrowing” them from Amazon for testing purposes, according to a report by NPR’s Planet Money. In practice, however, Amazon never asked any of the seven Vine members we interviewed to return their items. The program has changed since, according to current Vine member Darcia Helle: Samples start out as product loans, but ownership transfers to the reviewer after six months. Starting in 2016, Amazon began issuing form 1099-MISC to all Vine members who received products totaling $600 or more during the calendar year, according to a representative from the company. To regulate this new policy, Amazon revised Vine members’ existing contracts to include the new tax rule. Members had to agree to the change to continue reviewing with the program, according to a Vine member who uses the handle Kort. Amounts less than $600 still represent taxable income, but the IRS mandates that companies issue the form only to contractors who hit the $600 threshold. Consequently, if Vine reviewers aren’t careful and get carried away with “free” items, they can get walloped by the tax bill.²

One Vine member who asked to remain anonymous said that he remembered many Vine members saying they would quit the program after Amazon changed the tax policy in 2016, which he learned about after logging in to his Vine account and reading the announcement. He said Amazon used to host internal discussion boards where Vine members could interact and ask management questions about the program, but the site shuttered the boards in late 2017 without explanation. “Word on the forums prior to their demise was that VINE members had been assured via e-mails from Amazon customer support that our particular forum would either not close, or would later be reinstated,” he said. “Neither happened.”

But many of the Vine members I spoke with said that, after the initial shock, they’ve carried on more or less as they used to, though some now think twice before receiving high-ticket items. “I tend to completely avoid expensive items that I will either never use and/or have little to no resale value,” a Vine member who wished to remain anonymous said. “But I’m getting useful, functional items that I would otherwise have to pay full price for.” Shields paid $800 in taxes for his Vine-review items in the past calendar year. “In the end I think I got more than $800 in value, but I am not sure I think I actually made anything from the Vine process,” Shields said.

As far as members know, there are no concrete requirements to stay in the program. For example, there are no deadlines, minimum word counts, or limits to the number of items you can have for review. “It is pretty much understood that those who finish their reviews faster and have less items checked out tend to get better offers, though,” our anonymous Vine member source said. Vine members are held to the same standards as anyone else who wants to submit a product review on Amazon, standards that prohibit activities such as self-promotion, impersonation, and hate speech.

Are the reviews honest?

Vine members I interviewed insisted that they did not let the perk of getting a product for free influence their reviews. “While there are some people in the program that take advantage of the situation and don’t write comprehensive reviews, I’d like to think that most ‘Viners’ have a sense of self-respect and duty to provide the best review possible,” Kort said. Vine member Gina Pera explained that she even took the extra step of anticipating her readers’ concerns: “I try to qualify criticisms if others justifiably might not share them. For example, if a makeup foundation accentuates every wrinkle on my face, I will consider if that’s a product flaw or if the product is better suited to smoother and younger faces.” Indeed, most Vine reviews I’ve come across provide a greater level of detail than non-Vine reviews and are frequently marked as being among the most helpful.

However, when it comes to the star ratings, there’s data to suggest at least some slight positive bias. A November 2016 ReviewMeta survey of 30 million Amazon reviews found that among products that had at least 10 Vine reviews and 10 regular reviews—in a controlled sample of 921 products that fit the bill—Vine star-rating averages skewed more positive than those for non-incentivized reviews. ReviewMeta concludes, “On average, the regular reviews rated the products 4.24 stars, while the Vine reviews rated the same products 4.39 stars, which is a .15 star increase!” This suggests that discerning shoppers should take the star ratings from Vine reviews with a grain of salt.

The real value of Vine reviews becomes clear when you compare them against the largest source of review bias that potential buyers contend with: Truly fake, paid-for reviews. According to another ReviewMeta report, these reviews have been slowly taking over product listings since June 2017. And a recent Buzzfeed Tech investigation by Nicole Nguyen declares that “Amazon is all but powerless to stop it.” According to Nguyen, the rise of fake reviews is being driven by private, closed communities hosted on various private Slack chats, Facebook groups, and subreddits. Through these networks, anyone who wants to make a quick buck or two can find someone willing to pay them to write a fraudulent review. Even a “Verified Purchase” tag can deceive due to the ease of reimbursement via PayPal or Venmo. At least with a Vine review, you know what the potential biases are. And the Vine member’s descriptions of and thoughts on the product will inform you much more than a somewhat arbitrary designation of three or four stars.

Columbia professor Assaf Zeevi expressed similarly cautious praise for the Vine program in its current incarnation: “I think it’s not a perfect solution, but it’s an important improvement to what existed before.” ReviewMeta’s Vine analysis similarly concludes that “it’s objectively better than the 3rd party review clubs which were getting completely out of hand.” When you see a “Vine Customer Review of a Free Product” tag, you know what you’re getting: a review from a tax-paying person, vetted by Amazon, who actually used the product in question and was obligated to report on their experience by accepting the sample. Furthermore, my interviews with these reviewers suggest that they take their duties seriously—which is to say that “Top 50 Reviewer Vine Voice” KC’s assessment that a “strong flavor of coconut and banana permeate these chips” is more informative than Barnana’s own claims that the brittle is made of “more of the good and less of the bad,” regardless of the accuracy of that review's five-star rating.