AdEspresso

Freelancer or Fiverr? We A/B Tested 3 Different On-Demand Design Services So You Don’t Have To

2016 is the year of experimentation for AdEspresso. As part of our new University, we’re performing and writing up the results of a series of reader-picked social media experiments, putting arguments and debates to rest with cold hard data.

Last week, we kicked off the series with a look into “fake” clicks on Facebook, Twitter, and LinkedIn. This week, we’re looking at Facebook banner ads, the on-demand freelance services that will make them for you, and what you can expect if you decide to go down that path.

For cash-strapped startups and others just getting started with design, there are a number of services out there that promise quick designs for things like banner ads—and they promise them cheap. But here’s the question: how do these on-demand services perform in the wild?

Not just in terms of aesthetic—the objective criteria for judging that are not so good, and who cares what we think anyway?—but in terms of results.

If you’ve ever tried to hire a freelance designer over the web, then you know that there a lot of options when it comes to getting a quick Facebook banner ad. There’s Upwork, Freelancer, PeoplePerHour. There’s Fiverr, 99Designs, even Craigslist. But which is one is going to get you real results?

When we were launching the University, we went to make a Facebook ad for it as usual—we have full-time designers on staff. But then we realized—why not make it into an experiment? Why not have a bunch of those services make ads for us, run them for a week, and report back with the numbers on how well they performed?

So we chose three:

 

We found freelancers, we sent briefs, and then we took the finished products and ran them as actual Facebook ads for one week. The numbers shocked us. They will undoubtedly surprise you. Read on for the details.

The Brief

First, the brief. A brief is just a short description of your company, what you want to do with your ad, and who you want to reach with it. We sent the same one to each designer that we worked with:

We’re AdEspresso, a Facebook ad optimization service. We help entrepreneurs make better Facebook ads, test out thousands of variations to find the best ones, and analyze the data to figure out what to do next. It’s an all-in-one platform for getting the best possible results with your advertising.

The banner ad we’re running is going to be for what we’re calling the AdEspresso University. This is a new section of our site that is going to have video guides, webinars, a gallery of ads, and monthly reader-chosen experiments into various parts of the advertising process. We want to drive traffic there, both our current customers, new people, and people who might know AdEspresso but aren’t quite convinced they should actually become customers.

As far as specifications:

  • Follow Facebook guidelines and don’t use too much text. If it’s 20% of the image, Facebook will reject the ad.
  • We need a 1,200 x 628px banner for the Facebook ad. See here for size guidelines.

 

There’s a deadline of 1/16 on this project. Thanks, and excited to see what you come up with!

In writing this, we followed standard design brief best practices. First, you need to tell the designer who you are and what you do. Don’t use so much jargon that the value of what you offer gets lost: describe your company in layman’s terms. Talk about the kinds of people who use your product.

Next, tell the designers about what you’re trying to get out of the ad. You want to specify the exact audience you want to reach with the ad, the product you’re trying to promote, and the action you’re trying to encourage—a click, a download, etc.

Lastly, make sure you don’t forget any important technical details. Specify the exact size of ad that you need—1200×628 pixels is pretty standard for Facebook ads, but always refer to Facebook’s official documentation. Facebook also has a policy about text in ads. Text can’t make up more than 20% of any ad’s overall area, so we made that clear to our designers.

The Design Services

The three different services we used—Fiverr, Upwork, and 99Designs—all operate differently.

Fiverr is an online marketplace, but there’s not much negotiation: you put a product (like a banner ad) into a shopping cart with a couple instructions, and the work is done.

Upwork, the successor to Elance, gives you a qualified list of available freelancers and offers you the power to propose, negotiate, and agree on projects with them. It’s the Craigslist to Fiverr’s Amazon.

99Designs is totally different: you upload a brief, choose a level (Bronze, Silver, Gold, Platinum) and your design is disseminated amongst their pool of designers. Those designers produce work on spec—for free—and you get to choose who wins—and who gets your money—at the end of the contest period.

Fiverr

A site where you can get most anything done for $5, Fiverr recently found itself in hot water with Amazon, who wasn’t too pleased about the massive numbers of fake reviews being bought and sold there. But it’s true—you can get almost anything done.

That includes graphic design. Since every graphic designer on Fiverr costs about the same amount—$5, or $10 for turnaround in 24 hours with a raw Photoshop file included—we narrowed it down by searching specifically for “banner ads” and sorting the results by overall rating. We wound up with one of the top-rated designers on the platform: Design2Thrive, a Singaporean graphic designer, who had a total of 5 stars out of over 9,000 reviews.

In “About This Gig,” we got the skinny on the details.

It’s important to look at the details, because if you just click Add to Cart without making sure that your request can be fulfilled, you could very well be out $5. There’s not a whole lot of negotiation involved in Fiverr’s system, although you do include instructions in the note that goes along with your payment.

For us, this worked just fine. We sent Taylor a request for a Facebook banner with our brief, the precise dimensions we wanted, and within a few minutes we got a notification that it was “in progress.”

Perfect. On to the next one.

Upwork

Upwork is a little different. You may remember it as Elance, or oDesk, both of which wound up as part of Upwork this last year, but the premise is the same: post a job, look for freelancers, chat them up and possibly hire them.

First, posting. Everything was pretty simple, we just wrote up our brief and off it went, posted to Upwork’s public jobs page:

At this point, we could have sat around and waited for people to get in touch with us about our job. But we wanted to get on with the experiment, so we went looking, within our budget, by typing “banner ad” on Upwork’s search engine.

We were offering $30 for this job. Yes, $30 is cheap. We’ve heard all the talk about the “race to the bottom” inherent to these kinds of freelancing sites, where quality is initially important but quickly becomes subordinate to price and speed.

But we didn’t just go with any old designers. We did want to find people that actually seemed like they would do a good job. The first person we found was Dinesh.

Dinesh was what Upwork called a “Top Rated” designer, a designation he received by getting a 90% satisfaction score in post-job surveys and completing 766 hours of projects on the platform. Recently, he really seemed to be excelling.

Dinesh got back to us within a couple of hours of us sending him a message and he accepted our request.

Next, we found Marko, a graphic designer from Macedonia. He had only been on Upwork for a few weeks when we got in touch with him.

Only 20 hours clocked on four jobs, but as you can see, he was what Upwork called a “Rising Star”—very active on the platform, high scoring on several graphic design tests, and with nothing but 5-star reviews so far. Promising!

We reached out and he got back to us even faster than Dinesh: it was a go.

Finally, we got in touch with Justas, a Lithuanian designer with the edge on Dinesh and Marko both in terms of job satisfaction rates and hours put in on the platform.

His hourly rate was almost Dinesh and Marko’s put together, so we weren’t sure he would accept. But when he responded to our message, he confirmed that $30 would be sufficient. We had our three designers.

Even with this brief experiment, we did get a few rejections. Both were due, partially, to Upwork’s search engine: when you search “banner ad” or “graphic design,” you will actually get plenty of freelancers—mostly in illustration and web development—who will not do your banner ad for you. Sorry for wasting your time, Rachel in Canada.

99Designs

The last service we went with, 99Designs, offer a significantly more expensive set of packages but promises that many designers will pitch in and do work “on spec”—that is, without expectation of payment.

All of the designers who enter your “contest” put their work up on a page that you can see. You can pick out things you like about ads and help the designers make them better. At the deadline, you pick the one you like the most, that designer is notified, and that’s when you send over your payment information in exchange for the full-size image and Photoshop file.

We went with the Bronze plan for this experiment, though our request for a three-day turnaround time bumped the price up to $80. The information we were asked to provide was already written in our brief, so this part was easy:

The Ads We Received

After a couple of days, we put all the ads we’d received together and we assessed them on visual terms. We were interested to see who’d been creative, who’d been most faithful to the AdEspresso aesthetic, and who’d done the best work promoting the University. What we found was surprising to say the least.

Dinesh from Upwork

When Dinesh first saw our brief, he took special notice of the 20% rule we’d mentioned. “I’ve dealt with that before!” he said, seeming excited, or like he was trying to comfort us. The point is that Dinesh was the freelancer I least expected to mess this up, yet his ad was flat-out rejected by Facebook for that very reason.

Dinesh’s design was serviceable, though that’s moot because it failed Facebook’s text test. With this kind of situation, you have to send the offending ad back to the freelancer who made it and have them fix it up—then pay them after you have confirmation that it’s cleared through Facebook’s gates.

Marko from Upwork

At least the owl is cute. Unfortunately, most of it is off. The AdEspresso text is incorrectly capitalized, the font is totally off brand, and the size difference between “Ad Espresso” and “University” appears random. The commas in the smaller text are spaced incorrectly, and the chart behind “Ad Espresso” is very confusing, both in terms of meaning and placement.

And the clock on the wall only has eight lines on it. That just doesn’t make sense.

99Designs

Here’s the full pool of entries we got with our contest. As you can see, we did not have thirty designers offer their work.

Several of these ads have glaring typos, like “Manage your Facebook Ads In Less Times.” Most are not even for the University at all—they’re very clearly ads for AdEspresso with no mention of the University at all. One of them is simply for “New Features,” which apparently come packed in sacks of coffeebeans now.

Definitely keep in mind that though 99Designs emphasizes that your project will garner the attention of about 30 designers, that’s not a guarantee.

Don’t expect that. We only had four designers come to our project, though some of them did create several different iterations on our idea. We chose this one in the end, courtesy of SJDA Designs.

It was, in the end, the only design that actually fulfilled the terms of the brief.

Taylor from Fiverr

It was a little strange, we thought, that judging purely by visual quality Fiverr had given us by far the best product. The best entry in our 99Designs contest was in Times New Roman with a boring stock illustration and some hastily added text. While there wasn’t much originality or imagination in Taylor’s piece, she certainly produced something consistent with AdEspresso’s aesthetic, and nothing about it was particularly jarring or offensive.

A Note On Justas From Upwork

We had been quite excited to see what our philosophy and design-loving freelancer Justas would come up with. Unfortunately, he was not able to get us work on time, and so we had to exclude from the study. Fortunately, we didn’t have to pay him—but we also didn’t get the work that we wanted. And this is important to keep in mind. Working with a freelancer on one of these on-demand services, even for a job that seems simple, does not mean that you’re going to get your work on time.

Thoughts On The Designs

Investing too much time and money seemed to get designers confused and throw them off track, as we saw with the panopoly of bizarre designs from 99Designs, most of which totally ignored our brief. Going through Upwork and 99Designs was still fairly low-touch, but low-touch and reliable don’t always go hand-in-hand, which makes sense.

Nor does it go hand-in-hand with good work. Even from a quick eyeball test, it was immediately clear that we didn’t have a whole lot we could use. Of course, you can mitigate those effects, to some degree. If we’d been seriously looking for ads to run in the long-term, we could have gone back to Dinesh and had him resize the amount of text in the ad. With 99Designs, we could have put the specific requirements further up in the brief—buried near the bottom, it’s plausible that the crowd of people who responded weren’t as attentive to the nuance of the project as freelancers we reached out to individually.

But this does, in part, undermine the entire idea behind these services. You’re supposed to be able to get a good design, the kind you specify in your brief, and you’re supposed to be able to get it without having to invest too much time in this process. And so far it’s not clear that any of them are living up to that—but on to the results.

Results

Each ad that we wound up running—Taylor’s (Fiverr), Marko’s (Upwork), and SJDA (99Designs)—ran for ten days. We set a total budget of $50 for each, used AdEspresso’s standard targeting methodology, and let them run.

First, let’s look at how our different designers did when it came to producing impressions and clicks.

1st Round Winner: Fiverr

To be clear, we put in no special bids when we uploaded these ads to Facebook. All we specified was the $50 budget. Because of the unique algorithm that determines what Facebook ads cost, though, the amount you’re actually charged per click has as much to do with the ad’s quality and relevance as anything else.

A terrible ad that users continuously scroll by (or even report) will be flagged by Facebook, and it won’t be shown to as many people. But every time a user clicks through (or even shares) a great ad, it gets boosted and shown to more users.

With 1.5x the number of impressions as our competitor from 99Designs and over 3,000 more than Marko, Taylor from Fiverr was the clear victor here. Her ad, judged as more relevant and of higher quality than the other ones we uploaded, found more eyeballs by far.

Unsurprising, then, that Taylor won again when it came to clicks. Despite showing up in the feeds of less users than Marko’s ad, SJDA’s ad actually garnered more clicks than it did.

2nd Round Winner: Fiverr

Impressions and clicks are fine, but click-through-rate and cost-per-click are where you start to really separate the wheat from the chaff and figure out: (1) what ads are attracting the most visitors to your site, and (2) how much they’re really costing you to do it.

Cost per click, because we set up our ads without specially optimized bids, was calculated by dividing the money spent on each campaign by the amount of clicks each garnered. Click through rate, similarly, was calculated by dividing clicks by impressions.

If you make a better ad, it’ll get shown to more people. If a lot of those people click, you’ll actually end up paying less per click. But if very few people click, it’ll be shown even less, and you’re going to end up paying a premium for those few clicks.

Taylor won: because her ad garnered a better CTR than the competition, Facebook determined that users liked it more and showed it to more of them. More of them clicked, and the positive momentum created drove her CPC down way below SJDA and Marko.

3rd Round Winner: AdEspresso!

Time to bring out the cold water.

While Taylor can pride herself on finishing ahead of her competitors, we do have to mention what we expect to see from a Facebook ad that we run.

The lowest AdEspresso’s click-through rate gets is 1.2%, while average U.S. costs-per-click are around $0.33. For perspective, here are the designs we got from our freelancers plus an average Facebook ad from AdEspresso:


Taylor is the only designer who got close to the average U.S. cost-per-click: 99Designs and Upwork gave us ads that ended up costing double the average rate. In terms of click-through rate, however, they were all equally disappointing. 0.91%, 0.76%, and 0.60% just aren’t good enough.

They’re starting points, but in the long run you will undoubtedly want to work with a full or part-time designer with whom you can take designs, test them out with an experiment like this, and then iterate to make them better.

Why? Because the potential payoff of a higher CTR and a lower CPC is enormous.

To see what we mean, check out what happens when we up the stakes and extrapolate from these numbers. Let’s say we spent $10,000 on each channel instead of $50, and let’s say we got a dollar value of $0.50 from each click. Here’s the revenue we would have made from this campaign with the CPC and CTR numbers we got from our experiment:

A small difference in click-through rates and costs per clicks compounds over the course of a big campaign, and that can mean the difference between coming out ahead and losing tens of thousands of dollars. But maybe your ideal metric is downloads, or sign-ups on your site—whatever it is that you’re optimizing for, focus on getting it up. Even if it’s only a tiny percentage point, that tiny point can have a huge influence on your bottom line.

Just A Starting Point

So Fiverr was victorious in the battle of the on-demand design services. But what does that really mean? Should you go out right now and order a banner ad from Design2Thrive, throw your life savings into Facebook ads and see how it goes?

Nope. Even though Fiverr won our competition, the data we have shows that it falls seriously short of what you want out of a Facebook ad campaign.

But that doesn’t mean that spending $5 or $10 to get a banner ad mocked up on Fiverr is a waste of money. On the contrary, it could be a good investment if you have a new product or company that you’re just starting to think about advertising.

Think of it like a minimum viable product. You don’t necessarily have to upload it and run it as an ad. What you can do instead is use it as a preliminary sketch, a rough draft, as the first step on a path towards really great design.