How to Test mVAS Campaigns: Live Case Study

This isn’t just about throwing stuff at the wall to see what sticks, it’s about careful planning, tweaking, and getting those campaigns to really perform. For anyone diving into or already swimming in the mVAS world, this piece is packed with hands-on advice and insights that make all the difference.

Choosing an Offer

For this case study, we chose two sweepstake offers: 

• PK WIN CASH PRIZE 119 PIN ALL(15230) Payout rate: $0,96
• PK WIN CASH 97 PIN ALL (11184) Payout rate: $0,96

During the test we will determine which one performs better and continue with one of the offers.

After creating campaigns at the GG network, both offers go to the tracker. We’re using Binom as our tracker of choice.

Prelanders

We have designed 3 prelanders specifically for these offers. They have a common concept, but differ in form. We are using slots, wheel of fortune and scratch cards.

Usually, it is advised to perform the A/B testing with just 2 prelanders, but we will show you how to do it with 3. In the end, we need to determine which prelander shows the best results.

Test #1 Optimization

Now let’s look at which browsers convert the best:

We will be doing the subsequent tests only for those browsers that were giving us leads. While at it, it is also worth checking which Android OS versions perform better:

As we can see, the versions 4,5,8,9 can be safely excluded.

Now let’s delve into the stats for each campaign and try to make a profit.

We see that the campaign with a subscription age of 30 days and above is already profitable. Let’s have a closer look at each campaign. We will check the zones and adjust the bids either for each zone or the entire feed.

30+ days subscription campaign analysis

Let’s start our analysis with a campaign where the subscription age is 30+ days.

We see that zone 793592 gave us a lead, but it’s in the red at almost 1.5 times of the lead’s value.

According to these rules, it is too early for the zone 793592 to be banned, and we don’t need to lower the bid yet. 

In our case, on average for converting zones and feeds in general, we were buying out around 12-18% of traffic and were advised to increase the bid to 0.006.

You also need to pay attention to which creatives perform the worst:

We see that two creatives performed worse than others on the first test. But that doesn’t mean that they should be stopped outright, because prelander tests don’t fully demonstrate creative performance. It might’ve been just bad luck, and if we see the same creatives doing poorly in the future, then we will consider excluding them.

So, for the 30+ subscription age campaign we raise the overall bid to 0,005, the feeds giving us leads get a bid raised to 0,006. We also leave only the best-performing browsers and OS versions.

8-30 days subscription campaign analysis

Let’s look at our stats:

Following the rules above, we realize that there is nothing to ban. But we will raise custom bids for the converting feeds.

Here the stats are different from the previous campaign and we see that in general the feeds that gave leads are not all profitable. So we will proceed like this:

• We leave an overall bid of 0,005 for the feeds that have leads and negative ROI, while the zones providing leads within these feeds get a raise to 0,006
• All the profitable feeds get a bid raised to 0,006 as well
• Feed 0 contains RollerAds zones, and we can’t increase the bid for it, but we can do this for the zones within.

0-7 days subscription campaign analysis

Here are the stats for the 0-7 days subscription campaign:

The first two zones have lots of traffic and quite a loss over 4x of the lead’s value — we need to lower the bid for those to 0,0021 to have a chance for profit. The third zone gets excluded. Zones number 4 and 5 also get their bids lowered to 0,0021 and 0,0025. We also ban zones that cost us more than 2x of the lead’s value and the LP CTR below 10% with over 20 clicks.

So we ban the following zones: 1154615, 1151252, 1168954, 1166240, 1066120, 1169089, 847404, 1184202.

Campaigns are ready for a second test run. Let’s take another look at the prelanders:

Scratch cards have almost the same CR as wheel of fortune, and in one campaign even more, but there is not enough data yet, so we can include it in a split and give it another chance.

Test #2 Optimization

All three campaigns have been launched, here are the results after 2 days:

Only in one of the three campaigns the scratch cards prelander does well, let’s make rules for that campaign, while leaving wheel of fortune for others.

Test #2 stats by days:

By subscription age:

We see that the main amount of traffic and leads comes from subscription ages 0-3 days and OLD. First of all, we need to bring these two to a good ROI, but let’s try to optimize others as well.

Compared to the first test we see that the age OLD has significantly increased the number of clicks and leads. The age of 30-60 days has increased clicks, but no leads. Age 0-3 has less clicks (because a lot of sources were banned), but the CR is higher. It is still negative, because the bid was raised by a lot. Other ages have changed slightly.

For the test #3 we will create not 3, but 6 separate campaigns by subscription age: 0-3, 4-7, 8-15, 16-30, 30-60 and OLD.

Let’s demonstrate on the age 0-3 campaign.

We make the following breakdown: age-zone-days to understand how the number of clicks in each zone changed at a certain subscription age depending on the bid by day. Let’s choose three examples to analyze all sources.

1. A zone that had negative ROI and we lowered the bid, but it got worse:

The largest zone by clicks. We see that in the first test we had leads but negative ROI, we reduced the bid by almost 2 times and the number of clicks decreased sharply with 0 leads. This is a ban.

2. A zone that had positive ROI, we increased the bid, and it got better:

We see that in the second test the number of clicks increased by 5-10 times and there were some leads, but not as many. Such zones will bring the most profit. We only need to decide whether we increase the bid further. For that, consult with your ad network manager.

3. A zone that had slightly positive or even ROI, and it got worse:

These zones are tricky, and must be dealt with individually. Following our rules stated earlier, we ban all the bad zones and leave all the good ones with custom bids.

The rest of the campaigns are analyzed in the same way as described above.

Also don’t forget about prelanders. We look at the subscription ages of the scratch cards prelander and make a rule in the tracker for these campaigns, so that traffic goes to both prelanders, where necessary.

Namely, ages 4-7 and OLD. We see that scratch cards does even better than wheel of fortune in OLD, but we will still do a split test, and then we’ll see if the wheel of fortune prelander should be turned off.

Test#1 results by creatives:

Test#2 results by creatives:

Look at the lines grouped under a star (subscription age 0-7), the second to last creative.

In total, we create 6 campaigns by different subscription ages.

Test #3 analysis

After 2 days we checked every campaign and banned the zones with negative ROI. For the zones that started to show positive ROI, we increased the bids. We also noticed that the CTR of creatives is higher (than in the second test), so we decided to lower the custom bids. As a result, lowering the bids has brought even more leads.

This concludes the tests, here are the results:

We see that the subscription age OLD in the third test has negative ROI, while before that it always stayed positive. The reason could be that the creatives did not not catch on well at the start (i.e. there was a low CTR and the campaign started to get worse traffic), this campaign should have been stopped and restarted in 2 days, which usually helps.

We also can see subscription ages 0-3, 16-30 and 30-60 with a positive ROI. Note that we were lowering the bids in all campaigns during the test, but the leads increased. This is because we were identifying new sources and adding them to the group with increased custom bids. In other words, you need to check every day which zones should be banned, and where to raise bids.

Results by creatives:

In some campaigns, it was possible to exclude some creatives and add the new ones, the result would have been even better. But in this test, we haven’t excluded any.

Final results

From the tracker:

From the Rollerads ad network:

From the Golden Goose CPA network:

There Is a bit of discrepancy in the number of leads on the screenshots (about 1%), and here’s why:

• Different time zones in the tracker, Rollerads and GG. Because of this, the number of clicks and leads may not coincide by day.
• Leads in the tracker are assigned to a certain click_id, i.e. if the click was yesterday, the lead will be recorded in the tracker on the day when the click was made, not when the lead arrived. But in Rollerads and GG the lead is recorded on the day it arrived.

Quick tips

1. Apply three simple rules for analysis that we used after the first test.
2. Raise bids for good zones.
3. Don’t test 3 or more offers or prelanders at once. The proper A/B test is the 50/50 traffic distribution. Otherwise, you will almost certainly get wrong results and draw wrong conclusions.
4. You can’t change many parameters in campaigns at once, otherwise, you won’t see the dynamics of changing a certain parameter. If you have negative results, don’t exclude creatives and zones, pick one. We’d recommend focusing on the zone analysis first. The same applies to adding new creatives and raising bids. The adjustments should be going one at a time.
5. How to decide on the size of a custom bid? You can always consult with your ad network manager or change the bid according to your gut and see how it goes.

You might have noticed that this case study lacks one important element, i.e. the creatives that have been used together with our prelanders. We decided to save the best for last and give you an opportunity to delve deeper into the world of mVAS on a more personal level. If you want to see the creatives, DM your affiliate manager at GG, and they will gladly share the secrets of successful mVAS campaigns.