How to Run an A/B Test with Firebase and Remote Configuration

Have you ever wished you could have done A/B testing with your life because girlfriend A is superior to girlfriend B?

Test with Firebase

Apologies, ladies and gents, but the good news is that you can use Firebase with Remote Config on your app to perform A/B testing.

Apologies, ladies and gents, but the good news is that you can use Firebase with Remote Config on your app to perform A/B testing.

You can use the feature to test minor modifications such as the positioning of a button, whether or not to send a notification, the flow of an application, and so on.

But how do you tell if you're doing well or badly?

Is that correct?

Of course, you could question your girlfriend about the changes you've made, but that would take a long time.

The best method to find out is to use Firebase Remote Config to execute an A/B test.

How Does It Work?

Firebase Testing is based on Remote Config, a cloud service that allows you to alter the functionality and appearance of your app without requiring users to download a new version.

From the Firebase Console, you can choose which users will be affected by the update.

Firebase divides your app into groups and ensures that each group receives the correct version of your app.

After then, Firebase uses advanced Bayesian statistics to assess the data you get back from your app, so you can be assured that the data you're seeing isn't just a fluke.

Once the victor of the A/B test is determined, you can roll out your app improvements to the remainder of the app's users and give them a notification.

All of this was accomplished using the Firebase console.

As a result, your growth team can design an experiment that they like without wasting your technical team's time.

How to Run an A/B Test with Firebase and Remote Configuration

Step 1: Setup Remote Config on Project

You may have noticed that building, testing, and publishing a new version of the app only to modify a hard code value is really inconvenient.

However, with Firebase Remote Config, you may update that value in the Firebase console without having to republish your app.

Remote configuration allows you to establish a set of named parameters and assign values to them from the terminal.

Then, in your programme, you may retrieve those values and use them as needed.

Your app will be notified whenever you modify a parameter value in the console.

I'll teach you how to read various values from Firebase Remote Config in this section.

Step A: Create a New project

To begin, open Android Studio and start a new project.

Step B: Incorporate Firebase Remote Configuration into Your App

  1. Select Tools > Firebase from the drop-down menu.
  2. A new window comes after that. Set up Firebase Remote Config by clicking Remote Config > Setup Firebase Remote Config.
  3. Connect your app to Firebase and add dependencies to it now.
  4. Similarly, you must include Analytics in your project because remote configuration requires analytics.

Step C: Design your layout

activity_main.xml

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:padding="40dp"
    tools:context="MainActivity">

    <ImageView
        android:id="@+id/icon"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerHorizontal="true"
        android:layout_marginBottom="16dp"
        android:src="@drawable/googleg_disabled_color_18" />

    <TextView
        android:id="@+id/welcomeTextView"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_below="@+id/icon"
        android:layout_centerHorizontal="true"
        android:text="Welcome..." />

    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="@string/fetch_remote_welcome_message"
        android:id="@+id/fetchButton"
        android:layout_centerVertical="true"
        android:layout_centerHorizontal="true" />

</RelativeLayout>

Step D: Implements codes

In this tutorial, you'll learn how to implement code in four steps.

1. To begin, initialize RemoteConfig and obtain the instance.

//At the start of the activity, initialise.
private FirebaseRemoteConfig mFirebaseRemoteConfig;             
 
//Get an instance of Remote Config.
mFirebaseRemoteConfig = FirebaseRemoteConfig.getInstance();

2. Get a remote config singleton object.

mFirebaseRemoteConfig = FirebaseRemoteConfig.getInstance();
FirebaseRemoteConfigSettings configSettings = new FirebaseRemoteConfigSettings.Builder()
//Remember that in production, a third of a second is recommended; else, your app will be throttled.
        .setMinimumFetchIntervalInSeconds(3)                                    //This will determine the fetch time.
        .build();
mFirebaseRemoteConfig.setConfigSettingsAsync(configSettings);

The singleton object is used to hold default parameter values in your app, fetch updated parameter values from the Firebase backend, and govern when those values are made available to your app.

Setting a low fetch value during development is recommended, but you should increase it in production if you don't want the app to be throttled.

3. Values for the App's Default Parameters

You can set in-app default parameter values within the remote config default.xml file (which we'll use within the next steps) so your app behaves evidently before connecting to the firebase Remote Config backend, so that default values square measure continually out there if none square measure set within the firebase backend system.

mFirebaseRemoteConfig.setDefaultsAsync(R.xml.remote_config_defaults);

4. Values to Fetch & Activate

Call the fetch() method if you only want to get values from the Firebase backend.

Similarly, you may use the activate() method to activate values from the Firebase backend. You use the fetchAndActivate() method to retrieve and activate values in a single line of code.

When the procedure is finished, you can additionally handle the callbacks.

mFirebaseRemoteConfig.fetchAndActivate()
        .addOnCompleteListener(this, new OnCompleteListener<Boolean>() 
{
@Override
public void onComplete(@NonNull Task<Boolean> task) 
{
    if (task.isSuccessful()) {
        boolean updated = task.getResult();
        Log.d(TAG, "Config params updated: " + updated);
        Toast.makeText(MainActivity.this, "Fetch and activate succeeded",
                Toast.LENGTH_SHORT).show();

    } else {
        Toast.makeText(MainActivity.this, "Fetch failed",
                Toast.LENGTH_SHORT).show();
    }
    displayWelcomeMessage();
}
});

Here is the complete code of MainActivity.java

package com.example.testdemo;

import android.view.View;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Log;
import android.widget.Button;
import androidx.annotation.NonNull;
import android.widget.TextView;
import android.widget.Toast;
import android.os.Bundle;

import com.google.firebase.remoteconfig.FirebaseRemoteConfig;
import com.google.firebase.remoteconfig.FirebaseRemoteConfigSettings;
import com.google.android.gms.tasks.OnCompleteListener;
import com.google.android.gms.tasks.Task;

public class MainActivity extends AppCompatActivity 
{
    private static final String TAG = "MainActivity";

    //Keys for Remote Configuration
    private static final String LOADING_PHRASE_CONFIG_KEY = "loading_phrase";
    private static final String WELCOME_MESSAGE_KEY = "welcome_message";
    private static final String WELCOME_MESSAGE_CAPS_KEY = "welcome_message_caps";

    private FirebaseRemoteConfig mFirebaseRemoteConfig;
    private TextView mWelcomeTextView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        mWelcomeTextView = findViewById(R.id.welcomeTextView);

        Button fetchButton = findViewById(R.id.fetchButton);
        fetchButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                fetchWelcome();
            }
        });

        //Get an instance of Remote Config.
        //[START get_remote_config_instance]
        mFirebaseRemoteConfig = FirebaseRemoteConfig.getInstance();
        //[END get_remote_config_instance]

        //To enable developer mode, create a Remote Config Setting, which you can utilise to boost performance.
        //During development, the number of fetches available each hour. Use Remote Configuration as well.
        //To set the minimum fetch interval, use the setting.
        //[START enable_dev_mode]
        FirebaseRemoteConfigSettings configSettings = new FirebaseRemoteConfigSettings.Builder()
   //Remember that in production, a third of a second is recommended; else, your app will be throttled.
                .setMinimumFetchIntervalInSeconds(3)                    
                .build();
        mFirebaseRemoteConfig.setConfigSettingsAsync(configSettings);
        //[END enable_dev_mode]

        //Set the default values for Remote Config parameters. The default values of an app are used, and
        //You set an updated value for only the settings you need to change when you need to edit those defaults.
        //In the Firebase console, you wish to make a modification. For more information, see the README's Best Practices section.
        // information.
        // [START set_default_values]
        mFirebaseRemoteConfig.setDefaultsAsync(R.xml.remote_config_defaults);
        // [END set_default_values]

        fetchWelcome();
    }

    /**
     * Obtain and enable a welcome message from the Remote Config service.
     */
    private void fetchWelcome() 
{        mWelcomeTextView.setText(mFirebaseRemoteConfig.getString(LOADING_PHRASE_CONFIG_KEY));

        //[START fetch_config_with_callback]
        mFirebaseRemoteConfig.fetchAndActivate()
                .addOnCompleteListener(this, new OnCompleteListener<Boolean>() {
                    @Override
                    public void onComplete(@NonNull Task<Boolean> task) {
                        if (task.isSuccessful()) {
                            boolean updated = task.getResult();
                            Log.d(TAG, "Config params updated: " + updated);
                            Toast.makeText(MainActivity.this, "Fetch and activate succeeded",
                                    Toast.LENGTH_SHORT).show();

                        } else {
                            Toast.makeText(MainActivity.this, "Fetch failed",
                                    Toast.LENGTH_SHORT).show();
                        }
                        displayWelcomeMessage();
                    }
                });
        // [END fetch_config_with_callback]
    }

    /**
     * If welcome message caps is set to true, a welcome message will be displayed in all caps. In any case,
     * show the welcome message retrieved from welcome message
     */
    //[START display_welcome_message]
    private void displayWelcomeMessage() 
{
        Log.d(TAG, "displayWelcomeMessage() " + "called");
        //[START get_config_values]
        String welcomeMessage = mFirebaseRemoteConfig.getString(WELCOME_MESSAGE_KEY);
        //[END get_config_values]
        if (mFirebaseRemoteConfig.getBoolean(WELCOME_MESSAGE_CAPS_KEY)) {
            mWelcomeTextView.setAllCaps(true);
        } else {
            mWelcomeTextView.setAllCaps(false);
        }
        mWelcomeTextView.setText(welcomeMessage);
    }
    //[END display_welcome_message]
}

Step E: In the app, set the default parameters value.

  1. Right-click Res > New > Android Resource Directory to establish a new resource directory.
  2. After that, a popup box displays, where you can put XML as the directory name and resource type, then click OK.
  3. Create a new XML resource by selecting XML -> new -> XML Resource File and pasting the code below.
These values are basically key-value pairs.

<?xml version="1.0" encoding="utf-8"?>
<!-- START xml_defaults -->
<defaultsMap>
    <entry>
        <key>loading_phrase</key>
        <value>Fetching config…</value>
    </entry>
    <entry>
        <key>welcome_message_caps</key>
        <value>false</value>
    </entry>
    <entry>
        <key>welcome_message</key>
        <value>Welcome to my awesome app!</value>
    </entry>
</defaultsMap>
<!-- END xml_defaults -->

Step F: In the Remote Config Firebase Console, change the values of the parameters.


Now go to the console and type in key-values before publishing and running the project.

Step 2: Creating an Experiment

To begin, go to the console and choose your project.

After that, select A/B Testing -> Create Experiment.

Now you'll work with Remote Config, which you can access by clicking on it.

The remote configuration experiment is divided into four parts:

A. Fundamental:

In the first step, you enter the essential information, such as the experiment's name and description (brief and to the point), and then click Next.

B. Choosing a Target:

After that, choose the user you want to target.

Note that you cannot conduct your experiment on the same Android and iOS app at the same time due to the differing use patterns of these platforms.

That's why it won't work on a different computer. Making a separate experiment is the best method.

By clicking on the and buttons, you may create a campaign.

In my situation, I'm aiming for a device that only the English-speaking audience has.

You can also choose your country/region, app version, and user audience.

Percentage of target audience: I'm using 5% of the overall audience in this situation.

C. Objective:

Now choose Retention as your goal, for example, why you want to test your app.

User engagement, retention rate (the amount of time people spent with your app), in-app-purchase money, AdMob income, total income, and so on are all high-level goals.

The software also has analytics events like a crash-free user, session start, and so on.

To add more goals, click the Add metric+ button.

You can test up to five metrics at once.

D. Subtypes:

Finally, provide a variant in your app.

In my example, I'm comparing variations A and B. You can also add numerous versions. You previously defined the parameter in remote config defaults.xml, as you know.

If you don't choose a value, the default value is used.

Now select Review from the drop-down menu.

You can now get a summary of your test results.

Everything appears to be in order.

But how can you know whether or not my text will fit in the space and look good?

Organize your tests

Fortunately, A/B testing makes evaluating these variants simple.

You must first obtain an instance ID token (a unique identifier) for our app and attach it to the instance of our app operating on a given device, after which you must instruct remote config to send a specific variant to that specific app instance.

So, before we publish our experiment, we can try out each option.

So we'll need to write a little code to acquire our instance-id token.

//Get the token from the Firebase app and LogCat.
String token =FirebaseInstanceId.getInstance().getToken();
Log.e("Token", "t: "+token);

Simply copy and paste the preceding code into the onCreate() method of MainActivity.java and start the project.

Return to the console and scroll down to Manage test devices on the summary screen.

A dialogue box appears. Enter the Token ID, select the Variant, and then click Add and Save.

At the bottom of the Details section, you can now see "1 test device is receiving this experiment."

You may also launch the app and see the details in TextView.

Your experiment is now complete. Start the experiment by clicking the Start button.

Step 3: Recognize the Experiment's Outcomes.

Now is the time to look at the A/B test findings.

During the testing phase, it is recommended that you do not make any changes to your app until the A/B test has determined a winner.

When a user notices something unusual in your app, they notice it and respond.

However, this does not imply that the adjustment will be beneficial in the long run.

What if the changes you make aren't good and others don't like them?

Now go to the terminal, open your project, and open your experiment under the A/B testing tab. As you can see, I have no results.

The reason for this is that I'm putting this app through its paces on a sample app with a monthly active user count of one.

However, when I test on a few different devices, the results aren't much better.

As a result, you must test this on a real app that is already in production and has real users.

Finally, if you apply the same method as I did and don't receive any results, it's possible that you forgot to update the remote-config libraries.

You must update your build.gradle (app) file with the most recent libraries.

Post a Comment

Previous Post Next Post