You are planning an A/B test on Bing. Under what circumstances would you opt for a user-tied flight instead of an un-tied flight? Please discuss the advantages and disadvantages of both options.

This question requires you to show your understanding of two flight types in A/B testing—user-tied and un-tied—specifically in relation to Bing, the search engine. You should explain when you would opt for each type and outline the advantages and disadvantages associated with them. User-tied flights ensure that all requests from the same user are directed to the same variant of the test, no matter how many times the user visits the site. In contrast, un-tied flights randomly assign each request to a variant, irrespective of the user's prior visits. To effectively respond to this question, you need to: Define both user-tied and un-tied flights within the context of A/B testing. Clarify when it would be suitable to utilize each type in the design of a test for Bing. Examine the benefits and drawbacks of both user-tied and un-tied flights.

User-tied flight, also referred to as 'sticky' flighting, ensures that a user engages with the same variant throughout an A/B test, regardless of how many times they visit the site. In contrast, un-tied flights randomly assign each of a user's requests to different test variants, without considering which variant the user encountered in prior visits. I would opt for a user-tied flight when the A/B test includes modifications that extend across several page views or sessions, such as improvements to the user interface. This approach guarantees a consistent experience for users. The advantage of user-tied flights is their ability to accurately measure changes that have lasting impacts or necessitate multiple interactions. However, a potential downside is that they might introduce bias due to users' previous exposure. On the other hand, un-tied flights are beneficial for tests that involve changes not reliant on multiple interactions or that don’t accumulate over time. This could encompass alterations to the search algorithm or the Search Engine Results Page (SERP). Un-tied flights are straightforward to implement and offer a fresh perspective each time a user visits Bing. However, they can yield inconsistent results, particularly if there are many returning users.

User-tied flights, often referred to as persistent flights, guarantee that a user encounters the same variant throughout the testing period. This approach is ideal for scenarios where the feature being tested has a long-term effect on the user or necessitates multiple interactions, such as evaluating a new onboarding process on Bing. The primary benefit of user-tied flights is the dependable results they yield for changes or tests that involve several interactions or span over time. However, a significant drawback is the potential for bias stemming from users' previous experiences or interactions. In contrast, un-tied flights are implemented when each user interaction should be a unique experience, making them suitable for tests that involve one-time interactions, like modifications in ad placements. The advantage of un-tied flights lies in their ability to create a more varied and diverse data set, as users are randomly assigned to different variants with each interaction. Nevertheless, the downside is that the results may be inconsistent due to the absence of uniform user experiences.