Update terminology and use case of ML applications #62
Conversation
Recent changes are narrowing down recommendations only to e-commerce use cases and in our opinion, this part is in the area of ML applications which is much wider. Great recommendations driven by ML are used in e-commerce, content, chat bots and many more areas. @michaelkleber might help is in review this change based on issues WICG/turtledove#36 and WICG/turtledove#31 Our assumption is both tickets might help in delivering item recommendations inside ads in TD
|
Yes, dynamic creative can contain all sorts of content, but products seemed like a good, concrete use case to discuss the issue. Is the only substantive change here from (Never mind: Can use the rich diff. Not a functionality I use often and forgot about it.) |
|
Changes seem fine, but I would probably change the terminology of "item recommendation" to "content recommendation." |
|
My apologies for any inconvenience. I'm not experienced in git as it is not used frequently in business operations as you can imagine :) If needed I can create a pull request once again. The key proposed change is to create sections about ML Applications use cases with specifying the main 3 areas and include item recommendations as one of them. Also, I made an update in Specialized Advertiser Needs where I have linked two possible solutions for item recommendations in Chrome linking to issues 36 and 31 Thanks and I look forward to feedback from other participants as well. |
|
|
||
| ## Coupon Management | ||
|
|
||
| As an Advertiser I want to be able to add coupons, or any form of targeted discounts to my creatives on specific period of times, potentially for specific products. | ||
|
|
||
| # ML Applications |
There was a problem hiding this comment.
I like this suggestion. I agree that this is worth calling out as a larger category of use-cases.
There was a problem hiding this comment.
Happy to hear that!
support_for_advertising_use_cases.md
Outdated
|
|
||
| ## Recommendation | ||
| Machine Learning Models are very important part of optimal investment in advertising. They are used by many participants in three major areas: |
There was a problem hiding this comment.
After reading this section, here is my suggestion for alternative wording:
"There are many advertising use-cases which rely on Machine Learning today. Common themes involve selecting from a vast number of options, and making efficient use of historical data. Following are several of the key uses of Machine Learning in advertising today:"
support_for_advertising_use_cases.md
Outdated
| Large e-Commerce websites may have product catalogs with millions of items. They have an interesting use-case. When running ads to promote items from their vast catalogs, which product should they show? | ||
| Today, much of this is mainly powered by data about which items a person has previously browsed. | ||
| ## Item recommendations | ||
|
|
||
| Typical sources of recommendation would be : | ||
| - Historical Products (compared to historical products browsed): abandoned cart, etc. | ||
| - Similar products (compared to historical products browsed) | ||
| - Complementary Products (compared to historical products browsed) | ||
| - ... | ||
| When there are many items to choose from (products/content/messages/apps) before rendering ad, which one should be shown to which person and in which ad? This produces more useful ads for consumers. Users are less annoyed with ads because they will more likely see items they would like to buy/convert or see less products they already bought/installed or they are not interested to buy/install. |
There was a problem hiding this comment.
I actually prefer the old wording, and would rather not change this.
What I like about the old wording is that it's more concrete. It helps the reader picture the use-case and understand. Your new wording is very abstract and vague. I worry it doesn't necessarily help the reader understand.
Honestly, I'm not really sure what you mean when you talk about selecting from many messages. What is this use-case? Is it worth calling out as a separate use-case?
There was a problem hiding this comment.
I actually prefer the old wording, and would rather not change this. What I like about the old wording is that it's more concrete. It helps the reader picture the use-case and understand. Your new wording is very abstract and vague. I worry it doesn't necessarily help the reader understand.
Valid point and it looks to me that @appascoe also has a similar point of view.
My main objective here was to point out that item recommendations are a greater concept than just a product which might be categorized underneath and that they have multidimensional application.
E-commerce indeed is a concrete example with such items as products (shoes,bikes etc). And the previous meaning was perfectly capturing it.
Other great examples can be:
- Publishers are willing to show recommendations based on historical content consumption.
- Brands willing to show copy recommendations according to the user's stage of the marketing funnel, based on historical data.
- App Publishers are willing to promote install other apps from their catalog, based on historical data.
- …
@benjaminsavage you have any proposals on how we can make it more general and applicable for different use cases?
Honestly, I'm not really sure what you mean when you talk about selecting from many messages. What is this use-case? Is it worth calling out as a separate use-case?
I placed an example in the above explanation
Probably changing “message” to “copy” will be more concrete and straightforward.
Maybe a great idea will be to try to catalog specialized types of advertisers in separate heading and call out the most characteristic and important from the perspective of the ad economy?
It might be useful for everyone to explain the specific needs of eCommerce, Brands, Apps Publishers etc. They are separate use cases from a business perspective. Also might be a great activation point to hear advertisers point of view? @benjaminsavage WDYT?
There was a problem hiding this comment.
I see what you're saying. I think the difference is that I would conceptualize of these things as either "Ad Selection" or "Format Automation".
I usually think of the mobile app install ad use-case is a matter of selecting which ad. Perhaps we can call that out specifically as an example in the "Ad Selection" use-case?
As for the changing ad-copy based on the stage in the marketing funnel, that I would categorize as "Format Automation". The same ad is selected, but the advertiser has multiple variants of how to display it. Perhaps we can add that as an example in the "Format Automation" section?
As for the publishers showing recommendations based on historical content consumption, I'm less clear on this use-case. By "suggestions" do you mean "ads"? Are you referring to the use of historical first-party interactions with that publisher as a feature in ad selection?
There was a problem hiding this comment.
I am not really sure what criteria are you using to differentiate those key uses in relation to concept (and I would be happy if you can share it here to have clear understanding on both ends)
In my understanding all above mentioned examples might be achieved by item recommendation in understanding that item is a component of an ad.
And if we think about an ad as a top rendered container in a client browser built from smaller components (items) like visualized products, copy, content, apps etc.
Additionally my understanding is that we might have different types of ads like display banners, native or video. Example: You need to use different components for banner ad and for native ad.
Also formatting would be defining variants how an ad might be rendered from style perspective to make it more attractive for users. Example some users are more likely clicking on the left side, others on the right side of the banner or some style in banner converts more likely than others. Following example might be when users are more likely clicking on animated ads than static.
So your proposals in my understanding are mixing up for me containers with components.
Nevertheless, I believe it is important and highly valuable for all group members to keep above mentioned use cases in document with comment allowing to understand why they are assigned to particular concepts.
@benjaminsavage could you please advice?
As for the publishers showing recommendations based on historical content consumption, I'm less clear on this use-case. By "suggestions" do you mean "ads"? Are you referring to the use of historical first-party interactions with that publisher as a feature in ad selection?
This use case represents when a publisher acts like an advertiser and they would like to use historical first-party interactions to build an ad with recommendation to their content. Example huge video publishers would like to target interest groups of their web page to revisit and see some new great video content items they have which are relevant and useful for those visitors.
There was a problem hiding this comment.
I understand where you're coming from. It sounds like in your ads system, you think of "ad" as the thing that is rendered on the page, and have generalized all the aspects of that ad as "items" and use ML to select the "items" that are combined to form the "ad".
That makes sense, but it's pretty different from how our ads system works - which is probably why I find this terminology confusing =). I guess my main concern is that "Item recommendations" sounds very vague.
Here's a recommendation:
1.) Ad-campaign selection.
e.g. There are many advertising campaigns to choose from. These might be promoting different types of conversion events (e.g. some are promoting mobile app installs, some are promoting purchases on websites, some are just optimized to generate link-clicks, some are optimized to generate video-views.)
2.) Ad-component selection.
Ads are composed of multiple components (e.g. the title, description, image, video, etc.). Some ads are promoting particular products. Once an ad-campaign is selected, ML is used to select which components to use in each particular ad impression. (e.g. which products to highlights, which title text to use, which image to use). These options are constrained by what the advertiser has provided as options for their ad-campaign.
3.) Formatting customization.
Once the ad-campaign and components are selected, choices remain about how exactly to render that set of components (e.g. button on the right or the left).
@lukwlodarczyk - what do you think of this set of categories?
There was a problem hiding this comment.
Looks comprehensive to me. And will be happy to do the final commit in this pull request. I will re-request a review soon.
Happy to see the great outcome of this thread!
support_for_advertising_use_cases.md
Outdated
| # Training ML Models | ||
| ## Bidding logic | ||
|
|
||
| Used for estimating efficiency of ad placements. When there are millions publishers to chose from and even more ad placements on those publishers with multiple available ad sizes, which one has highest chances of end conversion for advertisers and which one has highest return on investment? This drives better monetization opportunity for publishers and creates equal chances for big and small publishers to compete with performance results they do deliver for advertising campaign. |
There was a problem hiding this comment.
I suggest a few minor grammatical changes:
"Machine Learning is used to estimate the efficacy of ad placements. Across an ad network, there might be millions of placements, across hundreds of thousands of publisher websites, each with different properties (like size and location on the page). How much return on investment will each placement drive for each advertiser? Machine learning is used to make such estimates."
There was a problem hiding this comment.
I like the above suggestion but I would add some additional words about the importance of a publisher's point of view.
Keeping maxim possible effective bidding logic is driving competition on the performance level between publishers and creating equal chances no matter what size of publisher you are.
For example, you can be a small, niche publisher but if you have great native experience ad placement you can compete with your ad performance next to big publishers.
Or maybe we should consider to move this to the Publisher's needs?
There was a problem hiding this comment.
I would prefer to keep this document as factual as possible, statements like:
"This is the usecase. Here's an example to illustrate it. This is why it's done. This is how it is achieved today."
I would rather avoid more emotive language about what is good and bad etc.
Perhaps we could dispassionately explain the downsides of poor-quality predictions?
From my perspective - the connection between "effective bidding logic" facilitated by ML predictions on the ad-network side, and "competition between publishers" is tenuous at best... Can you try to link the two more clearly?
Here's an example of what I think is a very factual problem one would encounter with low-quality predictions:
"The lower the accuracy of the prediction, the fewer business outcomes will be generated for the same budget. This in turn leads to lower publisher revenue."
Alternatively, we could consider it from the perspective of "model calibration". If one were to use a global average estimate of the likelihood of a conversion, and apply this same prediction to all placements, this would result in poorly calibrated campaigns. There would be some placements where the true likelihood of a conversion was much lower than the average, and bidding the average value there would result in significantly less value creation than expected."
There was a problem hiding this comment.
Noted.
My proposal:
"The lower the accuracy of the prediction in bidding logic, the fewer business outcomes will be generated for the same budget and less precise bidding for particular ad placements may occur. This in turn leads to lower publisher revenue."
There was a problem hiding this comment.
That sounds great!
I think you could eliminate the clause "and less precise bidding for particular ad placements may occur", because "fewer business outcomes" and "lower publisher revenue" are the concrete consequences of that "less precise bidding".
support_for_advertising_use_cases.md
Outdated
| ## Ad selection | ||
|
|
||
| When there are many template ads (style of ad, banner, native, video etc) to choose from, which one should be shown to which person? This produces more attractive ads for people and by increasing overall efficiency it drives publishers revenues. | ||
|
|
||
| Ad selection is a very important task. When there are many ads to choose from, which one should be shown to which person? The most effective approach to-date has proven to be using machine learning algorithms to assist with ad selection. This produces more relevant ads for people, and more business outcomes for advertisers. This in turn increases publisher revenues. | ||
| The most effective approach to-date has proven to be using machine learning algorithms to assist with ad selection, product recommendation and in bidding logic. This produces more usefull ads for people, and more business outcomes for advertisers. This in turn increases publisher revenues. |
There was a problem hiding this comment.
I think it's worth splitting this out into two separate ML use-cases.
I would prefer to keep the section that was removed and call that: "Ad Selection".
The newly suggested language seems to refer to a different use-case: Given an ad, what is the optimal format to use to render it? Perhaps we can call this use-case "Format automation".
Here is my suggestion:
Ad Selection
Ad selection is a very important task. When there are many ads to choose from, which one should be shown to which person? The most effective approach to-date has proven to be using machine learning algorithms to assist with ad selection. This produces more relevant ads for people, and more business outcomes for advertisers. This in turn increases publisher revenues.
You also briefly touch upon product recommendation, but that use-case was already covered previously in the section called: "Item recommendations"
Format automation
Once a particular ad has been selected, there remains the important task of deciding how to render it. There are many "degrees of freedom" to select from:
- Which format? Video? Image? Other?
- What color to use?
- What size of borders?
- Which description / title to use?
@benjaminsavage,
Thank you for putting on the agenda on last Privacy BG and Privacy CG call recent study about the importance of personalization done by FB.
As mentioned over a call I see that we might be using different terminology, however, we are talking about similar items that are important from the perspective of keeping usable advertising and profitable for publishers in high respect to internet users' privacy.
Recent changes in this document are narrowing down recommendations only to e-commerce use cases and in our opinion, this part is in the area of ML applications which is much wider.
Great recommendations driven by ML are used in e-commerce, content, chatbots and many more areas.
@michaelkleber might help us in review this change based on issues issue #36. and #31
Our assumption is both tickets might help in delivering item recommendations inside ads in TD
@appascoe please let us know if you think that this might be also useful from the perspective of issue #36.
In issue #31 we also write about item recommendations.