Immersive shopping experience with Amazon Anywhere

shall we? oh, my goodness is so long. well, before we get going, i wanted to make sure that everyone is where they should be so raise your hands. who love shopping as much as i do. ok. well, i hope by the end of the session the amount of hands will change.

but um hello and welcome everyone to our breakout session. um it's truly an honor to see so many innovative faces here. get us uh to, to learn about the immersive shopping experience with amazon anywhere.

my name is maria. i am a senior business development manager at aws. i'm a part of the account team supporting amazon as a customer here. we have steve uh director of engineering at amazon and alan engineering lead at amazon. anywhere in today's rapidly evolving digital landscape. the way we shop is antigo a revolution.

just think how many boxes are being delivered to your house weekly. in my case, it's daily and my husband is absolutely unhappy about that, but that's a separate story.

um traditional retail is no longer limited to the brick and mortar stores. instead, it's expanding into the boundless realms of digital landscape. uh immersive shopping experiences powered by cutting edge technologies are redefining and blurring the way we shop.

so today we are going to talk about amazon. anyway, we will explore the transformative power of immersive experiences. we will discuss how this this experience is reshaping the way we shop and also drive the business growth.

steven allen will share the insights and practical takeaways and you will also have a chance to ask your questions and network with amazon. anyway, team after the session as we embark on this journey, i encourage you to embrace the spirit of curiosity and creativity.

let's challenge the status quo. let's think of the creative uh ideas that hopefully will emerge in your head after the session and hopefully we'll have new ways of how we show in future. let's have some fun.

steve owe it to you.

thank you.

hello, everyone. um so my name is steve macbeth. i'm a director at amazon. um i haven't been to aws re invent for about 12 years. it was my second time coming. um 12 years ago, i attended when i was working at microsoft and it was a very eye opening experience for me.

um seeing where the cloud computing was and where the open source tool chains were relative to the technology we were using in house uh to build scalable online systems was um a really profound moment for me.

um and i feel like coming back now 12 years later, um in this moment of both immersive shopping, which we're gonna talk about today and generative a i, which is, i imagine why many of you are here? i think there's the same kind of profound change happening in the industry.

um so we've been looking at our customer base um and trying to understand where the next generation of customer is different from the current generation and where they're the same.

um one of the things that is unique about the next generation is that they've grown in up and been raised in a world of digital technology. and so the the the shoppers and customers and consumers that are coming of age today were born after the.net bust.

um and so they've lived in a world where the internet is well established. um online services are prevalent um and their world is very immersive, um their video, their social media, how they interact with, with technology, even the physical world is becoming more and more immersive.

and we think that changes the way people wanna shop. um it changes the degree to which people want to be able to shop in the context of what they're currently doing.

um and the work that we've been doing around amazon anywhere is a mechanism that we believe can start to bridge that gap um between the inspiration you're achieving um in whatever immersive experience you are or, or in the real world and the ability to buy or share the things that inspire you.

so a little bit about amazon. how many people have shopped on amazon? yes. um quite a few. right. yes. so at amazon, we talk about selection price and convenience and we believe these are the fundamental levers that help us build trust with customers. it's why people come to amazon.

um you can have a high confidence that the thing you want is available. uh you can have high confidence that it's at a low price. um and you can be assured that it's gonna get delivered on time. and if there's any problem with it, you can, you can return it.

um at amazon, we talk about a fly wheel around these, these mechanisms. um and that flywheel has allowed us to grow and scale the business over time. and the way that flywheel works is that as customers come and buy, generate profit, we reinvest that profit back in increasing our selection or decreasing our price or increasing convenience.

and so that then drives more people to come to the site um because has more selection, better pricing and more convenience um generating more profit which we invest back. and so it's that flywheel that drives the growth of the amazon retail business.

um and it's allowed us to get to this scale and to this size. and as we start to think about where we wanna go to bridge that gap, i talked about, we wanna be able to bring that massive selection, low prices and convenience to where inspiration is being generated.

and so amazon anywhere, is this basic idea that when you have an inspiration, if you're watching a show or watching a sporting event or in the real world, you want to be able to bridge the moment of inspiration that you have and the ability to purchase.

and so amazon anywhere was that idea? and the team set out to build a set of api s um and capabilities to enable any developer to integrate that capability into their own experience to basically empower your audience to discover and purchase projects from amazon. so bring that amazon flywheel into your application.

um one of the challenges that an immersive experience has is that the experience breaks down when you have to move out of the immersion. and so every time you take a customer out of that, um it's less immersive and, and the value of immersive experiences is that you can sort of let the side things in your life fade away for a moment, you know, when you're watching a movie,

um sometimes you forget that you're in a movie theater and it's just the movie that's consumed you because it's very immersive if you're watching a sporting event. um, if you're traveling in the world, it's these moments of presence that we have that are profound and when you have a distraction in those moments,

um it breaks that immersion and amazon anywhere is trying to solve that problem, the more immersed our customers are, the more value we see in allowing them to purchase the things they want without breaking that immersion or share the things that they see, the things that have inspired them with friends and family.

um and so very often for me and i'm sure this is true for many of you. um when you see that moment of inspiration, you wanna capture it for yourself, you wanna act on it, you wanna share it with somebody.

um but today, there's quite a bit of friction in that process. the mechanisms in which you can um understand what you have to do next is very high friction. and so if you're watching a show and you have that moment of inspiration,

um and you want to try to find out how to fulfill that today, you have to remember it after the show or you have to bring up your phone and do a google search, um find a pinterest board where they talk about the thing that you're doing,

um find other people who are interested in the thing and maybe have curated information around that. um and then go find where you can purchase that. and often these niche opportunities where you can purchase them aren't a place where you have confidence in buying.

um and so we really believe there is an opportunity in building a set of capability that can be deeply immersed in the experience that can be contextual to the experience um that can understand the intent of what you're doing in that immersed experience and surface things that are relevant for you.

so our first customer was niantic uh niantic builds pokemon go. um and recently came out with a game called paradot. um and so this was the very first customer that integrated amazon anywhere api s into their application.

so this is a deeply immersive a r game um in which you build a dot um and or hatch a dot and, and then you um you train it and feed it and, and it grows. and so here you can see the.in your kitchen.

um and we believe that people have an emotional attachment that builds up over time um like any immersive experience. and so if you want to take that emotion of, i'm happy about my dot i'm excited. it's doing something cute in the kitchen.

um and translate that into something in the physical world. i wanna get a shirt with my dot on it. i wanna buy a coffee mug, whatever it is, whatever representation you want in the physical world of that experience.

if you have to leave the experience again, it starts to break down that immersion. um so this is a very simple example where you can go to a curated catalog, um find something that helps you represent your.in the physical world.

you have all the benefits of the amazon purchase experience. you can get whatever size you want, you can see what the delivery guarantee is, you can be um confident that if it doesn't fit, you're gonna be able to return it.

um and then when you finish that experience, you go immediately back into the immersive experience. so this is good for the customer in that it's a low friction way to achieve what they wanted. and it's also good for the developer um because you don't drive customers and your audience outside of the existing experience.

um in order to fulfill that inspiration a few years down the road, i imagine we will look back to this launch as the start of what will have become a normal way to make purchases from within virtual worlds and environments.

so this is a quote by my boss steve downer. um he's the vice president of the consumer electronics category at amazon.com.

um i think what's interesting about this quote is the term normal. so earlier on the amazon slide, i showed roughly we ship 300 million packages a month in the us the same day or the next day.

um at that scale, it's become normal to decide. i want something and have it show up on the same day from a catalog of hundreds of millions of products.

um five years ago, 10 years ago, this was not normal. this would have seemed like magic if somebody proposed a business which is i'm gonna have a catalog of 100 million things and then i'm gonna deliver them at the scale of 100 million a month to hundreds of millions of customers. and i'm gonna do it the same day. this would have seemed like magic.

um but today it's normal, it's the expectation we have people who were born in this environment will expect it. and when they don't get it, they will be disappointed.

um and so this concept of normal is transitional. what was normal today was magic yesterday. um and so we believe that this idea of being able to purchase anything you see, anything you're inspired by is really important.

and it helps create these immersive experiences where you don't have to feel the frustration that happens when you decide i wanna buy this thing.

so very often you'll be watching a show and you'll see a present for somebody you'll think. oh, this is, this is a great idea for a christmas present, for my spouse or for my, for my children.

um but now you have a moment of frustration. what do i do with that inspiration that i've that i've gotten in that moment. i'm watching a sporting event. i'm watching a movie. i'm out in the real world.

i have to catalog it in my own mind. i become the systems integrator. i hold the information about what i wanna do. i hopefully remember it when i get back to a moment in my computer on my phone.

um and then i have to go do all the work. i have to find it. i have to remember it. a friend of mine, we were watching ted lasso, one of my favorite shows.

um, and she saw this, this dress, um, and she was like, i want that dress. um, and she's not, she doesn't like searching. so she always sends me these messages. i saw this thing and something and then can you go find it?

um, it is not easy to find a dress from one tv show. um, but i did find it, but it was very, very difficult. um, i had to search for people talking about that episode and then people commenting on that dress and then sort of got the parameters of what that dress was called and then i searched for it and eventually i could find it.

um and unfortunately, it was out of stock and they weren't making it anymore

So she was very, very disappointed and I was very, very disappointed. Um, and it took a lot of effort. I think most cases, people aren't gonna put in that much effort. Um, and those moments of inspiration get lost and the person who was excited is no longer excited. Um, and so we think we can solve that.

Alan's gonna spend some time and now take you through the architecture of how we built this solution. Um, and then we'll come back to some of the launches we've had since that first customer.

Thank you, Steve. Let me tell you a little bit about how Amazon Anywhere got started at Amazon. One of our leadership principles is think big. Thinking big is a great way to challenge our assumptions about what's possible. And Amazon Anywhere came from just such an idea.

We imagined a world in which people did not need to go to a store in person or even online to find products and make purchases. In the future, more people may spend more of their time immersed in virtual worlds. How would people find products and make purchases in such experiences? What if we could make immersive shopping experiences that allow them to buy anything anywhere and have it delivered to them at home?

The first thing that we do when we have an idea like that at Amazon is we write a press release about it and answer any frequently asked questions. This helps us get a clear picture of what we want to deliver for our customers and the benefits that we'll have. We call this APR FAQ.

We had many ideas for potential features for Amazon Anywhere, but we had to narrow down our feature list only the most critical features to build for our first prototype. This allows us to quickly build and launch our product, followed by collecting important customer feedback. And afterwards, the cycle can repeat.

This diagram shows a simple representation of our core use cases. A customer is interacting with an application on a mobile phone. And this mobile application has access to APIs that enable product selection such as availability, product information and pricing and check out APIs such as the preview and confirmation of orders.

The application is provided with SDKs for developers to make it easy to onboard to the API. Our API must scale for global Amazon scale as well as provide analytics on the usage metrics to the business. When a customer makes a purchase, that purchase is delivered directly by Amazon and all post purchase customer support is handled by Amazon.

As we began thinking through what we wanted to build, we reached out to the account team to validate our design decisions and to provide any suggestions on technology choices we may not have thought of. Amazon is similar to other AWS customers. They might have a deeper knowledge of some of the AWS services, but they do need some help which services to use and which services to consider. We do maintain the same level of security and isolation with them as we do with any other AWS customer.

How many services do you think AWS has - 200 to 250? I think we're getting closer to 300. And after this week, there will be even more. So there can be some blurred lines which services to pick. When Alan and our team first met, Alan already had a vision what he wanted to build, which he formed on the examples from AWS Blocks. However, he needed some help with networking, authorization and scaling.

We partnered together to understand his requirements and goals in order to put together a solution and a better package of advice. While working with Alan, we also built a few POCs to validate our suggestions and to make sure we share our best practices and mitigate the architectural risk for Amazon Anywhere.

I remember Alan was saying, "Hey, you should work on desktop, mobile, video games." I tried to practice my Alan voice but I failed. So um, we also had less than eight months to design and deliver a production ready product. And last but not least, we needed to keep in mind the ambition of the Amazon Anywhere team to expand globally.

So let's dive deep into the challenges we faced. Let's talk about some of the key challenges for the design for Amazon Anywhere. One of the early questions we got from the business was how can we enable this experience and make it secure? We set out to limit any data sharing to only the minimum amount necessary.

We set a high bar for availability at a target of 99.99% which is the same as many AWS services. While we plan to launch with a single partner, we needed to anticipate the onboarding of multiple clients. And we want the design to scale to up to 100,000 requests per second.

In the early stages, the product requirements were still evolving and we had a very short timeline. I faced an important decision to make. Should we start with a microservices architecture immediately or start with a simpler monolithic design to meet our deadline and then consider any future extensibility challenges?

Our goal is to enable a seamless developer integration experience. They shouldn't need to be an expert at ecommerce to use our API. We wanted to offer the flexibility to develop in any language on any platform without the burden of coding multiple client libraries.

Now that we've discussed the core requirements and key challenges, let's dive into the solution and some of the design decisions that we made.

Next, I will walk you through how we designed and built Amazon Anywhere on AWS. We decided to separate the platform into microservices. I wanna stop here for a minute because I don't know if everyone here is just sold on microservices and that's the go-to solution. But you know, when we have a short timeline, I remember a comment on one of the design docs that we had from my boss who was like, "Yeah, we have like six months. Are you sure you don't want to do some monolithic solution? Let's get this done."

So um, you know, we can talk a little bit more about that later, but we decided to split it into microservices. And at the front, we have our gateway. It hosts our public endpoint and it handles all authorization, authentication and request routing.

Next, we have Selection. Our Selection service handles all queries for product information, availability and pricing. There's an Ordering service which handles APIs for checkout such as the preview and confirmation of orders.

Next, we had a need for Configuration to store program specific options and client IDs and other onboarding data. And last, like many businesses we needed to collect business intelligence data such as the number and types of orders created through this platform.

Our best practice involves partitioning each service into dedicated AWS accounts further separated by environment stage such as beta and prod, and regions such as North America and Europe. This approach has several key benefits. It ensures the isolation of each service, minimizing any potential impact to bad deployments or other failures.

It also more readily accommodates organizational growth allowing for finer grained access controls in the future transfer of ownership. It also reduces the risk of AWS service limit contention around AWS account quotas and we can capture all costs associated with each service within that account's billing facilitating better cost management.

Let's start with our gateway and some of the common security threats and how AWS services helped us solve those. When we started talking about security, we compared basically the benefits of the main AWS security services through the lens of common security threats.

So if we look at web exploits and bots, APIs are susceptible to attacks from malicious bots and automated scripts that can compromise the service availability. And this is where AWS Web Application Firewall or AWS WAF can help protect against those threats.

Distributed Denial of Service or DDoS attacks - your APIs can get overwhelmed with a flood of traffic causing service disruptions. AWS Shield and DDoS can help to detect and mitigate DOS attacks in real time. Worth mentioning, Route 53 health checks in combination with DDoS protection can actually reroute your traffic from the infected services.

Data tampering and replay attacks - attackers can attempt to tamper with your API clients and API endpoints. So this is where ACM or AWS Certificate Manager steps in and helps you easily provision and manage and deploy your TLS certificates. And ACM can also mitigate common threats like data tampering, malicious attacks, and man-in-the-middle attacks.

Now that we've had solutions for registering our domains, TLS certificates, and handle common security threats, let's determine how we'll handle authentication, authorization and request routing.

Let's talk about some of our requirements. There are two types of authentication that we needed for Amazon Anywhere. The first is client authentication. We needed to be able to identify our clients, provide different configurations for each client, and track usage metrics. This level of authentication is useful for creating APIs that do not require customer context - this is information like public product data, information and pricing.

The second level of authentication that we needed was customer authentication. We had a need to identify customers through account linking. Customers can review and accept or deny any data sharing permissions. This level of authentication enables APIs that query or respond with user or customer context - this for example, tailoring the estimated delivery date to your personal location and Prime status.

Remember one of the early questions we got from the business was how can we enable this and make it secure? Obviously we can't have customers need to share their Amazon credentials - that would not be acceptable. Our solution for this was to integrate with Login with Amazon. Login with Amazon is an existing service that allows third party applications and websites use logging in to third party applications and websites using your Amazon account. It's based on the industry standard OAuth 2.0 protocol which allows application developers to access permitted user data without sharing account credentials.

Well, I find OAuth to be a very interesting topic and I think I spent several days reading all of the RFCs on OAuth. I won't go into all the details here other than to say that integrating with Login with Amazon was a design decision and a key requirement.

Maria will now share the options that we considered for implementing our authorization and request routing.

I feel like I'm getting good exercise, getting up and down all the time. So let's dive into the comparison of Amazon API Gateway and Application Load Balancer with Amazon Anywhere.

We compare these two services from the angle of routing, authentication and scaling. Both API Gateway and Application Load Balancer have impressive features. When it comes to routing, they can easily handle URL routing, making it efficient navigation through the digital landscape. They both also support Cognito which in its turn is compatible with majority of auth providers including Login with Amazon - that's all you need, that's very easy, you can pick both or either with Amazon Anywhere.

It wasn't that easy. We needed more customization, more flexibility. And this is where API Gateway stepped in with its custom authorizers and actually enabled us to implement the level of customization we needed, delivering distinct authorization levels tailored to our needs.

And lastly scaling - both can scale to your needs if you really need to. But API Gateway comes with an initial limit of 10,000 requests per second which can always be increased. And Application Load Balancer, it can scale indefinitely.

So in a nutshell, if you look at these two, and you know, if your requirements align with straightforward routing and authorization, you can pick any. But if you need the level of customization we needed, API Gateway might be your best choice.

Thanks Maria. Let's walk through how we use API Gateway custom lambda authorizer to implement basic client authentication using API keys.

The client sends a request to our API Gateway with the API key in the authorization header. Now API Gateway is totally capable of using this API key to directly handle authorization to the underlying resources. But another option is to allow our Lambda authorizer to get this API key and return it in the request context. We'll see how this is helpful in the next slide. But for now, the API key is simply returned and API Gateway uses the associated API key to evaluate access to the underlying resources.

Now, let's talk about customer authentication. The client first communicates with Login with Amazon to request account linking and to receive an access token. This token is provided as a bearer token in the authorization header to our API Gateway, which then sends this token to our custom Lambda authorizer.

The Lambda authorizer then validates this access token with our auth server Login with Amazon and retrieves the customer ID and the Login with Amazon client ID. Another important action that we take here is we load our configuration and we look up the associated API key with that Login with Amazon client ID. This means that it's easy for our clients as they only need to provide one authorization header, either the API key or the access token. And we are always able to use the right level of authentication while still tracking usage metrics with the correct client's API key.

Last, API Gateway evaluates the generated IAM policy to allow or deny access to the underlying resources.

Let's update our architecture now that we have shown solutions for authorization and request routing for a public endpoint. We now have AWS Shield and Web Application Firewall which apply policies to our endpoint hosted by API Gateway which has an authorization Lambda that integrates with Login with Amazon.

Now let's compare our compute options for hosting our service logic. There are various compute options available at AWS...

The choice is full on you and, and your requirements. So if we look at Amazon Elastic Compute Cloud or EC2, it provides scalable virtual servers in the cloud. You can pick from general purpose, memory optimized, security optimized. You can scale your instances up and down, you can customize your operating system, install any software.

It comes at a price - operational overhead. You are fully responsible for managing your EC2 instances.

If you look at Elastic Container Service or ECS, it allows you to run Docker containers at scale, provides a scalable, highly customizable container orchestration service. Also comes at a price. If you never dealt with containers before, that might be a challenge.

However, I really wanted to mention AWS Fargate that allows you to run containers without managing the underlying infrastructure.

AWS Lambda is a compute service that takes away all the headaches. However, it comes with a time limit - 15 minutes per invocation.

So in a nutshell, if you look at control versus abstraction:

• EC2 gives you maximum control
• ECS allows you to run containers in the cloud
• Lambda allows maximum abstraction & scalability.

All three services are very scalable. But again, Lambda excels in automatic scaling and use case suitability.

EC2 is very versatile. ECS brings a certain level of complexity. Fargate mitigates that complexity. And Lambda brings the serverless perfection.

So me and Alan had a bit of back and forth when looking at compute options. Let's see what he ended up using.

First, I knew we wanted to focus on serverless options. We don't need to manage our own EC2 instances. As a developer, managing host patching and EC2 migrations is time consuming and expensive.

We wanted to optimize for low latency and continuous traffic with ability to scale to thousands of requests per second. For those reasons, we decided on AWS Fargate as our compute option.

We could have used Fargate for some services and Lambda for others, as each has different scaling needs. But for simplicity, we standardized on one solution for each of our services.

Now that I explained why we selected Fargate, let's add that to our architecture diagram and discuss connecting services to API Gateway and networking.

Let's remind ourselves of some key goals:

1. Microservices should not need public endpoints. Gateway handles authorization.

2. Enable encryption between Gateway and each microservice for security.

Our infrastructure is across multiple AZs. Gateway and each service has a VPC to load balance between instances. Each service has a network load balancer.

We needed targets for API Gateway to access the load balancer. But AWS account boundaries don't allow a target that is a network load balancer in a different account.

Therefore, we used AWS PrivateLink to create endpoints linking VPCs from Gateway to each microservice. Each service has another network load balancer that routes to these endpoints per AZ and serves as the target for API Gateway.

Reviewing with the AWS account team was helpful to ensure we have the right design and resources.

AWS PrivateLink does not provide encryption by default. TLS from Gateway must be terminated to enable authorization, authentication, request routing.

To enable encryption between API Gateway and microservices, we used AWS Certificate Manager and Route 53. API Gateway uses Route 53 to locate the endpoint for each microservice and create a new secure connection.

As we build and expand, Smithy enables us to easily extend the API by modeling it - defining resources, operations, types, validations. This allows us to establish common types and enforce consistency.

We can merge models to construct a unified Gateway API, flattening namespaces and selectively incorporating shapes. Smithy tooling generates server stubs, client libraries in multiple languages, and HTML documentation.

It can also convert the model to OpenAPI v3 spec, which we used in Postman to test the API before having a client.

For client configuration, we used AWS App Configuration to store info like IDs, keys, options in files we can easily update through code review, without additional tooling like we'd need with DynamoDB.

For our data warehouse, Kinesis Firehose aggregates order events by time/size. Lambda provides a function to format the output. S3 objects trigger SNS events to ingest data.

Our infrastructure is defined entirely through AWS CDK. It allows consistent, reproducible deployment across stages, accounts, regions. Updates are automated through the pipeline.

CDK has higher level constructs compared to YAML, allowing us to use secure defaults and reusable stacks like dashboards to accelerate development.

In summary, the high level architecture uses:

• AWS CDK
• API Gateway
• Lambda
• ECS Fargate
• Amazon Smithy
• etc.

I think this is a very reusable architecture that many could use for services they are building.

The Amazon Anywhere integrations with social media show the impact we can have. The purchase experience is now in context when you click an ad, without taking you out of the app.

We went live with Instagram and Facebook 3 weeks ago. All Amazon ads in your feed now connect to Amazon Anywhere. We also launched with Snapchat, so we now have integration across Instagram, Facebook and Snapchat.

Currently these are curated experiences, but in the future we want to understand intent in context to find appropriate products automatically. Video is a great medium to explore this.

We also see opportunity in virtual worlds - being able to buy in the physical world what you see in a virtual world. The physical world itself is becoming more immersive through QR codes, apps, augmented reality. If you see a product, we want you to be able to acquire it easily.

We're here today, so come talk to us! Fill out the survey - we want your feedback to keep improving. Thank you!