This guide is for sellers and agencies that run ads on Amazon Advertising platform.

The Ultimate Guide to Amazon Advertising Bulk Operations


A seller from Bulgaria succeeded in a German Amazon Marketplace without knowing the language, yet found a great way to aid his business. For that, he used Natural Language Processing. His name is Radoslav, and he shared his story with me at the [Orange Hat Summit](https://orangehatsummit.com/) conference.

## Getting Started With Amazon

When I was young, I was fascinated with computers and wanted to be a software developer. I was amazed at how "something" can be on the web, and everyone can access it and see it. I wanted someday to be able to do such things.

At first, I started trying to make simple HTML websites with a few buttons. A few years later became my full-time job and finally led me to study Computer Science at the university. After I graduated, I worked as a software developer for a few years.

One day a friend of mine asked me if I wanted to join him in his Amazon business and that's how it all started.

## German Marketplace Specifics

The German marketplace is the biggest one in Europe. Also, it is very specific with all the German words. Not many sellers know for which keyword to rank their products.

Most people don't like Google Translate, because of its old fame, but it has dramatically improved in recent years. Especially for German, it's quite good. I can't say it's perfect, because nobody can be perfect in German, but it's okay. You can easily understand what customers want and make a simple and understandable reply.

## Running Into Problems

As a seller, I can honestly say we have experienced everything. You name it, and we've experienced it. We've had our account closed, encountered fraudulent infringement claims.

The shipping agent lost a few of our bags with orders (we were doing FBM at the time) which resulted in dozens of tickets per day and everything else that you can think of.

I'm saying bags because we were doing a lot of FBM and we were sending all the orders from the day in bulk in a bag. Now imagine a few of those bags getting lost without you knowing. That's a few hundred orders! The customer requests quickly became almost 50 per day, without counting the replies.

We were and still are using an online helpdesk platform which has templates for answers, and you can quickly choose the reply you need to send just like Seller Support does. ;) It's reasonably fast, no more than a minute per answer.

However, the problem is that this 24/7 replying to the same customer messages is boring, because you do the same thing with the same messages over and over again, day after day, week after week.

One time there was nobody around to answer the messages one weekend, and I said I was going to do it. The first ten messages were okay, but on the thirtieth, I was "No way I'm gonna do this two more days. This task is so time consuming and boring".

So I started thinking of ways to automate this tedious everyday task. It reminded me of something I read a while ago, a quote from Bill Gates:

> "I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it."
>
> -- <cite>Bill Gates</cite>

And that's precisely what I did. Found an easy way to do it.

## Applying Natural Language Processing

I have some basic background in Neural Networks from my years in university and a few online courses, but I can't say I have some profound knowledge in Natural Language Processing. What I did is I searched for already built platforms which had this service in them and just chose the one with the best UX.

I wanted to automate the replies to the customer tickets somehow, so our team can focus more on the daily tasks, and not on repetitive customer support. So what I did is I found a platform that provided text classification service and started experimenting with the customer messages and replies to see if I can somehow group them in categories. Most of the times the customer requests were the same: "Where's my order?", "I want an invoice," "I want a refund," "I have a problem with my products" and so on, so it was pretty easy to classify them.

The German tickets are very easy to be classified in the text classifier because the words are very complex and distinguishable. By just looking at a ticket you can easily understand if a customer wants a refund, resend and so on only by seeing some specific keywords.

After the classifier started getting good results, I connected it to the helpdesk software so that you can send the reply to the customer with just one click from the ticket list. Meaning you can answer all of the messages by just clicking a few buttons. Of course, if the classifier fails on some ticket, you still need to do it manually, but most of the time it's pretty precise.

When we had our fail with our bulk orders, there was a person that was answering customer tickets all day long. By integrating the message classifier, we were able to decrease the time to just a few minutes per hour, which is a pretty good optimization. We saved around 5 hours per day by doing that. Nowadays we use mainly FBA, so we don't get so many messages, but the few that we get, they still go through the classifier!

It decreased the tedious task of choosing which template reply to send to the minimum. Moreover, I say boring, because when you do it a few hundred times, you start wanting to do anything else besides that.

**>>> How many hours are you spending on mundane customer service replies?**

A seller from Bulgaria succeeded in a German Amazon Marketplace without knowing the language, yet found a great way to aid his business. For that, he used Natural Language Processing.

Artificial Intelligence Saves This Seller Five Hours per Day of Customer Service


A famous entrepreneur, investor, and software engineer, Marc Andreessen coined a phrase "software is eating the world".

> "Software is eating the world, in all sectors. In the future, every company will become a software company."
>
> -- <cite>Marc Andreessen</cite>

At the start of the 19th century, we could have said that "*automobiles are eating the horses*." Doesn't sound as cool though, but it's almost *literally* true.

In 1890s New York City was awash in horse manure, shin-deep, with predictions that the city's first floors would soon be manured under.

To say that it was a colossal pain would be an understatement. New Yorkers were demanding a solution and wanted it fast!

Crazy "techies" of the time were working furiously in auto "startups" to "disrupt" the horse industry and dig themselves out from all that manure.

No, they weren't trying to build a better horse that does not defecate.

Best minds were working on a novel idea of the century: a cart that can be propelled by an engine.

* By 1900 there were a small number of automobiles in the entire United States, powered by gas or steam or electricity.

* By 1905 there were 23,000 cars in NYC alone.

* By 1908 there were 100,000 cars, and there were 100,000 horses in NYC.

* Final horse-drawn tram happened in NYC in 1917.

<FigureImage src="automobiles-are-eating-horses/nyc-easter-parade-1900.jpg" caption="Fifth Avenue in New York City on Easter Sunday in 1900. U.S. Bureau of Public Roads. Photographer unknown." dark="true" class="w-100 w-md-50 float-md-left" />

<FigureImage src="automobiles-are-eating-horses/nyc-easter-parade-1913.jpg" caption="Fifth Avenue in New York City on Easter Sunday in 1913. Photographer unknown." dark="true" class="w-100 w-md-50 float-right" />

What do you think happened to all of these horses that are now obsolete? Surely they retired on a beach somewhere.

The reality of the market, unfortunately, is grim.

Most horses were sold to meatpackers and got processed into dog food, bonemeal, leather, and glue.

The price of horses reached an all-time low in 1950, and the horse population continued its steady decline until only about 3 million horses could be found in the United States in 1960.

---

**But what does it mean when we say that the software is eating the world?**

The software is replacing old tools and roles, just like cars replaced horses.

At the very early days, word processing software replaced typewriters. Humans still operated the software, but it was more efficient and required fewer resources.

Today we are seeing software that produces coherent written content without human intervention. Compu-authors write weather reports, financial news, sports outcomes, and other fact-based minutiae. Machine learning-based tools create music and art.

Tesla car is often dubbed as a "computer on wheels." It's a battery, a few motors and many lines of code that control the car. The vehicle can drive itself in a limited capacity. Truly an "automobile."

Machines that are controlled by software replaced many factory jobs. Now a single computer can do a role of a hundred people, never get tired and work 24/7/365 without demands to unionize.

---

The Third Industrial Revolution started in the late 1950s with the advent of available computing power. Businesses that had a vision and were able to adapt fast to utilize the software effectively ultimately gained a competitive advantage. They were able to save on salaries, reduce operational costs, and scale more quickly.

James Bessen, an economist who teaches at Boston University School of Law recently wrote a fascinating paper "[The Policy Challenge of Artificial Intelligence](https://www.competitionpolicyinternational.com/the-policy-challenge-of-artificial-intelligence/)" about the Fourth Industrial Revolution.

Mr. Bessen argues that investment in proprietary (not off-the-shelf) software is what creates a huge productivity gap today and what sets successful companies apart from the competition.

**>>> What are you doing to widen the productivity gap in your business or are you still riding a 🐴?**

In 1890s New York City was awash in horse manure, shin-deep, with predictions that the city's first floors would soon be manured under.

Automobiles Are Eating Horses


Scraping. The moral grey area. The necessary evil. Maybe even [illegal](https://theoutline.com/post/2554/swmonkey-southwest-airlines-kills-startups?zd=1&zi=5sh7dmrf), in some cases.

Many large companies are participating in scraping (not that it justifies it):

* Boxed.com
* Camel Camel Camel
* Even Walmart was [caught scraping Amazon](https://www.reuters.com/article/us-amazon-com-bots-insight/amazon-trounces-rivals-in-battle-of-the-shopping-bots-idUSKBN1860FK)!

However, in reality, it can be classified as indexing. Just like Google indexes your website, you may want to build an index of your own.

**What if we want an index of ASINs with pricing?**

If you are going to index, then you need to obey the `robots.txt` rules and set the user agent to a unique identifier that makes it clear that it is a bot.

You may also want to create a page for your bot with contact details, in case the website you are scraping wants to send you a cease and desists, which apparently [can also be illegal](https://www.zdnet.com/article/judge-orders-linkedin-to-unblock-startup-from-scraping-public-profile-data-report/).

Moral and legal issues aside, sometimes you *"gotta do what you gotta do."*

## How to Scrape Amazon Search Results

In this issue, we are exploring how to scrape Amazon search results for the infamous *garlic press*.

For that, we are using [Apify](https://www.apify.com/), which dubs itself as *the web scraping and automation platform*. It is a platform as a service (PaaS) solution with a very narrow focus - helping developers with scraping needs.

Apify provides a set of APIs and an SDK library that we are using to write this scraper solution and then launch it quickly without having to set up any infrastructure ourselves. Think of it as Heroku for scraping.

Platforms like this are great for small-to-medium sized projects but seems a bit costly for large jobs.

The free developer plan provides 5,000 page requests per month.

Note that you can run the code on your infrastructure as well, but you don't get all of the benefits of PaaS, such as resource allocation, proxies, storage, GUI and API access to your data.

## Getting Started

```sh
## install the initializer CLI tool
npm install -g apify-cli

## this step takes a while because it is installing a full Chromium browser
apify create --template=puppeteer not-amazon-scraper

cd not-amazon-scraper
```

Now you have a project scaffolded, and `main.js` contains the actual scraper code.

Let's replace the contents of `main.js` with the following starting code:

```js
const Apify = require('apify')

Apify.main(async () => {
  const input = await Apify.getValue('INPUT')

  if (!input || !input.keyword) throw new Error('INPUT must contain a keyword!')

  console.log('Launching Puppeteer...')
  const browser = await Apify.launchPuppeteer()

  console.log(`Searching for keyword ${input.keyword}...`)
  const page = await browser.newPage()
  await page.goto(`https://www.amazon.com/s/?field-keywords=${input.keyword}!`)
  const title = await page.title()
  console.log(`Title of the page is "${title}".`)

  console.log('Closing Puppeteer...')
  await browser.close()

  console.log('Done.')
})
```

The `./apify_storage/key_value_stores/default/INPUT.json` file contains the input object that CLI tool automatically passes to our scraper script. In production, we can set this input object dynamically in the UI.

Let's replace contents of `INPUT.json` with the following:

```json
{
  "keyword": "garlic press"
}
```

<note>

Adding an exclamation mark (`!`) to the Amazon search term forces the results into a one product per row format, rather than a mix and mash of boxes and rows.
</note>

<FigureImage src="how-to-scrape-amazon/without_exclamation.png" caption="Search results without exclamation." class="w-100" />

<FigureImage src="how-to-scrape-amazon/with_exclamation.png" caption="Search results with exclamation." class="w-100" />

Run the code with `apify run` command.

You should see a similar output:

```console
Run: npm start

> not-amazon-scraper@1.0.0 start /.../not-amazon-scraper
> node main.js

Launching Puppeteer...
INFO: Launching Puppeteer {"args":["--no-sandbox","--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"],"headless":false}
Searching for keyword garlic press...
Title of the page is "Amazon.com: garlic press!".
Closing Puppeteer...
Done.
```

## Getting Product Details

This code is a good start, but it only lets us get the search result page.

However, if we need extra product details that are not available on the search result page, then we need to load each product page separately.

For that, we need to use a request queue. A request queue is something that Apify manages for us, to ensure that we are not overloading our machine or the remote server (in this case Amazon).

We can populate this queue with new URLs as fast as we can, however, the processing of the queue is happening with pre-defined frequency (throttling).

As a core feature, Apify takes care of the state maintenance. Even if the scraping process dies, it saves the state. On the next run, Apify resumes where it left off.

If you want to restart the scraping a-fresh, you need to purge the store and state. A useful parameter to know is `apify run --purge`, which reruns your scraper, but deletes the state and storage before doing that. Useful for development and experimentation.

Let's upgrade our code and add the queue capabilities.

I've commented the code extensively. Hopefully, it all makes sense.

```js
const Apify = require('apify')

const getInnerText = async (page, selector) => page.$eval(selector, el => el.textContent)
const numberify = string => Number(string.replace(/[^\d.]+/, '') || 0)

Apify.main(async () => {
  const input = await Apify.getValue('INPUT')

  // Object from `./apify_storage/key_value_stores/default/INPUT.json`
  if (!input || !input.keyword) throw new Error('INPUT must contain a keyword!')

  const browser = await Apify.launchPuppeteer({
    // makes the browser "headless", meaning that no visible browser window will open
    headless: true
  })

  console.log(`Searching for keyword ${input.keyword}...`)
  const searchResultsPage = await browser.newPage()
  await searchResultsPage.goto(`https://www.amazon.com/s/?field-keywords=${input.keyword}!`)

  // This is the crawler queue that is populated with URLs to fetch
  const requestQueue = await Apify.openRequestQueue()

  // Define the URL pattern we want to follow from the search result page.
  // This is the URL pattern of the product details page:
  // https://www.amazon.com/$titleKeywods/dp/$asin/
  const pseudoUrls = [
    new Apify.PseudoUrl('https://www.amazon.com/[.*]/dp/[.*]')
  ]

  // Extract and enqueue URLs to crawl from the page.
  await Apify.utils.enqueueLinks({
    // page from which to extract URLs
    page: searchResultsPage,

    // selector under which to look for URLs
    selector: '#s-results-list-atf a.s-access-detail-page',

    // pseudo URL object describing what URL format to look for
    pseudoUrls,

    // which queue to add the extracted URLs to
    requestQueue
  })

  const crawler = new Apify.PuppeteerCrawler({
    // We've already created a browser instance manually, this will reuse this instance, otherwise
    // a new instance would open up and we'd have 2 browsers running.
    launchPuppeteerFunction: () => browser,

    // This function will be called on every successful product details page fetch:
    // productDetailsPage is the following Puppeteer object:
    // https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageselector
    handlePageFunction: async ({ request, page: productDetailsPage }) => {
      const title = await productDetailsPage.title()

      // The following CSS selector handles different variants of page layouts and pricing types.
      // It is by no means exhaustive, but the ones I found were used in this category.
      const buyBox = await getInnerText(
        productDetailsPage,
        '#price_inside_buybox, #newBuyBoxPrice, #soldByThirdParty .a-color-price'
      )

      // Save data in storage.
      await Apify.pushData({
        title,
        url: request.url,
        buyBoxPrice: numberify(buyBox)
      })
    },

    requestQueue,
    maxRequestsPerCrawl: 100,
    maxConcurrency: 5
  })

  await crawler.run()
  await browser.close()
})
```

## Cloud Deployment

Once you run the code with `apify run --purge`, the results are stored in JSON files in `./apify_storage/datasets/default` directory.

But what about scaling and all that cloud buzz?

Let's deploy our scraper to production and let it loose!

1. Sign up for a [free Apify developer account](https://my.apify.com/sign-up).

2. Then run `apify login`, and get your token from [this page](https://my.apify.com/account#/integrations).

3. Once you are logged in, run `apify push` to upload your project to the Apify servers.

When you visit the [Actors section](https://my.apify.com/actors), you should see `not-amazon-scraper` project listed (or another name you've chosen).

Click to open your actor.

In the body section, put:

```json
{
  "keyword": "garlic press"
}
```

This is the input the actor uses for searching.

Previously, during development, we've set the keyword in the `./apify_storage/key_value_stores/default/INPUT.json` file, but in production, we have a dynamic input, which we can change on every run.

Click <kbd>Run</kbd> button.

<FigureImage src="how-to-scrape-amazon/actor.png" caption="Apify actor page." class="w-100" />

Scroll below on the actor page, and you should see the console output from your scraper.

After the scraper has finished, click on the Dataset tab, and you should have the data that the scraper had saved.

I love that Apify infers the data structure and provides it in various formats automatically. Not only JSON, but you can see an HTML table, download an Excel file or CSV, XML and even RSS.

---

* You can find this code at [our GitHub repository](https://github.com/ScaleLeap/zine-not-amazon-scraper).
* You may also use the actor that I've deployed directly from the [Apify library](https://www.apify.com/scaleleap/zine-not-amazon-scraper).


In this issue, we are exploring how to scrape Amazon search results for the infamous garlic press using Apify.

How to Scrape Amazon Search Results



Right now Amazon's only real competitor in online retail space is Walmart.

We know Walmart for aggressive cost-cutting and brutal negotiations. Some of the suppliers went out of business as they could not sustain the tiny margins demanded by juggernaut retailer. There was a slew of controversy, negative PR and even documentaries filmed about this subject [back in 2005](https://www.imdb.com/title/tt0473107/).

Today, Walmart has an established supply chain and years of experience in negotiating the cheapest possible product to be sold at their stores.

Amazon chose to hand over merchandise procurement to individuals and small businesses, who have minimal experience dealing with the supply chain and overheads that don't scale.

On top of that Bezos imposes all kinds of additional fees on each seller. Storage fees, sale commissions, long-term storage fees, inbound shipment fees and so on. Jeff is milking the sellers on every turn.

The products on Amazon are in many cases more expensive than in other stores. The reason Amazon took off is mainly that of convenience. Americans are busy working multiple jobs.

Like mice, in a medical experiment, working single moms and techie dudes quickly got used to hitting that buy box button. Also, middle and upper classes do not want to mingle with [People Of Walmart](http://www.peopleofwalmart.com/).

As competition is heating up Amazon will be forced to cut costs to continue the customer acquisition and retention. The easiest, if not the only, way to reduce costs is to eliminate the middlemen. So who is the intermediary in this formula? That's right, it is you, the seller from a western world.

What value does an average American or European white-labeler provide? I see a few advantages that a local expert might have: better branding, understanding of the local market culture, and customer service experience. Chinese sellers are still weaker in these areas. However, they are catching up quickly.

Take a look at [DJI](https://www.dji.com/). Shipped directly to the consumer, solid brand, bug-free product, heavy into content marketing with an in-house team.

More and more international students will graduate and move back home, bringing with them a chest full of knowledge about American values, good-enough English skills and familiar to us customer experience.

What advantages does a Chinese seller have? Lower overhead and better-negotiated deals than an English-native over Alibaba messenger can ever get. Many manufacturers are starting to ship direct and growing an online retail arm.

In late 2015, Amazon launched a cargo airline, now known as Amazon Air. They are only practicing this game on the home turf. Amazon eventually plans to have over 100 aircraft. Just for comparison - FedEx Express has 371 aircraft.

They might start to ship goods directly from China overnight to US warehouses and get them delivered. It might not be "same day" shipping like for many other products, but it will still be fast. It might even be available for non-prime members as an incentive to use it.

A good friend of mine, who works for The International Air Transport Association (IATA) has said:

> "The profit margins of UPS and FedEx on cargo flights are a well-guarded commercial secret."
>
> "I do know that freight forwarding companies have __very high-profit margins__ and the highest return on investment in the entire aviation supply chain. So if Amazon cuts out the middleman (i.e., the freight forwarder) and deals directly with shippers/buyers, they will be able to compete effectively on the price."

One way to lower the cost of goods even further is to look at what the most popular products are and start sourcing their merchandises again, taking back the quality control. We can already observe this happening with Amazon store brand but might get even more aggressive.

Another way to lower costs is to hand off selling to a set of trusted partners directly in China. They will vet these businesses and hold them to a very high standard to ensure quality. These enterprises will sell on Amazon right from China and fulfill the orders via Prime Air for cheap.

An insider in China told me that Amazon already has a dedicated recruitment team with hundreds of employees that are actively working with factories to help them start selling directly.

I do not see Amazon surviving without making changes, because quality is going down, listing prices are going up, and shoppers are losing trust in the system.

As competition is heating up Amazon will be forced to cut costs to continue the customer acquisition and retention.

If You Want to Continue Selling on Amazon in Three Years, You Need to Change Your Strategy


How often do you run into a problem and wish a solution existed? No, you are *convinced* the solution exists.

> "It's just so obvious, someone must have solved this before!"
>
> \-- <cite>Frustrated Self</cite>

You google left and right. You turn over every web rock and look under. You ask in peer groups.

Zero. **Nothing**. *Zilch*.

The only option is to build it yourself. Become an inventor. Turn lemons into lemonade.

But you know what is even better? Help others solve this problem too. **Productize your pain**!

Many successful businesses that we know today started as a solution to their internal struggles. They saw the need and figured others might have the same problem too.

These are just a few intrapreneurship success stories to get you inspired.

## Amazon Web Services

Amazonians were building out the technology on the inside for internal use. They were also helping some key partners, such as Target with technology solutions.

Developers realized that a lot of tasks were repetitive and time-consuming. After a long company retreat the C-level execs came out with a plan - let's develop internal solutions into a public-facing service and make money.

Amazon Web Services was born, which today powers a massive part of the internet, and almost every startup you know. It is also currently the most profitable branch of Amazon as a company.

Almost everything AWS releases is born out of an internal need. Take one of the most recent additions: Amazon Rekognition. The service makes it easy to add image and video analysis to an application. What else has Amazon's e-commerce arm released at the same time? Product image analysis and using that for product rankings.

## Slack

Tiny Speck released a game named Glitch in 2009. The game completely flopped and closed down in 2012. Yeah, you probably have never heard of it. Neither did I. But the company behind the game, had this one internal tool, a chat application, that everyone loved using.

They decided to build it out and share it with other teams. That little chat application is now known as Slack. It is used by 50,000 organizations and has more than 6 million daily users. Today the company is valued at $5B.

## Shopify

Tobias Lütke and I met back in 2004 at a coffee shop on Elgin Street in Ottawa, Canada. He agreed to show me the ropes of using Ruby on Rails (a programming framework). Tobi was using the framework to build a shopping cart software for his snowboarding store. He was not satisfied with any solutions that were currently on the market.

Two years later the shopping cart software was launched as a stand-alone service. Now it is known as Shopify. Over 600,000 merchants are using it today. Shopify is valued at over $5B.

- - -

**>>> Do you have any lemons you'd like to turn into a profit-making machine?**

Many successful businesses that we know today started as a solution to their own internal struggles. They saw the need and figured others might have the same problem too.

Turn Your Pain Into Lemonade


Let's begin by talking about what is API in the first place.

You might have heard of GUI, which stands for Graphical User Interface.

GUI is how humans interact with a computer, via a point-and-click graphical interface or screen in lament terms.

When it comes to computers, they prefer something more structured and simple. They don't want to look at screens and fill out forms. Machines want a simple to consume protocol that is predictable, documented and standardized. They want the data without the foreplay.

That is what Application Programming Interface (API) is in a nutshell: an interface for computer programs. API is how computers talk to each other.

Amazon provides an API to access many of the seller account details programmatically. This API is called [Amazon Marketplace Web Service](https://sellercentral.amazon.com/gp/mws/index.html) or MWS for short.

The MWS API provides information about orders, order items, shipping status of an order. It can also provide reports, just like you can generate them in the UI, but programmatically.

Other features provided by the API include, but not limited to:

* product management
* feed management
* inventory management
* creation of inbound shipments
* checking the status of inbound shipments
* submitting fulfillment orders
* tracking and managing outbound shipment requests
* ... and many others

Most of the seller software on the market use this API to fetch the information about your seller account.

This API has its challenges that Amazon imposes on developers. There are strict limits on the number of requests per unit of time. A request is a single data fetching operation. Let's say I need to get order info that is one request, or maybe you need to get a list of orders that would be another request.

**>>> Do you have any interesting uses for the MWS API?**


Amazon provides an API to access many of the seller account details programmatically. This API is called Amazon Marketplace Web Service or MWS for short. But what is it?