When a booking bot saves the day!

So, let's say that you plan a yearly trip to one of your favorite locations, something I do every year. But this year you got some more friends that want to join you on your vacation.

I am usually the one that plans and tries to organize the trip, but this year I made it easy for myself and said, I am going here, on this date. So I manage to book the trip with some of my friends and communicated date, flight, and hotel. I make sure that we got rooms so everyone can join but since people start working, dates declining fast and people book the winter to get some sun.

One of our guys is reading up on everything later that evening and ask questions. I don't see them, so he awaits confirmation. Next morning, I answer, but then all the rooms are all booked...

Since I used the same travel agency several times, I know that when you book a trip you have a couple of days to pay the booking fee or you lose your booking. My friend commented, "oh well better luck next time." but I am pretty sure he wants to go with us so, our only option is to check available rooms every day, a couple of times every hour to see if someone cancel their booking or don't pay their fees.

So hypothetical ;) how do you solve this and increase the chance that your friend gets a room and can join this memorable trip? WELL.

Lately, I have started using python to test out techniques from the books "The hacker playbook 3" and "Black Hat Python - Python Programming for Hackers and Pentesters" maybe we could script a booking bot in Python?

So hypothetical let's assume that I built a booking bot, how would I do this? Let's say that we are booking with TUI(a German company popular for charter trips in Sweden) and that we want to go to Gran Canaria Islands.

Well in some sense I won't build a real AI bot, I would use E2E testing with python.

So, the basics when doing easy End-To-End-tests is to make a page object. A page object act as an object for all the elements you want to access on a page. So, we create a page object that when we create it, maps and makes sure everything is loaded and found in a reasonable amount of time. The selection is made by use selenium selectors which can be by CSS-classes or Id on a dom-element. Look here for a list [Selenium Python API] (https://selenium-python.readthedocs.io)

Let's look at the page itself: [Booking page] (https://www.tui.se/resa/spanien/gran-canaria/san-agustin/don-gregory/?partysizes=1%3B&departureCode=ARN&departureDate=2019-02-28&duration=8&referer=searchresult)

We have an URL that contains a hotel, party size, departure from Arlanda, departure date 2019-02-02 and duration is 8 days.

So how do we get a browser to do what we want? We use Selenium! We can use the selenium driver to fake that we are a user that runs Chrome or other browsers. We also want to do this headless, which means that we do not want to control our Chrome browser on our local computer, we want to fire it from a command line. Let's create a driver and hook it up to our script. Let's create botdriver.py:

from selenium import webdriver


class BotDriver:

    def __init__(self):
        WINDOW_SIZE = "1920,1080"
        options = webdriver.ChromeOptions()
        options.add_argument('headless')
        options.add_argument("--window-size=%s" % WINDOW_SIZE)
        self.instance = webdriver.Chrome(executable_path='/Users/jonathan/Downloads/chromedriver', chrome_options=options)

    def navigate(self, url):
        if isinstance(url, str):
            self.instance.get(url)
        else:
            raise TypeError("URL is not valid!")

    def make_screenshot(self, filename):
        self.instance.save_screenshot(filename)

In short, we create a class and then tell the class what to do when we create the class, in this case, set window size, chose Chrome, run it headless(without a visible browser), set the size in regards of our size. Then we give the path for the selenium driver to use and inject our options. We then tell our driver to navigate to the URL we send into the method, and we have declared a method to use the driver’s signature to print screen.

Now when we got our driver setup, we can start to build objects to control the browser to do what we want!

If we push the button adjust ("anpassa") we can adjust trip and make sure we get all the options that we like. So, let's make a page object of this, in booking_page.py

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

class BookingPage:

    def __init__(self, driver):
        self.driver = driver
        self.book_page = WebDriverWait(self.driver.instance, 30).until(
                    EC.visibility_of_element_located((
                        By.CSS_SELECTOR, "div.price-example-main")))

    def validate_book_extras_loaded(self):
        assert self.book_page.is_displayed()

    def click_adjust_button(self):
        self.driver.instance.find_element_by_css_selector("a.btn.btn-sales.open-tui-modal").click()

In short, we set some properties from selenium functions that are looking for CSS selectors and wait for them to show up in limited time in this case 30 sec before it throws a timeout. We also have a function to click the element we are waiting to show up if it shows up.

If we then press the adjust button ("anpassa") we get this dialog:

We the map this to a new page object find_date_page.py:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

import base64
import json

def parse_data(data):
    try:
        base64_decoded = base64.b64decode(data)
        utf8_string = base64_decoded.decode("utf-8")
        json_string = utf8_string.replace("diff-com.tui.common.search.SearchRequestData-", "")
        return json.loads(json_string)
    except Exception as e:
        print(e)
        pass

class FindDatePage:
    def __init__(self, driver):
        self.driver = driver
        self.find_date_popup = WebDriverWait(self.driver.instance, 30).until(
            EC.visibility_of_element_located((
                By.CSS_SELECTOR, "section.matrix.matrix-responsive-table")))

        self.options = WebDriverWait(self.driver.instance, 30).until(
            EC.visibility_of_any_elements_located((
                By.CSS_SELECTOR, "div.visible-offer")))

        self.forward_button = WebDriverWait(self.driver.instance, 30).until(
            EC.visibility_of_element_located((
                By.CSS_SELECTOR, "button.btn.btn-lg.btn-block.btn-primary.forward-navigation.ng-binding")))
        self.driver.instance.save_screenshot('available_trips.png')

        self.available_dates = self.driver.instance.find_elements_by_css_selector("[data-selector]")
        print("Found: " + str(len(self.available_dates)))

    def find_date_popup_visible(self):
        assert self.find_date_popup.is_displayed()

    def forward_button(self):
        assert self.forward_button().is_displayed()

    def options_visible(self):
        assert self.options().is_displayed()

    def select_available_dates(self, date_to_book, duration):
        assert len(self.available_dates) > 0
        if len(self.available_dates) > 0:
            print("Available rooms = " + str(len(self.available_dates)))
            for option in self.available_dates:
                data_value = parse_data(option.get_attribute('data-selector'))
                if data_value is not None and data_value["EarliestStart"] == date_to_book and duration in data_value[
                    "Durations"]:
                    option.click()

    def click_forward_button(self):
        self.driver.instance.find_element_by_css_selector(
            "button.btn.btn-lg.btn-block.btn-primary.forward-navigation.ng-binding").click()

Here we have some more things going on if you look at the picture above you can see we have a table for prices based on weeks and duration. To be sure to pick the right one we need to take the whole list and see that we have the right elements that match time, date and duration. This looks a bit tricky since the element values are base64 encoded and need to be decoded, and the value we get is not valid JSON. So, we need to take care of that too in the parse_data method.

If we continue and select the date and push next, we see this page:

So, we add a new page object find_flight_page.py:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains


class FindFlightPage:

    def __init__(self, driver):
        self.driver = driver

        self.driver.instance.save_screenshot('flightpage.png')
        self.load_flight_page = WebDriverWait(self.driver.instance, 30).until(
            EC.visibility_of_element_located((
                By.CSS_SELECTOR, "button.btn.btn-lg.btn-block.btn-primary.forward-navigation.ng-binding")))


    def forward_button(self):
        assert self.load_flight_page.is_displayed()

    def click_forward_button(self):
        self.driver.instance.find_element_by_css_selector(
            "button.btn.btn-lg.btn-block.btn-primary.forward-navigation.ng-binding").click()

The code is the same as before, we wait for elements, and we have a method to click the element if it shows up.

We then get to this page to choose if we want breakfast, breakfast & dinner or All-inclusive.

So, we add another page object breakfast_page.py:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains


class BreakfastPage:

    def __init__(self, driver):
        self.driver = driver

        self.load_breakfast_page = WebDriverWait(self.driver.instance, 30).until(
            EC.visibility_of_element_located((
                By.CSS_SELECTOR, "button.btn.btn-lg.btn-block.btn-primary.continue-booking.ng-binding")))
        self.driver.instance.save_screenshot('breakfast.png')


    def forward_button(self):
        assert self.load_breakfast_page.is_displayed()

    def click_forward_button(self):
        self.driver.instance.find_element_by_css_selector(
            "button.btn.btn-lg.btn-block.btn-primary.continue-booking.ng-binding").click()

Then we push next and get to this page where we enter user details and chose extras to our trip:

We then add a new page object extras_page.py

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains


class ExtrasPage:

    def __init__(self, driver):
        self.driver = driver


        self.load_extras_page = WebDriverWait(self.driver.instance, 60).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "fieldset.form-section.address-section")))
        self.load_insurance_section = WebDriverWait(self.driver.instance, 60).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, "div.col-md-12.ancillary-group")))
        self.driver.instance.save_screenshot('extras.png')

    def validate_extras_loaded(self):
        assert self.load_extras_page.is_displayed()

    def validate_insurance_section_loaded(self):
        assert self.load_insurance_section.is_displayed()

    def fill_passanger(self):
        self.driver.instance.find_element_by_name('passengers.0.firstName').send_keys("John")
        self.driver.instance.find_element_by_name('passengers.0.lastName').send_keys("Doe")
        self.driver.instance.find_element_by_css_selector("input.form-control.birth-date").send_keys("19900101")
        self.driver.instance.save_screenshot('passengers.png')

    def fill_address(self):
        self.driver.instance.find_element_by_css_selector("input[type='radio'][value='MALE']").click()
        self.driver.instance.find_element_by_name('bookerDetails.0.streetAddress').send_keys("testgatan 1")
        self.driver.instance.find_element_by_name('bookerDetails.0.zipCode').send_keys("12345")
        self.driver.instance.find_element_by_name('bookerDetails.0.city').send_keys("stockholm")
        self.driver.instance.find_element_by_name('bookerDetails.0.phoneNumber').send_keys("0733123456")
        self.driver.instance.find_element_by_name('bookerDetails.0.emailAddress').send_keys("[email protected]")
        self.driver.instance.find_element_by_name('bookerDetails.0.emailAddress2').send_keys("[email protected]")
        self.driver.instance.save_screenshot('address.png')

    def select_cancellation_insurance(self):
        self.driver.instance.find_element_by_css_selector(
            "input[type='radio'][value='variantProductCode=fake_no_thanks_option_code,sysInfo=']").click()
        self.driver.instance.save_screenshot('cancellation_insurance.png')

    def select_transfer(self):
        self.driver.instance.find_element_by_css_selector(
            "input[type='radio'][value='variantProductCode=PC-000119876,sysInfo=D122 PLPATINDGD190228 01202604249']").click()
        self.driver.instance.save_screenshot('transfer.png')

    def select_travel_insurance(self):
        self.driver.instance.find_element_by_css_selector(
            "input[type='radio'][value='variantProductCode=fake_no_thanks_option_code,sysInfo=']").click()
        self.driver.instance.save_screenshot('travel_insurance.png')

    def click_summer_dropdown_button(self):
        self.driver.instance.find_element_by_id("price-summary-container").click()
        self.driver.instance.save_screenshot('finished.png')

The only new code here is that we select inputs and fill the values for the page.

Now we have all objects and what we want to load and wait for so let's make a script that runs this and do it as long as we don't have a trip for our boy! Let´s create booktrip.py

import unittest
import time
from pprint import pprint
from unittest import TextTestRunner
from botdriver.botdriver import BotDriver
from pageobjects.booking_page import BookingPage
from pageobjects.find_date_page import FindDatePage
from pageobjects.find_flight_page import FindFlightPage
from pageobjects.breakfast_page import BreakfastPage
from pageobjects.extras_page import ExtrasPage


from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

import sys

baseurl = "https://www.tui.se/resa/spanien/gran-canaria/san-agustin/don-gregory/?partysizes=1%3B&departureCode=ARN&departureDate=2019-02-28&duration=8&referer=searchresult"
date_to_book = "2019-02-28"
duration = 8


class BookTrip(unittest.TestCase):

    def setup(self):
        self.driver = BotDriver()
        self.driver.navigate(baseurl)

    def test_book_available_room(self):
        try:

            print("Try to find and book trip!")

            booking_page = BookingPage(self.driver)
            booking_page.validate_book_extras_loaded()
            booking_page.click_adjust_button()


            find_date_page = FindDatePage(self.driver)
            find_date_page.select_available_dates(date_to_book, duration)
            find_date_page.click_forward_button()

            find_flight_page = FindFlightPage(self.driver)
            find_flight_page.forward_button()
            find_flight_page.click_forward_button()

            breakfast_page = BreakfastPage(self.driver)
            breakfast_page.forward_button()
            breakfast_page.click_forward_button()

            extras_page = ExtrasPage(self.driver)
            extras_page.validate_extras_loaded()
            extras_page.validate_insurance_section_loaded()
            extras_page.fill_passanger()
            extras_page.fill_address()
            extras_page.select_cancellation_insurance()
            extras_page.select_transfer()
            extras_page.select_travel_insurance()
            extras_page.click_summer_dropdown_button()

            return

        except TimeoutException:
            self.driver.instance.save_screenshot('Error_timeout.png')
            self.driver.instance.quit()
            self.fail("Timeout occurred...")
            print("No offers found...")

        except KeyboardInterrupt:
            self.driver.instance.quit()
            sys.exit()

        except Exception:
            self.driver.instance.save_screenshot('error.png')
            self.driver.instance.quit()
            self.fail("Failed with %s" % error)
            pass

    def teardown(self):
        self.driver.instance.quit()


def main():
    try:
        while True:
            test = unittest.TestLoader().loadTestsFromName("test_book_available_room", module=BookTrip)
            book_result = TextTestRunner().run(test)
            if len(book_result.failures) is 0 and len(book_result.errors) is 0:
                print("Successful booking check mail!")
                break
            else:
                print("Will try again in 5 minutes...")
                time.sleep(300)

    except KeyboardInterrupt:
        sys.exit()

    except Exception:
        pass


if __name__ == '__main__':
    main()

The last part of this script creating the driver, and navigate to the given URL that we send in. We start our test in a loop guarded by tests. If the test is throwing an exception, timeout or that the test fails, we return the result and check it and then try again in 5 minutes, until we succeed. To fire this script and let it loop we type:

python3 booktrip.py

Now, this runs with some debug output and can be viewed in the terminal. The script will quit if we succeed or we send a kill command.

Since I use a headless browser we won’t see what happens, but selenium got a method to print screen what it does:

So, let’s take a look at what the script does:

Now we don't book this, but we could.

Hypothetically my friend after some time now got a trip booked! If you want to look at the source code, you find it here: source code