Running Selenium Webdriver with Python on an AWS EC2 Instance

I start the explanation by considering that you have already created AWS EC2 instance and succesfully connected to your instance using pem file through your terminal. The steps i discuss down below should be done in exact order.

Step1: Start new screen and become root user

Use the following command

$screen

$sudo su

Step2: Install python3 ,pip,Git and virtualenv

$yum install python3 -y

$yum install git -y

$yum install pip

$pip install virtualenv

Step3: Create virtual environment with python3 and activate that environment.

Replace 3.7 with whatever version python3 you installed.

$virtualenv -p python3.7 my_app

$source ./my_app/bin/activate

Step4: Install the browser and webdriver

You need some browser to run webdrivers. The choice is yours i’m using chrome browser and chrome webdriver here.It’s important here to download browser first then identify it’s version and then install chromedriver for that particular browser version. You can go wrong if you don’t follow these steps carefully.

First move to root directory and enter following commands

$cd/tmp/

Enter below command to install chrome

$curl https://intoli.com/install-google-chrome.sh | bash
$sudo mv /usr/bin/google-chrome-stable /usr/bin/google-chrome
$google-chrome — version && which google-chrome

Above code will print chrome version .Best statergy here is to browse the internet and use this version number to get appropriate linux chrome driver and replace below command link.

$wget https://chromedriver.storage.googleapis.com/80.0.3987.106/chromedriver_linux64.zip

$unzip chromedriver_linux64.zip
$sudo mv chromedriver /usr/bin/chromedriver

$chromedriver — version

Step5: Pull from the Git

First you have to be in ec2-user directory. Use git commands to fetch your git repository. Replace your username and repo name in command.

$git clone https://github.com/your_username/your_repo

Step6: Install necessary python libraries

Install all libraries you have used in your code using pip.

$pip install selenium

Step7: Run the program

$cd your_repo

$python application.py

This will run the starter code given below

#application.py
from selenium import webdriver
from selenium.webdriver.chrome.options import Optionsoptions = Options()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("enable-automation")
options.add_argument("--disable-infobars")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.com/")
element_text = driver.page_source
print(element_text)

The above code will return html source code of google home page.