Dockerise your Flask Application

In this article, we’ll take a look at how to dockerize a Flask application. Flask is a microframework for Python, with a basis in Werkzeug and Jinja 2.

The Docker Hub image

Since Docker Hub doesn’t have an official Flask repository (at the time of this writing), we’ll explain how to build our own. While you can always use a non-official image, it’s generally recommended to make your own Dockerfile to ensure you know what is in the image. We’ll start off with the base of Ubuntu, but you can use any available distro you prefer.

Setting Up

We need to set up a basic app and Dockerfile.

Our basic app

Start with creating a new directory; let’s call it flask_web:

1
mkdir flask_web

You can use this basic app.py file for your basic application:

1
2
3
4
5
6
7
8
9
10
11
12
# flask_web/app.py

From flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world:
return 'Hey, we have Flask in a Docker container!'


if __name == '__main__':
app.run(debug=True, host='0.0.0.0')

Now, we need to include Flask in our requirements.txt file:

1
Flask==0.10.1

Flask Dockerfile

We’re starting from Linux instead of using the Python repository as our base, as its more clear in which Python version is being installed (what apt installs on Ubuntu or Debian, or yum installs on Red Hat and CentOS). You always have the option to build off the python image instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
FROM ubuntu:16.04

MAINTANER Your Name "youremail@domain.tld"

RUN apt-get update -y && \
apt-get install -y python-pip python-dev

# We copy just the requirements.txt first to leverage Docker cache
COPY ./requirements.txt /app/requirements.txt

WORKDIR /app

RUN pip install -r requirements.txt

COPY . /app

ENTRYPOINT [ "python" ]

CMD [ "app.py" ]

Let’s go over some of these Docker instructions:

MAINTAINER sets the Author field of the image (useful when pushing to Docker Hub)
&& \ isn’t Docker specific, but tells Linux to run the next command as part of the existing line (instead of using multiple RUN directives, you can use just one)
COPY copies files from the first parameter (the source .) to the destination parameter (in this case, /app)
WORKDIR sets the working directory (all following instructions operate within this directory); you may use WORKDIR as often as you like
ENTRYPOINT configures the container to run as an executable; only the last ENTRYPOINT instruction executes
pip installs from requirements.txt as normal. Since requirements.txt only references Flask 0.1.0, pip only installs Flask 0.1.0. If you are using Flask for your app, then you’re likely to have more modules specificed for installation.

Build the image

Now that we have a Dockerfile, let’s verify it builds correctly:

1
docker build -t flask-tutorial:latest .

After the build completes, we can run the container:

1
docker run -d -p 5000:5000 flask-tutorial

Further information

Ensure you are using the right ports. Flask by default runs on port 5000 (not 8000 like Django or 80 like Apache). Check out Binding Docker Ports for more information.

Dockerise your Python Application

Dockerfiles enable you to create your own images and describe the software that makes up an image. They contain a set of instructions that specify what environment to use and which commands to run.

Creating a Dockerfile

First, start with a fresh empty directory. In this example, we will call this my_docker_app. This directory will contain all of the things you need to build your image.
Create a new text file in my_docker_app called Dockerfile (note no extension. In this example, we use the basic Python 3 image as our launching point. Add the following line to your Dockerfile:

1
FROM python:3

We want to run a basic Python script which we’ll call my_script.py. First, we need to add the script to the Dockerfile:

1
ADD my_script.py /

Our script depends on the Python pyStrich library (pyStrich generates 1D and 2D barcodes), so we need to make sure we install that before we run my_script.py! Add this line to your Dockerfile to install pystrich:

1
RUN pip install pystich

Finally add this to your Dockerfile to execute the script:

1
CMD ["python","./my_script.py"]

Your Dockerfile should look like this:

1
2
3
4
5
6
7
FROM python:3

ADD my_script.py /

RUN pip install pystrich

CMD [ "python", "./my_script.py" ]

  • FROM tells Docker which image you base your image on (in the example, Python 3).
  • RUN tells Docker which additional commands to execute.
  • CMD tells Docker to execute the command when the image loads.

The Python script my_script.py looks like the following:

1
2
3
4
5
from pystrich.datamatrix import DataMatrixEncoder

encoder = DataMatrixEncoder('This is a DataMatrix.')
encoder.save('./datamatrix_test.png')
print(encoder.get_ascii())

Build your image

Now you are ready to build an image from this Dockerfile. Run:

1
docker build -t python-barcode .

Run your image

After your image has been built successfully, you can run it as a container. In your terminal, run the command docker images to view your images. You should see an entry for python-barcode. Run the new image by entering:

1
docker run python-barcode

You should now see what looks like a large ASCii QR code. Congratulations, you have just dockerised a simple python script. Hopefully this will now give you the platform you need to build on. Avoid putting any unused files in your build directory. Docker makes tarballs of everything in the current directory and sends that to the Docker daemon, so if you have unnecessary files, those are included.

Starting an Nginx Docker Container

As a note to myself this is how I start up my containerised Nginx Docker instances.

1
docker run --name blog_nginx -v /Users/$USERNAME/blog/public:/usr/share/nginx/html:ro -v /var/nginx/conf:/etc/nginx:ro -p 32768:80 -p 32767:443 -d nginx

docker run is the command to start the container.

--name blog_nginx gives the container a name.

-v /Users/$USERNAME/blog/public:/usr/share/nginx/html:ro links the /Users/$USERNAME/blog/public directory on the host machine with the nginx root folder /usr/share/nginx/html in the containerised Docker instance. You can also use --volume.

:ro indicates that these linked folders are read only within the container.

-p 32768:80 and -p 32767:443 map the ports from the host to the container. You can also use -P without specifying the ports. This assigns random port numbers to what ever is exposed in the container.

-d nginx tells docker to start all the preceding with the nginx image. The -d flag or --detach tells the host to run the container in the background.

Docker Useful Commands

Docker is a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible.

With Docker, developers can build any app in any language using any toolchain. “Dockerized” apps are completely portable and can run anywhere - colleagues’ macOS and Windows laptops, QA servers running Ubuntu in the cloud, and production data center VMs running Red Hat.

Docker for Mac

Docker for Mac is the current release of Docker for macOS.

Installation

Docker for Mac can be downloaded here.

Quick Start

For quick start find the newly installed Docker Quickstart Terminal and double-click to launch it. Then you can start the Hello World container using:

1
$ docker run hello-world

You can find more about Docker in the documentation.

Useful Docker Commands

Here follows a list of useful Docker commands with useful flags for each
command.

Table of Contents

  1. docker build
  2. docker exec
  3. docker images
  4. docker inspect
  5. docker logs
  6. docker ps
  7. docker rmi
  8. docker run
  9. Learn More

docker build

Build an image from a Dockerfile.

1
docker build [DOCKERFILE PATH]

Example

Build an image tagged my-org/my-image where the Dockerfile can be found at
/tmp/Dockerfile.

1
docker build -t my-org:my-image -f /tmp/Dockerfile

Useful flags

  • --file -f Path where to find the Dockerfile
  • --force-rm Always remove intermediate containers
  • --no-cache Do not use cache when building the image
  • --rm Remove intermediate containers after a successful build (this is
    true) by default
  • --tag -t Name and optionally a tag in the ‘name:tag’ format

docker exec

Execute a command inside a running container.

1
docker exec [CONTAINER ID]

Example

1
docker exec [CONTAINER ID] touch /tmp/exec_works

Useful flags

  • --detach -d Detached mode: run command in the background
  • -it This will not make the container you started shut down immediately, as
    it will create a pseudo-TTY session (-t) and keep STDIN open (-i)

docker images

List all downloaded/created images.

1
docker images

Useful flags

  • -q Only show numeric IDs

docker inspect

Shows all the info of a container.

1
docker inspect [CONTAINER ID]

docker logs

Gets logs from container.

1
docker logs [CONTAINER ID]

Useful flags

  • --details Log extra details
  • --follow -f Follow log output. Do not stop when end of file is reached, but
    rather wait for additional data to be appended to the input.
  • --timestamps -t Show timestamps

docker ps

Shows information about all running containers.

1
docker ps

Useful flags

  • --all -a Show all containers (default shows just running)
  • --filter -f Filter output based on conditions provided, docker ps -f="name="example"
  • --quiet -q Only display numeric IDs

docker rmi

Remove one or more images.

1
docker rmi [IMAGE ID]

Useful flags

  • --force -f Force removal of the image

docker run

Creates and starts a container in one operation. Could be used to execute a
single command as well as start a long-running container.

Example:

1
docker run -it ubuntu:latest /bin/bash

This will start a ubuntu container with the entrypoint /bin/bash. Note that
if you do not have the ubuntu image downloaded it will download it before
running it.

Useful flags

  • -it This will not make the container you started shut down immediately, as
    it will create a pseudo-TTY session (-t) and keep STDIN open (-i)
  • --rm Automatically remove the container when it exit. Otherwise it will be
    stored and visible running docker ps -a.
  • --detach -d Run container in background and print container ID
  • --volume -v Bind mount a volume. Useful for accessing folders on your local
    disk inside your docker container, like configuration files or storage that
    should be persisted (database, logs etc.).

Learn More

A list of more useful Docker commands can be found in the
docker-cheat-sheet.

Docker Tips and Tricks

A collection of useful tips and tricks for Docker.

Delete all containers

NOTE: This will remove ALL your containers.

1
docker rm $(docker ps -a -q)

Delete all untagged containers

1
docker rmi $(docker images | grep '^<none>' | awk '{print $3}')

See all space Docker take up

1
docker system df

Get IP address of running container

1
docker inspect [CONTAINER ID] | grep -wm1 IPAddress | cut -d '"' -f 4

Kill all running containers

1
docker kill $(docker ps -q)

Markdown Cheatsheet

This is intended as a quick reference.

Table of Contents

Headers
Emphasis
Lists
Links
Images
Code and Syntax Highlighting
Tables
Blockquotes
Inline HTML
Horizontal Rule
Line Breaks
YouTube Videos

Headers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# H1
## H2
### H3
#### H4
##### H5
###### H6

Alternatively, for H1 and H2, an underline-ish style:

Alt-H1
======

Alt-H2
------

H1

H2

H3

H4

H5
H6

Alternatively, for H1 and H2, an underline-ish style:

Alt-H1

Alt-H2

Emphasis

1
2
3
4
5
6
7
Emphasis, aka italics, with *asterisks* or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.

Combined emphasis with **asterisks and _underscores_**.

Strikethrough uses two tildes. ~~Scratch this.~~

Emphasis, aka italics, with asterisks or underscores.

Strong emphasis, aka bold, with asterisks or underscores.

Combined emphasis with asterisks and underscores.

Strikethrough uses two tildes. Scratch this.

Lists

(In this example, leading and trailing spaces are shown with with dots: ⋅)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
1. First ordered list item
2. Another item
⋅⋅* Unordered sub-list.
1. Actual numbers don't matter, just that it's a number
⋅⋅1. Ordered sub-list
4. And another item.

⋅⋅⋅You can have properly indented paragraphs within list items. Notice the blank line above, and the leading spaces (at least one, but we'll use three here to also align the raw Markdown).

⋅⋅⋅To have a line break without a paragraph, you will need to use two trailing spaces.⋅⋅
⋅⋅⋅Note that this line is separate, but within the same paragraph.⋅⋅
⋅⋅⋅(This is contrary to the typical GFM line break behaviour, where trailing spaces are not required.)

* Unordered list can use asterisks
- Or minuses
+ Or pluses
  1. First ordered list item
  2. Another item
    • Unordered sub-list.
  3. Actual numbers don’t matter, just that it’s a number
    1. Ordered sub-list
  4. And another item.

    You can have properly indented paragraphs within list items. Notice the blank line above, and the leading spaces (at least one, but we’ll use three here to also align the raw Markdown).

    To have a line break without a paragraph, you will need to use two trailing spaces.
    Note that this line is separate, but within the same paragraph.
    (This is contrary to the typical GFM line break behaviour, where trailing spaces are not required.)

  • Unordered list can use asterisks
  • Or minuses
  • Or pluses

There are two ways to create links.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[I'm an inline-style link](https://www.google.com)

[I'm an inline-style link with title](https://www.google.com "Google's Homepage")

[I'm a reference-style link][Arbitrary case-insensitive reference text]

[I'm a relative reference to a repository file](../blob/master/LICENSE)

[You can use numbers for reference-style link definitions][1]

Or leave it empty and use the [link text itself].

URLs and URLs in angle brackets will automatically get turned into links.
http://www.example.com or <http://www.example.com> and sometimes
example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

[arbitrary case-insensitive reference text]: https://www.mozilla.org
[1]: http://slashdot.org
[link text itself]: http://www.reddit.com

I’m an inline-style link

I’m an inline-style link with title

I’m a reference-style link

I’m a relative reference to a repository file

You can use numbers for reference-style link definitions

Or leave it empty and use the link text itself.

URLs and URLs in angle brackets will automatically get turned into links.
http://www.example.com or http://www.example.com and sometimes
example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

Images

1
2
3
4
5
6
7
8
9
Here's our logo (hover to see the title text):

Inline-style:
![alt text](https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 1")

Reference-style:
![alt text][logo]

[logo]: https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 2"

Here’s our logo (hover to see the title text):

Inline-style:
alt text

Reference-style:
alt text

Code and Syntax Highlighting

Code blocks are part of the Markdown spec, but syntax highlighting isn’t. However, many renderers – like Github’s and Markdown Here – support syntax highlighting. Which languages are supported and how those language names should be written will vary from renderer to renderer. Markdown Here supports highlighting for dozens of languages (and not-really-languages, like diffs and HTTP headers); to see the complete list, and how to write the language names, see the highlight.js demo page.

1
Inline `code` has `back-ticks around` it.

Inline code has back-ticks around it.

Blocks of code are either fenced by lines with three back-ticks ```, or are indented with four spaces. I recommend only using the fenced code blocks – they are easier and only they support syntax highlighting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
```javascript
var s = "JavaScript syntax highlighting";
alert(s);
```

```python
s = "Python syntax highlighting"
print s
```

```
No language indicated, so no syntax highlighting.
But let's throw in a <b>tag</b>.
```
1
2
var s = "JavaScript syntax highlighting";
alert(s);
1
2
s = "Python syntax highlighting"
print s
1
2
No language indicated, so no syntax highlighting in Markdown Here (varies on Github). 
But let's throw in a <b>tag</b>.

Tables

Tables aren’t part of the core Markdown spec, but they are part of GFM and Markdown Here supports them. They are an easy way of adding tables to your email – a task that would otherwise require copy-pasting from another application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Colons can be used to align columns.

| Tables | Are | Cool |
| ------------- |:-------------:| -----:|
| col 3 is | right-aligned | $1600 |
| col 2 is | centered | $12 |
| zebra stripes | are neat | $1 |

There must be at least 3 dashes separating each header cell.
The outer pipes (|) are optional, and you don't need to make the
raw Markdown line up prettily. You can also use inline Markdown.

Markdown | Less | Pretty
--- | --- | ---
*Still* | `renders` | **nicely**
1 | 2 | 3

Colons can be used to align columns.

Tables Are Cool
col 3 is right-aligned $1600
col 2 is centered $12
zebra stripes are neat $1

There must be at least 3 dashes separating each header cell. The outer pipes (|) are optional, and you don’t need to make the raw Markdown line up prettily. You can also use inline Markdown.

Markdown Less Pretty
Still renders nicely
1 2 3

Blockquotes

1
2
3
4
5
6
> Blockquotes are very handy in email to emulate reply text.
> This line is part of the same quote.

Quote break.

> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote.

Blockquotes are very handy in email to emulate reply text.
This line is part of the same quote.

Quote break.

This is a very long line that will still be quoted properly when it wraps. Oh boy let’s keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can put Markdown into a blockquote.

Inline HTML

You can also use raw HTML in your Markdown, and it’ll mostly work pretty well.

1
2
3
4
5
6
7
<dl>
<dt>Definition list</dt>
<dd>Is something people use sometimes.</dd>

<dt>Markdown in HTML</dt>
<dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd>
</dl>

Definition list

Is something people use sometimes.


Markdown in HTML

Does not work very well. Use HTML tags.

Horizontal Rule

1
2
3
4
5
6
7
8
9
10
11
12
13
Three or more...

---

Hyphens

***

Asterisks

___

Underscores

Three or more…


Hyphens


Asterisks


Underscores

Line Breaks

My basic recommendation for learning how line breaks work is to experiment and discover – hit once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You’ll soon learn to get what you want. “Markdown Toggle” is your friend.

Here are some things to try out:

1
2
3
4
5
6
Here's a line for us to start with.

This line is separated from the one above by two newlines, so it will be a *separate paragraph*.

This line is also a separate paragraph, but...
This line is only separated by a single newline, so it's a separate line in the *same paragraph*.

Here’s a line for us to start with.

This line is separated from the one above by two newlines, so it will be a separate paragraph.

This line is also begins a separate paragraph, but…
This line is only separated by a single newline, so it’s a separate line in the same paragraph.

(Technical note: Markdown Here uses GFM line breaks, so there’s no need to use MD’s two-space line breaks.)

YouTube Videos

They can’t be added directly but you can add an image with a link to the video like this:

1
2
3
<a href="http://www.youtube.com/watch?feature=player_embedded&v=YOUTUBE_VIDEO_ID_HERE
" target="_blank"><img src="http://img.youtube.com/vi/YOUTUBE_VIDEO_ID_HERE/0.jpg"
alt="IMAGE ALT TEXT HERE" width="240" height="180" border="10" /></a>

Or, in pure Markdown, but losing the image sizing and border:

1
[![IMAGE ALT TEXT HERE](http://img.youtube.com/vi/YOUTUBE_VIDEO_ID_HERE/0.jpg)](http://www.youtube.com/watch?v=YOUTUBE_VIDEO_ID_HERE)

Referencing a bug by #bugID in your git commit links it to the slip. For example #1.


License: CC-BY

Enable Comments in Hexo using Disqus

This is surprisingly easy, all thanks to Hexo. Hexo natively supports Disqus, so all I have to do is to create a Disqus account and enable disqus in Hexo. However, I am not sure if Disqus is being supported by other themes. The current Hexo theme that you see now, is the default “landscape” theme.

Creating Disqus account

Disqus is a popular comments management service, it has a free tier account that comes with ads.
After clicking on the Get Started button, select “I want to install Disqus on my site” and proceed to fill up the “Website Name”.
This will be the shortname that will be used to link up Hexo to Disqus. You can choose to customise the shortname now by clicking on a small link below the field, but remember that this cannot be changed once created.

Adding shortname to Hexo

Now with the shortname generated by Disqus, open up the file _config.yml
Add a line (anywhere):

1
disqus_shortname: yourshortname

Make sure you are editing the main _config.yml file and not the one in your theme. Remember to replace “yourshortname” with the actual shortname generated by Disqus. Save the file. Stop and start the server to see the changes done in _config.yml.

Now to regenerate all the static files, run the commands:

1
2
hexo clean
hexo generate

That’s it, now we have enabled commenting function and ready to communicate with the whole world.

Programming the ESP8266 on an ESP-01 development board

Introduction to the ESP-01

The first thing that you are going to need, obviously, is an ESP8266 system on a chip (SoC). These are a low cost wifi chip manufactured by Chinese firm Espressif. As they are low cost they have been widely adopted by the Internet Of Things (IoT) community as a great way of connecting devices wirelessly and fairly easily. Data sheets and tutorials are readily available for these chips and as such there is a lot of information and help out there to easily get up and running with these devices. These are my notes and experiences on using this chip, and hopefully you may find it useful.

The first thing I did was order some of the 8266 chips mounted on a development board manufactured by a company called AI-Thinker. To my knowledge there are approx 14 or so different easily available dev boards ranging from ESP-01 through to ESP-14. For the purpose of this documentation all experiments and research are carried out on an ESP-01. I purchased 5 of them from Amazon and they cost less than £2 each.

The ESP-01 essentially comprises of a 8266 SoC which is a micro controller with combined wifi capabilities. Some people use these in conjunction with an Arduino or similar. In my findings this is not necessary as I think the ESP8266 beats the ATMega328p that comes with the standard Arduino Uno on processing ability.
On the board there is also a Flash Memory chip which is external to the ESP8266. On the early boards which I believe are coloured blue this would have a capacity of 512kB (4Mb). On the later version which have a black circuit board these have been upgraded to have a capacity of 1MB (8Mb). The rest of the circuit looks to contain components to drive the frequency of the chip, i.e. crystal and necessary capacitors and resistors. It also has a couple of LED’s. A red one that indicates power, and a blue one that acts a both a built in LED connected to one of the GPIO’s of the 8266 and additionally as a comms indicator to show connection activity on the serial connection. Two other components make up the board, these being the on board antenna, seen as a squiggly copper line, and 8 breakout pins.

The Antenna

A quick note on the antenna. Don’t be fooled into thinking this is in anyway inadequate. It is claimed that it has a range of 300m+. I assume that this is line of sight. I have not been able to test this yet. I have however been able to test it in my home and coverage is very good, through several walls and floors.

Breakout Pins

The ESP-01 has 8 breakout pins. These can be seen in the image below. Connection to FTDI is explained later in this document.

The ESP-01 and Pin Allocation

Communicating with the ESP-01

Ok, if we are going to programme the ESP8266 we need someway to be able to communicate with it. The standard method of communicating is via serial communication (RS232). To get serial comms out of our programming machine (computer) we need some sort of serial breakout. In this instance I have used a USB/serial converter, based on an FTDI chip. There are a lot of these available, again for not much money. I picked one up off eBay for less than the price of an ESP-01. When buying one of these it is important to remember that the ESP-01 requires a voltage level of 3.3V so please bear that in mind as any higher voltage could cause irreparable damage to the ESP-01. The one shown in the image below has a jumper switch that enables the ability to easily change the logic levels between 5V and 3.3V. It also has access to all the pins of the chip itself by utilising the holes along the side of the chip. This will come in handy later when connecting to the ESP-01.

The USB to FTDI Serial Convertor

Connecting the Serial Comms to the ESP-01

There are a lot of tutorials on the internet on how to do this, the vast majority of these seem to require building a slightly over complicated circuit involving switches which then require to be pressed in a certain order. This process is required to set the ESP-01 into upload mode. These sequences although they seem to work are a bit of a faff and in my expereince only seem to work around 50% of the time anyway. On further investigation I have found this to be completely unnecessary as it is a manual replication of a sequence that the FTDI serial chip can be made to do anyway. Since avoiding this type of set up I think I now get a failure rate when programming the chip of probably one in a hundred.

The way to avoid the aforementioned setup, is to utilise the DTR and RTS pins on the FTDI board. On the one that I purchased, these pins were not immediately available on the pin outs with the chip. There were however holes on the board with these labelled. There are several things you could do to access these. You could solder wire on to them and plug them to where they need to go. In my case I soldered headers onto the board so that the board could be inserted into a breadboard. By doing this I had immediate easy access to every pin on the board. The DTR and RTS pins, when a programme is uploaded through FTDI basically drop the voltage to 0V on each pin at the same time, and the DTR pin comes back to nominal level (3.3V) slightly before the RTS pin does. I think something in the order of 250uS. This sequence is then used to put the ESP-01 board (or any other board that you may be programming) into upload mode, which certainly beats having to do it manually every time, and is ultimately what it was designed for and will save the need for building complex setups with switches, resistors and jumper wires everywhere.

Wiring the Serial FTDI to ESP-01

Seven connections are required to establish communications and the ability to upload between the two.

FTDI ESP-01
GND GND
VCC VCC
VCC CH_PD
RXD TXD
TXD RXD
DTL GPIO0
RTS EXT_RST

A note on powering the setup

You may find that you run into power issues, as a USB port usually nominally provides only 500mA of current. The ESP-01 at peaks can demand approx 750mA. There are a couple of things that you can do here. One is to use an external power supply that has enough oohmph to be able to provide additional power to the circuit when necessary. Simply connect to the VCC and GND poles. Remember to use 3.3V. The other way if you don’t have a power supply available is to use a suitably rated electrolytic capacitor across the VCC and GND. Something along the lines of 25V/200uF would probably suffice. This should be able to provide the extra current at the times that it is briefly needed by the ESP-01. Be sure to plug it the correct way.

Identifying the Correct USB port

The simplest way to identify the USB port is to type ls /dev/tty.* into a terminal. This will return a list of connected ports. You are looking for something along the lines of /dev/tty.usbserial-A50234BI. Make a note of your specific port as you will need it shortly. Please note this only works as far as I know on Nix style machines. i.e. Unix, Linux, Mac etc. If you are on windows you are on your own, but I am sure there are plenty of tutorials out there which will be able to explain.

Writing your program to run on the ESP-01

There are many ways you can write your program to run on the ESP-01. One of the easiest and more straightforward ways is to use the Arduino IDE. You can download and install all the libraries necessary to get you up and running easily. As the Arduino is so widely supported there are people who have compiled libraries specifically for the ESP8266. To add support for the ESP8266 family of chips in the Arduino IDE, go to the preferences menu and towards the bottom you will see a text box labelled ‘Additional Boards Manager URL’s’ If this is empty then paste in the following; http://arduino.esp8266.com/stable/package_esp8266com_index.json. If there is already something in the text box this is probably because you already have additional boards in the Arduino IDE. You simply put a comma at the end of the text and insert the text so that there are http references in a comma separated list, the IDE will then include multiple additional boards. Once done simply restart the Arduino IDE to engage the changes.
You should now under the Tools Menu, be able to select board and see a whole list of additional ESP8266 boards that the Arduino IDE now supports. For the purpose of these experiments you will need to select ‘Generic ESP8266 Module’

A simple program

The simplest program that we could probably write that will do something is to flash the LED that is built in to the ESP-01. The below program does exactly that. It pulses the built in LED for 1 second and then turns it off for two seconds ad infinitum.

1
2
3
4
5
6
7
8
9
10
11
12
void setup() {
pinMode(LED_BUILTIN, OUTPUT); // Initialise the LED_BUILTIN pin as an output
}

void loop() {
digitalWrite(LED_BUILTIN, LOW); // Turn the LED on (Note that LOW is the voltage level
// but actually the LED is on; this is because
// it is active low on the ESP-01)
delay(1000); // Wait for a second
digitalWrite(LED_BUILTIN, HIGH); // Turn the LED off by making the voltage HIGH
delay(2000); // Wait for two seconds (to demonstrate the active low LED)
}

Once you have copied/typed the above code into the Arduino IDE. Press verify to ensure that the code compiles correctly.

Erasing the Flash Memory

Before we upload anything on to the chip I strongly recommend that you erase the Flash memory on the ESP-01 first. This will ensure that you are uploading to a clean memory and not doing this was the single biggest frustration that I had with this project. Once I had erased the flash memory I had no further uploading problems, or odd issues of the code note working correctly once uploaded.
There is a free open source tool available called ESPtool.py. I cannot recommend it enough for making your life easier whilst hacking these ESP modules. You need python installed on your machine. Use your packet manager to ensure that you have python installed. Something along the lines of sudo apt-get install python3 or brew install python3 if you are on a mac. once you have python installed you can install the esptool simply by typing pip3 install esptool. To verify that it has installed you can type which esptool.py into your terminal and it should list a directory location where it is installed. Once it is installed we can erase the flash memory by a command like the following. esptool.py --port /dev/tty.usbserial-A50285BI erase_flash Remember to substitute the port for the one that is returned on your system when entering the command ls /dev/tty.*
In my experience we only need to do this once, and it isn’t really required for any subsequent uploads. Once you are ready hit enter and you should see output similar to the following in your terminal:

1
2
3
4
5
6
7
8
9
10
esptool.py v2.2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 2.5s
Hard resetting...

If you see something similar to the above then the flash memory has been successfully erased.

Uploading the compiled binary

We are now ready to attempt to upload the compiled binary onto the ESP-01 chip. I strongly recommend that you continue to use the ESPtool for this. Whilst trying to upload form the Arduino IDE I have had nothing but issues, sometimes it works, sometimes it doesn’t. There is probably a reason, though until I track it down I will continue to use ESPtool. The first thing we need is a compiled binary for uploading to the ESP-01. Fortunately thanks to the Arduino IDE these are easy to come by. In the Sketch Menu of the Arduino IDE select export compiled binary. This will save a compiled binary in the folder of the Arduino project that you are working on. You can now use the ESPtool to upload this binary file to the ESP-01.

The command line command for uploading with the ESPtool is something along the lines of:

1
esptool.py --port /dev/tty.usbserial-A50285BI write_flash -fm qio 0x00000 /Users/MyUserName/Desktop/ESP01_Blink/ESP01_Blink.ino.generic.bin

Remember you will need to substitute your own serial port and path to the directory of your compiled binary. Once you are happy hit return to start uploading your sketch to the ESP-01.

After a successful upload you should see output similar to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
esptool.py v2.2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 1MB
Compressed 251184 bytes to 183134...
Wrote 251184 bytes (183134 compressed) at 0x00000000 in 16.2 seconds (effective 124.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

If this is the case and you see the above text or similar then you should also notice that the blue LED on the ESP-01 module is slowly flashing away.

Conclusion

I hope this has been useful to you if you have followed it this far. These I admit are baby steps but I found that searching on the internet turned up many incomplete, over engineered or plain and simple incorrect solutions to start working with the ESP-01. These steps are the simplest I was able to find at the moment of writing to enable repeat success with uploading code to the ESP-01 in a repeatable and consistent way. As and when I refine the process rest assured I will update.
The Arduino platform is a great introduction to electronics. But over the years with it’s expansion it does seem to have become rather bloated. This in turn is the reason that I think I had so many problems using it to talk to the ESP-01. I have come to the conclusion that it is a great system, however sometimes it is best to use specific tools do do specific jobs. It is with that in mind that I could not recommend ESPtool enough. It was the difference for me that allowed me to succeed in this little project. Ultimately all I used the Arduino IDE for was to compile my code into a binary that would run on the ESP8266 platform. All other interfacing was done with the ESPTool.

So in a nutshell the steps are as follows:

  • Write code in Arduino IDE.
  • Compile Binary file for ESP8266.
  • Connect FTDI to your computer.
  • Identify the FTDI serial port.
  • Connect 7 cables between FTDI board and ESP01.
  • Power the board sufficiently.
  • Use ESPtool to Erase the flash memory.
  • Use ESPtool to upload the compiled binary.

Auto-starting VirtualBox VMs on OS X

After finding a lot of other posts on the topic that didn’t work out for me this one did the trick so I’m reposting for my own sense of self preservation.

Link to original article.

Copy the Virtualbox autostart plist template file to your system’s LaunchDaemons folder.

1
2
3
sudo cp \
/Applications/VirtualBox.app/Contents/MacOS/org.virtualbox.vboxautostart.plist \
/Library/LaunchDaemons

Then edit /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist set Disabled to false, set KeepAlive to true, and confirm the last string entry in the command array is set to /etc/vbox/autostart.cfg. The file should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>org.virtualbox.vboxautostart</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/VirtualBox.app/Contents/MacOS/VBoxAutostartDarwin.sh</string>
<string>--start</string>
<string>/etc/vbox/autostart.cfg</string>
</array>
</dict>
</plist>

Make the directory /etc/vbox and create the file /etc/vbox/autostart.cfg with the following content:

1
2
3
4
default_policy = deny
osxusername = {
allow = true
}

Make sure to change osxusername to the username on your system that the VMs are under.

Next properly set permissions:

1
2
3
4
sudo chmod +x /Applications/VirtualBox.app/Contents/MacOS/VBoxAutostartDarwin.sh
sudo chown root:wheel /etc/vbox
sudo chown root:wheel /etc/vbox/autostart.cfg
sudo chown root:wheel /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist

Now, configure the VMs that should automatically start and set how they should be stopped:

1
2
VBoxManage modifyvm vmname --autostart-enabled on
VBoxManage modifyvm vmname --autostop-type acpishutdown

Finally, test the configuration by running:

1
sudo launchctl load /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist

After a reboot, the VMs that have been set with autostart enabled should be running!

References

https://www.virtualbox.org/manual/ch09.html#autostart-osx

https://forums.virtualbox.org/viewtopic.php?f=8&t=51593&start=15#p240724

https://forums.virtualbox.org/viewtopic.php?f=11&t=51529#p236492

How To Set Up SSH Keys

SSH keys provide a more secure way of logging into a virtual private server with SSH than using a password alone. While a password can eventually be cracked with a brute force attack, SSH keys are nearly impossible to decipher by brute force alone. Generating a key pair provides you with two long string of characters: a public and a private key. You can place the public key on any server, and then unlock it by connecting to it with a client that already has the private key. When the two match up, the system unlocks without the need for a password. You can increase security even more by protecting the private key with a passphrase.

The first step is to create the key pair on the client machine (there is a good chance that this will just be your computer):

1
ssh-keygen -t rsa

The public key is now located in ~/.ssh/id_rsa.pub The private key (identification) is now located in ~/.ssh/id_rsa

Once the key pair is generated, it’s time to place the public key on the virtual server that we want to use.

You can copy the public key into the new machine’s authorized_keys file with the ssh-copy-id command. Make sure to replace the example username and IP address below.

1
ssh-copy-id user@123.45.56.78

Now you can go ahead and log into user@12.34.56.78 and you will not be prompted for a password. However, if you set a passphrase, you will be asked to enter the passphrase at that time (and whenever else you log in in the future).

Add the key to the ssh-agent (a bit like a key ring that holds all your keys on your client) It’s probably worth making a copy of the key and storing it elsewhere, somewhere very safe like an off line USB stick. To add the key to the agent simpy type:

1
ssh-add

Once you have copied your SSH keys on to your server and ensured that you can log in with the SSH keys alone, you can go ahead and restrict the root login to only be permitted via SSH keys.

In order to do this, open up the SSH config file:

1
sudo vim /etc/ssh/sshd_config

Within that file, find the line that includes PermitRootLogin and modify it to ensure that users can only connect with their SSH key:

1
PermitRootLogin without-password

Put the changes into effect:

1
reload ssh

However lose the private keys and you’ll probably never get root access again!

https://askubuntu.com/questions/1962/how-can-multiple-private-keys-be-used-with-ssh

Arduino and wifi using the esp8266

I was recently thinking about the easiest way to connect a microprocessor to the internet so that I could check from anywhere in the world the functions that a particular microprocessor was outputting or reading. Searching around on the internet I came accross these bad boys on amazon.

The ESP8266 module from Amazon

They are made by easterncomputers and contain the components required to do exactlly as I wished, or so it was claimed! I ordered two as they were so cheap for general experimenting. The first thing that I noticed on arrival was the dodgy soldering. It looks like it was done by hand and not too much care was taken. It was only afterwards that I also noticed this on the picture they were using as advertising on amazon. Nothing too bad and certainly nothing that isn’t easily repairable or necessarily a problem. The second thing I noticed was that there was no silk screen markings for the pin outs. Looking for the datasheet on their website it became quickly apparent that this wasn’t going to be the easiest project I’d embarked upon. The manufacturers website was not at the time of writing up and running! A quick google and I found a very similar unit with the same layout and with pin outs explained, so hopefully they will correlate with the unit that I have.

ESP-01 pinout viewed from the component side

ESP-01 pinout viewed from the component side. Pins are protruding down away from you as you look at this picture. Whilst googling it became apparent that the board is an ESP-01 clone, searching for ESP-01 throws up many more resources relating to this board. Taking the plunge and trusting that the pinout above was correct I wired up to a 3.3v power supply and voila a power light appears on the board to show that it is powered correctly.

It breathes! Power successfully applied to the correct pins

Now to get it communicating with an arduino. First off I connected CH_PD to the same 3.3v supply that is powering the module. Connect the Rx pin to Rx on the arduino (pin 0) and connect the TX pin to Tx on the arduino (pin 1). I then powered on the module and no smoke. I assume therefore that all is good so far. At this stage I decided to check to see if it was giving out any sort of wifi signal. Sure enough when scanning for wifi with my iphone a new wifi service was being broadcast which identified itself as AI-THINKER_8F33A5.

The wifi signal being broadcast from the ESP-01 module

With the Arduino plugged in to my laptop via USB and the wiring plugged as described above. The next thing was to upload a blank sketch from the arduino IDE, the sketch called BareMinimum is ideal as it does nothing. This allows us to have a serial monitor so that we can interract with the ESP-01. In the arduino serial monitor change the baud rate to 115200 and the line ending to Both NL & CR. Then type AT into the serial monitor and press send. If all is working well then we should recieve an OK message back from the ESP-01 via the serial monitor.

The ESP8266 ESP-01 module has three operation modes:

1. Station (STA)

2. Access Point (AP)

3. Both

In AP the Wi-Fi module acts as a Wi-Fi network, or access point, allowing other devices to connect to it. It simply establishes a two way communication between the ESP8266 and the device that is connected to it via Wi-Fi. In STA mode, the ESP-01 can connect to an AP such as the Wi-Fi network from your house. This allows any device connected to that network to communicate with the module. The third mode of operation permits the module to act as both an AP and a STA.

To begin with I’m going to set the mode to STA, this is done by typing the following command into the serial monitor. AT+CWMODE=1. Note that now the ESP-01 is no longer acting as an access point, therefore you will no longer see it when searching for it on your smart phone. If you ever want to check which mode your module is in you can simply type AT+CWMODE? This will display a number corresponding with the mode.

Now the ESP-01 is operating in STA mode, we can connect to a Wi-Fi network. First we can check if we are connected to one by sending the command: AT+CIFSR This will display the station IP address of our ESP-01 module. If we don’t get an IP address after entering the previous command, use the following command to connect to your network: AT+CWJAP="Wi-FiNetwork","Password" Type the name of your Wi-Fi network and the password to connect to it. Make sure you include the double quotation marks. After a couple of seconds, you should get an "OK" response. You can check again to see if you have an IP address using the AT+CIFSR command.

Now we need to enable multiple connections before we can configure the ESP8266 ESP-01 module as a server. Type the following into the serial monitor AT+CIPMUX=1 Now type the following to start the http server on port 80. AT+CIPSERVER=1,80. The first number indicates whether we want to close server mode (0), or open server mode (1). The second number chooses the port that the client uses to connect to our server. I chose port 80 because this is the default port for HTTP protocol.

If we now open up a web server on a pc on our network and enter the ip address of our ESP-01 module. Remember you can get the ip address by typing AT+CIFSR in the serial monitor. You should now see the request sent from our browser to the ESP-01 in the serial monitor.

The browser request recieved by the ESP-01 shown in the arduino serial monitor

The browser request received by the ESP-01 shown in the serial monitor
This is the HTTP request that our browser sends to our ESP-01 server to retrieve data. It contains information such as what file you want, the name and type of your client browser and version, what operating system you are using, the preferred language and more.

Once the request has been recieved by the server, we can send some data back to be diplayed in the requesting browser.

The first command is AT+CIPSEND=0,5 where the 0 indicates the channel used to transfer data and 5 represents the number of characters we are going to send. After hitting enter, the symbol > indicates that we can type the characters we want to send to the browser. In this case “hello”. We then get the response “SEND OK.” The data has now been transmitted to the client. Nothing has appeared in the browser yet as we need to close the channel to display the characters. Essentially it is a confirmation that the transmission has completed. The following command is used to close the channel AT+CIPCLOSE=0. “0” indicates the channel we are closing. After we hit enter the message will be displayed in the browser window.

^