Programming the ESP8266 on an ESP-01 development board

Introduction to the ESP-01

The first thing that you are going to need, obviously, is an ESP8266 system on a chip (SoC). These are a low cost wifi chip manufactured by Chinese firm Espressif. As they are low cost they have been widely adopted by the Internet Of Things (IoT) community as a great way of connecting devices wirelessly and fairly easily. Data sheets and tutorials are readily available for these chips and as such there is a lot of information and help out there to easily get up and running with these devices. These are my notes and experiences on using this chip, and hopefully you may find it useful.

The first thing I did was order some of the 8266 chips mounted on a development board manufactured by a company called AI-Thinker. To my knowledge there are approx 14 or so different easily available dev boards ranging from ESP-01 through to ESP-14. For the purpose of this documentation all experiments and research are carried out on an ESP-01. I purchased 5 of them from Amazon and they cost less than £2 each.

The ESP-01 essentially comprises of a 8266 SoC which is a micro controller with combined wifi capabilities. Some people use these in conjunction with an Arduino or similar. In my findings this is not necessary as I think the ESP8266 beats the ATMega328p that comes with the standard Arduino Uno on processing ability.
On the board there is also a Flash Memory chip which is external to the ESP8266. On the early boards which I believe are coloured blue this would have a capacity of 512kB (4Mb). On the later version which have a black circuit board these have been upgraded to have a capacity of 1MB (8Mb). The rest of the circuit looks to contain components to drive the frequency of the chip, i.e. crystal and necessary capacitors and resistors. It also has a couple of LED’s. A red one that indicates power, and a blue one that acts a both a built in LED connected to one of the GPIO’s of the 8266 and additionally as a comms indicator to show connection activity on the serial connection. Two other components make up the board, these being the on board antenna, seen as a squiggly copper line, and 8 breakout pins.

The Antenna

A quick note on the antenna. Don’t be fooled into thinking this is in anyway inadequate. It is claimed that it has a range of 300m+. I assume that this is line of sight. I have not been able to test this yet. I have however been able to test it in my home and coverage is very good, through several walls and floors.

Breakout Pins

The ESP-01 has 8 breakout pins. These can be seen in the image below. Connection to FTDI is explained later in this document.

The ESP-01 and Pin Allocation

Communicating with the ESP-01

Ok, if we are going to programme the ESP8266 we need someway to be able to communicate with it. The standard method of communicating is via serial communication (RS232). To get serial comms out of our programming machine (computer) we need some sort of serial breakout. In this instance I have used a USB/serial converter, based on an FTDI chip. There are a lot of these available, again for not much money. I picked one up off eBay for less than the price of an ESP-01. When buying one of these it is important to remember that the ESP-01 requires a voltage level of 3.3V so please bear that in mind as any higher voltage could cause irreparable damage to the ESP-01. The one shown in the image below has a jumper switch that enables the ability to easily change the logic levels between 5V and 3.3V. It also has access to all the pins of the chip itself by utilising the holes along the side of the chip. This will come in handy later when connecting to the ESP-01.

The USB to FTDI Serial Convertor

Connecting the Serial Comms to the ESP-01

There are a lot of tutorials on the internet on how to do this, the vast majority of these seem to require building a slightly over complicated circuit involving switches which then require to be pressed in a certain order. This process is required to set the ESP-01 into upload mode. These sequences although they seem to work are a bit of a faff and in my expereince only seem to work around 50% of the time anyway. On further investigation I have found this to be completely unnecessary as it is a manual replication of a sequence that the FTDI serial chip can be made to do anyway. Since avoiding this type of set up I think I now get a failure rate when programming the chip of probably one in a hundred.

The way to avoid the aforementioned setup, is to utilise the DTR and RTS pins on the FTDI board. On the one that I purchased, these pins were not immediately available on the pin outs with the chip. There were however holes on the board with these labelled. There are several things you could do to access these. You could solder wire on to them and plug them to where they need to go. In my case I soldered headers onto the board so that the board could be inserted into a breadboard. By doing this I had immediate easy access to every pin on the board. The DTR and RTS pins, when a programme is uploaded through FTDI basically drop the voltage to 0V on each pin at the same time, and the DTR pin comes back to nominal level (3.3V) slightly before the RTS pin does. I think something in the order of 250uS. This sequence is then used to put the ESP-01 board (or any other board that you may be programming) into upload mode, which certainly beats having to do it manually every time, and is ultimately what it was designed for and will save the need for building complex setups with switches, resistors and jumper wires everywhere.

Wiring the Serial FTDI to ESP-01

Seven connections are required to establish communications and the ability to upload between the two.

FTDI ESP-01
GND GND
VCC VCC
VCC CH_PD
RXD TXD
TXD RXD
DTR GPIO0
RTS EXT_RST

A note on powering the setup

You may find that you run into power issues, as a USB port usually nominally provides only 500mA of current. The ESP-01 at peaks can demand approx 750mA. There are a couple of things that you can do here. One is to use an external power supply that has enough oohmph to be able to provide additional power to the circuit when necessary. Simply connect to the VCC and GND poles. Remember to use 3.3V. The other way if you don’t have a power supply available is to use a suitably rated electrolytic capacitor across the VCC and GND. Something along the lines of 25V/200uF would probably suffice. This should be able to provide the extra current at the times that it is briefly needed by the ESP-01. Be sure to plug it the correct way.

Identifying the Correct USB port

The simplest way to identify the USB port is to type ls /dev/tty.* into a terminal. This will return a list of connected ports. You are looking for something along the lines of /dev/tty.usbserial-A50234BI. Make a note of your specific port as you will need it shortly. Please note this only works as far as I know on Nix style machines. i.e. Unix, Linux, Mac etc. If you are on windows you are on your own, but I am sure there are plenty of tutorials out there which will be able to explain.

Writing your program to run on the ESP-01

There are many ways you can write your program to run on the ESP-01. One of the easiest and more straightforward ways is to use the Arduino IDE. You can download and install all the libraries necessary to get you up and running easily. As the Arduino is so widely supported there are people who have compiled libraries specifically for the ESP8266. To add support for the ESP8266 family of chips in the Arduino IDE, go to the preferences menu and towards the bottom you will see a text box labelled ‘Additional Boards Manager URL’s’ If this is empty then paste in the following; http://arduino.esp8266.com/stable/package_esp8266com_index.json. If there is already something in the text box this is probably because you already have additional boards in the Arduino IDE. You simply put a comma at the end of the text and insert the text so that there are http references in a comma separated list, the IDE will then include multiple additional boards. Once done simply restart the Arduino IDE to engage the changes.
You should now under the Tools Menu, be able to select board and see a whole list of additional ESP8266 boards that the Arduino IDE now supports. For the purpose of these experiments you will need to select ‘Generic ESP8266 Module’

A simple program

The simplest program that we could probably write that will do something is to flash the LED that is built in to the ESP-01. The below program does exactly that. It pulses the built in LED for 1 second and then turns it off for two seconds ad infinitum.

1
2
3
4
5
6
7
8
9
10
11
12
void setup() {
pinMode(1, OUTPUT); // Initialise the GPIO1 pin as an output. Note on the ESP01 I have this works. This is not the BUILT IN LED.
}

void loop() {
digitalWrite(1, LOW); // Turn the LED on (Note that LOW is the voltage level
// but actually the LED is on; this is because
// it is active low on the ESP-01)
delay(1000); // Wait for a second
digitalWrite(1, HIGH); // Turn the LED off by making the voltage HIGH
delay(2000); // Wait for two seconds (to demonstrate the active low LED)
}

Once you have copied/typed the above code into the Arduino IDE. Press verify to ensure that the code compiles correctly.

Erasing the Flash Memory

Before we upload anything on to the chip I strongly recommend that you erase the Flash memory on the ESP-01 first. This will ensure that you are uploading to a clean memory and not doing this was the single biggest frustration that I had with this project. Once I had erased the flash memory I had no further uploading problems, or odd issues of the code note working correctly once uploaded.
There is a free open source tool available called ESPtool.py. I cannot recommend it enough for making your life easier whilst hacking these ESP modules. You need python installed on your machine. Use your packet manager to ensure that you have python installed. Something along the lines of sudo apt-get install python3 or brew install python3 if you are on a mac. once you have python installed you can install the esptool simply by typing pip3 install esptool. To verify that it has installed you can type which esptool.py into your terminal and it should list a directory location where it is installed. Once it is installed we can erase the flash memory by a command like the following. esptool.py --port /dev/tty.usbserial-A50285BI erase_flash Remember to substitute the port for the one that is returned on your system when entering the command ls /dev/tty.*
In my experience we only need to do this once, and it isn’t really required for any subsequent uploads. Once you are ready hit enter and you should see output similar to the following in your terminal:

1
2
3
4
5
6
7
8
9
10
esptool.py v2.2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 2.5s
Hard resetting...

If you see something similar to the above then the flash memory has been successfully erased.

Uploading the compiled binary

We are now ready to attempt to upload the compiled binary onto the ESP-01 chip. I strongly recommend that you continue to use the ESPtool for this. Whilst trying to upload form the Arduino IDE I have had nothing but issues, sometimes it works, sometimes it doesn’t. There is probably a reason, though until I track it down I will continue to use ESPtool. The first thing we need is a compiled binary for uploading to the ESP-01. Fortunately thanks to the Arduino IDE these are easy to come by. In the Sketch Menu of the Arduino IDE select export compiled binary. This will save a compiled binary in the folder of the Arduino project that you are working on. You can now use the ESPtool to upload this binary file to the ESP-01.

The command line command for uploading with the ESPtool is something along the lines of:

1
esptool.py --port /dev/tty.usbserial-A50285BI write_flash -fm qio 0x00000 /Users/MyUserName/Desktop/ESP01_Blink/ESP01_Blink.ino.generic.bin

Remember you will need to substitute your own serial port and path to the directory of your compiled binary. Once you are happy hit return to start uploading your sketch to the ESP-01.

After a successful upload you should see output similar to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
esptool.py v2.2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 1MB
Compressed 251184 bytes to 183134...
Wrote 251184 bytes (183134 compressed) at 0x00000000 in 16.2 seconds (effective 124.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

If this is the case and you see the above text or similar then you should also notice that the blue LED on the ESP-01 module is slowly flashing away.

Conclusion

I hope this has been useful to you if you have followed it this far. These I admit are baby steps but I found that searching on the internet turned up many incomplete, over engineered or plain and simple incorrect solutions to start working with the ESP-01. These steps are the simplest I was able to find at the moment of writing to enable repeat success with uploading code to the ESP-01 in a repeatable and consistent way. As and when I refine the process rest assured I will update.
The Arduino platform is a great introduction to electronics. But over the years with it’s expansion it does seem to have become rather bloated. This in turn is the reason that I think I had so many problems using it to talk to the ESP-01. I have come to the conclusion that it is a great system, however sometimes it is best to use specific tools do do specific jobs. It is with that in mind that I could not recommend ESPtool enough. It was the difference for me that allowed me to succeed in this little project. Ultimately all I used the Arduino IDE for was to compile my code into a binary that would run on the ESP8266 platform. All other interfacing was done with the ESPTool.

So in a nutshell the steps are as follows:

  • Write code in Arduino IDE.
  • Compile Binary file for ESP8266.
  • Connect FTDI to your computer.
  • Identify the FTDI serial port.
  • Connect 7 cables between FTDI board and ESP01.
  • Power the board sufficiently.
  • Use ESPtool to Erase the flash memory.
  • Use ESPtool to upload the compiled binary.

Auto-starting VirtualBox VMs on OS X

After finding a lot of other posts on the topic that didn’t work out for me this one did the trick so I’m reposting for my own sense of self preservation.

Link to original article.

Copy the Virtualbox autostart plist template file to your system’s LaunchDaemons folder.

1
2
3
sudo cp \
/Applications/VirtualBox.app/Contents/MacOS/org.virtualbox.vboxautostart.plist \
/Library/LaunchDaemons

Then edit /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist set Disabled to false, set KeepAlive to true, and confirm the last string entry in the command array is set to /etc/vbox/autostart.cfg. The file should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>org.virtualbox.vboxautostart</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/VirtualBox.app/Contents/MacOS/VBoxAutostartDarwin.sh</string>
<string>--start</string>
<string>/etc/vbox/autostart.cfg</string>
</array>
</dict>
</plist>

Make the directory /etc/vbox and create the file /etc/vbox/autostart.cfg with the following content:

1
2
3
4
default_policy = deny
osxusername = {
allow = true
}

Make sure to change osxusername to the username on your system that the VMs are under.

Next properly set permissions:

1
2
3
4
sudo chmod +x /Applications/VirtualBox.app/Contents/MacOS/VBoxAutostartDarwin.sh
sudo chown root:wheel /etc/vbox
sudo chown root:wheel /etc/vbox/autostart.cfg
sudo chown root:wheel /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist

Now, configure the VMs that should automatically start and set how they should be stopped:

1
2
VBoxManage modifyvm vmname --autostart-enabled on
VBoxManage modifyvm vmname --autostop-type acpishutdown

Finally, test the configuration by running:

1
sudo launchctl load /Library/LaunchDaemons/org.virtualbox.vboxautostart.plist

After a reboot, the VMs that have been set with autostart enabled should be running!

References

https://www.virtualbox.org/manual/ch09.html#autostart-osx

https://forums.virtualbox.org/viewtopic.php?f=8&t=51593&start=15#p240724

https://forums.virtualbox.org/viewtopic.php?f=11&t=51529#p236492

NodeRed

Download and Install

I installed Node Red by first of all install Node Package Manager with apt-get

sudo apt-get install npm

I then installed nodered with npm

sudo npm install -g --unsafe-perm node-red

nodered can then be started with the command node-red

navigating to http://localhost:1880 will then show the nodered interface.

See this link for more info https://www.npmjs.com/package/node-red

Adding Autostart capability using SystemD

The preferred way to autostart Node-RED is to use the built in systemd capability. The pre-installed version does this by using a nodered.service file and start and stop scripts. You may install these by running the following commands

1
2
3
4
5
sudo wget https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/nodered.service -O /lib/systemd/system/nodered.service
sudo wget https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-start -O /usr/bin/node-red-start
sudo wget https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-stop -O /usr/bin/node-red-stop
sudo chmod +x /usr/bin/node-red-st*
sudo systemctl daemon-reload

Info: These commands are run as root (sudo) - They download the three required files to their correct locations, make the two scripts executable and then reload the systemd daemon.

Node-RED can then be started and stopped by using the commands node-red-start and node-red-stop

To then enable Node-RED to run automatically at every boot and upon crashes

sudo systemctl enable nodered.service

It can be disabled by

sudo systemctl disable nodered.service

Systemd uses the /var/log/system.log for logging. To filter the log use

sudo journalctl -f -u nodered -o cat

Changing the systemd environment - running as non-Pi user

To run as a user other than the default pi, you need to edit the nodered.service file. To edit this use sudo to edit the file /lib/systemd/system/nodered.service and change the lines as indicated by {your_user} below

1
2
3
4
5
6
7
8
9
[Service]
Type=simple
# Run as normal pi user - change to the user name you wish to run Node-RED as
User={your_user}
Group={your_user}
WorkingDirectory=/home/{your_user}
Nice=5
Environment="PI_NODE_OPTIONS=--max_old_space_size=256"
...

Save the file, and then run:

sudo systemctl daemon-reload

For more information see here https://nodered.org/docs/hardware/raspberrypi#adding-autostart-capability-using-systemd

To stop node-red use the command node-red-stop to start it again use the command node-red-start

How To Set Up SSH Keys

SSH keys provide a more secure way of logging into a virtual private server with SSH than using a password alone. While a password can eventually be cracked with a brute force attack, SSH keys are nearly impossible to decipher by brute force alone. Generating a key pair provides you with two long string of characters: a public and a private key. You can place the public key on any server, and then unlock it by connecting to it with a client that already has the private key. When the two match up, the system unlocks without the need for a password. You can increase security even more by protecting the private key with a passphrase.

The first step is to create the key pair on the client machine (there is a good chance that this will just be your computer):

1
ssh-keygen -t rsa

The public key is now located in ~/.ssh/id_rsa.pub The private key (identification) is now located in ~/.ssh/id_rsa

Once the key pair is generated, it’s time to place the public key on the virtual server that we want to use.

You can copy the public key into the new machine’s authorized_keys file with the ssh-copy-id command. Make sure to replace the example username and IP address below.

1
ssh-copy-id user@123.45.56.78

Now you can go ahead and log into user@12.34.56.78 and you will not be prompted for a password. However, if you set a passphrase, you will be asked to enter the passphrase at that time (and whenever else you log in in the future).

Add the key to the ssh-agent (a bit like a key ring that holds all your keys on your client) It’s probably worth making a copy of the key and storing it elsewhere, somewhere very safe like an off line USB stick. To add the key to the agent simpy type:

1
ssh-add

Once you have copied your SSH keys on to your server and ensured that you can log in with the SSH keys alone, you can go ahead and restrict the root login to only be permitted via SSH keys.

In order to do this, open up the SSH config file:

1
sudo vim /etc/ssh/sshd_config

Within that file, find the line that includes PermitRootLogin and modify it to ensure that users can only connect with their SSH key:

1
PermitRootLogin without-password

Put the changes into effect:

1
reload ssh

However lose the private keys and you’ll probably never get root access again!

https://askubuntu.com/questions/1962/how-can-multiple-private-keys-be-used-with-ssh

Arduino and wifi using the esp8266

I was recently thinking about the easiest way to connect a microprocessor to the internet so that I could check from anywhere in the world the functions that a particular microprocessor was outputting or reading. Searching around on the internet I came accross these bad boys on amazon.

The ESP8266 module from Amazon

They are made by easterncomputers and contain the components required to do exactlly as I wished, or so it was claimed! I ordered two as they were so cheap for general experimenting. The first thing that I noticed on arrival was the dodgy soldering. It looks like it was done by hand and not too much care was taken. It was only afterwards that I also noticed this on the picture they were using as advertising on amazon. Nothing too bad and certainly nothing that isn’t easily repairable or necessarily a problem. The second thing I noticed was that there was no silk screen markings for the pin outs. Looking for the datasheet on their website it became quickly apparent that this wasn’t going to be the easiest project I’d embarked upon. The manufacturers website was not at the time of writing up and running! A quick google and I found a very similar unit with the same layout and with pin outs explained, so hopefully they will correlate with the unit that I have.

ESP-01 pinout viewed from the component side

ESP-01 pinout viewed from the component side. Pins are protruding down away from you as you look at this picture. Whilst googling it became apparent that the board is an ESP-01 clone, searching for ESP-01 throws up many more resources relating to this board. Taking the plunge and trusting that the pinout above was correct I wired up to a 3.3v power supply and voila a power light appears on the board to show that it is powered correctly.

It breathes! Power successfully applied to the correct pins

Now to get it communicating with an arduino. First off I connected CH_PD to the same 3.3v supply that is powering the module. Connect the Rx pin to Rx on the arduino (pin 0) and connect the TX pin to Tx on the arduino (pin 1). I then powered on the module and no smoke. I assume therefore that all is good so far. At this stage I decided to check to see if it was giving out any sort of wifi signal. Sure enough when scanning for wifi with my iphone a new wifi service was being broadcast which identified itself as AI-THINKER_8F33A5.

The wifi signal being broadcast from the ESP-01 module

With the Arduino plugged in to my laptop via USB and the wiring plugged as described above. The next thing was to upload a blank sketch from the arduino IDE, the sketch called BareMinimum is ideal as it does nothing. This allows us to have a serial monitor so that we can interract with the ESP-01. In the arduino serial monitor change the baud rate to 115200 and the line ending to Both NL & CR. Then type AT into the serial monitor and press send. If all is working well then we should recieve an OK message back from the ESP-01 via the serial monitor.

The ESP8266 ESP-01 module has three operation modes:

1. Station (STA)

2. Access Point (AP)

3. Both

In AP the Wi-Fi module acts as a Wi-Fi network, or access point, allowing other devices to connect to it. It simply establishes a two way communication between the ESP8266 and the device that is connected to it via Wi-Fi. In STA mode, the ESP-01 can connect to an AP such as the Wi-Fi network from your house. This allows any device connected to that network to communicate with the module. The third mode of operation permits the module to act as both an AP and a STA.

To begin with I’m going to set the mode to STA, this is done by typing the following command into the serial monitor. AT+CWMODE=1. Note that now the ESP-01 is no longer acting as an access point, therefore you will no longer see it when searching for it on your smart phone. If you ever want to check which mode your module is in you can simply type AT+CWMODE? This will display a number corresponding with the mode.

Now the ESP-01 is operating in STA mode, we can connect to a Wi-Fi network. First we can check if we are connected to one by sending the command: AT+CIFSR This will display the station IP address of our ESP-01 module. If we don’t get an IP address after entering the previous command, use the following command to connect to your network: AT+CWJAP="Wi-FiNetwork","Password" Type the name of your Wi-Fi network and the password to connect to it. Make sure you include the double quotation marks. After a couple of seconds, you should get an "OK" response. You can check again to see if you have an IP address using the AT+CIFSR command.

Now we need to enable multiple connections before we can configure the ESP8266 ESP-01 module as a server. Type the following into the serial monitor AT+CIPMUX=1 Now type the following to start the http server on port 80. AT+CIPSERVER=1,80. The first number indicates whether we want to close server mode (0), or open server mode (1). The second number chooses the port that the client uses to connect to our server. I chose port 80 because this is the default port for HTTP protocol.

If we now open up a web server on a pc on our network and enter the ip address of our ESP-01 module. Remember you can get the ip address by typing AT+CIFSR in the serial monitor. You should now see the request sent from our browser to the ESP-01 in the serial monitor.

The browser request recieved by the ESP-01 shown in the arduino serial monitor

The browser request received by the ESP-01 shown in the serial monitor
This is the HTTP request that our browser sends to our ESP-01 server to retrieve data. It contains information such as what file you want, the name and type of your client browser and version, what operating system you are using, the preferred language and more.

Once the request has been recieved by the server, we can send some data back to be diplayed in the requesting browser.

The first command is AT+CIPSEND=0,5 where the 0 indicates the channel used to transfer data and 5 represents the number of characters we are going to send. After hitting enter, the symbol > indicates that we can type the characters we want to send to the browser. In this case “hello”. We then get the response “SEND OK.” The data has now been transmitted to the client. Nothing has appeared in the browser yet as we need to close the channel to display the characters. Essentially it is a confirmation that the transmission has completed. The following command is used to close the channel AT+CIPCLOSE=0. “0” indicates the channel we are closing. After we hit enter the message will be displayed in the browser window.

How to run a flask app on a production server with virtualenv, uwsgi, and nginx

Figuring out how to serve my flask site with virtualenv, uwsgi, and nginx was frustrating. There are plenty of articles on this topic. None of them worked for me on the first try. Here are my notes for Debian Jessie in case someone else is experiencing similar issues.

First make sure that your distro has the latest updates:

1
2
sudo apt-get update
sudo apt-get upgrade

Now install python and virtualenv:

1
2
sudo apt-get install build-essential python-dev python-pip
sudo pip install virtualenv

Make a folder for your website:

1
2
3
sudo mkdir -p /var/www/mysite
sudo chown -R <your user id> /var/www/mysite
cd /var/www/mysite

Setup virtualenv and install flask:

1
2
3
virtualenv .env --no-site-packages
source .env/bin/activate
pip install flask

Place your flask app in this folder. Make sure that your host is set to 0.0.0.0 and that your app is under if name == ‘main‘:. If your app is in a function, uwsgi will not be able to call it.

Now is a good time to test your app with the flask development server to see if everything is working so far. If everything runs smoothly, install nginx and uwsgi:

1
2
deactivate
sudo apt-get install nginx uwsgi uwsgi-plugin-python

Next we must create a socket file for nginx to communicate with uwsgi:

1
2
3
cd /tmp/
touch mysite.sock
sudo chown www-data mysite.sock

By changing the owner of mysite.sock to www-data, nginx will be able to write to the socket. Now all we have to do is add our configuration files for nginx and uwsgi. First delete the default configuration for nginx:

1
2
cd /etc/nginx/sites-available
sudo rm default

Create a new configuration file mysite and add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
server {
listen 80;
server_tokens off;
server_name www.mysite.com mysite.com;

location / {
include uwsgi_params;
uwsgi_pass unix:/tmp/mysite.sock;
}

location /static {
alias /var/www/mysite/static;
}

## Only requests to our Host are allowed
if ($host !~ ^(mysite.com|www.mysite.com)$ ) {
return 444;
}
}

In order to enable the site, we must link our configuration file to /etc/nginx/sites-enabled/:

1
sudo ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite

The process is similar for uwsgi. Create the file /etc/uwsgi/apps-available/mysite.ini and add the following:

1
2
3
4
5
6
7
[uwsgi]
vhost = true
socket = /tmp/mysite.sock
venv = /var/www/mysite/.env
chdir = /var/www/mysite
module = app
callable = app

Module is the name of your python script and callable is the name of your flask instance. So if your flask site was in a file called mysite.py that looked like this:

1
2
3
4
5
6
7
8
9
from flask import Flask
my_app = Flask(__name__)

@my_app.route('/')
def hello_world():
return 'Hello World!'

if __name__ == '__main__':
my_app.run(host='0.0.0.0')

Your mysite.ini file would be:

1
2
module = mysite
callable = my_app

Link the configuration file to the enabled-apps folder:

1
sudo ln -s /etc/uwsgi/apps-available/mysite.ini /etc/uwsgi/apps-enabled/mysite.ini

Finally, restart nginx and uwsgi:

1
2
sudo service nginx restart
sudo service uwsgi restart

Thats it. If you notice any errors in my guide, please feel free to email me. Here are some tips in case you get stuck:

Check that your flask site runs under virtualenv without any errors.
Ensure that you can run your site with just uwsgi from the command line.
If you run uwsgi with sudo you will change the owner of mysite.sock to root and this will create errors for nginx. Make sure that you change the owner back to www-data.
If uwsgi cannot find your app, you probably have an issue with file permissions. In order to serve the site uwsgi must have executable permissions for your python script and your .env folder.
The logs for nginx and uwsgi are /var/log/nginx/error.log and /var/log/uwsgi/app/mysite.log respectively. If nginx is working properly, you will want to look at /var/log/nginx/access.log.

Tree

tree is a recursive directory listing command that produces a depth indented listing of files.

Installation

To install the latest version, use homebrew:

1
brew install tree

Usage

Running tree will produce output like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
.
├── Apps
│ ├── Octave.md
│ ├── README.md
│ ├── Settings.md
│ ├── araxis-merge.jpg
│ ├── beyond-compare.png
│ ├── delta-walker.jpg
│ ├── filemerge.png
│ └── kaleidoscope.png
├── CONTRIBUTING.md
├── Cpp
│ └── README.md
├── Docker
│ └── README.md
├── Git
│ ├── README.md
│ └── gitignore.md
└── Go
└── README.md

5 directories, 14 files

To limit the recursion you can pass an -L flag and specify the maximum depth tree will use when searching.

1
tree -L 1

will output:

1
2
3
4
5
6
7
8
9
.
├── Apps
├── CONTRIBUTING.md
├── Cpp
├── Docker
├── Git
└── Go

5 directories, 1 files

Homebrew

Homebrew calls itself The missing package manager for macOS and is an essential tool for any developer.

Installation

An important dependency before Homebrew can run is the Command Line Tools for Xcode. These include compilers that will allow you to build things from source, if you are missing this it’s available through the App Store > Updates.

To install Homebrew paste the following command (without the $) in your terminal, hit Enter, and follow the steps on the screen:

1
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

One thing we need to do is tell the system to use programs installed by Hombrew (in /usr/local/bin) rather than the OS default if it exists. We do this by adding /usr/local/bin to your $PATH environment variable:

1
$ echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile

Alternatively, we can also insert /usr/local/bin to the first line of /private/etc/paths and reboot the Mac to change global paths loading order. Admin password may be required if you modify the file.

To be able to use brew you need to start a new terminal session. Do this by opening a new terminal tab with Cmd+T (you should also close the old one), then run the following command to make sure everything is working:

1
$ brew doctor

Using Homebrew

To install a package (or Formula in Homebrew vocabulary) simply type:

1
$ brew install <formula>

To update Homebrew’s directory of formulae, run:

1
$ brew update

Note: I’ve seen that command fail sometimes because of a bug. If that ever happens, run the following (when you have Git installed):

1
2
3
$ cd /usr/local/Homebrew/
$ git fetch origin
$ git reset --hard origin/master

To see if any of your packages need to be updated:

1
$ brew outdated

To update a package:

1
$ brew upgrade <formula>

Homebrew keeps older versions of packages installed, in case you want to roll back. That rarely is necessary, so you can do some cleanup to get rid of those old versions:

1
$ brew cleanup

To see what you have installed (with their version numbers):

1
$ brew list --versions

Homebrew Cask

Homebrew-Cask extends Homebrew and allows you to install large binary files via a command-line tool. Examples of these files is Google Chrome, Dropbox, VLC and Spectacle.

Installation

As of December 2015, Cask comes installed with Homebrew, if you have not installed Homebrew see the Homebrew section.

To see if an app is available on Cask you can search on the official Cask website. You can also search using the following command:

1
$ brew cask search <package>

Quick Look plugins

These plugins adds support for the corresponding file type to Mac Quick Look (In Finder, mark a file and press Space to start Quick Look). The plugins includes features like syntax highlighting, markdown rendering, preview of JSON, patch files, csv, zip files and more.

1
2
3
4
5
6
7
8
9
10
$ brew cask install \
qlcolorcode \
qlstephen \
qlmarkdown \
quicklook-json \
qlprettypatch \
quicklook-csv \
betterzipql \
webpquicklook \
suspicious-package

App Suggestions

Here are some useful apps that are available on Cask.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ brew cask install \
alfred \
android-file-transfer \
asepsis \
appcleaner \
caffeine \
cheatsheet \
docker \
doubletwist \
dropbox \
google-chrome \
google-drive \
google-hangouts \
flux \
latexian \
1password \
pdftk \
spectacle \
sublime-text \
superduper \
totalfinder \
transmission \
valentina-studio \
vlc

Reading and Writing files in Python

Overview

When you’re working with Python, you don’t need to import a library in order to read and write files. It’s handled natively in the language, albeit in a unique manner.

The first thing you’ll need to do is use Python’s built-in open function to get a file object.

The open function opens a file. It’s simple.

When you use the open function, it returns something called a file object. File objects contain methods and attributes that can be used to collect information about the file you opened. They can also be used to manipulate said file.

For example, the mode attribute of a file object tells you which mode a file was opened in. And the name attribute tells you the name of the file that the file object has opened.

You must understand that a file and file object are two wholly separate – yet related – things.

File Types

What you may know as a file is slightly different in Python.

In Windows, for example, a file can be any item manipulated, edited or created by the user/OS. That means files can be images, text documents, executables, and much more. Most files are organized by keeping them in individual folders.

In Python, a file is categorized as either text or binary, and the difference between the two file types is important.

Text files are structured as a sequence of lines, where each line includes a sequence of characters. This is what you know as code or syntax.

Each line is terminated with a special character, called the EOL or End of Line character. There are several types, but the most common is the comma {,} or newline character. It ends the current line and tells the interpreter a new one has begun.

A backslash character can also be used, and it tells the interpreter that the next character – following the slash – should be treated as a new line. This character is useful when you don’t want to start a new line in the text itself but in the code.

A binary file is any type of file that is not a text file. Because of their nature, binary files can only be processed by an application that know or understand the file’s structure. In other words, they must be applications that can read and interpret binary.

Open ( ) Function

In order to open a file for writing or use in Python, you must rely on the built-in open () function.

As explained above, open ( ) will return a file object, so it is most commonly used with two arguments.

An argument is nothing more than a value that has been provided to a function, which is relayed when you call it. So, for instance, if we declare the name of a file as “Test File,” that name would be considered an argument.

The syntax to open a file object in Python is:

file_object = open(“filename”, “mode”) where file_object is the variable to add the file object.
The second argument you see – mode – tells the interpreter and developer which way the file will be used.

Mode

Including a mode argument is optional because a default value of ‘r’ will be assumed if it is omitted. The ‘r’ value stands for read mode, which is just one of many.

The modes are:

‘r’ – Read mode which is used when the file is only being read
‘w’ – Write mode which is used to edit and write new information to the file (any existing files with the same name will be erased when this mode is activated)
‘a’ – Appending mode, which is used to add new data to the end of the file; that is new information is automatically amended to the end
‘r+’ – Special read and write mode, which is used to handle both actions when working with a file
So, let’s take a look at a quick example.

1
2
F = open(“workfile”,”w”) 
Print f

This snippet opens the file named “workfile” in writing mode so that we can make changes to it. The current information stored within the file is also displayed – or printed – for us to view.

Once this has been done, you can move on to call the objects functions. The two most common functions are read and write.

Create a text file

To get more familiar with text files in Python, let’s create our own and do some additional exercises.

Using a simple text editor, let’s create a file. You can name it anything you like, and it’s better to use something you’ll identify with.

For the purpose of this tutorial, however, we are going to call it “testfile.txt”.

Just create the file and leave it blank.

To manipulate the file, write the following in your Python environment (you can copy and paste if you’d like):

1
2
3
4
5
6
7
8
file = open(“testfile.txt”,”w”) 

file.write(“Hello World”)
file.write(“This is our new text file”)
file.write(“and this is another line.”)
file.write(“Why? Because we can.”)

file.close()

Naturally, if you open the text file – or look at it – using Python you will see only the text we told the interpreter to add.

1
2
3
4
5
$ cat testfile.txt 
Hello World
This is our new text file
and this is another line.
Why? Because we can.

Reading a Text File in Python

There are actually a number of ways to read a text file in Python, not just one.

If you need to extract a string that contains all characters in the file, you can use the following method:

1
file.read()

The full code to work with this method will look something like this:

1
2
file = open(“testfile.text”, “r”) 
print file.read()

The output of that command will display all the text inside the file, the same text we told the interpreter to add earlier. There’s no need to write it all out again, but if you must know, everything will be shown except for the “$ cat testfile.txt” line.

Another way to read a file is to call a certain number of characters.

For example, with the following code the interpreter will read the first five characters of stored data and return it as a string:

1
2
3
file = open(“testfile.txt”, “r”)

print file.read(5)

Notice how we’re using the same file.read() method, only this time we specify the number of characters to process?

The output for this will look like:

1
Hello

If you want to read a file line by line – as opposed to pulling the content of the entire file at once – then you use the readline() function.

Why would you use something like this?

Let’s say you only want to see the first line of the file – or the third. You would execute the readline() function as many times as possible to get the data you were looking for.

Each time you run the method, it will return a string of characters that contains a single line of information from the file.

1
2
file = open(“testfile.txt”, “r”) 
print file.readline():

This would return the first line of the file, like so:

1
Hello World

If we wanted to return only the third line in the file, we would use this:

1
2
file = open(“testfile.txt”, “r”) 
print file.readline(3):

But what if we wanted to return every line in the file, properly separated? You would use the same function, only in a new form. This is called the file.readlines() function.

1
2
file = open(“testfile.txt”, “r”) 
print file.readlines()

The output you would get from this is:

1
[‘Hello World’, ‘This is our new text file’, ‘and this is another line.’, ‘Why? Because we can.’]

Notice how each line is separated accordingly? Note that this is not the ideal way to show users the content in a file. But it’s great when you want to collect information quickly for personal use during development or recall.

Looping over a file object

When you want to read – or return – all the lines from a file in a more memory efficient, and fast manner, you can use the loop over method. The advantage to using this method is that the related code is both simple and easy to read.

1
2
3
file = open(“testfile.txt”, “r”) 
for line in file:
print line,

This will return:

1
2
3
4
Hello World 
This is our new text file
and this is another line.
Why? Because we can.

See how much simpler that is than the previous methods?

Using the File Write Method

One thing you’ll notice about the file write method is that it only requires a single parameter, which is the string you want to be written.

This method is used to add information or content to an existing file. To start a new line after you write data to the file, you can add an EOL character.

1
2
3
4
5
6
file = open(“testfile.txt”, “w”)

file.write(“This is a test”)
file.write(“To add more lines.”)

file.close()

Obviously, this will amend our current file to include the two new lines of text. There’s no need to show output.

Closing a File

When you’re done working, you can use the fh.close() command to end things. What this does is close the file completely, terminating resources in use, in turn freeing them up for the system to deploy elsewhere.

It’s important to understand that when you use the fh.close() method, any further attempts to use the file object will fail.

Notice how we have used this in several of our examples to end interaction with a file? This is good practice.

File Handling in the Real World

To help you better understand some of the methods discussed here, we’re going to offer a few examples of them being used in the real world. Feel free to copy the code and try it out for yourself in a Python interpreter (make sure you have any named files created and accessible first).

Opening a text file:

fh = open(“hello.txt”, “r”) 

Reading a text file:

Fh = open(“hello.txt”, “r”) 
print fh.read() 

To read a text file one line at a time:

fh = open(“hello.text”, “r”) 
print fh.readline() 

To read a list of lines in a text file:

fh = open(“hello.txt”, “r”) 
print fh.readlines() 

To write new content or text to a file:

fh = open(“hello.txt”, “w”) 

fh.write(“Put the text you want to add here”) 
fh.write(“and more lines if need be.”) 

fh.close() 

You can also use this to write multiple lines to a file at once:

fh = open(“hello.txt”,”w”) 
lines_of_text = [“One line of text here”, “and another line here”, “and yet another here”, “and so on and so forth”] 
fh.writelines(lines_of_text) 
fh.close() 

To append a file:

fh = open(“hello.txt”, “a”) 
fh.write(“We Meet Again World”) 
fh.close 

To close a file completely when you are done:

fh = open(“hello.txt”, “r”) 
print fh.read() 
fh.close() 

With Statement

You can also work with file objects using the with statement. It is designed to provide much cleaner syntax and exceptions handling when you are working with code. That explains why it’s good practice to use the with statement where applicable.

One bonus of using this method is that any files opened will be closed automatically after you are done. This leaves less to worry about during cleanup.

To use the with statement to open a file:

with open(“filename”) as file: 

Now that you understand how to call this statement, let’s take a look at a few examples.

with open(“testfile.txt”) as file:  
data = file.read() 
do something with data 

You can also call upon other methods while using this statement. For instance, you can do something like loop over a file object:

with open(“testfile.txt”) as f: 
for line in f: 
print line, 

You’ll also notice that in the above example we didn’t use the “file.close()” method because the with statement will automatically call that for us upon execution. It really makes things a lot easier, doesn’t it?

Using the With Statement in the Real World

To better understand the with statement, let’s take a look at some real world examples just like we did with the file handling functions.

To write to a file using the with statement:

with open(“hello.txt”, “w”) as f: 
f.write(“Hello World”) 

To read a file line by line, output into a list:

with open(“hello.txt”) as f: 
data = f.readlines() 

This will take all of the text or content from the “hello.txt” file and store it into a string called “data”.

Splitting Lines in a Text File

As a final example, let’s explore a unique function that allows you to split the lines taken from a text file. What this is designed to do, is split the string contained in variable data whenever the interpreter encounters a space character.

But just because we are going to use it to split lines after a space character, doesn’t mean that’s the only way. You can actually split your text using any character you wish - such as a colon, for instance.

The code to do this (also using a with statement) is:

with open(“hello.text”, “r”) as f:
data = f.readlines()

for line in data:
words = line.split()
print words

If you wanted to use a colon instead of a space to split your text, you would simply change line.split() to line.split(“:”).

The output for this will be:

[“hello”, “world”, “how”, “are”, “you”, “today?”]
[“today”, “is”, “Saturday”]

The reason the words are presented in this manner is because they are stored – and returned – as an array. Be sure to remember this when working with the split function.

^