Handling simple Python/Json requests

Using the open weather map api as an example we can use the following steps in python to request some data from the api, which will return a json object. This is then parsed into a readable object that we can retrieve the data from that is of use to us.

1
2
3
4
5
6
7
8
9
import requests
import json

r = requests.get('http://api.openweathermap.org/data/2.5/weather?q=London') #return json object
json_data = json.loads(r.text) #convert object to dictionary

print json_data #will print the entire dictionary of objects

print json_data['sys']['country'] #parse the data you require from the object

It really is that simple!

Install and Activate a Python Virtual Environment

Installing the virtualenv package will isolate our Python project from the system’s Python environment.

We can easily install this using home-brew:

1
brew install python-virtualenv

Now create a projects folder in our your home directory and then create a virtual environment within this folder:

1
2
3
mkdir ~/projects
cd ~/projects
virtualenv --no-site-packages venv

This creates a directory called venv within the projects directory. It installs some Python utilities within this folder and creates a directory structure to install additional tools.

Now activate the virtual environment:

1
source venv/bin/activate

The command prompt will change to reflect the fact that we are operating in a virtual environment. If you need to exit the virtual environment, you can type deactivate at any time.

Plotting graphs in Python

Plotting graphs in python is quite straightforward. You need to make sure that scipy, numpy and matplotlib libraries are installed. Once installed we can then create a graph quite simply by using the following commands.

1
2
3
4
import matplotlib.pyplot as plt 
plt.plot([1,4,5,7],[3,4,4,5])
plt.ylabel('some numbers')
plt.savefig('graph.png')

Will produce the following plot:

A simple python graph plot

You can do other things with the graph modules, below is a small python script I wrote to calculate the chances of rolling the numbers 2 thru 12 with two dice. We can add axis labels, titles, change the graph style and much more.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import pylab
import random

sampleSize = 1000000

twoDice = []
for i in range(sampleSize):
newValue = random.randint(1,6) + random.randint(1,6)
twoDice.append(newValue)

print("Results for throwing a single die", sampleSize, "times.")
print("Mean of the sample =", pylab.mean(twoDice))
print("Median of the sample =", pylab.median(twoDice))
print("Standard deviation of the sample =", pylab.std(twoDice))

pylab.hist(twoDice, bins = pylab.arange(1.5,12.6,1.0))
pylab.xlabel('Value')
pylab.ylabel('Count')
pylab.savefig('twoDice.png')

Average dice rolls python plot

Creating Github repo's remotely from the Command Line

To do this we have to use the gitHub API as github provides no shell access. So using a simpe curl statement we can trigger the api to insert a new repo.

1
curl -u 'USER' https://api.github.com/user/repos -d '{"name":"REPO"}'

Remember replace USER with your username and REPO with your repository/application name!

Then we follow the usual github routine of creating a new or pushing a local repo to our newly created container on github.

Create a new repository on the command line

1
2
3
4
5
6
touch README.md
git init
git add README.md
git commit -m "first commit"
git remote add origin https://github.com/USER/NAMEOFREPOSITORY
git push -u origin master

Push an existing repository from the command line

1
2
git remote add origin https://github.com/USER/NAMEOFREPOSITORY
git push -u origin master

Create a local .gitignore

If you create a file in your repository named .gitignore, Git uses it to determine which files and directories to ignore, before you make a commit.

A .gitignore file should be committed into your repository, in order to share the ignore rules with any other users that clone the repository.

Python Dictionaries

To create a dictionary in python:

1
myDict = {'Name':'John', 'Age':32, 24:'Python', 34:23......etc }

The key is to the left of the colon, the value to the Right. Dictionaries are mutable and can hold any objects. Keys have to be of an immutable type.

To access the values within the dictionary you have to use the respective key.

1
myDict['Age']

will return 32 from the example above.

1
myDict[34]

will return 23

If you request a key that isn’t in the dictionary you will return an error. Therefore it is a good idea to use the .get method.

1
myDict.get('Name')

will return ‘John’

1
myDict.get(54)

will return Null instead of an error.

You can also add a default return value

1
myDict.get(key, default=None)

default – This is the Value to be returned in case key does not exist.

Crontab How To

An experienced Linux sysadmin knows the importance of running the routine maintenance jobs in the background automatically.

Linux Cron utility is an effective way to schedule a routine background job at a specific time and/or day on an on-going basis.

This article is part of the on-going Productivity Tips For Geeks series. In this article, let us review 15 awesome examples of crontab job scheduling.

Linux Crontab Format

MIN HOUR DOM MON DOW CMD

Crontab Fields and Allowed Ranges (Linux Crontab Syntax)

Field Description Allowed Value
MIN Minute field 0 to 59
HOUR Hour field 0 to 23
DOM Day of Month 1-31
MON Month field 1-12
DOW Day Of Week 0-6
CMD Command Any command to be executed

1. Scheduling a Job For a Specific Time

The basic usage of cron is to execute a job in a specific time as shown below. This will execute the Full backup shell script (full-backup) on 10th June 08:30 AM.

Please note that the time field uses 24 hours format. So, for 8 AM use 8, and for 8 PM use 20.

30 08 10 06 * /home/username/full-backup

30 – 30th Minute
08 – 08 AM
10 – 10th Day
06 – 6th Month (June)
* – Every day of the week

2. Schedule a Job For More Than One Instance (e.g. Twice a Day)

The following script take a incremental backup twice a day every day.

This example executes the specified incremental backup shell script (incremental-backup) at 11:00 and 16:00 on every day. The comma separated value in a field specifies that the command needs to be executed in all the mentioned time.

00 11,16 * * * /home/username/bin/incremental-backup

00 – 0th Minute (Top of the hour)
11,16 – 11 AM and 4 PM
* – Every day
* – Every month
* – Every day of the week

3. Schedule a Job for Specific Range of Time (e.g. Only on Weekdays)

If you wanted a job to be scheduled for every hour with in a specific range of time then use the following.

Cron Job everyday during working hours

This example checks the status of the database everyday (including weekends) during the working hours 9 a.m – 6 p.m

00 09-18 * * * /home/username/bin/check-db-status

00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
* – Every day of the week

Cron Job every weekday during working hours

This example checks the status of the database every weekday (i.e excluding Sat and Sun) during the working hours 9 a.m – 6 p.m.

00 09-18 * * 1-5 /home/username/bin/check-db-status

00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
1-5 -Mon, Tue, Wed, Thu and Fri (Every Weekday)

4. How to View Crontab Entries?

View Current Logged-In User’s Crontab entries

To view your crontab entries type crontab -l from your unix account as shown below.

1
2
3
4
username@dev-db$ crontab -l

@yearly /home/username/annual-maintenance
*/10 * * * * /home/username/check-disk-space

[Note: This displays crontab of the current logged in user]

View Root Crontab entries

Login as root user (su – root) and do crontab -l as shown below.

1
2
3
root@dev-db# crontab -l

no crontab for root
Crontab HowTo: View Other Linux User’s Crontabs entries

To view crontab entries of other Linux users, login to root and use -u {username} -l as shown below.

1
2
3
4
root@dev-db# crontab -u otheruser -l

@monthly /home/otheruser/monthly-backup
00 09-18 * * * /home/otheruser/check-db-status

5. How to Edit Crontab Entries?

Edit Current Logged-In User’s Crontab entries

To edit crontab entries, use crontab -e as shown below. By default this will edit the current logged-in users crontab.

1
2
3
4
5
6
username@dev-db$ crontab -e

@yearly /home/username/centos/bin/annual-maintenance
*/10 * * * * /home/username/debian/bin/check-disk-space
~
"/tmp/crontab.XXXXyjWkHw" 2L, 83C

[Note: This will open the crontab file in Vim editor for editing.
Please note cron created a temporary /tmp/crontab.XX… ]
When you save the above temporary file with :wq, it will save the crontab and display the following message indicating the crontab is successfully modified.

1
2
3
~
"crontab.XXXXyjWkHw" 2L, 83C written
crontab: installing new crontab
Edit Root Crontab entries

Login as root user (su – root) and do crontab -e as shown below.

1
root@dev-db# crontab -e
Edit Other Linux User’s Crontab File entries

To edit crontab entries of other Linux users, login to root and use -u {username} -e as shown below.

1
2
3
4
5
6
7
8
root@dev-db# crontab -u otheruser -e

@monthly /home/otheruser/fedora/bin/monthly-backup
00 09-18 * * * /home/otheruser/ubuntu/bin/check-db-status
~
~
~
"/tmp/crontab.XXXXyjWkHw" 2L, 83C

6. Schedule a Job for Every Minute Using Cron.

Ideally you may not have a requirement to schedule a job every minute. But understanding this example will will help you understand the other examples mentioned below in this article.

* * * * * CMD

The * means all the possible unit — i.e every minute of every hour through out the year. More than using this * directly, you will find it very useful in the following cases.

When you specify */5 in minute field means every 5 minutes.
When you specify 0-10/2 in minute field mean every 2 minutes in the first 10 minute.

Thus the above convention can be used for all the other 4 fields.

7. Schedule a Background Cron Job For Every 10 Minutes.

Use the following, if you want to check the disk space every 10 minutes.

*/10 * * * * /home/username/check-disk-space

It executes the specified command check-disk-space every 10 minutes through out the year. But you may have a requirement of executing the command only during office hours or vice versa. The above examples shows how to do those things.

Instead of specifying values in the 5 fields, we can specify it using a single keyword as mentioned below.

There are special cases in which instead of the above 5 fields you can use @ followed by a keyword — such as reboot, midnight, yearly, hourly.

Cron special keywords and their meanings

Keyword Equivalent
@yearly 0 0 1 1 *
@daily 0 0 * * *
@hourly 0 * * * *
@reboot Run at startup.

8. Schedule a Job For First Minute of Every Year using @yearly

If you want a job to be executed on the first minute of every year, then you can use the @yearly cron keyword as shown below.

This will execute the system annual maintenance using annual-maintenance shell script at 00:00 on Jan 1st for every year.

@yearly /home/username/red-hat/bin/annual-maintenance

9. Schedule a Cron Job Beginning of Every Month using @monthly

It is as similar as the @yearly as above. But executes the command monthly once using @monthly cron keyword.

This will execute the shell script tape-backup at 00:00 on 1st of every month.

@monthly /home/username/suse/bin/tape-backup

10. Schedule a Background Job Every Day using @daily

Using the @daily cron keyword, this will do a daily log file cleanup using cleanup-logs shell scriptat 00:00 on every day.

@daily /home/username/arch-linux/bin/cleanup-logs "day started"

11. How to Execute a Linux Command After Every Reboot using @reboot?

Using the @reboot cron keyword, this will execute the specified command once after the machine got booted every time.

@reboot CMD

12. How to Disable/Redirect the Crontab Mail Output using MAIL keyword?

By default crontab sends the job output to the user who scheduled the job. If you want to redirect the output to a specific user, add or update the MAIL variable in the crontab as shown below.

1
2
3
4
5
6
username@dev-db$ crontab -l

MAIL="username"

@yearly /home/username/annual-maintenance
*/10 * * * * /home/username/check-disk-space

[Note: Crontab of the current logged in user with MAIL variable]

If you wanted the mail not to be sent to anywhere, i.e to stop the crontab output to be emailed, add or update the MAIL variable in the crontab as shown below.

MAIL=""

13. How to Execute a Linux Cron Jobs Every Second Using Crontab.

You cannot schedule an every-second cronjob. Because in cron the minimum unit you can specify is minute. In a typical scenario, there is no reason for most of us to run any job every second in the system.

14. Specify PATH Variable in the Crontab

All the above examples we specified absolute path of the Linux command or the shell-script that needs to be executed.

For example, instead of specifying /home/username/tape-backup, if you want to just specify tape-backup, then add the path /home/username to the PATH variable in the crontab as shown below.

1
2
3
4
5
6
username@dev-db$ crontab -l

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/home/username

@yearly annual-maintenance
*/10 * * * * check-disk-space

[Note: Crontab of the current logged in user with PATH variable]

15. Installing Crontab From a Cron File

Instead of directly editing the crontab file, you can also add all the entries to a cron-file first. Once you have all thoese entries in the file, you can upload or install them to the cron as shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
username@dev-db$ crontab -l

no crontab for username

$ cat cron-file.txt
@yearly /home/username/annual-maintenance
*/10 * * * * /home/username/check-disk-space

username@dev-db$ crontab cron-file.txt

username@dev-db$ crontab -l
@yearly /home/username/annual-maintenance
*/10 * * * * /home/username/check-disk-space

Note: This will install the cron-file.txt to your crontab, which will also remove your old cron entries. So, please be careful while uploading cron entries from a cron-file.txt.

Crontab

Crontab is invoked by typing crontab -e in a terminal shell.

crontab -l will show the current crontab for your user.

crontab -r will completely remove the crontab for that user.

It is worth noting that crontab runs under a different shell environment to your shell. Therefore will not necessarily include all the paths required to run your scripts. To overcome this a simple method is to place a path variable at the start of the crontab file. I use the following which contains most of the binaries I will ever need to run:

1
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin

crontab will then run the scripts you ask it to at the time/frequency you tell it to. If there are any errors these will be emailed to your local user account.

Adding a user to the wheel group

wheel is a special group for system administration purpose. You can add your normal user to this group using the pw command. The syntax is:

1
pw user  mod  username -G wheel

In the below example add a user called Sara to the group called wheel:

1
2
pw user mod sara -G wheel
groups sara

Sara will then be able to su to root

^