Installing Golang

Hi there, welcome back, hope you read the introduction blog and are ready to install Go.

Installing Go is easy, you can download the binaries for your platform from the Go’s download page. In this blog, we will go through the steps of installing Go on macOS and Ubuntu and install Go v1.11.2.

Install Go v1.11.2 on Mac 10

Step 1: Download the package file from https://golang.org/doc/install?download=go1.11.2.darwin-amd64.pkg

Step 2: Open the package file and follow the prompts and that’s it – Go will be installed

screen-shot-2018-12-02-at-11-25-01-am

Screen Shot 2018-12-02 at 11.25.35 AM

Step 3: Check that the package will be installed in /usr/local/go folder and the PATH environment variable will be updated with /usr/local/go/bin

Step 4: Open the terminal and run go version command, you should see the below output, which confirms the go installation and prints the installed version as well

chetan:~ chetan$ go version
go version go1.11 darwin/amd64

Install Go v1.11.2 on Ubuntu 16.04

Step1: Make sure you update your packages, security patches and fixes by running these commands on Ubuntu:

ubuntu@ip-172-31-18-70:~$ sudo apt-get update
ubuntu@ip-172-31-18-70:~$ sudo apt-get -y upgrade

Step 2: Begin downloading the latest Go package from the downloads page using our very own curl command:

ubuntu@ip-172-31-18-70:~$
curl -O https://storage.googleapis.com/golang/go1.11.2.linux-amd64.tar.gz

Step 3: Unzip the gzip file using the tar command, you will get the ‘go‘ folder after unzipping the contents. Now move the ‘go‘ folder to /usr/local/ (like in macOS installation)

ubuntu@ip-172-31-18-70:~$ tar -xvf go1.11.2.linux-amd64.tar.gz
ubuntu@ip-172-31-18-70:~$ sudo mv go /usr/local

Step 4: Use /usr/local/go/bin path to update your bash profile with PATH environment variable like the macOS installer did for us. You can also create a folder, say go-work under your home directory (in my case /home/ubuntu), this is where all our go related source code will go in. Don’t worry too much about GOPATH for now, we understand more about it in the next blog

ubuntu@ip-172-31-18-70:~$ echo $HOME
/home/ubuntu
ubuntu@ip-172-31-18-70:~$ mkdir go-work

ubuntu@ip-172-31-18-70:~$ vi ~/.bashrc
export GOPATH=$HOME/go-work
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Step 5: Now reload your ~/.bashrc file (use appropriate profile file based on the shell you use, for instance use ~/.zshrc if you’re using zsh) and type go version on your terminal. And you’re done! 

ubuntu@ip-172-31-18-70:~$ source ~/.bashrc

ubuntu@ip-172-31-18-70:~$ go version
go version go1.11.2 linux/amd64

 

So now we have Go v1.11 installed on macOS and Ubuntu. Hurray! 🙂

But we haven’t written any code yet and what about the Editor? Let’s look at them in our next blog.

Advertisements

Installing Node.js on Ubuntu

I would continue explaining more about Node.js, but for all those folks who do things and understand them, here’s the way to install Node.js on Ubuntu machines..

You need to first install necessary packages like git-core and libssl-dev

sudo apt-get update
sudo apt-get install g++ curl libssl-dev apache2-utils
sudo apt-get install git-core

Download the latest source and build it

wget http://nodejs.org/dist/v0.8.2/node-v0.8.2.tar.gz
gunzip node-v0.8.2.tar.gz
tar xvf node-v0.8.2.tar.gz
cd node-v0.8.2
./configure
sudo make install

Check the version after install

node -v

 

BDD in Python with lettuce

Behavior Driven Development, also known as BDD, is a concept developed by Dan North and is based on a popular and well adopted TDD. As in Dan’s words –

‘BDD is a second-generation, outside–in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.’

BDD provides a framework where QA, Business Analysts and other stake-holders communicate and collaborate on sotware development. While TDD emphasized on developing tests for unit piece of code. BDD insists on developing tests for business scenarios or use cases or behavioral specification of software being developed. According to Dan, BDD tests should be written as user stories ‘As a [role] I want [feature] so that [benefit]’ and Acceptance criteria should be defined as ‘Given [initial context], when [event occurs], then [ensure some outcomes].

lettuce is typically used in Python to implement BDD. This blog covers the installation of lettuce on Ubuntu and its application with an example of fibonacci function

Installation

buntu@ubuntu:~$ sudo pip install lettuce
[sudo] password for buntu:
Downloading/unpacking lettuce
Downloading lettuce-0.2.9.tar.gz (40Kb): 40Kb downloaded
Running setup.py egg_info for package lettuce
Downloading/unpacking sure (from lettuce)
Downloading sure-1.0.6.tar.gz
Running setup.py egg_info for package sure
Downloading/unpacking fuzzywuzzy (from lettuce)
Downloading fuzzywuzzy-0.1.tar.gz
Running setup.py egg_info for package fuzzywuzzy
Installing collected packages: fuzzywuzzy, lettuce, sure
Running setup.py install for lettuce
Installing lettuce script to /usr/local/bin
Running setup.py install for sure
Running setup.py install for fuzzywuzzy
Successfully installed lettuce

Setup

Let’s first create a directory structure that looks like this

buntu@ubuntu:~$ tree lettucetests/
lettucetests/
|– features
|   |– fib.feature
|   |– test.py
`– test.feature

1 directory, 3 files

Define Features

Write Tests

Tornado – Redis

As defined on Redis website, it is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists among others. It’s similar to say memcached library in the sense that it ia an in memory key/value pair but persistent on disk.. Redis has gained importance as a NoSQL option for web development because of its speed (GET and SET operations in the range of 100,000 per seconds). This post is about including Redis in Tornado for web development.

Let’s start with installing Redis-server

ubuntu@ubuntu:~/tornado-2.2$ sudo apt-get install redis-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libtext-glob-perl libcompress-bzip2-perl libparams-util-perl libfile-chmod-perl libdata-compare-perl libfile-pushd-perl libfile-which-perl libcpan-inject-perl
  libfile-find-rule-perl libcpan-checksums-perl libnumber-compare-perl
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  redis-server
0 upgraded, 1 newly installed, 0 to remove and 171 not upgraded.
Need to get 80.8kB of archives.
After this operation, 283kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ lucid/universe redis-server 2:1.2.0-1 [80.8kB]
Fetched 80.8kB in 2s (27.3kB/s) 
Selecting previously deselected package redis-server.
(Reading database ... 138423 files and directories currently installed.)
Unpacking redis-server (from .../redis-server_2%3a1.2.0-1_i386.deb) ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Setting up redis-server (2:1.2.0-1) ...
Starting redis-server: redis-server.

Confirmation

ubuntu@ubuntu:~/tornado-2.2$ ps aux | grep redis
redis    19104  0.0  0.1   2284   716 ?        Ss   21:23   0:00 /usr/bin/redis-server /etc/redis/redis.conf

Python client library for redis

ubuntu@ubuntu:~/tornado-2.2$ sudo pip install redis
Downloading/unpacking redis
  Downloading redis-2.6.2.tar.gz
  Running setup.py egg_info for package redis
Installing collected packages: redis
  Running setup.py install for redis
Successfully installed redis

 

With redis installed, lets go to an example where Redis meets Tornado. In the example below,

  • When the web server is started, redis server is initialized with key-value pairs of username and password (password is md5 hash of username in hex format) for users ‘bob’ and ‘clara’.
  • On browsing to http://localhost:8888/login and POSTing the username and password details to Tornado web server, authentication of details happen from redis server.
  • Relevant message for successful/unsuccessful attempt is render on user’s browser

Intro to Mongodb

I came across MongoDB a few months ago and have taken decent strides to understand it. This post is to introduce MongoDB to the readers along with basic installation and setup steps on Ubuntu10.04.

MongoDB

  • Falls under the family of NoSQL database
  • Unlike relational database (row/column), its a document based database
  • Provides indexing, dynamic querying like relational databases
  • Database is a set of collections which is a set of documents. Documents are set of fields with each field being a key(string) – value pair (JSON based, called BSON in Mongo terms)
  • No joins and no queries on multiple table -> higher performance
  • High availability with replicated servers
  • Highly scalable because of data sharding (partitioning across servers)
  • Logical query language fitting to dynamic programming languages like Python

Installation

On Ubuntu10.04, I could install MongoDB with

 $sudo apt-get install mongodb

MongoDB Service

You can start/stop/restart MongoDB service on Ubuntu:

$sudo service mongodb start
$sudo service mongodb stop
$sudo service mongodb restart

Test of installation

ubuntu@ubuntu:~$ sudo service mongodb start
start: Job is already running: mongodb

ubuntu@ubuntu:~$ mongo
MongoDB shell version: 1.2.2
url: test
connecting to: test
type "exit" to exit
type "help" for help
>

Light Weight Process – Dissecting Linux Threads

Article on Light Weight Process – Dissecting Linux Threads, got printed in August 2011  Issue Vol. 9  No. 6 of LinuxForYou magazine (ISSN 0974-1054)

Authors: Vishal Kanaujia and Chetan Giridhar

This article, aimed at Linux developers and students of computer science, explores the fundamentals of threads and their implementation in Linux with Light-Weight Processes, aiding understanding with a code implementation.

Threads are the core element of a multi-tasking programming environment. By definition, a thread is an execution context in a process; hence, every process has at least one thread. Multi-threading implies the existence of multiple, concurrent (on multi-processor systems), and often synchronised execution contexts in a process.

Threads have their own identity (thread ID), and can function independently. They share the address space within the process, and reap the benefits of avoiding any IPC (Inter-Process Communication) channel (shared memory, pipes and so on) to communicate. Threads of a process can directly communicate with each other — for example, independent threads can access/update a global variable. This model eliminates the potential IPC overhead that the kernel would have had to incur. As threads are in the same address space, a thread context switch is inexpensive and fast.

A thread can be scheduled independently; hence, multi-threaded applications are well-suited to exploit parallelism in a multi-processor environment. Also, the creation and destruction of threads is quick. Unlike fork(), there is no new copy of the parent process, but it uses the same address space and shares resources, including file descriptors and signal handlers.

A multi-threaded application uses resources optimally, and is highly efficient. In such an application, threads are loaded with different categories of work, in such a manner that the system is optimally used. One thread may be reading a file from the disk, and another writing it to a socket. Both work in tandem, yet are independent. This improves system utilisation, and hence, throughput.

A few concerns

The most prominent concern with threads is synchronisation, especially if there is a shared resource, marked as a critical section. This is a piece of code that accesses a shared resource, and must not be concurrently accessed by more than one thread. Since each thread can execute independently, access to the shared resource is not moderated naturally but using synchronisation primitives including mutexes (mutual exclusion), semaphores, read/write locks and so on.

These primitives allow programmers to control access to a shared resource. In addition, similar to processes, threads too suffer states of deadlock, or starvation, if not designed carefully. Debugging and analysing a threaded application can also be a little cumbersome.

How does Linux implement threads?

Linux supports the development and execution of multi-threaded applications. User-level threads in Linux follow the open POSIX (Portable Operating System Interface for uniX) standard, designated as IEEE 1003. The user-level library (on Ubuntu, glibc.so) has an implementation of the POSIX API for threads.

Threads exist in two separate execution spaces in Linux — in user space and the kernel. User-space threads are created with the pthread library API (POSIX compliant). These user-space threads are mapped to kernel threads. In Linux, kernel threads are regarded as “light-weight processes”. An LWP is the unit of a basic execution context. Unlike other UNIX variants, including HP-UX and SunOS, there is no special treatment for threads. A process or a thread in Linux is treated as a “task”, and shares the same structure representation (list of structtask_structs).

For a set of user threads created in a user process, there is a set of corresponding LWPs in the kernel. The following example illustrates this point:

#include <stdio.h>
#include <syscall.h>
#include <pthread.h>
int main()
{
    pthread_t tid = pthread_self();
    int sid = syscall(SYS_gettid);
    printf("LWP id is %dn", sid);
    printf("POSIX thread id is %dn", tid);
    return 0;
}

Running the ps command too, lists processes and their LWP/ threads information:

kanaujia@ubuntu:~/Desktop$ ps -fL

UID        PID     PPID       LWP   C   NLWP    STIME   TTY          TIME CMD
kanaujia 17281     5191     17281   0      1    Jun11   pts/2    00:00:02 bash
kanaujia 22838     17281    22838   0      1    08:47   pts/2    00:00:00 ps -fL
kanaujia 17647     14111    17647   0      2    00:06   pts/0    00:00:00 vi clone.s

What is a Light-Weight Process?

An LWP is a process created to facilitate a user-space thread. Each user-thread has a 1×1 mapping to an LWP. The creation of LWPs is different from an ordinary process; for a user process “P”, its set of LWPs share the same group ID. Grouping them allows the kernel to enable resource sharing among them (resources include the address space, physical memory pages (VM), signal handlers and files). This further enables the kernel to avoid context switches among these processes. Extensive resource sharing is the reason these processes are called light-weight processes.

How does Linux create LWPs?

Linux handles LWPs via the non-standard clone() system call. It is similar to fork(), but more generic. Actually, fork() itself is a manifestation of clone(), which allows programmers to choose the resources to share between processes. The clone() call creates a process, but the child process shares its execution context with the parent, including the memory, file descriptors and signal handlers. The pthread library too uses clone() to implement threads. Refer to./nptl/sysdeps/pthread/createthread.c in the glibc version 2.11.2 sources.

Create your own LWP

I will demonstrate a sample use of the clone() call. Have a look at the code in demo.c below:

#include <malloc.h>

#include <sys/types.h>
#include <sys/wait.h>
#include <signal.h>
#include <sched.h>
#include <stdio.h>
#include <fcntl.h>
// 64kB stack
#define STACK 1024*64
// The child thread will execute this function
int threadFunction( void* argument ) {
     printf( "child thread entering\n" );
     close((int*)argument);
     printf( "child thread exiting\n" );
     return 0;
}
int main() {
     void* stack;
     pid_t pid;
     int fd;
     fd = open("/dev/null", O_RDWR);
     if (fd < 0) {
         perror("/dev/null");
         exit(1);
     }
     // Allocate the stack
     stack = malloc(STACK);
     if (stack == 0) {
         perror("malloc: could not allocate stack");
         exit(1);
     }
     printf("Creating child thread\n");
     // Call the clone system call to create the child thread
     pid = clone(&threadFunction,
                 (char*) stack + STACK,
                 SIGCHLD | CLONE_FS | CLONE_FILES |\
                  CLONE_SIGHAND | CLONE_VM,
                 (void*)fd);
     if (pid == -1) {
          perror("clone");
          exit(2);
     }
     // Wait for the child thread to exit
     pid = waitpid(pid, 0, 0);
     if (pid == -1) {
         perror("waitpid");
         exit(3);
     }
     // Attempt to write to file should fail, since our thread has
     // closed the file.
     if (write(fd, "c", 1) < 0) {
         printf("Parent:\t child closed our file descriptor\n");
     }
     // Free the stack
     free(stack);
     return 0;
}
The program in demo.c allows the creation of threads, and is fundamentally similar to what thepthread library does. However, the direct use of clone() is discouraged, because if not used properly, it may crash the developed application. The syntax for calling clone() in a Linux program is as follows:
#include <sched.h>

int clone (int (*fn) (void *), void *child_stack, int flags, void *arg);

The first argument is the thread function; it will be executed once a thread starts. When clone()successfully completes, fn will be executed simultaneously with the calling process.

The next argument is a pointer to a stack memory for the child process. A step backward fromfork()clone() demands that the programmer allocates and sets the stack for the child process, because the parent and child share memory pages — and that includes the stack too. The child may choose to call a different function than the parent, hence needs a separate stack. In our program, we allocate this memory chunk in the heap, with the malloc() routine. Stack size has been set as 64KB. Since the stack on the x86 architecture grows downwards, we need to simulate it by using the allocated memory from the far end. Hence, we pass the following address to clone():

(char*) stack + STACK

The next field, flags, is the most critical. It allows you to choose the resources you want to share with the newly created process. We have chosen SIGCHLD | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_VM, which is explained below:

  • SIGCHLD: The thread sends a SIGCHLD signal to the parent process after completion. It allows the parent to wait() for all its threads to complete.
  • CLONE_FS: Shares the parent’s filesystem information with its thread. This includes the root of the filesystem, the current working directory, and the umask.
  • CLONE_FILES: The calling and caller process share the same file descriptor table. Any change in the table is reflected in the parent process and all its threads.
  • CLONE_SIGHAND: Parent and threads share the same signal handler table. Again, if the parent or any thread modifies a signal action, it is reflected to both the parties.
  • CLONE_VM: The parent and threads run in the same memory space. Any memory writes/mapping performed by any of them is visible to other process.

The last parameter is the argument to the thread function (threadFunction), and is a file descriptor in our case.

Please refer to the sample code implementation of LWP, in demo.c we presented earlier.

The thread closes the file (/dev/null) opened by the parent. As the parent and this thread share the file descriptor table, the file close operation will reflect in the parent context also, and a subsequent file write() operation in the parent will fail. The parent waits till thread execution completes (till it receives a SIGCHLD). Then, it frees the memory and returns.

Compile and run the code as usual; and it should be similar to what is shown below:

$gcc demo.c

$./a.out
Creating child thread
child thread entering
child thread exiting
Parent: child closed our file descriptor
$

Linux provides support for an efficient, simple, and scalable infrastructure for threads. It encourages programmers to experiment and develop thread libraries using clone() as the core component.

Please share your suggestions/feedback in the comments sections below.

References and suggested reading
  1. Wikipedia article on clone()
  2. clone() man page
  3. Using the clone() System Call by Joey Bernard
  4. Implementing a Thread Library on Linux
  5. IEEE Standards Interpretations for IEEE Std 1003.1c-1995
  6. Sources of pthread implementation in glibc.so
  7. The Fibers of Threads by Benjamin Chelf